Experiment Prioritization: ICE, PIE, and What Actually Works

Every experimentation program hits the same wall: you have more test ideas than you have traffic and time to run them. Prioritization is what separates a program that ships meaningful wins from one that spreads itself so thin that nothing reaches significance. The frameworks people reach for, ICE and PIE, are useful, but the framework matters far less than the discipline of forcing every idea through the same filter so that the loudest voice does not simply set the roadmap.

Here is how to prioritize experiments in a way that actually works.

Why prioritization is really about traffic

The deepest reason prioritization matters is statistical, not organizational. Every test needs a certain amount of traffic to reach a trustworthy result, and traffic is finite. If you run ten tests at once across your traffic, each gets a tenth of it, and most will never reach the sample size they need to detect a real effect. You end up with ten inconclusive tests instead of two or three clear answers.

So prioritization is not just about picking good ideas; it is about concentrating your finite traffic on the tests most worth running, so those tests actually conclude. This is why prioritization and statistics are inseparable, a link I make in statistical significance for product managers. A perfectly good test idea on a page without enough traffic is not worth running, because it can never reach power. Prioritization is the discipline of spending traffic where it pays.

ICE: impact, confidence, ease

ICE scores each candidate on three dimensions, usually one to ten.

Impact. If this test wins, how much could it move the number, accounting for how much traffic or revenue sits behind the page? A win on your highest-traffic template is worth more than the same win on a page nobody visits.

Confidence. How strong is the evidence that there is a real problem here and that your change will fix it? A hypothesis backed by analytics, recordings, and survey data scores higher than a hunch. This is where hypothesis-driven testing pays off, because a well-evidenced belief is exactly what raises confidence, a connection I draw in hypothesis-driven experimentation.

Ease. How little effort does it take to build and run? A test that needs a week of engineering scores lower than one a marketer can ship today.

You score each idea, combine the three, and sort. The value of ICE is speed and simplicity: a team can score a backlog quickly and get a defensible order.

PIE: potential, importance, ease

PIE is a close cousin, tuned slightly for CRO.

Potential. How much room for improvement does the page have? A page that already converts well has less headroom than a leaky one.

Importance. How valuable is the traffic hitting the page? High-intent, high-volume, or high-cost traffic makes a page more important to get right.

Ease. As in ICE, how hard is it to implement?

PIE and ICE overlap heavily. Potential and importance together do much the same work as ICE’s impact, split into “how broken is it” and “how much does the traffic matter.” Use whichever split helps your team reason more clearly. The differences between them are not worth arguing about.

The framework matters less than the discipline

Here is the honest truth after running a program at scale: the specific framework matters far less than the fact that you have one and apply it consistently. The real value of ICE or PIE is not the precision of the scores, which are subjective estimates anyway. It is that every idea passes through the same filter, which strips out the two forces that otherwise decide roadmaps: the loudest opinion in the room and the shiniest new idea.

Without a consistent filter, the executive’s pet idea jumps the queue, the idea someone mentioned most recently gets built, and prioritization becomes politics. With a filter, every idea, including the executive’s, gets scored on the same axes, and the conversation shifts from “who wants this” to “how does this score.” That shift is the entire point. The scores do not need to be exact; they need to be applied to everyone equally.

Score honestly, especially on confidence

The axis teams game most is confidence, because it is the easiest to inflate for an idea you already like. Guard against it by tying confidence to actual evidence. A high confidence score should require pointing to the analytics drop-off, the session recordings, the survey responses, or the prior test that supports the belief. “I have a good feeling about this” is not confidence; it is preference wearing a number.

This discipline also improves your idea pipeline over time, because it rewards doing the diagnostic work that produces well-evidenced hypotheses. Teams that score confidence honestly end up investing more in research, because research is how you earn a high confidence score, and better research produces better tests. The scoring quietly pulls the whole program toward evidence.

Concentrate, do not scatter

The most important practical rule that falls out of prioritization: concentrate traffic rather than scattering it. It is almost always better to run one well-powered test on your highest-value template than five underpowered tests spread across pages that will never reach significance. The scattered approach feels productive, more tests running, but produces fewer answers, because most of those tests die inconclusive.

Prioritization gives you the ranked list; concentration is what you do with it. Run the top one or two or three that your traffic can actually support at full power, get clean answers, then move to the next. A slower cadence of conclusive tests beats a frantic cadence of inconclusive ones every time, because only the conclusive ones teach you anything or bank a real win. This is the same concentrate-not-scatter principle that governs landing page optimization at scale.

Keep the backlog alive

Prioritization is not a one-time ranking; it is a living process. New evidence arrives constantly, from analytics, from support tickets, from the results of tests you just ran. A test that concludes changes the priority of everything related to it, a win opening up adjacent hypotheses, a loss closing off a direction. So re-score on a cadence, add new ideas as evidence surfaces them, and let priorities reflect what you have learned rather than what seemed important a quarter ago.

A healthy backlog is never empty and never stale. When a test ends, the next best-evidenced test should be ready to launch, already scored, rather than requiring a fresh brainstorm. The state of your backlog, how full and how well-evidenced it is, is a good read on the health of the whole program, a point I make in the A/B testing guide.

Common prioritization mistakes

• No framework at all. Prioritizing by whoever argues hardest, which is not prioritization.
• Treating scores as precise. Debating whether an idea is a 7 or an 8 misses the point; the scores are rough sorts, not measurements.
• Inflating confidence. Scoring ideas you like as high-confidence without the evidence to back it.
• Ignoring traffic. Prioritizing a test on a page that can never reach the sample size it needs.
• Scattering. Running everything at once so nothing concludes.
• A stale backlog. Failing to re-score as tests conclude and evidence arrives.

Avoid these and prioritization does its real job: it turns a chaotic pile of ideas and opinions into a disciplined, evidence-weighted queue that spends your scarce traffic where it pays, which is what lets a program produce wins that actually add up.

A worked scoring, and why the order surprised us

Numbers on a page make the discipline concrete. Take three real candidates from a backlog. First, a full redesign of the homepage hero, high impact if it works, but the evidence is a stakeholder’s hunch and it needs two weeks of design and engineering. On ICE: impact 8, confidence 3, ease 2. Second, a headline test on the highest-traffic pricing template, backed by exit surveys naming price confusion, buildable in a day. Impact 7, confidence 8, ease 9. Third, reordering the steps in a low-traffic onboarding flow, plausible but thinly evidenced, moderate effort. Impact 5, confidence 4, ease 5.

Summed, the pricing headline test scores far above the glamorous homepage redesign, and that inversion is the whole lesson. The redesign feels like the important work, it is what the room wants to talk about, but its confidence is a hunch and its ease is terrible, so it commits weeks of effort to a bet you cannot yet justify. The headline test is unglamorous and scores highest because it is well-evidenced and cheap, meaning you learn fast and bank a win or a lesson within days. Prioritization consistently surfaces this pattern: the highest-scoring tests are rarely the flashiest, and a program that trusts its scores over its instincts ships more learning per week than one that chases the exciting idea.

Sequence for momentum, not just for score

Raw score gives you a ranked list, but the order you actually run tests in should account for more than the numbers. Early in a program, or when trust in experimentation is low, deliberately front-load a few high-confidence, high-ease tests even if they are not the very top of the list. A couple of quick, clean wins builds the credibility and the habit that let you later spend weeks on a harder, higher-impact test without the organization losing patience. Momentum is a real input, and ignoring it is how technically-correct prioritization still fails politically.

There is also a dependency dimension the score misses. Some tests, win or lose, unlock a whole branch of follow-up hypotheses, while others are dead ends whatever the result. A foundational test that will teach you something you will use ten more times is worth sequencing early even at a slightly lower score, because its learning compounds into everything downstream. Read the ranked list as a strong default, then adjust the running order for momentum and for which tests unlock the most future work. The score tells you what is worth testing; judgment tells you what to test first.

Make the backlog visible to everyone

Prioritization only defuses politics if the scoring is transparent. A ranked backlog that lives in one person’s head, or in a spreadsheet nobody else opens, invites exactly the end-runs the framework is supposed to prevent, because a stakeholder who cannot see why their idea sits at number fourteen will simply lobby to jump the queue. When the whole backlog is visible, with each idea’s scores and the evidence behind its confidence, the conversation changes: someone who wants their idea prioritized has to argue about the score, on the same axes as everyone else, which is precisely the debate you want.

Visibility also spreads the discipline. When people can see that high-confidence scores require pointing at real evidence, they start bringing evidence with their ideas, because they have watched hunches score low and lose. The backlog becomes a teaching tool, quietly training the organization to think in impact, evidence, and effort rather than in volume and enthusiasm. A prioritization framework hidden from view is just one person’s opinion with numbers attached; a visible one is a shared language for deciding what to build, which is the real prize.

The short version

• You always have more ideas than traffic; prioritization decides where traffic goes.
• ICE (impact, confidence, ease) and PIE (potential, importance, ease) are near-equivalent; pick one.
• The framework matters less than applying it consistently to every idea, including the boss’s.
• Tie confidence to real evidence, which pulls the program toward research.
• Concentrate traffic on tests that can win; do not scatter.
• Keep the backlog alive, re-scoring as evidence and results arrive.

Prioritization is not bureaucracy. It is how you make sure your finite traffic produces conclusive answers instead of a pile of inconclusive ones.

---

I am Deepanshu Grover, a Growth Product Manager in Paris. I prioritized and ran the experimentation program that lifted conversion 34% across 200+ pages at Chegg. If your test backlog is chaos, connect on LinkedIn or get in touch.