Statistical Significance for Product Managers, Without the Jargon

You do not need a statistics degree to run clean experiments, but you do need to respect a handful of ideas, because the ways teams fool themselves with A/B tests are almost all statistical. This is significance, power, sample size, and the common traps, explained in plain language for product and growth managers who have to make real decisions from test results. No formulas, just the intuition you need to not deceive yourself.

What significance actually means

When a platform says a result is “statistically significant,” it is answering a narrow question: if there were really no difference between the variants, how surprising would this result be? Significance is a measure of surprise under the assumption that nothing is going on.

The common threshold is 95% confidence, which corresponds to a p-value of 0.05. Loosely, it means that if the variant truly made no difference, you would see a result this extreme only about 5% of the time by chance. It does not mean there is a 95% chance the variant is better, which is the near-universal misreading. It means the result would be unlikely if the variant did nothing.

The practical takeaway: significance tells you the result is probably not pure luck. It does not tell you the effect is large, that it will last, or that it matters to the business. Those are separate questions you still have to ask.

The false positive you are guarding against

The reason for the whole apparatus is to guard against the false positive, declaring a winner that is really just noise. Random variation means that even two identical variants will show some difference in any given test. Significance testing is how you decide whether the difference you see is large enough, relative to the noise, to be worth believing.

Set the threshold too loose and you will ship a stream of “winners” that are really luck, then wonder why your aggregate conversion never improves the way your test log says it should. This is the quiet killer of experimentation programs: a pile of significant-looking wins that do not add up, because too many of them were false. Respecting significance is how you make sure the wins you bank are real.

Power and sample size, the parts teams skip

Significance gets all the attention, but power is the concept that determines whether your test can succeed at all, and it is the one teams skip. Power is the probability that your test will detect a real effect if there is one. A test with low power will often miss a genuine improvement, showing “no significant difference” not because there is none, but because the test was too small to see it.

Power depends on three things: your baseline conversion rate, the size of the effect you want to detect, and your sample size. The practical consequence is that you must estimate your needed sample size before you start. If a page converts at 4% and you want to reliably detect a 10% relative improvement, the math dictates roughly how many visitors each variant needs. Run with fewer and you are gambling; a flat result tells you nothing, because the test never had the power to see the effect in the first place.

This is why prioritization and statistics are linked: a test on a low-traffic page may never reach the sample size it needs, which is a reason to concentrate traffic on higher-value pages, a point I make in experiment prioritization. Power is also why “we ran it for a week and saw nothing” is often meaningless, the week simply did not deliver enough visitors.

Do not peek and stop

The most common and most damaging statistical mistake is peeking at a running test and stopping it the moment it crosses significance. It feels efficient. It badly inflates your false positive rate, because a test wandering around will cross the significance line by chance at some point if you keep checking, and if you stop the instant it does, you have selected for a fluke.

The fix depends on your platform’s method. With a traditional fixed-horizon approach, decide the sample size and duration in advance and do not act on the result until you reach it, no matter how tempting the mid-test numbers look. Some modern platforms use sequential or “always-valid” statistics designed precisely so you can monitor continuously without inflating error; if yours does, you may peek, but you must know that is the method you are on. The cardinal sin is using fixed-horizon statistics while behaving as if you can stop whenever you like. Know your method and follow its rules.

Run full business cycles

Even with the right sample size, timing matters. User behavior on a Tuesday is not user behavior on a Sunday, and behavior in the first week of a month can differ from the last. If you run a test for three days, you may be measuring a weekday artifact rather than a durable effect.

The rule of thumb is to run at least one full week, and ideally two, so that every day of the week is represented and any weekly cycle averages out. This holds even if you hit your sample size faster, because reaching the number on a burst of atypical weekend traffic can still mislead you. Let the test span whole cycles so the result reflects normal behavior, not a slice of it.

Watch the base rate and the winner’s curse

Two subtler effects trip up experienced teams. The first is the base rate: a 34% relative lift on a page converting at 2% is a very different result from the same relative lift on a page converting at 40%, both in what it takes to achieve and in what it means. Always read a lift alongside the baseline it is measured from, because a percentage in isolation can mislead.

The second is the winner’s curse: the tests that show the biggest wins are also the ones most likely to have gotten lucky, so the effect you measured is often larger than the effect you will actually get when you ship. This is why a surprising, large win deserves more scrutiny than a modest one, and why re-testing your most important or most surprising wins before betting real money on them is prudent. The bigger and more surprising the result, the more you should hold it to a higher bar.

Significance is not importance

A result can be statistically significant and business-irrelevant. With enough traffic, a 0.1% lift can reach significance, but it may not be worth the complexity of shipping and maintaining the change. Statistical significance answers “is this real?”, not “is this worth doing?”. You still have to weigh the size of the effect against the cost of the change and its effect on everything downstream.

The reverse also holds: an effect can be real and important but not yet significant because the test lacked power. Do not treat “not significant” as “no effect”; treat it as “not enough evidence yet,” which might mean running longer or accepting that the page lacks the traffic to ever prove it. Significance is one input to a decision, not the decision itself, which is part of what it means to own the number, not the task.

Measure the downstream, not just the immediate metric

A final trap that is as much about metrics as statistics: a variant can significantly lift the immediate metric and harm the thing you actually care about. More signups that activate worse, more clicks that convert worse, more checkouts that refund more. A statistically clean win on the wrong metric is still a loss.

Always tie a test to a downstream outcome, not just the surface conversion, and watch that downstream number for harm even when the headline metric wins. The cleanest experiment in the world, read on the wrong metric, will confidently lead you the wrong way. This ties back to writing a proper hypothesis, which names the metric and the mechanism you expect, covered in hypothesis-driven experimentation.

The one calculation worth doing before you start

There is exactly one piece of math I insist a product manager do before launching a test, and it takes two minutes with any online calculator: estimate the sample size you need. You feed in three numbers you already have or can decide, your current conversion rate, the smallest lift you would care about detecting, and your confidence and power targets, and it tells you how many visitors each variant needs. That single number changes how you plan, because it tells you up front whether the test is even feasible on the traffic you have.

The reason this matters more than any other calculation is that it prevents the most common waste in experimentation: running a test that never had a chance. If the calculator says you need 40,000 visitors per variant to detect a 5% lift, and the page gets 2,000 visitors a week, you now know the test would take months, and you can decide before you start whether to run it longer, target a bigger effect, or pick a different page. Skipping this step is how teams end up three weeks into a test that could never have concluded, and calling the flat result “no effect” when it was really “no chance.” Do the calculation first, every time.

Multiple comparisons: the more you test, the more luck you find

A subtler trap catches teams as their programs grow: the more things you test at once, the more false winners you will find by pure chance. Statistics at a 95% threshold means roughly one in twenty truly-null tests will look significant anyway. Run one test and that risk is small. Run a variant with five different goals, or one experiment split across ten segments hunting for a subgroup that “responds,” and you have given chance many rolls of the dice, so a false positive somewhere becomes likely rather than rare.

This is why segment-hunting after a flat overall result is so dangerous. If the headline was flat but “mobile users in Germany converted 30% better,” that is very often noise dressed as insight, because you tested many segments and reported the one that looked good. The defenses are simple: decide your primary metric and key segments before the test, treat anything you find by slicing afterward as a hypothesis to confirm in a fresh test rather than a result to act on, and be honest with yourself about how many comparisons you actually made. The number of chances you gave luck is the number you have to account for.

Statistics is really about honesty

Step back from the individual rules and the common thread is not mathematics, it is honesty with yourself. Every trap in this piece, peeking and stopping, running underpowered tests, hunting for a flattering segment, trusting the biggest wins, reading the wrong metric, is a way of letting yourself believe something the evidence does not actually support. The math is just the tool that keeps you honest, a fixed standard you agree to in advance so that you cannot quietly move the goalposts once you see a number you like.

That is why a product manager does not need the formulas but does need the discipline. The formulas are handled by the platform; the discipline, deciding the sample size before you start, waiting for the test to finish, pre-committing to your metric and segments, holding surprising wins to a higher bar, is entirely on you, and no tool can supply it. Teams that internalize this stop arguing about test results, because they have agreed in advance on the rules for what counts as real. That agreement, more than any single calculation, is what lets a group of people learn from experiments together instead of each seeing in the data whatever they hoped to find.

The short version

• Significance measures surprise under “no difference”; it does not mean 95% chance the variant is better.
• It guards against false positives, the wins that are really luck.
• Power and sample size decide whether your test can detect a real effect; estimate them up front.
• Do not peek and stop unless your platform’s statistics allow it; know your method.
• Run full business cycles, watch the base rate, and distrust the biggest wins a little.
• Significance is not importance, and always check the downstream metric.

You do not need the math. You need the discipline to not fool yourself, and these few rules are that discipline.

---

I am Deepanshu Grover, a Growth Product Manager in Paris. I ran the experimentation program that lifted conversion 34% across 200+ pages at Chegg. If your team argues about test results, connect on LinkedIn or get in touch.