Is Your A/B Test Actually Significant?

You ran the test. Variant B is up 8%, the tool says p = 0.03, and there's a little green checkmark next to the word Significant. You ship it.

Here's the uncomfortable part: that green checkmark almost certainly doesn't mean what you think it means. Not because the math is wrong — because of when you looked.

The habit that breaks almost every test

You didn't run this test the way the statistics assume you did.

A classic significance test — the p < 0.05, fixed-horizon kind every dashboard shows you — makes one quiet assumption: you decided the sample size in advance, you looked exactly once, at the end, and then you stopped. That's the deal that makes p < 0.05 mean "5% chance this is a fluke."

Nobody runs tests that way. You watched it. On day two it was flat, day three B pulled ahead, you refreshed it on your phone over coffee, and the moment it crossed the line you called it. That's the honest, human way to run a test — and it silently voids the guarantee.

The name for this is peeking, and it's not a rounding error. When you monitor a fixed-horizon test continuously and stop the first time it hits p < 0.05, your real false-positive rate isn't 5%. Depending on how often you look, it climbs toward 20–40%+. (This isn't folklore — it's the finding behind Optimizely's own move to sequential testing, and the Johari et al. paper literally titled "Peeking at A/B Tests.") Roughly: given enough looks, a coin-flip difference will eventually wander across the 0.05 line by pure chance, and you'll be standing there ready to declare victory when it does.

So the failure isn't that your tools lie. It's that they answer a question — "is this significant if you looked once, at a pre-set sample size?" — that has nothing to do with how you actually used them.

Why "just wait for significance" is bad advice

The usual fix you'll hear is: stop peeking — pick your sample size up front and don't look until you hit it.

It's technically correct and practically useless. It means:

• Committing to a fixed run length before you know the effect size (the thing the test exists to measure).
• Sitting on a variant that's obviously harming users because "the test isn't done."
• Ignoring the single most useful thing about running online — that you can watch.

Telling a builder to not look at their own live experiment is like telling a pilot to not look at the altimeter until landing. The real answer isn't to look less. It's to use a test that's valid under looking.

Tests you're allowed to watch

There's a whole family of methods — anytime-valid or always-valid inference — built for exactly this. The short version: instead of a p-value that's only honest at one pre-chosen moment, you compute a quantity that stays honest no matter when you stop. You can check it every hour, stop the instant you're convinced, and still hold your error rate.

The mechanics have a few names — e-values, sequential probability ratio tests, always-valid confidence sequences — but the property that matters is one sentence: you may stop whenever you like, and the guarantee still holds. No pre-registered sample size, no peeking penalty, no pretending you didn't look.

The trade-off is honest and worth stating: to buy that freedom, these tests are a touch more conservative at any single moment than a fixed-horizon test would be if you'd genuinely only looked once. You pay a small tax in exchange for permission to do the thing you were always going to do anyway. That's a good trade. A slightly slower "yes" you can trust beats a fast "yes" that's wrong a third of the time.

The five-minute integrity checklist

Before you believe any test — sequential or not — run these:

1. Sample Ratio Mismatch (SRM). You split 50/50, but the arms have 12,000 vs. 13,400 users. That gap should be near-impossible by chance. If your split is off, stop — something in the assignment or logging is broken, and every downstream number is contaminated. SRM is the smoke alarm; check it first, every time.
2. Did you slice after the fact? "It's not significant overall, but it's significant for Android users in Canada on Tuesdays." Every extra segment you test is another lottery ticket for a false positive. If you're going to slice, your significance bar has to get stricter as you add comparisons (this is what false-discovery-rate correction is for) — not stay at 0.05 per slice.
3. What's the smallest effect you'd actually act on? If a 0.5% lift wouldn't change a single decision, don't run a test powered to detect it — you'll just wait forever or fool yourself. Decide the effect that matters before you look at the result.
4. Is this a novelty spike? New things get clicked because they're new. A week-one win can be pure curiosity that decays to nothing. Let it run past the novelty, or your "winner" is a fad.
5. Can you even see why it moved? A test tells you that B beat A. It rarely tells you why — and "why" is what you actually need to not get fooled next time. If B won because it accidentally hid a confusing button, you learned something. If you can't tell, you're one layer short.

The honest version of "data-driven"

Being data-driven was never about producing a number. It's about producing a number you're allowed to believe — one whose guarantee survives contact with how you actually work.

That's the whole design principle behind how we build Drengr: the test math should be honest about the thing humans really do — watch, and stop when convinced — instead of quietly assuming you didn't. An answer you can check beats a confident answer you can't, every time.

Your last "significant" result might have been real. The point is that, run the usual way, you have no way to know — and that not-knowing is the exact thing a real test is supposed to remove.