Live demo · Data science
Did your A/B test actually win?
A lift number on its own says nothing — it has to survive the question of how often noise alone would produce it. This reads an experiment three ways: the frequentist test, the Bayesian posterior, and a simulation of the mistake that quietly breaks more experiments than any of them.
What the data knows about each rate
Posterior distributions for the two conversion rates. Width is uncertainty — more traffic narrows a curve. The more they overlap, the less the data separates the arms.
The peeking problem
Here is the part that costs teams real money. Below, both arms are identical — same true conversion rate, no effect whatsoever. Each simulated experiment is checked a fixed number of times as traffic arrives, and called a winner the first time it crosses significance. A correctly-run test is wrong 5% of the time. Checking daily is wrong far more often.
Press Run simulation to simulate A/A experiments in your browser.
Full read-out
Method & assumptions
Significance is a two-sided two-proportion z-test with a pooled standard error:
The confidence interval on the absolute difference uses the unpooled (Wald) standard error, since it describes the difference rather than testing the null that there isn't one.
P(B beats A) is Bayesian: each arm gets a Beta(1+conversions, 1+non-conversions) posterior — a uniform Beta(1,1) prior updated by the data — and the two are compared by Monte Carlo with 60,000 draws. That sampling adds roughly ±0.2pp of noise, so the figure moves slightly between runs. It answers a different question from the p-value: not "would noise alone do this?" but "given what I've seen, how likely is B genuinely better?"
The peeking simulation runs A/A experiments — both arms drawing from the same true rate, so every "winner" is a false positive:
- Traffic arrives in equal blocks; the test is evaluated after each block.
- The run stops the first time p < α, exactly as a team watching a dashboard would.
- 2,000 runs per peek count. Blocks are drawn from a normal approximation to the binomial when both np and n(1−p) exceed 10, and exactly otherwise.
One look gives you the 5% you asked for. Ten looks roughly quadruples it, because each look is another chance to cross the line. This is why sequential testing exists — alpha spending, group-sequential bounds, always-valid inference. If a test is monitored continuously, a fixed-horizon p-value is not the right tool.
Assumptions worth stating, because they are usually the thing that's actually wrong:
- Visitors are independent and each counted once. Repeat visitors and shared devices break this.
- Conversion is binary. Revenue-style metrics are heavy-tailed and need a different test.
- The normal approximation needs a reasonable number of conversions per arm; the read-out warns when counts get too small for it.
- No correction for multiple metrics. Testing ten metrics at 5% gives you about a 40% chance of at least one false winner.
How this is built
Pure client-side: no API, no backend, no build step, no chart or stats library. The container serves static files from the Python standard library, so there are no dependencies to pin and nothing that can fail at container start. It runs scale-to-zero on Cloud Run and costs nothing while idle.
The statistics are implemented from scratch — the normal CDF via an Abramowitz–Stegun series, its inverse via Acklam's rational approximation, log-gamma via Lanczos for the Beta densities, and Beta sampling through the Marsaglia–Tsang gamma method. The Monte Carlo runs in the browser on click.
Both charts are hand-rolled SVG. The two-series palette was validated for colourblind separation and contrast against this background rather than picked by eye, the series are distinguished by a legend and direct labels as well as colour, and every plotted number is also in the read-out table.
Built by Man Patel — more work at manpatel.me/work.