An A/B test simulator: thousands of virtual tests with a known truth show how often a test finds a real effect, how many false wins an A/A test, peeking and 20 metrics produce, and how much a “win” overstates the uplift, the winner's curse. Five scenarios, the p-value path of one test, CSV and code.
Your numbers stay in the browser and are never sent anywhere
With several metrics, none of them has an effect: B equals A.
Runs
seed 1 · 1,000 tests
α = 0.05 · two-sided test
Significance level α
Choose it only if you decided in advance that B cannot be worse and a negative result does not interest you.
The test finds the effect in 82 % of cases.
The power is sufficient. Among the “wins”, the mean observed uplift is +16.9 %, 1.1 times the true one: this is the winner's curse.
Win rate (power)
81.5 %
815 of 1,000
True effect
+15.0 %
2.00 → 2.30 %
Mean uplift among wins
+16.9 %
overstated 1.1 times
Visitors per group
37,800
2,700 × 14 days
Checks per test
1
at the end
Metrics
1
one primary
02
Observed uplift across all tests
Observed uplift across all tests
Red marks tests found significant; the dashed line is the true effect
03
One test day by day
One test day by day
p-value after each day; below the red line means “significant”
In this test the p-value dropped below α on day 11 and stayed there until the end of the test.
04
Check in Python / R
Check in Python / R
import numpy as np
from scipy import stats
rng = np.random.default_rng(1)
pa, pb, per_day, days, runs = 0.02, 0.023, 2700, 14, 1000
def z(a, b, n):
p = (a + b) / (2 * n)
return (b - a) / n / np.sqrt(p * (1 - p) * 2 / n)
# одна проверка в конце: доля побед и проклятие победителя
n = per_day * days
a = rng.binomial(n, pa, runs)
b = rng.binomial(n, pb, runs)
win = np.abs(z(a, b, n)) > stats.norm.ppf(0.975)
up = b / a - 1
print(win.mean(), up[win & (up > 0)].mean())
05
Scenarios
06
Frequently asked questions
To see how often a test is wrong with your parameters. A thousand virtual tests with a known truth take a fraction of a second and show the power, the false win rate and the cost of peeking. Formulas say the same thing, but a histogram of a thousand results convinces faster: you see how differently one and the same effect can look.
A test of two identical versions. In the simulator it is the “no effect” scenario: at α = 0.05 about 5% of tests still show a “significant” difference, and 10,000 runs give 5.0%. In practice an A/A test checks that traffic splitting and metric counting work: noticeably more than 5% wins or unbalanced groups mean there is a bug to find.
Among tests found significant, the observed uplift is systematically larger than the true one: the ones that pass are those lucky with the noise. At 82% power the overstatement is small, 17.0% instead of 15%. A small test for a small effect finds it in 6% of cases, and among those “wins” the mean uplift is 44.5% instead of 5%, almost nine times larger.
p < 0.05 happens without an effect too, in 5% of tests: that is the price of the threshold. Beyond that, the rate grows with how the test is run: checking every day for 28 days gives about 28% false wins, and 20 metrics without a correction give 64%. The simulator shows all three cases on the same parameters.
That is the power. If the sample is sized for the effect, about 80%: the “proper test” scenario, +15% with 2,700 visitors a day per group for 14 days, finds the effect in 82% of runs. A test for +5% at 300 a day over the same 14 days finds it in 6% of cases, almost as rarely as a false win.
A single well-planned one, broadly yes: the sample was sized in advance, you looked once at the end, and there is one primary metric. Yet even such a test finds a non-existent effect in 5% of cases and overstates a real one. Important decisions are worth confirming with a repeat test or by watching the metric after rollout.