A/B tests · AdvancedPeeking and sequential testing
Peeking at an A/B test: a simulation shows how many false wins daily checks produce, about 25% instead of 5% with 20 looks. An interim look plan with O’Brien–Fleming, Pocock or Lan–DeMets boundaries, the corridor, sample size growth and a stop-or-continue decision at the current look. Python and R code.
Your numbers stay in the browser and are never sent anywhere
An A/A test: the versions are identical and there is no effect. We look at the result every day and stop at the first significant one.
How many looks
Runs
seed 1 · 5,000 A/A tests · 20 days
Peeking inflates the error 4.8 times.
There is no effect, but if you look 20 times and stop at the first significant result, a “win” turns up in 25.6 % of tests instead of 5%. Every look is one more chance for a random spike.
Looked 20 times
25.6 %
false wins
One look at the end
5.3 %
false wins, just as α promises
The more often you look, the more false wins
Looked once5.3 %
Looked 2 times8.7 %
Looked 5 times14.8 %
Looked 10 times20.1 %
Looked 20 times25.6 %
02
Interim look plan
Looks K
Boundaries
Strict boundaries early and nearly 1.96 at the end: you can stop early only for a very large effect. The sample grows the least.
One boundary at every look: easier to stop early, but the last look is noticeably stricter than 1.96 and the sample grows more.
An O’Brien–Fleming-type α spending function: looks can happen at any share of the sample, for example at 30% and 70%.
Comma-separated, ascending, the last one 100. For example, 30, 70, 100.
α = 0.05 · power 80% · two-sided test
Significance level α
Power
Choose it only if you decided in advance that B cannot be worse and a negative result does not interest you.
First-look boundary
4.562
z, absolute
Last-look boundary
2.040
a fixed test uses 1.96
The sample grows by
+2.8 %
over a fixed test at 80% power
Corridor
z leaving the corridor at look k means stopping with a win or a loss
LookDataBoundary zp thresholdα spent
120 %4.562< 0.0010.0000
240 %3.2260.00130.0013
360 %2.6340.00840.0089
480 %2.2810.0230.0256
5100 %2.0400.0410.0500
03
Check now
Stop: B is better.
At look 3 of 5, z = 2.652 crossed the boundary ±2.634. Uplift +15.0 %.
Modern methods
mSPRT and “always valid” p-values let you look at the result whenever you like without inflating the error: the boundary grows with every new observation. This is how sequential reports in large experimentation platforms work. The price is slightly more data than a fixed test needs.
04
Check in Python / R
Check in Python / R
import numpy as np
from scipy import stats
# A/A-тесты: 5000 прогонов, 20 дней, смотрим каждый день
rng = np.random.default_rng(1)
base, per_day, runs = 0.1, 500, 5000
a = rng.binomial(per_day, base, (runs, 20)).cumsum(1)
b = rng.binomial(per_day, base, (runs, 20)).cumsum(1)
n = per_day * np.arange(1, 21)
pp = (a + b) / (2 * n)
z = (b - a) / n / np.sqrt(pp * (1 - pp) * 2 / n)
print((np.abs(z) > stats.norm.ppf(0.975)).any(1).mean()) # ложных побед при 20 проверках
05
Examples
06
Frequently asked questions
Every look at the result is one more chance to catch a random spike. If you stop at the first significant result, you pick the luckiest moment of the noise, not a real effect. The p < 0.05 threshold promises 5% false wins only for a single look at the end; with daily looks that guarantee no longer holds, even though each number looks honest.
In an A/A test, where there is no effect, 20 daily looks find a “win” in about 25% of tests instead of 5%: a simulation of 20,000 runs gives 24.9%. Five looks give about 15%, two about 9%. The simulation above recalculates this for your conversion and traffic, and “Run again” changes the seed.
A z threshold for each interim look, set before the start. It is very strict early and almost ordinary at the end: for 5 looks the first boundary is 4.562 and the last is 2.040 instead of 1.96. You can stop early only for a very large effect, but the overall error stays at 5% and the sample barely grows.
Usually 2 to 5. More looks let you stop earlier for a strong effect, but the sample and the complexity grow, and the gain barely changes after five. It is convenient to tie the looks to the weeks of the test. If the data shares are unequal, say 30% and 70%, use Lan–DeMets boundaries, which you can set here too.
For O’Brien–Fleming with 5 looks, by about 2.8% over a fixed test at 80% power, and by 0.8% with 2 looks. Pocock grows more noticeably: by 22.8% with 5 looks. In return, Pocock lets you stop early more often. The growth is measured against the sample of an ordinary test with the same α and power.
The mixture sequential probability ratio test: its boundary grows with every new observation, so you can look at the result whenever and as often as you like without inflating the error. It underlies the “always valid” p-values in the sequential reports of large experimentation platforms. The price is slightly more data than a fixed test needs.