Contents · 9 sections

01Peeking

What people do

They start a test and check the p-value every day. As soon as it drops below 0.05, they declare a win.

Live demonstration
Checks per test
26%
false wins in an A/A test with 20 checks: the groups do not differ at all
0% · line is α 5%30%
Each extra check is one more chance to catch noise. The numbers come from the Sequential test.Conditions: baseline 10%, 500 visitors a day per group, 20 days, α = 5%, 5,000 A/A tests for each number of checks.
Open in the Sequential test →
How to do it right

Decide in advance when to look. Either a fixed duration with one check at the end, or a sequential test with a stopping rule: group boundaries or the simple rule.

02Stopping at the first significance

What people do

On day six the test showed significance, so they stop it and ship B.

Live demonstration
10.10.01threshold 0.05d1d7d14d21d28day 6 · p = 0.04 → “win”day 28 · p = 0.41
Cumulative p-value by day in a test with no real difference. On day 6 it fell to 0.04, and by day 28 it was back at 0.41. The series is an illustration.
Significance by day →
How to do it right

Significance in the middle of a test is not a result. Wait for the sample calculated in advance; if you need to stop earlier, use a sequential test.

03A partial week

What people do

They start the test on Monday and sum up on Friday: “it's obvious already”.

Live demonstration
+9%Mon
+8%Tue
+7%Wed
+6%Thu
+8%Fri
−10%Sat
−12%Sun
B's lift over A by day of the week; red with hatching means B loses. A Monday to Friday test would show +7.6%, the full week +2.3%. The series is an illustration.
Test duration →
How to do it right

Keep a test to whole weeks: 7, 14 or 28 days. The duration calculator rounds up to weeks by itself.

04An unfair traffic split (SRM)

What people do

They look only at conversions and do not check whether traffic split evenly.

Live demonstration
4·10⁻⁷
p-value of the split check: 50,800 and 49,200 visitors with a 50/50 plan
A skew of only 0.8 pp (50.8% instead of 50%), but over 100,000 visitors chance almost never produces one (χ² = 25.6). The traffic split is broken, so the result cannot be trusted.
SRM check →
How to do it right

Check SRM in the first days and at the end. At p < 0.001 look for the cause (a redirect, a cache, bots) and do not use the result.

05Many metrics without a correction

What people do

They watch 20 metrics and celebrate the one that lit up.

Live demonstration
Metrics
64.2%
the chance that at least one of 20 metrics lights up when there is no effect anywhere
0% · line is 5%100%
1 − 0.95^20 = 0.642.
Multiple comparisons →
How to do it right

One primary metric, chosen in advance. For the rest, a multiple comparisons correction (Holm or Benjamini–Hochberg).

06Effect dilution

What people do

They change the payment page but analyse everyone in the test, even those who never reached payment.

Live demonstration
See the change
×4.7
how many times more traffic you need if you analyse everyone assigned rather than only those who saw the change
Base 2%, effect +15% among those who saw the change. If 20% see it, across everyone assigned the effect is diluted to +3.0%.
Sample size with dilution →
How to do it right

Analyse only those who saw the change: “triggered analysis”. The condition: “saw it” is defined the same way in A and B.

07A small test for a small effect

What people do

They run a test on modest traffic “for a month” and expect to find +5%.

Live demonstration
+72%
the smallest effect such a test can detect at all
≈ 143 visitors a day, 2% conversion, 4 weeks: 2,000 per group. Real improvements are usually +2…+10%, so this test will not see them.
Minimum effect →
How to do it right

Calculate the minimum detectable effect first. If it exceeds a realistic one, test bolder changes or do not test at all.

08Significant but useless

What people do

They see p < 0.05 and ship without looking at the effect size.

Live demonstration
+0.3%
the lift is significant, p = 0.01, but the interval is +0.07…+0.53%
−1%+3%
The bar is the effect interval, the red line is the usefulness threshold of +2%: anything less does not pay for development. The whole interval is below the threshold.
Confidence interval →
How to do it right

Decide in advance which effect pays for the change, and compare the whole interval with it, not just the p-value.

09Checklist

Before launch

During

After

Ticked 0 of 15 · ticks are not saved or printed

Idea: Evan Miller, “How Not To Run An A/B Test” (evanmiller.org). The text, examples and mistakes 3–8 are our own.

Frequently asked questions

As long as the sample calculated in advance requires, and at least a full week. Round the duration to whole weeks: behaviour differs on weekdays and weekends, and a partial week shifts the result. Work out the duration before launch, not when you get tired of waiting.
All guides
A/B testing: how to run a testSeven steps from hypothesis to conclusion, a hypothesis template, 12 common mistakes and a glossary8 minHow many of your A/B test wins are falseWhy every third win is false at α = 5%: a grid of 100 tests, a table and a simulation8 min
seqSequential test/tools/ab-test/sequentialSRMSRM check/tools/ab-test/srm×Multiple comparisons/tools/ab-test/multiple-comparisonsnSample size/tools/ab-test/sample-size