Contents · 6 sections
01The paradox: 5% errors, yet a third of wins are false
A team runs 40 tests a year and checks them fairly: α = 5%, 80% power. It seems that one win in twenty is wrong. In fact it is every third win, if only one idea in ten works.
Change the numbers and the conclusion recalculates.
Of 40 tests a year: about 5.0 expected wins, about 1.8 of them false.
02Where it comes from: few working ideas
Take 100 tests. Ten work, and the test catches 8 of them. Ninety do not work, and at α = 5% another 4–5 of them “win” by chance. For 8 real wins there are 4–5 false ones: non-working ideas outnumber working ones nine to one.
03What the share depends on
Most of all on the share of working ideas. Test settings help, but less:
| Ideas that work, π | α 5%, power 80% | α 1% | power 90% |
|---|---|---|---|
| 5% | 54.3% | 19.2% | 51.4% |
| 10% | 36.0% | 10.1% | 33.3% |
| 20% | 20.0% | 4.8% | 18.2% |
| 30% | 12.7% | 2.8% | 11.5% |
| 50% | 5.9% | 1.2% | 5.3% |
Table 1. The π = 10% row is the article's example.
04Check with a simulation
You need not take the formula on trust: let's run 1,000 tests. In 100 the effect is real, in 900 there is none. Each test is checked at α = 5% and 80% power.
A run of 1,000 tests, 100 with a real effect, seed 8: 118 wins, 44 of them false, 37%.
Each run uses a new random seed, so the share wanders around 36%. The printed page shows one run.
05What to do
- Repeat the winning tests. With two wins in a row, about 3% are false.
- α = 1%, when you have enough traffic. The sample grows about one and a half times, and there are three times fewer false wins.
- More power. Work out the sample in advance with the sample size calculator.
- Estimate π from your test log: π ≈ (share of wins among all tests − α) / (power − α). With 13% wins, α = 5% and 80% power that is about 10%.
06Calculate your own share
The calculator on the A/B testing page opens with the numbers from the calculation above; enter your own α, power and number of tests.
Idea: Evan Miller, “The Low Base Rate Problem” (evanmiller.org). The text and calculations are our own.