bringer.ru
ruen

A/B tests · PlanningA/B test sample size

A sample size calculator: how many visitors each group needs for the test to detect the effect. For conversion and average order value, unequal groups, up to five variants, CUPED, a sensitivity table, the formula and Python and R code.

Your numbers stay in the browser and are never sent anywhere

01Calculation02Curve03Sensitivity04Formula05Examples06Questions07Related
01

Calculation

Metric
The control’s conversion now, before the test
Effect to detect (MDE)
Variants
Traffic split
CUPED — a pre-test metric exists
Cuts noise using each user’s history
α = 0.05 · power 80% · two-sided test
Visitors needed per group
36,693

Total 73,386 · 2 equal groups · effect +15 % (from 2.00 to 2.30%)

of them ≈ 734 will convert in A and 844 in B

02The smaller the effect, the larger the sample
Visitors per group (log scale) against the relative effect
10,000100,000+5%+10%+20%+30%+40%36,693
03Sensitivity table · per group
EffectPower 80%Power 90%
+5 %315,206421,971
+10 %80,682108,009
+15 %36,69349,121
+20 %21,10928,258
+30 %9,79813,116

The row with your effect is highlighted. Half the effect needs roughly four times the visitors.

04Formula
n = (z1−α/2 · √(p̄q̄(1 + 1/k)) + z1−β · √(p1q1 + p2q2/k))² / (p2 − p1)²
p₁ = 0.0200, p₂ = 0.0230, k = 1, z = 1.960 and 0.842 → n = 36,693

z₁₋α/s is the critical value for α and the test direction, z₁₋β the one for power, k how many times group B is larger than A. The result is rounded up for each group: falling short by even one visitor drops power below the target.

Check in Python / R
from math import ceil, sqrt
from scipy.stats import norm

p1, p2 = 0.02, 0.023
alpha, power, k = 0.05, 0.8, 1
z_a = norm.ppf(1 - alpha / 2)
z_b = norm.ppf(power)
pb = (p1 + k * p2) / (1 + k)
n = (z_a * sqrt(pb * (1 - pb) * (1 + 1 / k))
     + z_b * sqrt(p1 * (1 - p1) + p2 * (1 - p2) / k)) ** 2 / (p2 - p1) ** 2
print(ceil(n))  # 36693
05

Examples

06

Frequently asked questions

You need four numbers: the control’s baseline conversion, the effect you want to catch, the significance level α and the power. The calculator uses the formula of R’s power.prop.test and rounds up for each group. Example: 2% conversion, an expected +15% uplift, α = 0.05 and 80% power need 36,693 visitors per group, 73,386 in total. A one-sided test needs 28,903 per group.
нед.

Test duration

How many weeks to wait and what it costs
MDE

MDE

The smallest effect the test can see
p

Statistical significance

Is it significant, the uplift interval
7

A/B testing guide

7 steps, 12 mistakes and a glossary
A/B

A/B test calculator

Sample size, significance, test picker

Updated