An MDE calculator: which effect your A/B test can see with your traffic and timeline. Absolute and relative MDE, a plain-language verdict, a week-by-week table, the formula and Python and R code.
Your numbers stay in the browser and are never sent anywhere
from math import sqrt
from scipy.optimize import brentq
from scipy.stats import norm
p1, n = 0.02, 2000
alpha, power = 0.05, 0.8
z_a, z_b = norm.ppf(1 - alpha / 2), norm.ppf(power)
def need(p2): # выборка на группу для p1 → p2
pb = (p1 + p2) / 2
return (z_a * sqrt(2 * pb * (1 - pb)) + z_b * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
p2 = brentq(lambda p: need(p) - n, p1 * 1.000001, 0.999999)
print(p2, p2 / p1 - 1) # 0.034408
04
Examples
05
Frequently asked questions
MDE, the minimum detectable effect, is the smallest change a test reliably notices at the chosen α and power. Effects below the MDE are likely to be missed, and the test reports “not significant”. For example, at 2% conversion, 1,000 visitors a week and 4 weeks the MDE is +72%: conversion has to reach 3.44% for the test to see it with 80% power.
It depends on what you test. For copy, colour or a button, real effects are 1–5%; for a new offer, a price or a redesigned page, 10–30%. If the test’s MDE is above +30%, it will see only very large changes, and small tweaks stay invisible. The calculator says so in its verdict: a sensitive test, good for noticeable changes, or sees only big ones.
More traffic or time, fewer variants, CUPED or a less noisy metric. MDE falls slowly: it is inversely proportional to the square root of the sample, so halving it takes roughly four times the visitors. At 2% conversion and 1,000 visitors a week, a test sees +72% in 4 weeks and about +40% in 12. The week-by-week table shows what each extra week buys.
It is one effect written two ways. Absolute is in percentage points: conversion growing from 2% to 2.3% is +0.3 pp. Relative is a share of the baseline: the same shift is +15%. A/B testing usually talks about the relative one because it compares easily between pages with different conversion rates. The calculator shows both side by side.
Most often the real effect was smaller than the MDE, and the test could not see it. “Not significant” does not mean “no effect” — it means the sample was too small to tell it from noise. Work out the MDE for your traffic and duration: if it is +40% and you hoped for +5%, the test was doomed from the start. That is a reason to plan the sample before launch.
The longer the test, the more visitors and the lower the MDE, but the gain is slow. At 2% conversion and 1,000 visitors a week, one week sees only very large shifts, 4 weeks see from +72%, 12 weeks about +40%. The reverse question — how long to wait for a given effect — is answered on the sample size page.