A/B tests · ResultsSRM check: sample ratio mismatch

0 days · no mismatch on any day

Sample ratio mismatch: the actual traffic split differs from the planned one. With a 50/50 plan and 50,800 against 49,200 the gap looks small, yet under a fair split it happens with a probability of 4·10⁻⁷. It signals a technical fault: some visitors of one group were lost or counted differently. The check is a chi-square test of the actual counts against those expected from the plan.

The check runs on every test, often every day. At 0.05 one fair test in twenty would raise a false alarm, and people would stop trusting the signal. A threshold of 0.001 catches real faults, whose p is usually far smaller, and rarely errs. Between 0.001 and 0.01 the calculator says “borderline”, a reason to go through the checklist of causes.

Most often a redirect or a slow variant: some people leave before analytics counts them. Next come caching and CDN, bots, a bug in group assignment, report filters, variants switched on at different times, return visits landing in the other group, and a variant broken in one browser. The checklist on the page walks through all eight.

No, not until you find the cause. A mismatch means the groups differ by more than the page version: for example, slow phones that never finished loading are missing from B. The conversion gap may then come from that rather than from the change. Find and fix the cause, then rerun the test; dropping “extra” visitors from the larger group does not help.

Enter the planned shares next to the visitor counts, in percent or any units — the calculator normalises them. With a 20/80 plan and 20,600 against 79,400 the expected counts were 20,000 and 80,000; chi-square is 22.5 and p = 2·10⁻⁶, a mismatch. The deviation is only three percent of the expected count, but on a hundred thousand visitors it cannot be chance.

Every day, especially early in the test: an assignment fault is easier to catch before it burns all the traffic. A mismatch check is not peeking: it looks at visitor counts, not conversion, and does not affect the verdict on the winner. Paste visitors by day and the calculator marks the days when the mismatch began.