Analytics · A/B testsA/B test calculator

Enter visitors and conversions for both versions under “Check a result”. If the p-value is below the significance level, usually 0.05, and the uplift interval does not include zero, the difference is significant. Example: 1,600 purchases out of 80,000 against 1,696 out of 80,000 is 2.00% against 2.12%, a +6% uplift, but p = 0.091 and the interval runs from −0.9% to +13.4%. The effect is not proven yet.

It depends on the baseline conversion and on the effect you want to detect. At a 2% conversion and a +15% uplift you need 36,693 visitors per group at a 0.05 significance level and 80% power. If you only expect +6%, it takes 219,938 per group: half the effect needs roughly four times the traffic. Try your own numbers under “Plan a test”.

As long as it takes to collect the sample, but always in whole weeks and at least one: people behave differently on weekdays and weekends. Two groups of 36,693 make 73,386 visitors; at 5,000 a day that is 14.7 days, rounded up to 15 and then to three whole weeks. At 1,000 visitors a week the same test takes 74 weeks — better redesign it for a bigger effect.

It answers one question: how often would we see a difference this large or larger if the versions were really the same. p = 0.03 means “in 3% of cases”. The smaller p is, the worse chance explains the result. But a p-value is not the probability that B beats A, nor the size of the effect: with a million visitors even a useless 0.3% uplift turns out significant.

Only by a plan of interim checks written before the start, with stricter boundaries for the early looks. If you look at the result every day and stop at the first significance, about 24% of tests produce false wins instead of the promised 5%: random swings cross the threshold sooner or later. Otherwise wait for the planned end date and look at the result once.

Nowhere: every calculation runs in the browser, and visitors, conversions and results are never sent to a server. The page address does not change as you type either, so the numbers end up neither in browser history nor in visit statistics. That is also why recalculation is instant: the page computes on its own and waits for no server. Close the tab and what you entered is gone.