what does the day-by-day view show

How the cumulative uplift and its interval changed from the first day to the last. The “By day” tab shows the day the result first became “significant” and whether significance survived to the end — this is what the peeking trap looks like. It also shows the novelty effect: in the example the first week gave +16.9% and the other three +2.9%. And the days with a traffic mismatch, where the split broke.

should I use a one-sided or a two-sided test

Two-sided if you need to notice both a rise and a drop, which is almost always. A one-sided test checks only one direction and is honest only if the direction was chosen before launch. Choosing it after the fact, once B is ahead, doubles the risk of a false win: from 5 to 10%. The calculator shows both: for a button at 2.00% vs 2.12%, two-sided p = 0.091, one-sided 0.046.

why is a +6% uplift not significant

Because the test is too small for such an effect. At a 2% conversion rate and 80,000 visitors per group, the uplift interval runs from −0.9% to +13.4%: it includes zero, so no effect is not ruled out. To catch +6% reliably with 80% power, you need about 220,000 visitors per group. Small effects need a great deal of traffic.

how do I compare more than two variants

Compare each variant with the control and adjust the p-values for the number of comparisons, by default with the Holm method. Without a correction, two comparisons almost double the chance of a random win. Example: three banners converting at 2.0, 2.2 and 2.1% on 40,000 views each — without a correction B is “significant” (p = 0.049), with it p = 0.097 and there is no winner.

what if there are few conversions

When a group has fewer than 10 conversions or an expected count below five, the normal approximation of the z-test is unreliable. The calculator then switches to Fisher’s exact test by itself and shows a “little data” badge. Example: a landing page with 10 leads of 200 against 22 of 200 — the z-test gives p = 0.027, Fisher 0.041. The conclusion is the same, but the uplift interval is wide: from +7% to +353%.

A/B tests · ResultA/B test statistical significance

A statistical significance calculator: visitors and conversions of A and B (up to 10 variants) → whether the result is significant, the uplift interval, the p-value and a summary for your manager. Step-by-step solution, Python and R code.

Your numbers stay in the browser and are never sent anywhere

01Data 02Numbers 03Charts 04Trust 05Examples 06Questions 07Related

α = 0.05 · power 80% · two-sided test

Daily data · 28 days · example

5 October 2026 — 1 November 2026 · date · visitors A · conversions A · visitors B · conversions B

Day by day

Cumulative uplift of B vs A by day

The line is the uplift, the band the 95 % interval. While the band crosses zero, the difference is not proven.

Day 5 (9 October 2026) — “it became significant”. Stopping the test here, you would have shipped B with an uplift that was gone by the end of the test.

All days together: A 1,563 / 78,280 · B 1,652 / 77,750 → uplift +6.4 % [−0.6 %; +14.0 %], p = 0.075

Novelty and mismatch

Novelty effect

Week 1

+16.9 %

Days 8–28

+2.9 %

The uplift dropped 5.9 times: people reacted to novelty, not to an improvement. Judge without the first week.

Uplift by day of the week

Mon+8 %

Tue+6 %

Wed+7 %

Thu+7 %

Fri+4 %

Sat+8 %

Sun+5 %

Traffic mismatch by daydays with a mismatch: 2

DateABB / Ap

5 OctMon2,8512,8510.0 %1

6 OctTue3,0263,030+0.1 %0.96

7 OctWed3,0883,081−0.2 %0.93

8 OctThu3,0063,004−0.1 %0.98

9 OctFri2,7962,798+0.1 %0.98

10 OctSat2,4572,462+0.2 %0.94

11 OctSun2,3462,343−0.1 %0.97

12 OctMon2,8512,8510.0 %1

13 OctTue3,0263,030+0.1 %0.96

14 OctWed3,0883,081−0.2 %0.93

15 OctThu3,0063,004−0.1 %0.98

16 OctFri2,7962,518−9.9 %1.4·10⁻⁴mismatch

17 OctSat2,4572,216−9.8 %4.2·10⁻⁴mismatch

18 OctSun2,3462,343−0.1 %0.97

19 OctMon2,8512,8510.0 %1

20 OctTue3,0263,030+0.1 %0.96

21 OctWed3,0883,081−0.2 %0.93

22 OctThu3,0063,004−0.1 %0.98

23 OctFri2,7962,798+0.1 %0.98

24 OctSat2,4572,462+0.2 %0.94

25 OctSun2,3462,343−0.1 %0.97

26 OctMon2,8512,8510.0 %1

27 OctTue3,0263,030+0.1 %0.96

28 OctWed3,0883,081−0.2 %0.93

29 OctThu3,0063,004−0.1 %0.98

30 OctFri2,7962,798+0.1 %0.98

31 OctSat2,4572,462+0.2 %0.94

1 NovSun2,3462,343−0.1 %0.97

Test data

VariantVisitorsConversionsConversion · CI 95 %

Acontrol2.00 %1.91…2.10 %

Bvariant2.12 %2.02…2.22 %

Not proven

The difference is not proven.

B may differ from A by anything from −0.9 % to +13.4 % — a zero effect is not ruled out. To catch a +6.0 % effect reliably (power 80%), you need ≈ 220,000 visitors per group — you have 80,000.

Test

A one-sided test gives p = 0.046 — significant.

Many calculators, ABTestGuide among them, use a one-sided test by default — so there the same test may come out “significant”.

The link will contain the visitor and conversion counts and the settings — share it deliberately.

A2.00 %

B2.12 %

interval where the true conversion lies

What to do next

1Continue to ≈ 220,000 per group or change the hypothesis
2The test cannot tell an effect smaller than +10 % — small tweaks need more traffic
3Do not switch to a one-sided test after the fact

Numbers

B vs A

Conversion uplift

+6.0 %CI 95 %: −0.9 % … +13.4 %

uplift

Conversion difference

+0.120 ppCI: −0.019 … +0.259 pp

Δ, pp

Probability of chance

0.091threshold α = 0.05

p, two-sided

Difference in standard errors

1.690significance threshold ±1.96

Standard error of the difference

0.00071for the interval, from separate rates

Needed per group for this effect

219,938to catch +6.0 % with 80% power

Charts

Where the true conversions may lie

If B is right of the threshold, the difference is significant

Conversion AConversion Bsignificance zone

Uplift vs the control

An interval crossing zero means the effect is not proven

Can you trust the test?

Trust

high so far

Traffic split fairly (SRM)auto

p = 1 for an equal split — no mismatch.

Enough eventsauto

At least 1,600 conversions per group — the approximation holds.

Did the test run for whole weeks?

Answer — trust depends on it.

Did you stop as soon as it became significant?

Answer — this is the most common mistake.

Did you look at more than one metric?

Answer if you looked at several.

Was the test size calculated in advance?

Answer — it affects how to read “not significant”.

Answer 4 more questions

Check in code

from statsmodels.stats.proportion import proportions_ztest

z, p = proportions_ztest([1696, 1600], [80000, 80000])
print(round(z, 4), round(p, 4))  # 1.6897 0.0911

Experiment report

Hypothesis, data, verdict, interval and diagnostics — for Confluence, Notion or print.

Hypothesis · optionalMetric · optional

Worked examples

What we see · “Buy” button

A +6% uplift looks convincing, but the interval includes zero: a two-sided test gives p = 0.091. The test is too small for such an effect — it needs ≈ 220,000 per group. A one-sided test would give p = 0.046, but choosing it after seeing the data is fitting.

Frequently asked questions

Significance tells whether chance can explain the difference between A and B. The calculator computes the p-value: if the versions did not differ, how often a difference this large or larger would appear. If p is below the chosen threshold α, usually 0.05, the difference is called statistically significant. Significance says nothing about the size of the effect — the uplift interval does.

Updated 31 October 2026