how does an A/B test differ from A/B/n and multivariate tests

An A/B test has two variants: the control and one change. An A/B/n test compares several variants with the control at once, say three headlines, so it needs a multiple-comparisons correction and more traffic. A multivariate test, MVT, changes several elements at the same time and tests every combination: 2 headlines × 2 images × 2 buttons make 8 groups. It reveals interactions but needs a great many visitors.

how do I write an A/B test hypothesis

Use the template “If we [change], then [metric] grows by [amount], because [reason]”. For example: if we add reviews to the product card, then purchase conversion grows by 10%, because shoppers hesitate less. The amount feeds the sample size calculation, and the reason makes even a losing test teach you something. The guide has a builder: fill in three fields and copy the sentence.

which metric should an A/B test use

One primary metric that the decision rests on. It should be close to money yet frequent enough: purchase conversion rather than revenue per year. Secondary metrics explain why the primary one moved. Guardrail metrics — load speed, bounces, returns, complaints — must not drop even if the primary metric grows. Fix the metrics before the start: one chosen afterwards is just fitting the data.

what is an A/A test and why run one

It is a test of two identical versions. There is no real difference between them, so it tests the system itself: whether traffic splits fairly and whether the metrics are counted correctly. At a 5% significance level about 5% of such tests should come out “significant”, and none should show a traffic mismatch. If false wins are clearly more frequent, the split or the counting is broken, and ordinary tests cannot be trusted yet.

what are the most common A/B testing mistakes

Peeking and stopping at the first significance: if you check the result every day, about 24% of tests produce false wins instead of 5%. Then come partial weeks, traffic mismatch, twenty metrics without a correction and a test too small for a small effect: +6% on a 2% conversion needs about 220 thousand visitors per group. All 12 mistakes with numbers and links to calculators are in the section above.

A/B tests · GuideA/B testing: how to run a test

An A/B test, also called a split test, is an experiment that splits traffic into groups and compares versions of a page. Here are 7 steps, a hypothesis template, 12 common mistakes and a glossary of 25 terms — with calculators built in.

Your numbers stay in the browser and are never sent anywhere

01What it is 027 steps 0312 mistakes 04Glossary 05Tools 06Questions 07Related

What an A/B test is

A/B

A/B test

Traffic is split at random into two groups: A sees the current version, B the changed one. One metric is compared.

Example: A green “Buy” button against a red one.

A/B/n

A/B/n test

Several variants against one control. Needs a multiple-comparisons correction and more traffic.

Example: Three landing headlines against the current one.

A/A

A/A test

Two identical versions. Tests the system: the split, the metrics and the false-win rate — it should be about 5%.

Example: Before the first test on a new platform.

MVT

Multivariate test

Several elements change at once and every combination is tested. Shows interactions but needs a lot of traffic.

Example: 2 headlines × 2 images × 2 buttons = 8 variants.

A test in 7 steps

Hypothesis
One sentence: what changes, which metric moves and why. A hypothesis without a “because” is a lottery, not a test.
If wethenbecause
If we add reviews to the product card, then purchase conversion grows by 10%, because shoppers hesitate less.
Metric
One primary metric decides. Secondary metrics explain why it moved. Guardrails — speed, bounces, complaints — must not drop even if the primary one grows.
MDE and sample size
What is the smallest gain worth shipping? The smaller it is, the more visitors you need: half the effect takes four times the traffic.
Conversion, %
Gain, +%
Per group
36,693
α = 0.05 · power 80% · two-sided
Sample size
Duration
Sample ÷ traffic, rounded up to whole weeks and at least one week. Set the end date before the start.
Launch and SRM
On day one, check that traffic splits as planned. A mismatch signals a technical fault, and then the results cannot be trusted.
Analysis
Only after the planned end date. Look at the uplift interval, not just the p-value, and go through the “Can you trust the test?” checklist.
Significance
Decision and report
Ship, reject or rerun with another hypothesis. Write down the hypothesis, data, conclusion and caveats — in six months it will save an argument.
Experiment report

12 common mistakes

01Peeking

20 checks instead of one raise false wins from 5% to about 24%.

02Stopping at the first significance

“Significant on day 5” — by the end the uplift was gone. Noise got shipped.

Significance

03A partial week

Monday to Thursday: weekdays give +8%, weekends −3%, the result is biased.

04Traffic mismatch

50,800 / 49,200 with a 50/50 plan — p = 4·10⁻⁷. The result cannot be trusted.

05Many metrics, no correction

With 20 metrics at least one is “significant” in 64% of tests even with no effect.

06A small test for a small effect

+6% on a 2% conversion needs 220,000 per group. With 80,000 the result is a coin toss.

Sample size

07Novelty effect

+17% in week one and +2.5% after: people reacted to new, not to better.

Significance

08Seasons and sales

A test during a sale: a different audience and motivation, the conclusion does not carry over to normal days.

09Different audiences in the groups

B was shown only on mobile — you compared devices, not page versions.

10Changing the test midway

B's layout was fixed on day 10 — one test now holds two hypotheses.

11Revenue means with outliers

One 500,000 order decides the test. You need winsorizing or a bootstrap.

12“Significant but useless”

+0.3% on a million visitors is significant but does not pay for the work. Look at the effect size.

Glossary

25 terms

A/B/n test: Several variants against one control. Needs a multiple-comparisons correction.
Multivariate test (MVT): Several elements change at once and every combination is compared. Needs a lot of traffic.
Null hypothesis: The assumption that the versions do not differ. The test looks for data that fit it poorly.
Significance level α: The accepted share of false wins, usually 5%. The threshold the p-value is compared with.
Type I error: A false win: the test found an effect that is not there. Its rate is set by α.
Type II error: A miss: the effect is real but the test did not see it. Its rate is 1 − power.
Standard error: How much an estimate wobbles from sample to sample. Shrinks as 1/√n.
z statistic: The difference measured in standard errors. Above 1.96 in absolute value is significant at α = 0.05.
Novelty effect: A short-lived spike of interest in something new. First-week uplift often fades.
Practical significance: Whether the effect is big enough to pay for the change. Statistical significance does not promise that.
Bootstrap: Resampling your own data with replacement. Gives an interval without assuming normality.
Delta method: A way to get the variance of a ratio metric, such as revenue per order, at the user level.
Bayesian test: Answers how likely B is to beat A and how much you could lose by choosing it.
A/A test: A test of two identical versions. Checks that the split and the metrics are honest: about 5% of such tests should come out “significant”.
MDE: The smallest effect the test will reliably detect at the chosen power. Smaller effects will be missed.
p-value: If the versions did not differ, a difference this large or larger would show up in p·100% of cases. It is not the probability that B is better.
Power: The probability of detecting an effect that is really there. Usually 80%: one real effect in five is missed.
Confidence interval: The range where the true effect most likely lies. If it includes zero, the difference is not proven.
SRM: Sample ratio mismatch: the groups did not get the planned shares. A sign of a technical fault — the results cannot be trusted.
Control: The current version the variants are compared with. Labelled A.
Uplift: The relative change of the variant against control: (p_B − p_A) / p_A.
Peeking: Checking the result every day and stopping at the first significance. False wins rise to 20–30% instead of 5%.
CUPED: Adjusting the metric for each user's behaviour before the test. Cuts noise and the sample you need.
FDR: The share of false discoveries among all “significant” results. The Benjamini–Hochberg correction keeps it in check.
Guardrail metric: A metric that must not drop — speed, bounces, complaints — even when the main metric grows.

What to run a test with

Varioqub

Yandex's experiment platform: traffic split, flags and reports on Metrica metrics.

Google Optimize

Shut down in September 2023. Mentioned for history: many old guides still point to it.

Your own setup

Flags in code and a deterministic split by user ID — for instance by a hash of the ID. Compute the metrics here — the data never leaves the browser.

Frequently asked questions

An A/B test, or split test, is an experiment: visitors are split at random into groups, one sees the current version of a page and the other a changed one. After a duration calculated in advance, one metric such as purchase conversion is compared, and statistics tells whether the difference is chance. The random split matters: only it guarantees the groups differ by version alone, not by audience, time or channel.

Updated 31 October 2026