An online Bayesian A/B test: the probability that B beats A, the expected loss of choosing each variant and a credible interval for the uplift. Up to 10 variants, conversions or means, a uniform prior or one from past data, a stopping rule by loss threshold, and a comparison with the classical test on the same data. Python and R code.
Your numbers stay in the browser and are never sent anywhere
Visitors and conversions per variant, up to 10. The first is the control, A. Two variants are calculated exactly, more by Monte Carlo with 20,000 draws.
Order value, revenue per user, time on site: n, mean and standard deviation for two groups. The posterior is a normal approximation.
VariantVisitorsConversions
A
B
Prior distribution
Strength is how many visitors the past “weighs”: at 2,000 it pulls the estimate as much as 2,000 new observations.
You can stop the test when the expected loss of choosing the leader is below the threshold: how much you are willing to lose if it is actually worse.
n must be a whole number of at least 2, the mean a number, SD zero or more
You can stop the test when the expected loss of choosing the leader is below the threshold: how much you are willing to lose if it is actually worse.
02
Result
You can choose B.
The probability that B beats A is 95.4 %. If B turns out worse, we lose 0.0013 pp on average, below the threshold of 0.010 pp.
Probability that B beats A
95.4 %
exact formula
Loss if you choose B
0.0013 pp
0.06 % of B's conversion
Loss if you choose A
0.1213 pp
6.06 % of A's conversion
Uplift interval of B over A
−0.9 % … +13.4 %
credible interval, 95%
Posterior conversion of B
2.121 %
Beta(1,697; 78,305)
Prior
Beta(1; 1)
uniform
You can choose B.
The probability that B beats A is 97.9 %. If B turns out worse, we lose 0.37 on average, below the threshold of 5.00.
Probability that B beats A
97.9 %
exact formula
Loss if you choose B
0.37
0.012 % of the mean
Loss if you choose A
100.37
3.346 % of the mean
Difference B − A
3.88 … 196.12
+0.1 % … +6.5 %
A Bayesian result is easier to explain, but it does not cancel out peeking and SRM: if you stop at the first good number or traffic split unevenly, the conclusion is just as unreliable.
Where the true conversion of each variant most likely lies
Posterior distributions
Where the true mean of each group most likely lies
04
Comparison
Two-sided z-test: p = 0.091, so the difference is not proven. The Bayesian probability is close to 1 − p of the one-sided test (95.4%): both approaches see the same data but phrase the conclusion differently.
Bayes and the classical test on the same data
QuestionClassicalBayesian
What it answersHow unusual the data are if there is no effectHow likely it is that B is better
Decisionp < αLeader's loss < threshold
PeekingBreaks αAlso harmful, but less so
05
Check in Python / R
Check in Python / R
import numpy as np
rng = np.random.default_rng(7)
a = rng.beta(1601, 78401, 200_000)
b = rng.beta(1697, 78305, 200_000)
print((b > a).mean()) # P(B > A)
print(np.maximum(a - b, 0).mean()) # потери при выборе B
06
Examples
07
Frequently asked questions
A classical test answers how unusual the data are if there is no effect: that is the p-value. A Bayesian one answers how likely it is that B beats A and how much you could lose by choosing it. For 1,600 and 1,696 conversions of 80,000, the classical two-sided p = 0.091, “not proven”, while the Bayesian probability is 95.4%. There is no contradiction: these answer different questions about the same data.
The share of plausible states of the world in which B's true conversion is higher than A's, given your data and the chosen prior. For 1,600 and 1,696 of 80,000 it is 95.4%. It is not the probability of a big win: B may be better by hundredths of a percent. That is why the uplift interval and expected loss sit next to it.
How much conversion drops on average if the chosen variant is actually worse. It accounts for both the chance of an error and its size. In the button example, choosing B risks a loss of 0.0013 pp of conversion, and choosing A 0.121 pp. That makes loss handier than probability: it weighs a high chance of a tiny error against a small chance of a large one fairly.
The uniform Beta(1, 1) if there is no history: with thousands of visitors it barely affects the result. If there is, use the past conversion and a “strength” in visitors. A past 5% at strength 2,000 gives Beta(100, 1,900) and pulls the estimate towards 5%: for 50 and 62 conversions of 1,000, the probability that B is better drops from 87.7 to 75.8%.
More often than at a classical one, but not for free. If you look every day and stop at the first “95%”, the share of wrong decisions grows too, if more slowly than with p-values. Bayes does not protect against a traffic mismatch either: with SRM the data are spoiled for any method. Set the stopping rule, a loss threshold and a minimum duration, before the start.
When the expected loss of choosing the leader is below a threshold: how much you are willing to lose if it is actually worse, say 0.01 pp of conversion. In the button example the loss of B is 0.0013 pp, so you can choose it. With 50 and 62 conversions of 1,000 it is 0.063 pp, too early, although the probability is already 87.7%. Keep the test for a week or two anyway, for seasonality.