A/B tests · ResultsOrder value and revenue in an A/B test
How to compare order value and revenue in an A/B test: Welch's t-test on summary data, a bootstrap on revenue per user, winsorizing outliers, Mann–Whitney as a check and an ARPU breakdown into conversion and order value. For revenue per order, the delta method with a bootstrap check. Python and R code.
Your numbers stay in the browser and are never sent anywhere
n, mean and standard deviation per group, from a report or an export. Welch's t-test is used: group variances do not have to match.
Revenue per user as a column, from Excel or an export, with zeros for users who did not buy. Adds a bootstrap, Mann–Whitney as a check and a revenue breakdown.
Revenue per order when users were randomized: one line per person, “revenue; orders”. The delta method gives the variance, a bootstrap checks it.
A
B
n must be a whole number of at least 2, the mean a number, SD zero or more
Outliers: winsorizing
Log transform ln(1 + x)
Enter at least three users per group with more than zero orders
α = 0.05 · two-sided test
Significance level α
Choose it only if you decided in advance that B cannot be worse and a negative result does not interest you.
B brings in more.
The mean in B is higher by 100 (+3.3 %), likely between +4 and +196; p = 0.042.
Difference of means
+100
CI: +4 … +196B − A
Uplift
+3.3 %
over 3,000 in Auplift
Probability of chance
0.042
Welch's t-testp
t statistic
2.039
df = 3981.5t
Standard error
49.0
SE
Degrees of freedom
3981.5
Welch–Satterthwaitedf
The difference is not proven.
The mean in B is higher by 24,905 (+2024.8 %), likely between −27,303 and +77,113; p = 0.33, so a zero difference is not ruled out.
The data contain an outlier of 500,000: on its own it shifts the mean and inflates the spread. Turn on 99% winsorizing.
Difference of means
+24,905
CI: −27,303 … +77,113B − A
Uplift
+2024.8 %
over 1,230 in Auplift
Probability of chance
0.33
Welch's t-testp
t statistic
0.998
df = 19.0t
Standard error
24944.1
SE
Mann–Whitney as a check
0.85
robust to outliersp
B brings in more.
Revenue per order in B is higher by 164 (+12.5 %), likely between +34 and +294; p = 0.014.
Difference of ratios
+164
CI: +34 … +294B − A
Uplift
+12.5 %
over 1,312 in Auplift
Probability of chance
0.014
delta method, z-testp
z statistic
2.465
per userz
Standard error
66.5
SE
Revenue per order A / B
1,312 / 1,476
12 and 12 usersR
Revenue per order = revenue / orders, but users were randomized. Orders of one person are related, so a naive t-test over orders understates the spread and finds an “effect” more often. The delta method computes the variance at the user level.
02
Bootstrap
A bootstrap needs per-user values: choose “Raw” or “Ratio”.
03
Revenue breakdown
Revenue breakdown
ARPU = conversion × buyer's order value
Revenue breakdownABΔ
Purchase conversion50.0 %50.0 %0.0 %
Buyer's order value2,46052,270+2024.8 %
ARPU1,23026,135+2024.8 %
Revenue per user moved mostly because of the buyer's order value, not the share of buyers.
The breakdown needs per-user values with zeros: choose “Raw”.
With summary data, use Welch's t-test: you need the number of buyers, the mean and the standard deviation in each group. For 3,000 ± 1,500 against 3,100 ± 1,600 with 2,000 buyers each, t = 2.04, p = 0.042, and the difference is between 3.9 and 196.1. If you have revenue for every user, add a bootstrap: it does not assume normality.
Winsorize them: set values above the 99th percentile of the pooled sample to the threshold. In the “whale” example, one order of 500,000 inflates B's mean 21-fold, yet the t-test finds nothing: p = 0.33. After winsorizing, one value is clipped, the uplift is +8.9% and p = 0.81: there is no difference, and that is the honest answer.
Resampling your own data: 10,000 times we draw users with replacement in each group and compute the difference of means. The spread of those differences gives an interval without assuming a normal distribution, which matters for revenue with a long tail. The seed is fixed, so running it again gives the same interval.
A metric such as revenue per order when users, not orders, were randomized. The variance of such a ratio cannot be computed over orders, because those observations are not independent. The delta method linearizes the ratio and computes the variance at the user level. In the example, revenue per order grew by 12.5%, p = 0.014, and a bootstrap over users gives the same interval.
Orders of one person are related: someone who buys often also spends in their own way. Treating every order as a separate observation makes the t-test see more independent data than there is, understate the spread and find a non-existent effect more often. The unit of analysis has to match the unit of randomization, the user.
ARPU is revenue per user, including those who bought nothing: ARPU = purchase conversion × the buyer's order value. Decisions about money are made on ARPU, while order value explains why it moved. The breakdown on this page shows both factors and says which of them produced the uplift.