A/B tests · ResultsPoisson rate comparison
An online Poisson test: compare event counts over different periods, such as tickets, orders or errors. An exact conditional test, the rate ratio and difference with intervals, and a daily check for bursts of events.
Your numbers stay in the browser and are never sent anywhere
01
Calculator
Try
Period / groupEventsLength
A
B
The difference is not proven.
“After release” has 25% more events (from −2% to +60%), exact p = 0.077.
What to do next: To detect a difference this size with 80% power, observe about 71 days in each period.
Rate · Before release
4.00
per day · CI 3.32…4.78
Rate · After release
5.00
per day · CI 4.23…5.87
Ratio B / A
1.250
CI 0.977…1.603
Difference B − A
+1.00
per day · CI −0.07…+2.07
Exact p
0.077
conditional binomial
z-test
z = 1.822
p = 0.068 · approximate
The link holds only the numbers and the unit: group names and daily data stay out of it.
02
About
A rate, not a share
Conversion is the share of people who acted. Here we count events: one customer can contact support three times.
Periods of different length
The test compares rates per unit of time, so 30 and 14 days can be compared.
Events must be independent
Promotions and outages create bursts of events. The “By day” mode checks the spread.
03
Check in Python / R
A/B testing: how to run a testSeven steps from hypothesis to conclusion, a hypothesis template, 12 common mistakes and a glossary8 minHow many of your A/B test wins are falseWhy every third win is false at α = 5%: a grid of 100 tests, a table and a simulation8 minHow not to run an A/B testEight mistakes with live demonstrations and a printable checklist12 min
05
Frequently asked questions
Updated