An online Bonferroni correction and five more methods side by side: Holm, Hochberg, Šidák, Benjamini–Hochberg and Benjamini–Yekutieli. Paste a list of hypotheses with p-values and the calculator shows the adjusted p-values, which results stay significant and where every number comes from, and explains FWER and FDR in plain words. Python and R code.
Your numbers stay in the browser and are never sent anywhere
A name and p separated by a semicolon or a tab — you can paste two columns from Excel. A line with just a number works too.
Method
Significance level α
The simplest and strictest: p × m. Controls the probability of at least one false finding.
Like Bonferroni, but step by step, and never weaker. A good default for A/B/n.
Steps from the bottom up: more powerful than Holm unless the hypotheses are negatively related.
More precise than Bonferroni for independent tests: 1 − (1 − p)^m.
Controls the share of false results among those found (FDR), for exploring many metrics.
FDR under any dependence between hypotheses, more cautious than BH.
02
Result
2 of 4 are significant.
After the Holm correction; without a correction 4 would look “significant”, and the difference is exactly the chance findings the correction protects you from.
Adjusted p-values
The chosen method is highlighted; “significant” means the adjusted p is below α
Every p is multiplied by the number of hypotheses m = 4, capped at 1.
The p-values in ascending order are multiplied by m, m − 1, …, 1 (m = 4); then a running maximum is taken from the top down, so an adjusted p cannot be smaller than the one before it.
The same multipliers as Holm (m = 4), but the running minimum is taken from the bottom up, which makes the method more powerful.
Adjusted p = 1 − (1 − p)^4: the probability of at least one such p among 4 independent tests.
The p of rank r is multiplied by m / r (m = 4); then a running minimum from the bottom up. This keeps the share of false results among those found in check.
Like BH, but with an extra factor Σ 1/k = 2.083 for m = 4, the price of allowing any dependence between hypotheses.
#HypothesispMultiplierp × multiplierAdjusted
1Price0.005× 4.0000.0200.020 · significant
2Button0.010× 3.0000.0300.030 · significant
3Copy0.030× 2.0000.0600.060
4Colour0.040× 1.0000.0400.060
04
False findings
Chance of at least one false finding
Without a correction, α = 0.05: 1 − (1 − α)^m
You have 4 hypotheses: without a correction, at least one false “win” turns up with probability 19% (if the tests are independent).
FWER or FDR
FWER is the probability of at least one false finding. Bonferroni, Holm, Hochberg and Šidák control it: use it when any error is costly, such as a rollout or medicine.
FDR is the share of false results among all those found. Benjamini–Hochberg and Benjamini–Yekutieli control it: for exploring many metrics, where a couple of false findings is acceptable.
05
Check in Python / R
Check in Python / R
from statsmodels.stats.multitest import multipletests
p = [0.01, 0.04, 0.03, 0.005]
reject, p_adj, _, _ = multipletests(p, alpha=0.05, method="holm")
06
Examples
07
Frequently asked questions
Each test at α = 0.05 carries a 5% chance of a false “win”. If you check 10 metrics, at least one turns out “significant” by chance with probability 40%, and with 20 metrics 64%. A correction recalculates the p-values so that the overall chance of an error is back near the stated α. Without it, chance findings look like discoveries and end up in the report.
Holm: it is never weaker than Bonferroni and controls the probability of any false finding just the same. Bonferroni multiplies every p by the number of hypotheses, Holm does it step by step, by m, m − 1 and so on. For p = 0.01, 0.04, 0.03, 0.005 Bonferroni gives 0.04, 0.16, 0.12, 0.02 and Holm 0.03, 0.06, 0.06, 0.02. Two are significant either way, but Holm sits closer to the threshold.
FDR is the expected share of false results among all those found. The Benjamini–Hochberg method keeps it at α: out of twenty “significant” metrics, one on average is false. That is milder than controlling FWER, where even one error is ruled out, and more useful for exploring many metrics. For the same example BH gives 0.02, 0.04, 0.04, 0.02, so all four are significant.
As many as you like, if one primary metric for the decision is chosen before the start. The rest are secondary: they are tested with a correction and treated as hypotheses for future tests, not as a reason to ship. The trouble starts when the metric is picked after the test, the one that “went up”: with 20 metrics such a metric turns up two times out of three even with no effect.
Paste a list of p-values, one hypothesis per line, with a name before a semicolon. The calculator recalculates them with six methods at once, highlights the chosen one and shows the steps: ascending order, the multiplier and the running maximum or minimum. Adjusted p-values are compared with the same α as ordinary ones. In Python, multipletests from statsmodels does this; in R, p.adjust.
Yes: three variants against the control are three tests. Without a correction, the chance of at least one random “winner” is 14% instead of 5%, almost three times higher. Holm works well for comparisons with a control. For example, p = 0.021, 0.048, 0.19 give two “winners” without a correction and none after Holm: 0.063, 0.096, 0.19.