Why Bonferroni Results Depend on the Testing Family
In a genomic analysis, a researcher might test many gene-level associations, variants, or prespecified contrasts. A p-value that would be small for one isolated test may be less compelling when many hypotheses are examined: with more tests, there are more opportunities for at least one false rejection. Bonferroni control handles this by comparing each p-value with a smaller cutoff, determined by the number of tests in the family.
The family is the collection of hypotheses for which a single familywise error guarantee is intended. Its size, \(m\), is the number of tests in that collection, not the number of p-values that happen to be small, and not the number of rejections ultimately made. Here, assume the family and its size are specified before looking at the observed p-values. If tests are divided into separate families, each family has its own size and adjustment.
A p-value is valid for a null hypothesis if, whenever that null is true, \(P(p_i\leq t)\leq t\) for every \(t\) in \([0,1]\). Bonferroni control requires valid p-values for the true null hypotheses and a correctly specified family. It does not require the tests to be independent. The guarantee is that the probability of one or more false rejections among the family is at most \(\alpha\), under any configuration of true and false nulls.
Thresholds and Adjusted P-Values
There are two equivalent ways to apply the procedure. The threshold method compares every raw p-value with \(\alpha/m\). The adjusted-value method multiplies each raw p-value by \(m\), then caps the result at one, and compares it with \(\alpha\). Both use the same family size and give the same decisions. The cap is needed because p-values are reported on the scale from zero to one.
For \(mp_i\leq1\), the adjusted value \(mp_i\) is the smallest familywise significance level in \([0,1]\) at which the raw Bonferroni rule rejects test \(i\). If \(mp_i>1\), no level in \([0,1]\) makes that test pass the raw cutoff: even at \(\alpha=1\), the cutoff is \(1/m\), which is less than \(p_i\). In that case the adjusted value is capped at one as a conventional p-value-scale summary; it is not a level at which rejection occurs.
Why the Familywise Guarantee Holds
Let \(I_0\) be the set of true null hypotheses in the family, and let \(m_0\) be the number of those true nulls. A familywise error occurs if at least one true null is rejected. For any test in \(I_0\), validity of its p-value gives \(P(p_i\leq\alpha/m)\leq\alpha/m\). The union bound then gives the following justification, without an independence assumption:
The final inequality follows because \(m_0\leq m\). This argument also explains why the denominator is the size of the full family, even if some hypotheses are false: in general, the analyst does not know which nulls are true. The guarantee may be conservative, particularly when tests are strongly dependent, but the bound remains valid under dependence when the individual p-values are valid.
Worked Example: Applying One Threshold to Six Tests
Suppose a hypothetical genomic screen has six prespecified gene-level tests, with raw p-values \(0.004,\ 0.013,\ 0.013,\ 0.041,\ 0.180,\) and \(0.720\). Use familywise \(\alpha=0.05\). Assume each test supplies a valid p-value and that all six tests form the intended family.
Threshold calculation. The family size is \(m=6\), so the Bonferroni cutoff is \(0.05/6=0.008333\). Only \(0.004\) is at or below that cutoff. The two tied values \(0.013\) are both above it, as are the remaining values. Thus, reject the null for the test with p-value \(0.004\), and do not reject the other five.
Adjusted-value calculation. Multiply each raw p-value by six and cap at one: \(6(0.004)=0.024\); \(6(0.013)=0.078\) for each tied result; \(6(0.041)=0.246\); \(6(0.180)=1.080\), capped at \(1\); and \(6(0.720)=4.320\), capped at \(1\). The adjusted values are \(0.024,\ 0.078,\ 0.078,\ 0.246,\ 1,\ 1\). Comparing them with \(0.05\) gives the same single rejection.
The tie does not require a ranking rule: equal raw p-values in the same family receive equal adjusted values and the same decision. The familywise conclusion is that, under the stated validity assumptions, the probability of making one or more false rejections among these six tests is controlled at no more than \(0.05\). This does not mean that a particular rejection has a 5% probability of being false.
Different Family Sizes, Ties, and Capping
The adjustment is performed separately for each declared family. A raw p-value alone is not enough to determine its Bonferroni-adjusted value: the family size matters. For example, the same raw p-value can pass in a smaller family and fail in a larger one. Conversely, if a single family was specified, it is not appropriate to change the size separately for each test or to omit tests because their results appear uninteresting.
Worked Example: The Same Result in Two Family Sizes
Consider two hypothetical analyses, each using familywise \(\alpha=0.05\). Family A contains four prespecified tests; Family B contains nine. In each family, one test has raw p-value \(0.012\). Assume the p-values are valid and each family is the intended testing family.
For Family A, the threshold is \(0.05/4=0.0125\). Since \(0.012\leq0.0125\), reject that test's null. Its adjusted p-value is \(\min(1,4(0.012))=0.048\), which is at most \(0.05\).
For Family B, the threshold is \(0.05/9=0.005556\). Since \(0.012>0.005556\), do not reject. Its adjusted p-value is \(\min(1,9(0.012))=0.108\), also above \(0.05\). The observed raw p-value is identical, but the familywise decision differs because the family sizes differ.
These are not contradictory conclusions: each controls error for a different declared family. Family size must reflect the scientific testing plan, not be chosen after seeing which choice yields a desired decision.
Worked Example: Adjusted Values Above One
A hypothetical family contains eight tests. One test has raw p-value \(0.19\). Its adjusted value before capping is \(8(0.19)=1.52\), so the reported Bonferroni-adjusted p-value is \(\min(1,1.52)=1\). At \(\alpha=0.05\), the raw cutoff is \(0.05/8=0.00625\), and \(0.19\) is not small enough to reject.
More generally, for this test the raw cutoff at any permitted level \(\alpha\in[0,1]\) is at most \(1/8=0.125\), which is still below \(0.19\). Thus no familywise significance level in \([0,1]\) yields rejection for this test. The adjusted value of one is a cap, not the smallest significance level at which this test is rejected.
Practical Checks and Common Pitfalls
A reliable calculation keeps the family definition, raw results, threshold, and decisions together. A useful workflow is to record the family size once, calculate one cutoff, and then check each test against it. Alternatively, compute one adjusted value per test and compare each with the same \(\alpha\). If sorting p-values for display, preserve the mapping from each value to its hypothesis; sorting is not required by Bonferroni.
State which hypotheses share the familywise error guarantee and count all tests in that family to obtain \(m\).
Use the prespecified \(\alpha\), commonly \(0.05\), and calculate the common raw-p-value threshold \(\alpha/m\).
Either compare each \(p_i\) with \(\alpha/m\), or calculate \(\min(1,mp_i)\) and compare with \(\alpha\). Check that both routes give identical decisions.
Give the family size, \(\alpha\), the method, and the rejected hypotheses. Describe the control as a bound on the probability of one or more false rejections, not as a probability that an individual finding is false.
Common errors include dividing \(\alpha\) by the number of observed rejections rather than by \(m\), adjusting tests in a family using different denominators without a prespecified reason, and treating ties as if only one tied test can be rejected. Another error is to report \(mp_i>1\) as a value greater than one rather than capping the adjusted p-value at one. A related interpretive error is to say that an adjusted value of one means the test would be rejected at familywise level one. When \(mp_i>1\), the raw Bonferroni rule does not reject at any \(\alpha\) in \([0,1]\).
Bonferroni's main practical advantage is simplicity: it gives a transparent, single cutoff and a direct adjustment that remains valid under arbitrary dependence when the p-values are valid. Its limitation is that it can lose power, especially for large, correlated test collections, because every test faces the same stringent threshold. Other multiplicity procedures can use additional structure to improve power, but they have their own assumptions or decision rules; the Bonferroni calculation itself does not rank tests or adapt its cutoff to the observed p-values.
A small adjusted p-value is evidence against its particular null under the model used to obtain the raw p-value, with multiplicity accounted for over the stated family. It is not an estimate of the probability that the null is true, and a rejection does not by itself establish a causal effect or biological importance. In genomic work, the family definition, p-value validity, effect-size estimates, and scientific context all remain essential.
Check Your Understanding
Answer the questions, then check the solutions below. Assume valid p-values and correctly specified families wherever a Bonferroni calculation is requested.
- A family has ten tests and \(\alpha=0.05\). What is the raw-p-value threshold?
- For \(m=5\), raw p-values \(0.008\) and \(0.014\) are observed. Calculate their adjusted values and decisions at \(\alpha=0.05\).
- Two tests in the same family have tied raw p-values \(0.02\). Must they receive the same adjusted value and decision? Explain.
- For \(m=4\), a test has raw p-value \(0.30\). What is its adjusted value, and is there any \(\alpha\in[0,1]\) at which the raw Bonferroni rule rejects it?
- Does Bonferroni control require independence of the tests? State the condition that matters for the individual p-values.
Solutions. (1) The threshold is \(0.05/10=0.005\). (2) The adjusted values are \(\min(1,5(0.008))=0.040\) and \(\min(1,5(0.014))=0.070\). At \(0.05\), reject the first null and not the second. (3) Yes. In the same family each adjusted value is \(\min(1,m(0.02))\), so the values and decisions match. (4) The adjusted value is \(\min(1,4(0.30))=1\). No level in \([0,1]\) rejects it, since the largest raw cutoff is \(1/4=0.25<0.30\). (5) Independence is not required. Each p-value must be valid under its true null, meaning \(P(p_i\leq t)\leq t\) for all \(t\in[0,1]\).