Adjusted P-Values for Bonferroni Tests

Why Adjust P-Values?

When several contrasts are tested, each test has its own marginal p-value. But even if every null hypothesis is true, the chance of at least one false rejection can exceed the level used for any single test. Bonferroni adjustment provides a direct way to report results while controlling the probability of one or more false rejections across the whole family.

The adjustment is simple: multiply each marginal p-value by the number of tests, then cap the result at one. Its justification does not require the tests to be independent. What matters is that each p-value is valid for its own null hypothesis and that the family of tests is specified.

Definitions and Normal-Theory Contrast Tests

Definition: A family is a finite, prespecified collection of $m$ null hypotheses $H_1,\ldots,H_m$, each tested using a valid marginal p-value $p_i$. The Bonferroni-adjusted p-value for test $i$ is $$ p_{i,\mathrm{adj}}=\min\{m p_i,1\}. $$ The familywise error rate is the probability of rejecting at least one true null hypothesis in the family.

In a normal linear model with fixed design, one common setting is $Y=X\beta+\varepsilon$, where $X$ has rank $r$, $\varepsilon\sim N(0,\sigma^2 I)$, and the residual degrees of freedom are $\nu=n-r>0$. For a nonzero contrast vector $c_i$, the contrast is $c_i^\top\beta$, estimated by $c_i^\top\hat\beta$. Under the null $H_i:c_i^\top\beta=\delta_i$, the statistic

T_i=\frac{c_i^\top\hat\beta-\delta_i} {s\sqrt{c_i^\top(X^\top X)^{-1}c_i}}

has a central $t$ distribution with $\nu$ degrees of freedom, under the stated model conditions. Here $s^2$ is the usual residual mean square. For a two-sided alternative, the marginal p-value is $p_i=2\{1-F_{t_\nu}(|T_i|)\}$, where $F_{t_\nu}$ is the central $t_\nu$ distribution function. A right-sided alternative uses $p_i=1-F_{t_\nu}(T_i)$; a left-sided alternative uses $p_i=F_{t_\nu}(T_i)$. The direction of a one-sided test must be specified before interpreting the observed statistic.

The adjustment itself does not depend on how the marginal p-values were obtained. More generally, it applies whenever each $p_i$ is valid: for every parameter value satisfying $H_i$ and every $u\in[0,1]$, $P(p_i\leq u)\leq u$. Under a continuous exact null distribution, equality holds. This validity condition is what the error-control argument uses.

Worked Examples

Worked Example: Adjusting Three Two-Sided Contrast Tests

Suppose three prespecified two-sided contrasts are tested using $t$ statistics with $\nu=2$ residual degrees of freedom. The observed absolute statistics are $10$, $8$, and $0.5$. For $t>0$, the two-sided $t_2$ p-value is

p(t)=1-\frac{t}{\sqrt{t^2+2}}.

For the first contrast, $p_1=1-10/\sqrt{102}\approx0.00985$, so $p_{1,\mathrm{adj}}=\min(3(0.00985),1)\approx0.02956$.

For the second, $p_2=1-8/\sqrt{66}\approx0.01527$, so $p_{2,\mathrm{adj}}=\min(3(0.01527),1)\approx0.04580$.

For the third, $p_3=1-0.5/\sqrt{2.25}=1-1/3=2/3$, so $p_{3,\mathrm{adj}}=\min(3(2/3),1)=1$.

At familywise level $\alpha=0.05$, the adjusted p-values for the first two contrasts are below $0.05$, so those null hypotheses are rejected. The third is not rejected. This decision is protected across all three tests, whether or not their statistics are dependent.

Worked Example: Why the Adjusted Value Is Capped at One

Consider a family of $m=4$ two-sided tests, again with $\nu=2$, and observed absolute statistics $2$, $1$, $0.5$, and $0$. The raw p-values are computed as follows:

For $t=2$, $p_1=1-2/\sqrt{6}\approx0.18350$, giving $4p_1\approx0.73401$, which is already below one.

For $t=1$, $p_2=1-1/\sqrt{3}\approx0.42265$, giving $4p_2\approx1.69060$. Thus $p_{2,\mathrm{adj}}=1$, not $1.69060$.

For $t=0.5$, $p_3=2/3$, so $4p_3=8/3$ and $p_{3,\mathrm{adj}}=1$. For $t=0$, $p_4=1$, so $p_{4,\mathrm{adj}}=1$.

A p-value is reported on the unit interval, so values above one are not retained. The cap also preserves the decision rule: an adjusted value of one cannot lead to rejection at any conventional level below one.

Worked Example: Adjusting Prespecified One-Sided Tests

Suppose two directional contrast tests are specified in advance, both with a right-sided alternative, and both use $t_2$ statistics. The observed statistics are $5$ and $2$. For a positive $t$, the right-tail probability is

P(t_2\geq t)=\frac{1}{2}\left(1-\frac{t}{\sqrt{t^2+2}}\right).

For $T_1=5$, the raw p-value is $p_1=\frac12(1-5/\sqrt{27})\approx0.01887$. With $m=2$, the adjusted value is $p_{1,\mathrm{adj}}\approx0.03775$.

For $T_2=2$, the raw p-value is $p_2=\frac12(1-2/\sqrt{6})\approx0.09175$, and $p_{2,\mathrm{adj}}\approx0.18350$.

At $\alpha=0.05$, only the first null is rejected. These right-sided p-values are appropriate only if the directions were set before observing the statistics. Replacing a two-sided p-value with a one-sided value after seeing which direction the estimate points would not preserve the stated testing procedure.

Why the Adjustment Controls Familywise Error

The key result is a direct consequence of Bonferroni’s inequality. It applies to a finite family with arbitrary dependence among its test statistics.

Theorem: Let $H_1,\ldots,H_m$ be a finite family, and suppose each $p_i$ is valid under its own null. Reject $H_i$ when $p_{i,\mathrm{adj}}\leq\alpha$, for $0<\alpha<1$. Then, for every parameter configuration, the probability of rejecting at least one true null is at most $\alpha$.

Proof. Fix any parameter configuration, and let $I_0$ be the set of indices of true null hypotheses. Since $\alpha<1$, the adjusted rejection condition is equivalent to $p_i\leq\alpha/m$. Therefore, the event of at least one false rejection is

\bigcup_{i\in I_0}\{p_i\leq\alpha/m\}.

The union bound gives

P\left(\bigcup_{i\in I_0}\{p_i\leq\alpha/m\}\right) \leq \sum_{i\in I_0}P(p_i\leq\alpha/m) \leq \sum_{i\in I_0}\frac{\alpha}{m} =\frac{|I_0|}{m}\alpha \leq\alpha.

The second inequality uses validity of each p-value for its true null. No assumption about independence was used. The bound holds for every configuration of true and false nulls, which is why it gives strong familywise error control. $\square$

Adjusted P-Values and the Decision Rule

For $0<\alpha<1$, the adjusted p-value rule and the Bonferroni threshold rule give exactly the same decisions:

Result: For every test $i$, $$ p_{i,\mathrm{adj}}\leq\alpha \quad\Longleftrightarrow\quad p_i\leq\frac{\alpha}{m}. $$

To verify this, if $p_i\leq\alpha/m$, then $mp_i\leq\alpha<1$, so $p_{i,\mathrm{adj}}=mp_i\leq\alpha$. Conversely, if $p_{i,\mathrm{adj}}\leq\alpha<1$, the capped value cannot equal one; hence $p_{i,\mathrm{adj}}=mp_i$, which implies $p_i\leq\alpha/m$. Thus reporting adjusted p-values is a convenient way to report the result of testing every marginal p-value at the shared Bonferroni threshold. $\square$

The adjusted p-value also has a useful threshold interpretation. It is the smallest familywise level at which that particular test would be rejected under the Bonferroni procedure: the test is rejected at level $\alpha$ exactly when $p_{i,\mathrm{adj}}\leq\alpha$. It is not necessarily the p-value for a single omnibus test of the joint null that all contrasts are zero. Each adjusted value pertains to one hypothesis within the family.

Common Mistakes and Practical Reporting

A frequent error is to compare every raw p-value with $\alpha$ and then claim familywise control. That is the unadjusted marginal rule; it does not use the Bonferroni threshold and can have familywise error above $\alpha$. Another error is to multiply by the number of contrasts but forget the cap: a reported adjusted value such as $1.69$ should be reported as $1$.

The family size must also be the number of tests in the stated family, not the number that happen to look promising after examining the results. If four contrasts were prespecified, use $m=4$ for each adjustment even if only two are ultimately discussed. Conversely, the adjustment is not automatically defined by every contrast one could imagine; the family should match the inferential claims being made.

A clear report identifies the contrasts, states whether tests are one-sided or two-sided, gives the marginal and adjusted p-values (or clearly identifies which are reported), specifies $m$, and states the familywise level used for decisions. Bonferroni control may be conservative, particularly when many tests are carried out or when their outcomes are strongly dependent. Conservatism does not invalidate the control guarantee; it means that the actual probability of any false rejection can be smaller than the target bound.

Define the family.
List the prespecified hypotheses and count them to obtain $m$.

Compute valid marginal p-values.
Use the test distribution and the alternative direction specified for each contrast.

Adjust and cap.
For each test, calculate $\min(mp_i,1)$.

Make and report decisions.
Reject at familywise level $\alpha$ exactly when the adjusted value is at most $\alpha$.

Check Your Understanding

Use the definitions and results above to answer these questions.

A family contains five tests, and one raw p-value is $0.012$. What is its Bonferroni-adjusted p-value, and is it rejected at familywise level $0.05$?
For a family of $m$ tests, explain why the raw-p-value decision threshold is $\alpha/m$, rather than $\alpha$.
What does the cap at one do when $mp_i>1$, and why is it appropriate?
Does the familywise error proof require independence among the contrast tests? Identify the probability inequality that supplies the bound.
Why must the direction of a one-sided contrast test be specified before interpreting the observed statistic?