Bonferroni and Holm for Selected ANCOVA Comparisons

Why Adjusted Comparisons Need Multiplicity Control

An ANCOVA can produce several scientifically relevant comparisons between treatment groups after adjusting for a covariate. If each comparison is tested at significance level \(\alpha\), the chance of making at least one false rejection across the whole collection can exceed \(\alpha\). Bonferroni and Holm are two ways to control that familywise error rate while retaining a prespecified set of adjusted-mean contrasts.

The family must be defined clearly: it is the collection of null hypotheses for which a joint error guarantee is wanted. Here, “prespecified” means the contrasts are chosen from the study question or analysis plan, rather than selected because their observed results look interesting. Both procedures use the individual p-values from the ANCOVA contrasts; neither changes the fitted adjusted means or their estimated standard errors.

Definition: For a family of \(m\) hypothesis tests, the familywise error rate is the probability of rejecting at least one true null hypothesis in that family. Strong familywise error control at level \(\alpha\) means this probability is at most \(\alpha\), for any configuration of true and false null hypotheses.

Model and Contrast Tests

For the comparisons here, suppose the ANCOVA model has a quantitative response, a categorical treatment with \(g\) groups, and one covariate. A common-slope model can be written as

Y_{ij}=\mu+\tau_i+\beta(x_{ij}-x^*)+\varepsilon_{ij}, \qquad \varepsilon_{ij}\sim N(0,\sigma^2).

Here \(Y_{ij}\) is the response for participant \(j\) in group \(i\), \(x_{ij}\) is that participant’s covariate value, and \(x^*\) is the reference covariate value. The adjusted mean for group \(i\) is its fitted mean at \(x^*\). The usual model-based tests assume independent errors, a linear covariate relationship, a common covariate slope across treatment groups, normally distributed errors, and a common error variance. The covariate target and the contrast family should be meaningful for the study.

A contrast between adjusted means has coefficients \(c_1,\ldots,c_g\) whose sum is zero. Its estimated difference is \(\hat{\theta}=\sum_i c_i\hat{\mu}_{i,\mathrm{adj}}\). For a null hypothesis \(H_0:\theta=0\), the fitted ANCOVA covariance matrix supplies \(SE(\hat{\theta})\); the test statistic is \(t=\hat{\theta}/SE(\hat{\theta})\), with the model’s residual \(df\). The two-sided p-value is the probability, under \(H_0\), of a \(t\)-statistic at least as extreme in either direction as the observed value.

Worked Example: Getting a Contrast p-Value

Suppose a hypothetical ANCOVA compares three treatments on symptom score, measured in points, adjusting for baseline score at a prespecified reference value. The fitted model has residual \(df=20\). Its assumptions are taken as reasonably supported for this illustration: participants are independent, residual plots show no substantial curvature or unequal spread, the groups have no important slope interaction, and the residual distribution is approximately normal. The planned contrast is adjusted mean A minus adjusted mean B.

Suppose the estimated adjusted means are 18.4 and 14.2 points. The estimated contrast is \(18.4-14.2=4.2\) points, and the covariance-based standard error is 1.68 points. Thus,

t=\frac{4.2\ \text{points}}{1.68\ \text{points}}=2.50.

For a two-sided test with 20 residual degrees of freedom, the resulting p-value is approximately \(0.0212\). This is the unadjusted p-value for this one contrast. It does not by itself account for testing other contrasts in the family.

Bonferroni: Divide the Error Budget

For \(m\) tests and a desired familywise level \(\alpha\), Bonferroni tests each null hypothesis at level \(\alpha/m\). Equivalently, if \(p_i\) is an unadjusted p-value, its Bonferroni adjusted p-value is

p_{i,\mathrm{Bonf}}=\min(1,mp_i).

Reject \(H_{0i}\) when \(p_{i,\mathrm{Bonf}}\leq\alpha\). The cap at 1 ensures that an adjusted p-value remains a probability-scale quantity. Bonferroni is easy to apply and does not require the tests to be independent.

Justification: If all \(m\) null hypotheses are true, the probability of at least one false rejection is no greater than the sum of the individual rejection probabilities, which is \(m(\alpha/m)=\alpha\). If only some nulls are true, the same union-bound argument applied to those true nulls gives a bound no larger than \(\alpha\). Thus Bonferroni controls the familywise error rate under any dependence among valid tests.

For two-sided contrast intervals, Bonferroni also gives a simultaneous confidence procedure. Construct each interval at confidence level \(1-\alpha/m\). Under the ANCOVA model, the interval for contrast \(i\) is

\hat{\theta}_i \ \pm\ t_{1-\alpha/(2m),\,df}\,SE(\hat{\theta}_i).

The probability that all \(m\) intervals cover their respective true contrasts is at least \(1-\alpha\). This is a familywise, simultaneous statement; it does not mean each interval has exactly \(1-\alpha\) coverage.

Worked Example: Bonferroni p-Values and Intervals

Continue with a hypothetical, prespecified family of three pairwise adjusted-mean contrasts from the same ANCOVA, with \(df=20\). Suppose their estimates, standard errors, and two-sided unadjusted p-values are as follows: A minus B is 4.2 points, \(SE=1.68\), \(p=0.0212\); A minus C is 3.0 points, \(SE=1.58\), \(p=0.0720\); and B minus C is \(-1.2\) points, \(SE=1.50\), \(p=0.4331\). These values correspond approximately to \(t=2.50,\ 1.90,\ -0.800\), respectively. Assume the ANCOVA conditions described above are adequate.

There are \(m=3\) comparisons, so the Bonferroni test level for each is \(0.05/3=0.01667\). The adjusted p-values are

\begin{aligned} p_{\mathrm{Bonf,A-B}}&=\min(1,3(0.0212))=0.0636,\\ p_{\mathrm{Bonf,A-C}}&=\min(1,3(0.0720))=0.216,\\ p_{\mathrm{Bonf,B-C}}&=\min(1,3(0.4331))=1. \end{aligned}

At familywise \(\alpha=0.05\), none of the three adjusted p-values is at most 0.05, so none is rejected using Bonferroni control.

For simultaneous 95% intervals, each individual interval has confidence level \(1-0.05/3=0.98333\), or about 98.33%. The two-sided critical value is approximately \(t_{0.991667,20}=2.613\). For A minus B, for example,

4.2\pm 2.613(1.68) =4.2\pm4.39 \approx(-0.19,\ 8.59)\ \text{points}.

Using the same critical value, the A minus C interval is \(3.0\pm2.613(1.58)\), or approximately \((-1.13,7.13)\) points. The B minus C interval is \(-1.2\pm2.613(1.50)\), or approximately \((-5.12,2.72)\) points. Each interval includes zero, consistent with the Bonferroni test decisions. Collectively, the procedure has at least 95% coverage for all three contrasts.

Holm: Order the Evidence and Test Step by Step

Holm’s procedure is a step-down method. Order the \(m\) unadjusted p-values from smallest to largest, denoting them \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\). At rank \(i\), compare \(p_{(i)}\) with the threshold

\frac{\alpha}{m-i+1}.

Starting at rank 1, reject while each p-value is no greater than its threshold. At the first p-value that exceeds its threshold, stop: that hypothesis and all hypotheses with larger ordered p-values are not rejected. The procedure does not skip a failed rank and continue testing later ranks.

Holm adjusted p-values can be calculated directly. First calculate \((m-i+1)p_{(i)}\) at each rank. Then take the running maximum, cap at 1, and map the results back to their original hypotheses:

p_{(i),\mathrm{Holm}} = \min\left(1,\ \max_{1\leq j\leq i}\{(m-j+1)p_{(j)}\}\right).

The running maximum makes adjusted p-values nondecreasing in the ordered list, as they must be. At a chosen \(\alpha\), rejecting when a Holm adjusted p-value is at most \(\alpha\) gives the same decisions as the step-down rule.

Justification: Suppose \(t\) null hypotheses in the family are true, and at least one true null is rejected by Holm. Consider the smallest p-value among the true nulls, and let its overall rank be \(r\). At most \(m-t\) false-null p-values can precede it, so \(r\leq m-t+1\). Its Holm rejection threshold is therefore \(\alpha/(m-r+1)\leq\alpha/t\). If Holm rejects any true null, this smallest true-null p-value must consequently be at most \(\alpha/t\). Each true-null p-value has probability at most \(\alpha/t\) of being that small; by the union bound, the probability that any of the \(t\) true-null p-values is at most \(\alpha/t\) is no greater than \(t(\alpha/t)=\alpha\). This proves strong familywise error control without an independence assumption.

Worked Example: Holm Adjusted p-Values

Suppose a different hypothetical ANCOVA has four prespecified adjusted-mean contrasts, all tested two-sided using a model that meets the stated ANCOVA assumptions. Their ordered unadjusted p-values are \(0.004,\ 0.018,\ 0.031,\ 0.200\). At familywise \(\alpha=0.05\), Holm’s thresholds, in order, are

\frac{0.05}{4}=0.0125,\quad \frac{0.05}{3}=0.01667,\quad \frac{0.05}{2}=0.025,\quad \frac{0.05}{1}=0.05.

The first p-value, \(0.004\), is below \(0.0125\), so reject its null. The second p-value, \(0.018\), exceeds \(0.01667\), so stop. The second and all later hypotheses are not rejected. The fact that \(0.031\) is below the last threshold, \(0.05\), does not change the decision: the procedure stopped at the second rank.

The Holm adjusted p-values follow from the rank multipliers and running maximum:

\begin{aligned} i=1:&\quad 4(0.004)=0.016,\\ i=2:&\quad \max(0.016,\ 3(0.018))=0.054,\\ i=3:&\quad \max(0.054,\ 2(0.031))=0.062,\\ i=4:&\quad \max(0.062,\ 1(0.200))=0.200. \end{aligned}

Thus the ordered Holm adjusted p-values are \(0.016,\ 0.054,\ 0.062,\ 0.200\). Only the first is at most 0.05, matching the step-down decisions. The sequential Holm thresholds describe a multiple-testing rule; they should not be mistaken for a ready-made collection of simultaneous confidence intervals. Bonferroni intervals, by contrast, follow directly from using the same per-contrast confidence level \(1-\alpha/m\).

How the Procedures Differ

Bonferroni gives every comparison the same test threshold, \(\alpha/m\). Holm uses a stricter threshold for the smallest p-value and progressively less stringent thresholds for later ranks, provided the earlier steps passed. This step-down feature means Holm adjusted p-values are never larger than the corresponding Bonferroni adjusted p-values, and Holm is at least as powerful in the sense that it can reject every hypothesis Bonferroni rejects, and sometimes more.

Feature	Bonferroni	Holm
How hypotheses are handled	Each uses level \(\alpha/m\)	Order p-values and test sequentially
Adjusted p-value	\(\min(1,mp_i)\)	Rank multiplier with a running maximum
Simultaneous intervals	Directly uses confidence level \(1-\alpha/m\) per contrast	Step-down tests do not directly specify a fixed set of intervals by that rule
Dependence requirement	No independence requirement	No independence requirement

Both procedures protect the family, not each comparison in isolation and not every possible contrast one might invent after examining the data. If the set of contrasts was expanded or selected after seeing results, the family and selection process must be addressed rather than treating the final set as though it had been prespecified.

Common Mistakes and Practical Use

Multiplying by the wrong number. Bonferroni uses the number \(m\) of hypotheses in the stated family, not the number of groups or the number of significant results.
Continuing after a failed Holm step. Once an ordered p-value exceeds its threshold, stop. A later p-value cannot be rejected by the step-down rule.
Reversing Holm’s rank logic. If there are \(t\) true nulls, the smallest p-value among them has rank at most \(m-t+1\), because only the \(m-t\) false-null p-values can come before it. This bound is what makes its Holm threshold no larger than \(\alpha/t\).
Calling an adjusted p-value the probability that a null is true. It is a multiplicity-adjusted measure of evidence under the null model, not the probability the null hypothesis is correct.
Reporting adjusted estimates as if they changed. Multiplicity adjustment changes the testing or interval procedure, not the fitted contrast estimate or its units.

In practice, state the contrast family, identify the adjustment method, and report adjusted-mean differences in the response’s units alongside adjusted p-values or the relevant simultaneous intervals. A small adjusted p-value is evidence against the corresponding null in the context of the fitted ANCOVA; it is not, by itself, a measure of clinical importance.

Check Your Understanding

Use the definitions and procedures above to answer the following questions.

For a prespecified family of five tests with \(\alpha=0.05\), what is the Bonferroni per-test threshold?
A contrast has unadjusted p-value \(0.012\) in a family of four. Calculate its Bonferroni adjusted p-value.
For four ordered Holm p-values \(0.006,\ 0.014,\ 0.030,\ 0.080\), identify the first rank at which the procedure stops when \(\alpha=0.05\).
Why does Holm control familywise error even when the contrast tests are dependent?
What does a simultaneous 95% Bonferroni interval statement say about the full set of contrasts?