Pairwise Tests and Multiplicity Across Contrasts

Why Multiple Contrast Tests Need Attention

A regression analysis may answer several related questions at once. For example, after adjusting for other predictors, an analyst might compare the mean outcome for each pair of three groups. Each comparison is a linear contrast of regression coefficients, and each can be tested with a \(t\) statistic. But if every test is judged separately at significance level 0.05, the chance of making at least one false rejection across the set can exceed 0.05.

The issue is not that any single \(p\)-value is calculated incorrectly. It is that many opportunities are being given for a small \(p\)-value to occur by chance. Bonferroni and Holm adjustments address this by controlling the error rate across a defined family of tests. They are useful for planned weighted comparisons as well as for pairwise contrasts.

Definition: A family is the set of related hypothesis tests for which a joint error rate is to be controlled. The familywise error rate is the probability of rejecting at least one true null hypothesis in that family. A procedure controls the familywise error rate at level \(\alpha\) if this probability is no greater than \(\alpha\).

For \(k\) groups, testing every distinct pair gives \(m=k(k-1)/2\) two-sided comparisons. For example, four groups produce six pairwise contrasts. The number of tests is determined by the questions included in the family, not by the number of coefficients displayed in a regression table. A family should be defined based on the scientific questions and analysis plan, rather than narrowed after looking at the results.

Setting Up a Family of Regression Contrasts

Suppose the regression model has coefficient vector \(\boldsymbol{\beta}\). A contrast is a specified linear combination \(c_j^\mathsf{T}\boldsymbol{\beta}\), where \(c_j\) is a vector of fixed contrast weights. For contrast \(j\), the null hypothesis is \(H_{0j}:c_j^\mathsf{T}\boldsymbol{\beta}=0\). The estimated contrast, its standard error, and the regression \(t\) distribution are used to obtain a \(p\)-value \(p_j\). A two-sided test compares the contrast with zero in either direction.

The ordinary contrast test requires that the regression model and standard error be appropriate for the data. For the usual exact \(t\) inference, the model errors are independent, have conditional mean zero and constant variance, and are normally distributed; the test uses the regression degrees of freedom. These conditions apply to each contrast test. The multiplicity procedures then adjust how the set of \(p\)-values is judged. Contrasts from the same fitted model are often dependent, but the Bonferroni and Holm procedures described here remain valid without assuming that the tests are independent.

Key distinction: A raw \(p\)-value describes evidence for one contrast considered on its own. A multiplicity-adjusted \(p\)-value or decision accounts for the fact that the analysis includes a family of tests.

Bonferroni Adjustment

With \(m\) tests and desired familywise level \(\alpha\), the Bonferroni procedure tests each null hypothesis at level \(\alpha/m\). Equivalently, it rejects test \(j\) when its raw \(p\)-value is at most \(\alpha/m\). An adjusted \(p\)-value reports the same decision using the original familywise level:

p_{j,\mathrm{Bonf}}=\min(mp_j,1).

The cap at 1 is necessary because a \(p\)-value cannot exceed 1. A test is rejected at familywise level \(\alpha\) when \(p_{j,\mathrm{Bonf}}\leq\alpha\).

The justification uses the union bound. Let \(A_j\) be the event that test \(j\) falsely rejects its null hypothesis. The probability of at least one false rejection is no greater than the sum of the individual false-rejection probabilities. If each test uses level \(\alpha/m\), then

P\left(\bigcup_{j=1}^{m} A_j\right) \leq \sum_{j=1}^{m} P(A_j) \leq m\left(\frac{\alpha}{m}\right) =\alpha.

This bound does not require independent tests. It can be conservative, particularly when many tests are strongly related, but it gives a simple way to control the familywise error rate.

Worked Example: Three Pairwise Comparisons

A hypothetical regression compares adjusted mean assessment scores for groups A, B, and C, controlling for a specified covariate. The outcome is measured in points. The planned family contains all three pairwise comparisons. Suppose the model conditions for the usual contrast \(t\) tests are reasonable, the tests use \(df=40\), and the estimates are as follows.

Contrast	Estimate (points)	SE (points)	\(t\)	Unadjusted two-sided \(p\)
A minus B	2.50	1.00	2.50	0.01662
A minus C	4.00	1.00	4.00	0.0002659
B minus C	1.50	1.00	1.50	0.1415

For A minus B, the test statistic is \(t=2.50/1.00=2.50\), giving \(p=2P(T_{40}\geq2.50)=0.01662\). For A minus C, \(t=4.00/1.00=4.00\), giving \(p=2P(T_{40}\geq4.00)=0.0002659\). For B minus C, \(t=1.50/1.00=1.50\), giving \(p=2P(T_{40}\geq1.50)\approx0.1415\). The point estimates are consistent: the estimated difference A minus C is 2.50 plus 1.50, or 4.00 points.

Here \(m=3\) and \(\alpha=0.05\), so the Bonferroni per-test threshold is \(0.05/3=0.01667\). The adjusted values are:

\begin{aligned} p_{\mathrm{Bonf},\,A-B}&=3(0.01662)=0.04986,\\ p_{\mathrm{Bonf},\,A-C}&=3(0.0002659)=0.0007977,\\ p_{\mathrm{Bonf},\,B-C}&=3(0.1415)=0.4245. \end{aligned}

At familywise level 0.05, the A-minus-B and A-minus-C null hypotheses are rejected, while the B-minus-C null hypothesis is not rejected. The estimated adjusted mean score is higher for A than for B by 2.50 points and higher for A than for C by 4.00 points. These are associations in the fitted regression model, not evidence that group membership caused the differences.

Holm’s Step-Down Procedure

Holm’s procedure also controls the familywise error rate at \(\alpha\), and it is never less powerful than the ordinary Bonferroni procedure for the same set of tests. It orders the raw \(p\)-values from smallest to largest, then tests them sequentially using increasingly less stringent thresholds. If the ordered values are \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\), compare \(p_{(i)}\) with \(\alpha/(m-i+1)\). Continue rejecting as long as each comparison passes; at the first failure, stop and do not reject that hypothesis or any hypothesis with a larger ordered \(p\)-value.

Order the results.
Sort the \(m\) raw \(p\)-values from smallest to largest, retaining which contrast belongs to each value.

Compare in sequence.
Compare \(p_{(1)}\) with \(\alpha/m\), \(p_{(2)}\) with \(\alpha/(m-1)\), and continue through \(p_{(m)}\) with \(\alpha\).

Stop at the first failure.
Reject hypotheses for all preceding successful comparisons. Do not reject the first one that fails or any later hypothesis.

Holm-adjusted \(p\)-values can also be reported, so each contrast can be compared directly with \(\alpha\). For the ordered raw values, calculate \((m-i+1)p_{(i)}\), take the running maximum of these quantities to keep the adjusted values in order, and cap each result at 1:

p_{(i),\mathrm{Holm}} = \min\left(1,\max_{1\leq j\leq i}\{(m-j+1)p_{(j)}\}\right).

Worked Example: Applying Holm to the Pairwise Tests

Use the same hypothetical three-comparison family. Order its raw \(p\)-values: A minus C has \(0.0002659\), A minus B has \(0.01662\), and B minus C has \(0.1415\). At \(\alpha=0.05\), Holm’s sequential thresholds are \(0.05/3=0.01667\), \(0.05/2=0.025\), and \(0.05/1=0.05\).

First, \(0.0002659\leq0.01667\), so reject the A-minus-C null hypothesis. Second, \(0.01662\leq0.025\), so reject the A-minus-B null hypothesis. Third, \(0.1415>0.05\), so stop; do not reject the B-minus-C null hypothesis. For these values the Holm-adjusted \(p\)-values are:

\begin{aligned} p_{\mathrm{Holm},\,A-C}&=3(0.0002659)=0.0007977,\\ p_{\mathrm{Holm},\,A-B}&=\max(0.0007977,2(0.01662))=0.03324,\\ p_{\mathrm{Holm},\,B-C}&=\max(0.03324,0.1415)=0.1415. \end{aligned}

Comparing these adjusted values with 0.05 gives the same decisions as the step-down comparisons. In this example, Holm is less conservative than Bonferroni for A minus B: its adjusted \(p\)-value is 0.03324 rather than 0.04986. Both procedures still control the familywise error rate.

Worked Example: Holm’s Stopping Rule for Four Tests

Suppose a planned family of four regression contrasts has raw two-sided \(p\)-values \(0.004\), \(0.012\), \(0.021\), and \(0.200\), already listed from smallest to largest. At familywise level 0.05, the Holm thresholds are \(0.0125\), \(0.01667\), \(0.025\), and \(0.05\).

The first three comparisons pass: \(0.004\leq0.0125\), \(0.012\leq0.01667\), and \(0.021\leq0.025\). The fourth fails because \(0.200>0.05\). Thus the first three null hypotheses are rejected and the fourth is not. Their Holm-adjusted \(p\)-values are \(0.016\), \(0.036\), \(0.042\), and \(0.200\), respectively. For comparison, Bonferroni-adjusted values are \(0.016\), \(0.048\), \(0.084\), and \(0.800\). At level 0.05, Bonferroni rejects only the first two, whereas Holm rejects the first three.

Choosing and Reporting the Family

The adjustment depends on what counts as the family. If the question is whether any pair of four groups differs, all six pairwise tests naturally form one family. Testing only a pre-specified subset may be appropriate when those are the comparisons that address the scientific question. The family should be stated clearly, including which outcome, model, and contrasts it contains. Quietly treating each outcome or each subgroup as a separate family after seeing the results can make the reported error control misleading.

Bonferroni is straightforward and can be applied to any set of valid \(p\)-values. Holm is also broadly applicable and usually has at least as much power. Neither procedure makes a weakly designed study informative, establishes that a difference is practically important, or turns an association into a causal effect. Statistical significance should be considered alongside the estimated contrast, its standard error, the outcome’s units, and the scientific context.

Common Mistakes

Using 0.05 separately for every test and calling the whole family a 5% procedure. That controls each test’s individual Type I error rate, not necessarily the familywise error rate. Use a multiplicity adjustment when the familywise rate is the goal.
Dividing alpha by the number of groups instead of the number of tests. For all distinct pairwise comparisons among \(k\) groups, the number is \(k(k-1)/2\). For example, four groups give six comparisons, not four.
Continuing Holm after a failed comparison. Holm is step-down: once an ordered \(p\)-value exceeds its threshold, stop rejecting. A later, larger \(p\)-value cannot pass after that failure.
Calling an adjusted \(p\)-value the probability that the null hypothesis is true. It is not. It is a multiplicity-adjusted measure used for decisions under a specified familywise error procedure.
Defining the family after seeing which results are small. The scope of the family should follow the planned scientific questions, not be changed to make selected findings look more convincing.

Key takeaway: First specify the family of regression contrasts. Bonferroni compares every raw \(p\)-value with \(\alpha/m\); Holm orders the \(p\)-values and applies sequential thresholds. Both control the chance of at least one false rejection in the family at no more than \(\alpha\).

Check Your Understanding

Answer each question using the ideas and procedures in this tutorial.

How many distinct pairwise comparisons are there among five groups, and what Bonferroni threshold would be used for a familywise level of 0.05?
For three tests with raw \(p\)-values 0.006, 0.020, and 0.040, determine the Holm decisions at familywise level 0.05. Show each threshold and identify whether the stopping rule is reached.
Why does the union-bound argument for Bonferroni not require the contrast tests to be independent?
A contrast estimate is 3.2 points with standard error 1.1 points. What is its \(t\) statistic, and what additional information is needed to obtain its raw \(p\)-value?
Explain why defining a smaller family only after inspecting the results can make a reported familywise error guarantee misleading.