Why Use Simultaneous Intervals?
A confidence interval for one regression contrast summarizes uncertainty about that particular quantity. But an analysis may ask about several contrasts at once: for example, differences between several adjusted group means, or slopes under several conditions. If each contrast receives its own ordinary 95% interval, those intervals do not generally provide 95% confidence that all the contrasts are covered together. Simultaneous intervals address that joint goal.
The adjustment is useful when the set of contrasts was selected for a common analysis and the conclusions will be considered together. Its cost is usually wider intervals. The size of that cost depends on the number of contrasts, the degrees of freedom, and the interval method.
Familywise Coverage and the Bonferroni Method
Suppose the analysis concerns \(m\) fixed regression contrasts, denoted \(\theta_1,\ldots,\theta_m\). For each contrast, let \(\hat{\theta}_j\) be its estimate and \(SE(\hat{\theta}_j)\) its standard error. A family has simultaneous coverage when every interval in the family covers its corresponding contrast. Its familywise coverage probability is the probability that all \(m\) intervals cover at once.
The formula allocates a total error probability of \(\alpha/m\) to each contrast, split equally between its two tails. For each contrast, the interval therefore has marginal confidence level \(1-\alpha/m\), not \(1-\alpha\). The collection has familywise coverage of at least \(1-\alpha\). The guarantee is “at least” because the method can be conservative.
The intervals use each contrast’s own standard error. That standard error must reflect the covariance among the regression coefficient estimates used to form the contrast; it is not generally correct to combine coefficient standard errors as though the estimates were independent. A contrast’s standard error is obtained from the regression fit using the contrast’s coefficients and their covariance.
For the usual exact \(t\)-based intervals, assume the fitted linear regression model is appropriate, its errors are independent, normally distributed with mean zero and constant variance, and the relevant contrast estimates and standard errors are calculated from that model. Under these assumptions, each standardized contrast estimate has the stated \(t\) distribution with residual \(df\). If these assumptions are not reasonable, the nominal \(t\)-based coverage may not be reliable.
Why the Adjustment Controls Joint Error
The key justification is that the chance of at least one interval failing to cover is no larger than the sum of the separate failure probabilities. This probability inequality does not require the contrast estimates to be independent.
The first inequality is the union bound: the probability of a union of events is no greater than the sum of their probabilities. Each two-sided Bonferroni interval has noncoverage probability at most \(\alpha/m\), so the chance that any interval misses is at most \(\alpha\). Taking the complementary event gives probability at least \(1-\alpha\) that all intervals cover simultaneously.
This result also explains why correlation among contrasts does not invalidate Bonferroni coverage. The contrasts may be strongly related; the union bound still applies. However, if the intervals’ errors overlap substantially, the sum of their individual error probabilities can overstate the actual chance of any failure, making Bonferroni more conservative than necessary.
Worked Example: Three Adjusted Mean Contrasts
Suppose a hypothetical regression analysis compares three prespecified adjusted mean differences in test score, measured in points. The estimates and standard errors come from one fitted model with \(df=20\). Assume the linear model conditions for exact \(t\)-based inference are reasonable: the conditional mean is adequately modeled, errors are independent and normally distributed with constant variance, and the reported contrast standard errors use the fitted coefficient covariance matrix.
| Contrast | Estimate (points) | Standard error (points) |
|---|---|---|
| Group A minus Group B | 2.4 | 0.8 |
| Group A minus Group C | -1.1 | 1.1 |
| Group B minus Group C | 0.5 | 0.6 |
For familywise 95% confidence, \(\alpha=0.05\) and \(m=3\). The Bonferroni critical value is \(t_{1-0.05/(2\cdot3),20}=t_{0.991667,20}\approx2.613\).
For Group A minus Group B, the margin of error is \(2.613(0.8)=2.090\) points, so the interval is \(2.4\pm2.090=[0.310,4.490]\) points.
For Group A minus Group C, the margin is \(2.613(1.1)=2.874\) points, giving \(-1.1\pm2.874=[-3.974,1.774]\) points.
For Group B minus Group C, the margin is \(2.613(0.6)=1.568\) points, giving \(0.5\pm1.568=[-1.068,2.068]\) points.
Together, these intervals have at least 95% familywise coverage under the stated model assumptions. For comparison, a separate two-sided 95% interval uses \(t_{0.975,20}\approx2.086\), giving a margin of \(2.086(0.8)=1.669\) points for the first contrast. That separate interval is narrower, but it does not provide the same 95% joint coverage guarantee for all three contrasts.
Comparing Separate and Simultaneous Intervals
A separate, or unadjusted, two-sided \(100(1-\alpha)\%\) interval for one contrast uses \(t_{1-\alpha/2,df}\). It targets that one contrast’s marginal coverage. When several such intervals are reported together, the probability that all cover is generally less than their individual confidence level. Its precise value depends on the joint distribution of the estimates; it cannot be inferred by simply treating the intervals as independent.
A Bonferroni family instead uses \(t_{1-\alpha/(2m),df}\), which is at least as large as the ordinary two-sided critical value for \(m>1\). For the same estimate and standard error, the Bonferroni interval is therefore at least as wide. The added width buys a guarantee about the family as a whole, rather than a separate guarantee for each interval considered alone.
Worked Example: Contrasts With Covariance-Based Standard Errors
Consider a hypothetical regression model comparing adjusted mean scores for three programs. Three prespecified contrasts are Program A minus B, A minus C, and B minus C. Their estimates and standard errors, calculated from the full covariance matrix of the fitted regression coefficients, are shown below. The model has \(df=18\). Assume the regression mean structure and the independent, normal, constant-variance error conditions are reasonable.
| Contrast | Estimate (points) | Standard error (points) |
|---|---|---|
| A minus B | 4.6 | 1.2 |
| A minus C | -1.3 | 0.7 |
| B minus C | 2.0 | 1.0 |
With \(m=3\) and familywise confidence 95%, the critical value is \(t_{1-0.05/(2\cdot3),18}\approx2.639\).
For A minus B, the margin is \(2.639(1.2)=3.167\), so its interval is \(4.6\pm3.167=[1.433,7.767]\) points. For A minus C, the margin is \(2.639(0.7)=1.847\), yielding \(-1.3\pm1.847=[-3.147,0.547]\) points. For B minus C, the margin is \(2.639(1.0)=2.639\), yielding \(2.0\pm2.639=[-0.639,4.639]\) points.
A useful check is that the margin changes with the standard error even though the critical value is shared. The three intervals have at least 95% familywise coverage, provided the stated model assumptions hold. This calculation does not require the three contrasts to be independent; their covariance has already been accounted for in each standard error, and the familywise guarantee follows from the union bound.
How Family Size Changes the Width
Increasing \(m\) decreases the tail probability allowed for each individual interval. The critical value \(t_{1-\alpha/(2m),df}\) then increases, so the intervals widen. The effect can be substantial when many contrasts are included, especially with limited residual degrees of freedom. The family must be defined to match the set of contrasts for which a joint claim is intended; choosing a smaller family after seeing which contrasts look favorable would not support the same planned interpretation.
Worked Example: Four Contrasts in One Family
Suppose a hypothetical regression analysis estimates four prespecified contrasts, all measured in outcome-score points. Each estimate has standard error \(1.0\) point, and the model has \(df=24\). Assume the regression conditions for exact \(t\)-inference are reasonable. The estimates are \(1.2\), \(3.4\), \(-0.8\), and \(4.1\) points.
For familywise 95% confidence, \(m=4\), so \(t_{1-0.05/(2\cdot4),24}=t_{0.99375,24}\approx2.700\). Each interval has margin \(2.700(1.0)=2.700\) points. The four intervals are therefore \([1.2-2.700,1.2+2.700]=[-1.500,3.900]\), \([3.4-2.700,3.4+2.700]=[0.700,6.100]\), \([-0.8-2.700,-0.8+2.700]=[-3.500,1.900]\), and \([4.1-2.700,4.1+2.700]=[1.400,6.800]\) points.
A separate 95% interval for one contrast would instead use \(t_{0.975,24}\approx2.064\), giving a margin of about \(2.064\) points. The larger Bonferroni margin reflects the goal of covering all four contrasts at once with probability at least 0.95, not a change in the estimated contrasts or their standard errors.
Interpreting the Intervals and Avoiding a Common Mistake
A correct interpretation describes both the contrasts and the family. For example: “Using this procedure, the four intervals have at least 95% familywise confidence for the four prespecified regression contrasts.” In repeated use under the model assumptions, at least 95% of collections constructed this way would cover all four true contrasts. This does not mean there is a 95% probability that the fixed true contrasts lie in these particular calculated intervals.
A common mistake is to calculate a separate 95% interval for every contrast and then claim that the entire set has 95% confidence. Each interval has 95% marginal confidence, but that statement alone does not imply 95% familywise coverage. The correction is to use a simultaneous method such as Bonferroni when the conclusion concerns all the contrasts jointly, and to report the family the adjustment covers.
Bonferroni is simple and makes no independence assumption, but it may produce wider intervals than a method that uses the dependence structure among contrasts. The trade-off is a dependable, straightforward familywise guarantee. When reporting results, state the number and identity of contrasts in the family, the confidence level, the method, and the units of each interval.
Check Your Understanding
Try these questions before reviewing the calculations above.
- For four two-sided Bonferroni intervals at familywise 95% confidence, what tail probability is used for each end of each interval?
- Why does the Bonferroni familywise coverage guarantee remain valid when the regression contrasts are dependent?
- A researcher reports three separate 95% intervals. What is the issue with claiming that all three cover their true contrasts with 95% confidence?
- For the same contrast estimate and standard error, why is a Bonferroni interval generally wider than a separate 95% interval?
- What information should be included when interpreting a collection of simultaneous contrast intervals?