Reading Adjusted Comparisons and Intervals

Why Adjusted Results Need Careful Reading

A comparison table may show several differences between group means, each with a p-value or confidence interval. Reading any one row in isolation can be misleading: when many hypotheses are tested, the chance of at least one false rejection across the set increases. Multiplicity adjustments address that risk, but their meaning depends on which comparisons were grouped together and what the reported adjustment controls.

This tutorial focuses on interpreting adjusted comparisons and intervals, rather than choosing a particular comparison method. The essential questions are: What is the family? What error rate or coverage does the method control for that family? And what does the result say about a specific comparison, as distinct from the family as a whole?

Definition: A family of comparisons is the specified collection of hypotheses or parameter comparisons for which a joint error guarantee is intended. A familywise error rate is the probability of making at least one false rejection in that family when the corresponding null hypotheses are true. A simultaneous confidence procedure constructs intervals with a stated coverage guarantee for all parameters in the family together.

For example, if four treatment groups are compared in every pair, the family contains six pairwise mean differences. If only three scientifically planned contrasts are under consideration, the family may contain just those three contrasts. The family is not automatically “all comparisons that could have been made.” It must be identified from the analysis plan and the question being answered. Changing the family can change the adjustment and its conclusions.

What an Adjusted p-Value Means

A raw p-value is calculated for an individual null hypothesis under a specified test. An adjusted p-value accounts for testing within a family using a stated multiplicity procedure. For that procedure, it can be read as the smallest familywise significance level at which the hypothesis would be rejected. It is not the probability that the null hypothesis is true, nor is it a measure of the size or practical importance of a difference.

For the Bonferroni procedure, with \(m\) hypotheses and raw p-value \(p_j\) for hypothesis \(j\), the adjusted p-value is:

p_{j,\mathrm{adj}}=\min(1,mp_j).

Equivalently, at familywise significance level \(\alpha\), reject an individual null hypothesis if \(p_j\leq \alpha/m\). The p-value form and the significance-threshold form give the same decision. Bonferroni controls the familywise error rate without requiring the tests to be independent, though it may be conservative.

Worked Example: Interpreting Bonferroni-Adjusted Results

In a hypothetical study of four training formats, an analyst planned four comparisons, each testing whether a specified mean outcome difference is zero. The raw two-sided p-values are \(0.006\), \(0.021\), \(0.034\), and \(0.120\). Apply Bonferroni adjustment and decide which comparisons are rejected at familywise level \(0.05\). State the family and the conclusion.

Method and family. The family is the four planned comparisons, so \(m=4\). For each row, multiply the raw p-value by 4 and cap the result at 1:

\begin{aligned} 0.006(4)&=0.024,\\ 0.021(4)&=0.084,\\ 0.034(4)&=0.136,\\ 0.120(4)&=0.480. \end{aligned}

The adjusted p-values are therefore \(0.024\), \(0.084\), \(0.136\), and \(0.480\). Comparing each with \(0.05\), only the first comparison is rejected.

Check by the equivalent threshold. The raw-p-value cutoff is \(0.05/4=0.0125\). Of the four raw p-values, only \(0.006\) is no greater than \(0.0125\). This confirms the adjusted-p-value decision.

Conclusion. There is familywise-significant evidence of a nonzero difference for the first planned comparison. The other three comparisons are not rejected at this familywise level; that does not establish that their true differences are zero.

Common mistake avoided: Comparing every raw p-value directly with \(0.05\) would reject the first three, which is not the Bonferroni familywise decision. The adjustment is tied to the four-comparison family.

Holm Adjustment: A Step-Down Interpretation

Holm’s procedure also controls familywise error without requiring independence, and is often less conservative than applying the same Bonferroni cutoff to every test. Order the \(m\) raw p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). At familywise level \(\alpha\), compare them in order with \(\alpha/m,\alpha/(m-1),\ldots,\alpha\). Reject in sequence until the first comparison that fails; do not reject that hypothesis or any of the remaining ones.

The adjusted p-values are formed in the same order and then made nondecreasing:

p_{(i),\mathrm{Holm}} = \min\left(1,\ \max_{1\leq j\leq i}\{(m-j+1)p_{(j)}\}\right).

The running maximum matters: without it, the adjusted values could decrease as the raw p-values increase, contradicting the ordered step-down decisions. The formula reports each hypothesis’s Holm-adjusted p-value in the original raw-p-value order after mapping the results back to the hypotheses.

Worked Example: Applying Holm’s Procedure to Planned Contrasts

A hypothetical materials study has four planned contrasts among mean durability scores. Their raw p-values, in the order the contrasts were specified, are \(0.018\), \(0.200\), \(0.006\), and \(0.041\). Find the Holm-adjusted p-values and determine the rejections at familywise level \(0.05\).

Method. The family is these four planned contrasts, so \(m=4\). Sort the raw p-values, keeping track of their original contrast labels:

\begin{array}{c|c|c} \text{Rank }i & p_{(i)} & \text{Original order}\\ 1&0.006&3\\ 2&0.018&1\\ 3&0.041&4\\ 4&0.200&2 \end{array}

The sequential cutoffs are \(0.05/4=0.0125\), \(0.05/3\approx0.0167\), \(0.05/2=0.025\), and \(0.05\). The first p-value, \(0.006\), passes \(0.0125\). The second, \(0.018\), exceeds \(0.0167\), so the procedure stops: reject only the first-ranked hypothesis, which is contrast 3.

To calculate adjusted p-values in sorted order, first calculate the rank-specific products:

\begin{aligned} 4(0.006)&=0.024,\\ 3(0.018)&=0.054,\\ 2(0.041)&=0.082,\\ 1(0.200)&=0.200. \end{aligned}

These products are already nondecreasing, so the running maxima are \(0.024\), \(0.054\), \(0.082\), and \(0.200\). Returning to the original contrast order gives adjusted p-values \(0.054\), \(0.200\), \(0.024\), and \(0.082\). Only contrast 3 has an adjusted p-value below \(0.05\), agreeing with the step-down decision.

Common mistake avoided: Testing the second-ranked p-value against \(0.05\) would incorrectly reject it. Holm’s second cutoff is \(0.05/3\), and failure at that step stops the sequence.

Simultaneous Confidence Intervals

An ordinary confidence interval for one contrast gives a coverage guarantee for that parameter under its stated model and procedure. When several contrasts are reported, separate ordinary intervals do not automatically have the same confidence level for the whole collection. Simultaneous intervals are constructed to control uncertainty across the family.

A familywise 95% simultaneous procedure aims for at least 95% probability that all intervals in the family cover their corresponding true parameters together, under the model and procedure. It does not mean that there is a 95% probability the fixed true parameter lies in each observed interval. Nor does it mean that every interval individually has exactly 95% coverage.

For a one-way fixed-effects ANOVA with independent normal errors of common variance, let \(\mu_1,\ldots,\mu_k\) be the group means. A pairwise comparison is a difference such as \(\mu_a-\mu_b\). A simultaneous procedure may produce one interval for each difference, with a joint guarantee for the specified family. For Tukey intervals, the usual family is all pairwise differences among the \(k\) means. Other procedures can define other families. The interpretation must follow the family and method actually used.

Interpretation: If a simultaneous interval for \(\mu_a-\mu_b\) lies entirely above zero, it supports a positive difference \(\mu_a>\mu_b\) within the stated familywise procedure. If the interval contains zero, that comparison is not distinguished from zero at the corresponding simultaneous level. An interval containing zero is not proof of equality.

Worked Example: Reading a Tukey Interval Table

In a hypothetical experiment, three independent groups each contain nine observations. A one-way normal-errors model with a common variance is used. The group sample means are \(18.4\), \(14.2\), and \(17.3\) units for groups A, B, and C, respectively. The ANOVA residual mean square is \(9\), with 24 residual degrees of freedom. A software report gives 95% Tukey simultaneous intervals for all three pairwise differences, using a common half-width of \(3.53\). Interpret the individual comparisons and the family-level guarantee.

Define the family. With three groups, all pairwise differences form a family of \(\binom{3}{2}=3\) comparisons: A minus B, A minus C, and B minus C. The Tukey intervals are simultaneous for this full family.

Calculate each estimated difference and interval. The estimates are:

\begin{aligned} \bar{x}_A-\bar{x}_B&=18.4-14.2=4.2,\\ \bar{x}_A-\bar{x}_C&=18.4-17.3=1.1,\\ \bar{x}_B-\bar{x}_C&=14.2-17.3=-3.1. \end{aligned}

Subtracting and adding the reported half-width \(3.53\) gives:

\begin{aligned} \mu_A-\mu_B &: 4.2\pm3.53=[0.67,\,7.73],\\ \mu_A-\mu_C &: 1.1\pm3.53=[-2.43,\,4.63],\\ \mu_B-\mu_C &: -3.1\pm3.53=[-6.63,\,0.43]. \end{aligned}

The A-minus-B interval is entirely positive, so the procedure supports a higher mean for A than B. The other two intervals contain zero, so they do not establish a difference from zero at this simultaneous level. This does not demonstrate that A and C, or B and C, have equal population means.

Family-level interpretation. Under the specified model and Tukey procedure, the collection of three intervals has the stated 95% simultaneous coverage. This is a joint guarantee for the family, not a 95% probability statement about the fixed means after observing the data.

Common mistake avoided: Declaring all three differences significant because the overall group means are not identical would confuse a general group effect with evidence for each pair. Each individual comparison is read from its own interval.

Why the Family and the Adjustment Matter

The basic justification for Bonferroni adjustment is the union bound. A familywise error occurs if at least one individual null hypothesis is falsely rejected. The probability of that union cannot exceed the sum of the individual probabilities. If each of \(m\) tests has Type I error probability at most \(\alpha/m\) under its null, then:

P(\text{at least one false rejection}) \leq \sum_{j=1}^{m}P(\text{false rejection for }H_j) \leq m\frac{\alpha}{m} =\alpha.

This argument does not require independence. The same logic explains Bonferroni simultaneous intervals. If each individual interval misses its parameter with probability at most \(\alpha/m\), the probability that at least one interval misses is at most \(\alpha\). Thus, with probability at least \(1-\alpha\), all intervals cover together. This guarantee can be conservative because the individual failure events may overlap.

An adjustment is meaningful only relative to a defined family. A table of six pairwise comparisons among four groups has a different family from a table of two prespecified contrasts, even if some rows happen to be identical. If the family was expanded after inspecting results, the nominal guarantee may not describe the full selection process. State the family in words before drawing conclusions.

Identify the target parameters.
Write the group differences or contrasts being assessed, including their direction if relevant.

Specify the family.
List which hypotheses or intervals are included in the joint error or coverage guarantee.

Name the procedure and level.
Distinguish, for example, Bonferroni or Holm adjusted p-values from Tukey simultaneous intervals.

Interpret each row and the whole set.
For a specific contrast, use its adjusted p-value or interval. Separately report the family-level guarantee and avoid claiming that nonrejection proves equality.

Adjusted p-values and simultaneous intervals often tell compatible stories when they are obtained from the same procedure and target the same family at matching levels. But do not assume that any adjusted p-value table and any interval table are exact duals: the methods, hypotheses, or families may differ. Check the report’s stated procedure before combining their conclusions.

These distinctions are aligned with the SOA VEE Mathematical Statistics syllabus because they connect multiple-comparison error control with inference about individual contrasts. In an applied analysis, report the estimated difference and its uncertainty as well as the decision; statistical significance alone does not communicate the magnitude or practical importance of an effect.

Key takeaway: An adjusted p-value or simultaneous interval is interpretable only after identifying its comparison family and procedure. A significant individual comparison supports evidence about that specified difference; it does not by itself establish a general ordering of all groups.

Check Your Understanding

For each problem, identify the relevant family before interpreting the result.

A researcher tests three planned contrasts with raw p-values \(0.010\), \(0.030\), and \(0.090\), using Bonferroni adjustment. Find the adjusted p-values and state which contrasts are rejected at familywise level \(0.05\). (Answer: Adjusted p-values \(0.030\), \(0.090\), and \(0.270\); reject only the first contrast.)
Four group means produce a 95% simultaneous interval for a specified difference of \([-1.2,\,4.8]\). What can be concluded about that difference at the simultaneous level? (Answer: The interval includes zero, so the procedure does not distinguish the difference from zero; it does not establish equality.)
For a family of three hypotheses, the ordered raw p-values are \(0.008\), \(0.020\), and \(0.070\). Apply Holm’s procedure at familywise level \(0.05\), and calculate the adjusted p-values. (Answer: Cutoffs are \(0.0167\), \(0.025\), and \(0.05\); reject the first two only. Adjusted p-values in sorted order are \(0.024\), \(0.040\), and \(0.070\).)
A study has five groups and reports all pairwise Tukey intervals. How many comparisons are in the family, and what does a 95% simultaneous guarantee mean? (Answer: \(\binom{5}{2}=10\) comparisons; under the method’s assumptions, the collection of all ten intervals covers all ten true differences together with at least 95% probability over repeated sampling.)
A report gives a Holm-adjusted p-value of \(0.047\) for one contrast and a raw p-value of \(0.032\) for another. At familywise level \(0.05\), which result is directly eligible for rejection based on the values given, and what limitation should be noted? (Answer: The contrast with adjusted p-value \(0.047\) is rejected; the other cannot be judged from its raw p-value alone without its adjusted value or the full Holm step-down results.)

What You'll Learn