Why a Contrast Family Can Be More Useful Than a List of Pairwise Tests
In a multi-arm trial, the questions of interest need not all compare two arms. Investigators may want to test whether response changes linearly across ordered doses, whether the middle dose departs from the average of two endpoint doses, or whether a particular dose differs from control. Each question defines a distinct hypothesis about the arm means. When several such hypotheses support the trial’s conclusions, their tests belong to a prespecified multiplicity family.
The key is to define the scientific questions and their contrasts before examining the results. The family is not limited to all pairwise comparisons, nor should it be expanded or narrowed after seeing which results look promising. Bonferroni and Holm procedures can control family-wise error for a planned mix of contrasts, including correlated contrasts.
The zero-sum condition makes a contrast a comparison among means: adding the same constant to every arm mean leaves \(L\) unchanged. Coefficients may be scaled without changing a two-sided test of \(L=0\), although the estimate and its standard error scale by the same amount. The coefficient pattern determines the question being tested; its interpretation must match the arithmetic.
Contrasts for Trends, Curvature, and Selected Comparisons
Suppose three equally spaced dose groups have means \(\mu_0,\mu_1,\mu_2\), for control, a lower dose, and a higher dose. The coefficients \((-1,0,1)\) define the endpoint difference \(\mu_2-\mu_0\). The coefficients \((1,-2,1)\) define a different question: whether the middle mean departs from the average of the endpoint means.
Thus, \((1,-2,1)\) is twice the difference between the average of the two outer means and the middle mean. It is not the endpoint difference, and its sign indicates whether the middle mean is below or above that endpoint average. For four equally spaced groups, coefficients such as \((-3,-1,1,3)\) give a linear trend contrast. Its value summarizes a trend across all four means; it is not simply one selected dose-versus-control difference.
A family can include, for example, a trend contrast and one or more selected comparisons. The individual tests may be statistically dependent because they use some of the same arm data. Bonferroni and Holm do not require the tests to be independent, provided the p-value for each null hypothesis is valid under that null.
Worked Example: Interpreting a Curvature Contrast
Consider hypothetical mean symptom-score reductions, in points, for control, low dose, and high dose: \(0\), \(6\), and \(16\). Assume each arm has 16 independent observations from a normal distribution with known standard deviation \(8\) points. We calculate the contrast with coefficients \((1,-2,1)\).
Estimate and interpretation. The contrast estimate is \(1(0)-2(6)+1(16)=4\) points. Equivalently, \(2[(0+16)/2-6]=2(8-6)=4\) points. The endpoint average is 8 points, which is 2 points above the low-dose mean; the contrast is twice that difference.
Standard error and test. With independent arm means and known common standard deviation \(\sigma=8\), the standard error of a contrast estimate is \(\sigma\sqrt{\sum c_i^2/n_i}\). Here, \(SE=8\sqrt{(1^2+(-2)^2+1^2)/16}=8\sqrt{6/16}=4.899\) points. For a two-sided test of \(H_0:\mu_0-2\mu_1+\mu_2=0\), the statistic is \(z=4/4.899=0.8165\), giving a two-sided normal p-value of approximately \(0.414\). In this one-test illustration, there is not sufficient evidence at the nominal \(0.05\) level to reject the null of no curvature contrast. This does not establish that the three means follow a straight line.
The calculation relies on the stated independent normal observations and known standard deviation. If those assumptions or the variance specification are not justified, this particular standard error and normal reference distribution are not justified either.
Define the Family Before Applying an Adjustment
A family is the set of null hypotheses for which a joint error guarantee is sought. In this setting, it is the set of prespecified contrast hypotheses tied to a defined set of trial claims. The family need not contain every possible contrast or every pairwise comparison. It should include all planned tests whose false rejections could contribute to the same family of confirmatory conclusions.
Before collecting or unblinding outcome data, record for each contrast its arm order, coefficient vector, direction of the alternative, and role in the family. Check that the coefficients sum to zero, and express the contrast in words. For ordered doses, also verify that the dose spacing and coefficient pattern support the intended trend interpretation. A linear trend contrast for equally spaced doses does not automatically represent a linear trend when dose spacing is unequal.
Worked Example: Building a Mixed Contrast Family
Suppose four hypothetical groups have means \(0,3,7,11\) units in the order control, low dose, middle dose, and high dose. The investigators prespecify three questions: a linear trend across equally spaced dose levels, high dose versus control, and middle dose versus low dose. These are three hypotheses, so \(m=3\) for adjustment.
Trend: Use coefficients \((-3,-1,1,3)\). They sum to \(-3-1+1+3=0\), and the estimated contrast is \(-3(0)-1(3)+1(7)+3(11)=37\) units. This is a trend contrast across the ordered means, not a direct high-dose-versus-control estimate.
High dose versus control: Coefficients \((-1,0,0,1)\) sum to zero and give \(-1(0)+0(3)+0(7)+1(11)=11\) units.
Middle versus low dose: Coefficients \((0,-1,1,0)\) sum to zero and give \(0(0)-1(3)+1(7)+0(11)=4\) units.
The family contains these three prespecified questions, not every contrast one could form from four means. The same data could support many other contrasts, but adding them after inspecting the means would change the family in response to the results.
Bonferroni and Holm for the Prespecified Family
Let \(p_1,\ldots,p_m\) be valid p-values for the \(m\) prespecified contrast hypotheses, and let \(\alpha\) be the desired family-wise error rate (FWER), the probability of at least one false rejection in the family. Bonferroni rejects a hypothesis when its p-value is at most \(\alpha/m\). Equivalently, its adjusted p-value is \(\min(1,mp_j)\). A union bound gives the guarantee: under any configuration of true and false nulls, the probability of one or more false rejections is no greater than the sum of the individual true-null rejection probabilities, and that sum is at most \(\alpha\).
Holm’s procedure is step-down. Order the p-values as \(p_{(1)}\le\cdots\le p_{(m)}\). Starting at the smallest, compare \(p_{(i)}\) to \(\alpha/(m-i+1)\). Reject while each comparison passes; stop at the first failure and do not reject that hypothesis or any with a larger p-value. Holm controls strong FWER under arbitrary dependence when each p-value is valid. To see the basis of the guarantee, let \(m_0\) be the number of true nulls. If any true null is rejected, the smallest p-value among the true nulls must be no larger than \(\alpha/m_0\): at the rank of that smallest true-null p-value, the Holm denominator is at least \(m_0\). The probability that any of the \(m_0\) true-null p-values is at most \(\alpha/m_0\) is at most \(\alpha\), by the union bound.
Worked Example: Adjusting Three Contrast Tests
Continue with the four-group hypothetical trial above. Assume 25 independent normally distributed observations per group and a known common standard deviation of \(10\) units. These assumptions give the standard error \(SE=10\sqrt{\sum c_i^2/25}\) for each contrast and permit two-sided standard normal tests. For the trend, high-versus-control, and middle-versus-low contrasts, respectively, the estimates are \(37\), \(11\), and \(4\) units.
Test statistics and raw p-values. For the trend, \(SE=10\sqrt{20/25}=8.944\), so \(z=37/8.944=4.137\), with two-sided \(p\approx0.000035\). For high dose versus control, \(SE=10\sqrt{2/25}=2.828\), so \(z=11/2.828=3.889\), with \(p\approx0.00010\). For middle versus low dose, \(SE=10\sqrt{2/25}=2.828\), so \(z=4/2.828=1.414\), with \(p\approx0.1573\).
Bonferroni. With \(m=3\) and \(\alpha=0.05\), the common threshold is \(0.05/3=0.01667\). The adjusted p-values are approximately \(3(0.000035)=0.000105\), \(3(0.00010)=0.00030\), and \(3(0.1573)=0.4719\). Thus, the trend and high-dose comparison are rejected, but the middle-versus-low comparison is not.
Holm. The ordered p-values are approximately \(0.000035,0.00010,0.1573\). Compare them in order with \(0.05/3=0.01667\), \(0.05/2=0.025\), and \(0.05/1=0.05\). The first two pass, while \(0.1573>0.05\), so the procedure stops. Holm also rejects the trend and high-dose comparison only. The Holm adjusted p-values, in this order, are approximately \(0.000105\), \(0.00020\), and \(0.1573\), using the running maximum of the ordered scaled p-values.
The conclusion concerns these prespecified contrast hypotheses and this family. It does not establish that the dose trend is the only meaningful pattern, nor that untested contrasts have no effects.
Diagnostics, Alternatives, and Limitations
Check the contrast family and its calculations before interpreting adjusted results. A useful audit is to write each coefficient vector next to its verbal question, verify its sum, and independently calculate the estimate from the arm means. Confirm that test direction, standard error, and reference distribution match the prespecified analysis. If a contrast is rescaled by a nonzero factor \(a\), its estimate is multiplied by \(a\) and its standard error by \(|a|\); the magnitude of its two-sided test statistic remains unchanged, although its sign reverses for negative \(a\).
Bonferroni and Holm are practical when the family is modest in size and a simple, dependence-robust guarantee is desirable. Bonferroni uses one threshold for every test; Holm is generally at least as powerful because its early step-down thresholds are less stringent while retaining FWER control. Procedures using the joint distribution of correlated contrast statistics may be more efficient when their assumptions and implementation are appropriate. The choice of procedure does not replace a clear family definition.
- Wrong: “The contrast \((1,-2,1)\) compares the two endpoint means.” Correction: It is twice the difference between the average endpoint mean and the middle mean. The endpoint difference uses \((-1,0,1)\).
- Wrong: “Only the smallest observed p-values belong in the family.” Correction: Include the planned hypotheses according to the prespecified claims, whether or not their results are favorable.
- Wrong: “A trend test makes separate dose comparisons unnecessary.” Correction: A trend and a selected pairwise comparison test different hypotheses; include both if both support confirmatory claims.
- Wrong: “Correlated tests invalidate Holm.” Correction: Holm controls FWER under arbitrary dependence for valid individual p-values. Dependence may affect efficiency, but it is not a requirement of independence.
These adjustments cannot repair an invalid contrast standard error, an unsuitable model, or p-values that are not valid under their null hypotheses. Nor do adjusted p-values measure the size or clinical importance of an effect. Report contrast estimates and their units alongside inferential results, and distinguish the planned claims from exploratory analyses.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- For means \(\mu_1,\mu_2,\mu_3\), what does the contrast \((1,-2,1)\) represent?
- A family has four prespecified hypotheses and \(\alpha=0.05\). What is the Bonferroni threshold?
- Ordered p-values are \(0.006,0.018,0.032\) for a family of three at \(\alpha=0.05\). Which hypotheses does Holm reject?
- Why should a dose-trend contrast and a selected dose-versus-control contrast both be included if both support planned confirmatory claims?
Solutions. (1) The contrast is \(\mu_1-2\mu_2+\mu_3=2[(\mu_1+\mu_3)/2-\mu_2]\): twice the difference between the average endpoint mean and the middle mean. (2) \(0.05/4=0.0125\). (3) Holm’s thresholds are \(0.05/3=0.01667\), \(0.05/2=0.025\), and \(0.05\). The first passes, the second passes, and the third passes because \(0.032\le0.05\), so all three are rejected. (4) They test distinct hypotheses: an overall ordered-dose trend and a particular arm difference. If both contribute to the planned family of claims, both belong in the family before results are examined.