Planned Contrasts and Multiplicity Burden

Why Plan Contrasts Before Looking at Results?

A study with several conditions can support many comparisons. With four conditions, for example, there are six pairwise comparisons, along with other possible questions such as whether two doses differ from a control or whether the average response to two conditions differs from another condition. Testing every conceivable comparison can make results harder to interpret and reduces the chance of detecting any one effect when error rates are controlled across the tests.

A planned contrast makes the scientific question explicit as a weighted comparison of condition means. Prespecifying a focused set of such questions helps align the analysis with the study’s aims. It also makes the multiplicity burden visible before results are examined. The goal is not to choose as few tests as possible regardless of the science; it is to define a defensible family of questions and understand the consequences of its size.

Definition: For condition means \(\mu_1,\ldots,\mu_K\), a contrast is a linear combination \(\theta=\sum_{k=1}^{K}c_k\mu_k\) whose coefficients satisfy \(\sum_{k=1}^{K}c_k=0\). The coefficient vector \(c\) specifies the comparison. A contrast family is the set of hypotheses whose results are intended to support one joint inferential claim or decision.

For example, the contrast \(\mu_3-\mu_1\) has coefficients \((-1,0,1,0)\) when there are four conditions. The contrast \((\mu_1+\mu_2)/2-\mu_3\) has coefficients \((1/2,1/2,-1,0)\): it compares the average of conditions 1 and 2 with condition 3. The zero-sum condition distinguishes a comparison of means from a linear combination that also includes an overall level.

Specify the Question and Its Family

A planned contrast should describe a scientific comparison, not merely a calculation that looks interesting after results are available. State which conditions enter the contrast, their weights, the direction of any one-sided hypothesis, and the set of tests that will be treated as a family. For two-sided tests, a contrast typically tests \(H_0:\theta=0\) against \(H_a:\theta\ne 0\). If direction matters scientifically, the one-sided alternative must be justified and specified in advance.

The family depends on the inferential goal. If the primary question is whether each of three treatments differs from a control, those three treatment-versus-control tests can form a focused family. If the goal is to identify any difference among four conditions, all six pairwise comparisons may be relevant. These are different sets of claims, so they have different multiplicity burdens. Neither family is automatically correct for every study; the choice should follow the scientific objectives and be documented before examining the outcomes.

In single-cell studies, the biological replicate for a condition comparison is generally the donor or other independently assigned experimental unit, not each cell. A donor-level model or an appropriate model for repeated or clustered observations supplies the contrast estimate and its uncertainty. The multiplicity calculation does not repair pseudoreplication, confounding, or an incorrectly specified model.

Worked Example: Estimating a Prespecified Contrast

Suppose a hypothetical experiment has four conditions, A through D, with 12 independent donors per condition. A donor-level response is summarized on a log-expression scale. The fitted one-way model gives condition means \(0.00\), \(0.20\), \(1.10\), and \(0.50\), respectively, and a pooled residual standard deviation of \(0.80\). The planned comparison is condition C minus condition B.

The contrast coefficients, in A–D order, are \(c=(0,-1,1,0)\), which sum to zero. Its estimated value is \(\hat{\theta}=0(0.00)-1(0.20)+1(1.10)+0(0.50)=0.90\) log-expression units.

Under the stated one-way model assumptions of independent donors, normally distributed errors with a common variance across conditions, and equal group sizes, the standard error is calculated using the pooled residual standard deviation:

SE(\hat{\theta})=0.80\sqrt{\frac{1}{12}+\frac{1}{12}} =0.3265986.

There are 48 donors and four fitted condition means, so the residual degrees of freedom are \(df=48-4=44\). The test statistic for \(H_0:\theta=0\) is

t=\frac{0.90}{0.80\sqrt{1/12+1/12}}=2.7557\approx 2.76.

For a two-sided \(t\) test with 44 degrees of freedom, the raw p-value is approximately \(0.00849\). This is evidence against a zero C-versus-B difference under the model, but it is not yet an adjustment for any other contrasts in the planned family. The estimated difference is 0.90 log-expression units; it is not a statement that the condition caused the difference.

Multiplicity: What Changes When the Family Grows?

When multiple hypotheses are tested, the probability of at least one false rejection can exceed the per-test significance level. The family-wise error rate (FWER) is the probability of making one or more false rejections within the defined family. Bonferroni control limits this probability without requiring the tests to be independent: if there are \(m\) tests, test each at level \(\alpha/m\).

Theorem: If each of \(m\) null hypotheses is tested at level \(\alpha/m\), then the probability of at least one false rejection among the true null hypotheses is at most \(\alpha\). Equivalently, a Bonferroni-adjusted p-value for test \(i\) is \(p_{i,\mathrm{adj}}=\min(1,mp_i)\), where \(p_i\) is its raw p-value.

The guarantee follows from the union bound. If \(A_i\) is the event of falsely rejecting the \(i\)th true null hypothesis, the probability of any false rejection is the probability of \(\bigcup_i A_i\), which cannot exceed the sum of the individual probabilities. Each test has probability at most \(\alpha/m\) of a false rejection under its null, so the sum is at most \(\alpha\). This argument works whether the test statistics are independent or correlated.

P\left(\bigcup_{i=1}^{m}A_i\right) \leq \sum_{i=1}^{m}P(A_i) \leq m\frac{\alpha}{m} =\alpha.

Worked Example: Focused Versus Expanded Contrast Families

Return to the hypothetical four-condition study and its C-versus-B raw p-value of \(0.00849\). Suppose the prespecified primary family consists of the three treatment-versus-control comparisons, and C-versus-B is not one of them; for a direct illustration, instead define the focused family as three prespecified comparisons that includes C-versus-B. At \(\alpha=0.05\), the Bonferroni adjusted p-value for this contrast in that family is

p_{\mathrm{adj}}=\min(1,3\times 0.00849)=0.02547\approx0.0255.

The adjusted value is below 0.05, so this contrast is significant under the three-test Bonferroni procedure. If the inferential goal is broadened to all six pairwise comparisons among the four conditions, the same raw p-value gives

p_{\mathrm{adj}}=\min(1,6\times 0.00849)=0.05094\approx0.0509.

That adjusted value is above 0.05, so the contrast does not meet the six-test Bonferroni criterion. The raw evidence for C versus B has not changed; the interpretation changes because the family of claims is larger. The focused family must be justified by the study question, not selected after seeing which family makes this result significant.

Bonferroni is simple and robust, but it can be conservative, particularly when tests are strongly correlated or the family is large. Holm’s step-down procedure also controls FWER and is at least as powerful as ordinary Bonferroni: order the raw p-values from smallest to largest and compare the \(i\)th value with \(\alpha/(m-i+1)\), stopping at the first failure. For all pairwise comparisons under suitable equal-variance assumptions, Tukey procedures can use the joint distribution of the comparisons; for several treatments compared with one control, Dunnett procedures target that family. These methods may use structure that Bonferroni ignores. Their assumptions and target family still need to match the design and scientific claim.

How Multiplicity Affects Power

Power is the probability that a specified test rejects its null hypothesis under a stated alternative. A stricter per-test threshold generally lowers power if the sample size, effect size, and standard error are held fixed. With Bonferroni control, increasing \(m\) decreases the threshold from \(\alpha/m\), so detecting a particular contrast can become harder. This is the multiplicity burden: additional planned claims require stronger evidence for each claim or more information from the study.

Worked Example: Approximate Power Under Two Family Sizes

For planning, suppose the true C-versus-B difference is \(0.90\) log-expression units and its standard error is expected to be \(0.3266\). Use a normal approximation for the test statistic, treating the standard error as fixed. The standardized effect is approximately \(0.90/0.3266=2.756\). The following calculation estimates the power for this one contrast, not the probability of detecting at least one effect in the whole family.

For a single two-sided test at level 0.05, the critical value is approximately 1.960. Under the assumed alternative, the statistic is approximately normal with mean 2.756 and variance 1, so the power is

P(|Z|>1.960\mid \text{alternative}) \approx \Phi(2.756-1.960)+\Phi(-2.756-1.960) \approx \Phi(0.796)+\Phi(-4.716) \approx 0.787.

For a three-test Bonferroni family, each two-sided test uses level \(0.05/3\), giving a critical value of approximately 2.394. The same calculation gives \(\Phi(2.756-2.394)+\Phi(-2.756-2.394)\approx\Phi(0.362)\approx0.641\). For a six-test family, the per-test level is \(0.05/6\), with critical value approximately 2.638; the approximate power is \(\Phi(2.756-2.638)+\Phi(-2.756-2.638)\approx\Phi(0.118)\approx0.547\). The probabilities in the negative tails are negligible at the displayed precision.

Thus, under these planning assumptions, per-contrast power falls from about 0.787 for one test to 0.641 for three and 0.547 for six. These are normal-approximation values, not exact power calculations for a small-sample \(t\) test. A final design analysis should use the sampling distribution and variance-estimation procedure appropriate to the planned model.

Diagnostics, Alternatives, and Limitations

Before interpreting adjusted results, check that each proposed contrast is estimable from the design matrix and that its coefficients express the intended comparison. A contrast can be mathematically written down but not estimated if condition effects are aliased with batch or another design factor. Earlier material on rank and alias checks is directly relevant here: adjustment for multiplicity cannot make a non-estimable comparison identifiable.

Also inspect whether the model assumptions used for the contrast standard error are plausible. In a simple pooled-variance model, assess residual behavior and whether a common variance is reasonable. For more complex donor-level models, use diagnostics and uncertainty estimates appropriate to that model. Unbalanced groups, repeated measures, missing donors, and correlation between observations can change the standard error formula; do not reuse the equal-size, independent-group formula without justification.

A common mistake is to report whichever contrasts happen to have small raw p-values and then treat each as a prespecified primary result. That is data-dependent selection: the nominal p-values do not account for searching across comparisons. The correction is to define the family from the scientific objective before inspecting results, then use a procedure that controls the desired error rate for that family. If exploratory screening is the goal, state that purpose and use an approach suited to it rather than presenting selected findings as confirmatory.

A second limitation is that the contrast family is only one dimension of multiplicity. If many genes are tested for each contrast, there may also be multiplicity across genes and across contrast-gene pairs. Define which claims are being controlled together and apply an inferential strategy that addresses the full intended scope. A prespecified contrast does not by itself solve multiplicity across thousands of features.

Key takeaway: Define contrasts by the scientific questions, verify that they are estimable, and specify the family before examining results. A larger family can lower adjusted significance and power even when the raw estimate and p-value for a particular contrast stay unchanged.

Check Your Understanding

Try each problem, then compare your reasoning with the solutions below.

For four condition means, write the coefficients for comparing the average of conditions A and B with condition C. Does the contrast involve condition D?
A planned family has four tests and one raw p-value is 0.018. What is its Bonferroni adjusted p-value, and does it pass a 0.05 FWER threshold?
Explain why choosing a two-test family only after noticing that it makes a result significant is not a valid prespecified analysis.
Why does adding correlated comparisons still increase or maintain the Bonferroni multiplicity burden, even though the tests are not independent?

Solutions. (1) The coefficients in A–D order are \((1/2,1/2,-1,0)\). They sum to zero, so they define a contrast; condition D has coefficient zero and is not involved. (2) The adjusted value is \(\min(1,4\times0.018)=0.072\), which exceeds 0.05 and therefore does not pass the Bonferroni criterion. (3) The family has been selected using the observed results, so the nominal error control for that two-test family does not account for the earlier search over possible families. The scientific family should be set in advance or the selection process must be handled by a suitable inferential method. (4) Bonferroni’s guarantee follows from the union bound, which does not assume independence. Correlation may make the bound conservative, but it does not invalidate the bound or justify treating the tests as independent single tests.