Why the Independence Assumption Fails
When several treatment arms are compared with one control arm, each estimated treatment effect uses the same estimated control mean. That reuse links the estimates: if the observed control mean happens to be high, every treatment-minus-control estimate is shifted downward; if it is low, each estimate is shifted upward. The treatment groups may be separate, and their outcomes may be independent, while their estimated contrasts are still dependent.
This matters whenever the estimates are considered jointly. Using the correct variance for each estimate but setting their covariance to zero can distort the uncertainty of a combined estimate, a difference between treatment effects, or a joint event involving several contrasts. The direction of the distortion depends on the signs of the contrasts being combined.
Assumptions and the Covariance Calculation
For the model-based calculation in this tutorial, assume fixed sample sizes; independent participants across arms; independent outcomes within each arm; and finite outcome variances. Write the outcome variance in arm \(i\) as \(\sigma_i^2\), with control arm indexed by \(0\). Then \(\operatorname{Var}(\bar X_i)=\sigma_i^2/n_i\). The independent-arm assumption implies that means from distinct arms have covariance zero. It does not make two contrasts that reuse one of those means independent.
For two different treatment arms, expand the covariance using the definition of each contrast. Terms involving a treatment mean and a mean from another arm vanish under the assumptions; the common control term remains.
The positive sign is important. Both contrasts subtract the same random quantity, so a fluctuation in the control mean moves them in the same direction. Each contrast also has its own marginal variance:
For two treatment arms, the joint covariance matrix therefore has the form below. A diagonal-only matrix is not the covariance matrix of these shared-control estimates unless the control mean has zero variance, which is not a realistic finite-sample assumption.
If the outcomes in each arm are normally distributed, this contrast vector is jointly normal under the stated independent-arm model. Without normal outcomes, the covariance calculation still holds, but exact joint normality does not follow; it may be an approximation when sample sizes support a central limit argument. In either case, knowing that the estimates are correlated does not by itself determine every joint probability: the assumed joint distribution also matters.
Worked Example: Two Equal-Variance Treatment Arms
Suppose a hypothetical trial measures a continuous outcome in points. There are 50 control participants and 40 participants in each of two treatment arms. Assume independent participants across arms, independent outcomes within arms, and a common outcome standard deviation of 10 points. Calculate the covariance, correlation, and joint covariance matrix for the two treatment-minus-control estimates.
The assumed standard deviation gives a variance of \(10^2=100\) points squared in each arm. For either treatment estimate, the variance is
Both estimates use the same control mean, so their covariance is
Their correlation is the covariance divided by the product of their standard errors:
Thus the covariance matrix is \(\begin{pmatrix}4.5&2.0\\2.0&4.5\end{pmatrix}\) points squared. Treating the estimates as independent would replace each off-diagonal 2.0 by zero. That replacement preserves the two marginal variances but misrepresents the joint uncertainty.
What Goes Wrong in a Joint Calculation
A direct way to expose the error is to consider a linear combination of estimates. For any constants \(a\) and \(b\), expanding the variance gives
The independence shortcut drops the final term. If both coefficients are positive, the shared-control covariance increases the variance relative to that shortcut. If one coefficient is negative, the covariance term is negative, and dropping it instead overstates the variance. Consequently, it is not correct to say that ignoring shared-control covariance always makes uncertainty too small.
Worked Example: Unequal Variances and a Difference Between Effects
Consider hypothetical outcome data measured in points, with \(n_0=30\), \(n_1=24\), and \(n_2=20\). Assume independent sampling across arms and outcome variances \(\sigma_0^2=144\), \(\sigma_1^2=100\), and \(\sigma_2^2=196\) points squared. Calculate the variance of the difference \(\hat\tau_1-\hat\tau_2\), first correctly and then under the mistaken independence assumption.
The shared-control mean has variance \(144/30=4.8\) points squared. The two contrast variances and their covariance are
Here \(a=1\) and \(b=-1\), so the covariance contribution is negative:
Equivalently, \(\hat\tau_1-\hat\tau_2=(\bar X_1-\bar X_0)-(\bar X_2-\bar X_0)=\bar X_1-\bar X_2\), so its variance is \(100/24+196/20=13.9667\) points squared. If the contrasts were incorrectly treated as independent, the calculation would give \(8.9667+14.6=23.5667\) points squared. The error occurs because the common control term cancels in the difference; pretending the estimates are independent loses that cancellation.
Effect on a Combined Estimate
The same covariance also matters when a summary averages treatment effects. A shared control contribution is present in every estimate, so it is counted repeatedly in the marginal variances but not independently each time. The covariance terms are what account for that shared source of uncertainty.
Worked Example: Averaging Two Treatment Effects
Suppose a hypothetical trial has \(n_0=60\) control participants and \(n_1=n_2=40\) participants in two treatment arms. Assume independent participants and a common outcome standard deviation of 12 points. Find the variance of the equally weighted average \((\hat\tau_1+\hat\tau_2)/2\), and compare it with the independence calculation.
The variance of each treatment mean is \(144/40=3.6\) points squared, and the variance of the control mean is \(144/60=2.4\) points squared. Thus each contrast variance is \(3.6+2.4=6.0\) points squared, while the covariance between them is \(2.4\) points squared. The correct variance is
Using independence would omit the covariance term and yield \((6.0+6.0)/4=3.0\) points squared. To check the correct result directly, the average equals \((\bar X_1+\bar X_2)/2-\bar X_0\). Its variance is \(3.6/4+3.6/4+2.4=4.2\) points squared. The independence shortcut understates uncertainty here because the two coefficients are both positive.
Diagnostics, Alternatives, and Limits
Before making a joint calculation, check whether the estimates share a group mean. If they do, identify the coefficient of that mean in each contrast. For two treatment-minus-control contrasts, both coefficients are \(-1\), giving a positive covariance. More generally, the shared contribution to covariance is the product of the two coefficients times the variance of the shared mean. This is a useful sign check, not a substitute for specifying the full contrast and covariance structure.
A practical diagnostic is to compare the proposed covariance matrix with the structure implied by the data and contrasts. Ask whether the same participants or estimated group means appear in more than one contrast, whether the off-diagonal entries have the expected signs, and whether the resulting matrix is a plausible covariance matrix. A matrix with zero off-diagonals simply because treatment arms contain different participants has ignored the shared control. Conversely, inserting a shared-control covariance is not justified if the comparisons actually use separate control groups.
| Approach | What it represents | When it is appropriate |
|---|---|---|
| Separate marginal analyses | Each treatment-control estimate and its own variance, considered individually | For a question about one contrast at a time; it does not supply a valid joint covariance calculation by itself |
| Joint shared-control covariance | Marginal variances and the covariance induced by reusing the control mean | For joint calculations under the stated model, using the actual shared control group |
| Independent contrasts | Zero covariance between estimates | When the estimates truly have no shared random component and are independent under the model, such as comparisons using independent groups rather than one common control |
This calculation is model-based: it relies on the arm means having the stated covariance structure. Other sampling or randomization structures may create additional dependence, so the design and analysis model must agree. Unequal group variances are allowed in the formulas above; they change marginal variances, but the shared-control covariance remains \(\sigma_0^2/n_0\) under independent arm sampling. If the control outcomes are clustered, repeated, or otherwise dependent, their mean variance must reflect that structure rather than automatically using \(\sigma_0^2/n_0\).
Common Mistake
- Wrong reasoning: “The two treatment arms contain different participants, so their treatment-control estimates are independent.”
- Correction: Independence of treatment-arm means only removes their covariance with each other. Both contrasts also contain the same random control mean, so their covariance is generally \(\operatorname{Var}(\bar X_0)\), not zero.
- Another pitfall: “Ignoring covariance always understates uncertainty.” That is false. For a difference of two same-direction treatment effects, the covariance term is subtracted and independence can overstate variance; for their positively weighted average, it is added and independence can understate variance.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Under independent sampling across arms, what is the covariance between two treatment-minus-control estimates if \(\operatorname{Var}(\bar X_0)=3.2\)?
- Two contrasts have variances 5 and 7, and covariance 2. What is the variance of their sum?
- Why can treatment-arm outcomes be independent while the corresponding treatment-control estimates are not?
- In what situation could setting the covariance between two treatment-control estimates to zero be appropriate?
Solutions. (1) The covariance is \(3.2\), in the squared units of the outcome. (2) Use \(\operatorname{Var}(A+B)=\operatorname{Var}(A)+\operatorname{Var}(B)+2\operatorname{Cov}(A,B)\): \(5+7+2(2)=16\), in the squared units of the estimates. (3) Each estimate subtracts the same random control mean; that shared component induces covariance even when the treatment means are independent. (4) Zero covariance can be appropriate when the contrasts do not share a random component and are independent under the specified sampling model, for example when they use genuinely separate, independently sampled control groups.