Why Unequal Treatment Variances Matter
When two treatment arms have different outcome variances, the number assigned to each arm affects the precision of the estimated difference in their mean outcomes. A larger sample in either arm reduces that arm’s contribution to uncertainty, but the size of each contribution also depends on that arm’s variance. Thus, two designs with the same total sample size can yield different standard errors even when the estimand and estimator are unchanged.
This tutorial compares fixed allocations using the difference in arm means. It addresses precision, not which allocation ratio should be chosen. In particular, a comparison of variance formulas does not by itself establish which design is preferable overall: practical constraints and other design goals may also matter.
The arm variances are variances of the observed outcome under each treatment. They can reflect both variation among patients and variation in treatment effects across patients. The formula does not require the treatment effect to be identical for everyone; it requires that the arm-specific means and variances describe the target populations and that the sampling assumptions below are reasonable.
Deriving the Variance Formula
Assume that outcomes are independently sampled within each arm, that the two arm samples are independent, and that allocation sizes are fixed. Each arm’s sample mean is unbiased for its arm mean. For independent observations with finite variance, the variance of a sample mean is the population variance divided by its sample size:
For any two random variables \(U\) and \(V\), \(\operatorname{Var}(U-V)=\operatorname{Var}(U)+\operatorname{Var}(V)-2\operatorname{Cov}(U,V)\). Independence of the arm samples makes the covariance of their sample means zero. Therefore,
This is an exact variance identity under the stated sampling model; normal outcomes are not needed for the identity itself. If outcomes are normally distributed within each arm, the sample means are normally distributed. Otherwise, a normal approximation for the difference may be useful when the arm sample sizes are sufficiently large and the outcome distributions are not so skewed or heavy-tailed that the approximation is poor.
In practice, population variances are usually unknown. Estimate them with arm-specific sample variances \(s_1^2\) and \(s_0^2\). The estimated standard error that does not impose equal variances is
For inference with estimated variances, the Welch two-sample procedure uses this standard error and approximates the reference degrees of freedom by
These degrees of freedom are generally not an integer. The Welch procedure uses a \(t\) reference distribution with the calculated \(df\); this is an approximation, rather than an exact result for all outcome distributions and sample sizes.
Comparing Fixed Allocation Ratios
Suppose the total sample size is \(N=n_1+n_0\), and define the proportion allocated to arm 1 as \(r=n_1/N\), where \(0<r<1\). Substituting \(n_1=rN\) and \(n_0=(1-r)N\) into the variance formula gives
This expression separates the total sample size from the allocation proportion. At a fixed \(N\), increasing the sample size in arm 1 lowers the contribution \(\sigma_1^2/n_1\), but necessarily reduces the sample size in arm 0 and increases \(\sigma_0^2/n_0\). Which change dominates depends on both arm variances. Equal allocation is not guaranteed to have the smallest variance when those variances differ.
Worked Example: Comparing Three Allocations With Known Variances
Consider a hypothetical continuous outcome measured in squared units, with \(\sigma_1^2=25\), \(\sigma_0^2=9\), and a fixed total sample size of \(N=100\). Compare allocations of 50/50, 70/30, and 30/70 between arms 1 and 0. Assume independent outcomes in the two arms and that these variances apply to the populations being sampled.
With 50 participants in each arm,
With 70 in arm 1 and 30 in arm 0,
With 30 in arm 1 and 70 in arm 0,
For these specified variances and this total sample size, the 70/30 allocation has a smaller variance than the equal allocation, while the 30/70 allocation has a larger one. The direction is understandable: the arm with the larger outcome variance contributes more uncertainty per participant, so assigning it more participants reduces that contribution. This calculation compares the three stated designs; it does not establish a universally best ratio.
Estimating Precision When Variances Are Unknown
In a real trial, the arm variances must usually be estimated from observed outcomes. When they differ, using a pooled variance that assumes a common population variance can misrepresent the standard error. The arm-specific standard error and Welch degrees of freedom retain the separate variance contributions. Their accuracy depends on the sample variances being informative; estimates can be unstable when one or both arm samples are small.
Worked Example: Welch Standard Error and Confidence Interval
Suppose a hypothetical study reports \(n_1=30\), \(n_0=70\), \(\bar{x}_1=12.4\), \(\bar{x}_0=9.1\), \(s_1^2=225\), and \(s_0^2=100\). Outcomes are measured in the same units in both arms. Assume independent observations, and use the Welch procedure rather than assuming equal arm variances.
The estimated contrast is \(12.4-9.1=3.300\) outcome units. The estimated variance contributions are \(225/30=7.500\) and \(100/70=1.428571\) squared outcome units. Thus,
Using the unrounded contributions in the Welch degrees-of-freedom formula gives
For a two-sided 95% confidence interval, the \(t\) critical value at \(df\approx40.48\) is approximately \(2.0203\). The interval is
Under the Welch model and approximation, this procedure produces intervals that cover the true mean contrast in about 95% of repeated samples. This particular interval includes zero, so these data do not rule out a zero mean contrast at the two-sided 5% level. The interval is wide, indicating considerable uncertainty about the contrast’s magnitude.
Assumptions, Diagnostics, and Alternatives
The calculation is most directly justified when the arm samples are independent and the allocation counts are fixed. In a randomized trial, the intended population and treatment assignment define the comparison, but dependence can arise if outcomes are clustered by site, household, or provider. If observations are dependent, the simple sum of variance contributions may not apply without accounting for that dependence. Likewise, substantial missing outcomes or deviations from the planned design can change the analysis population and the relevant uncertainty.
Useful checks include reviewing the outcome distributions separately by arm, looking for extreme observations, and comparing the arm sample sizes and sample variances. A few outliers can strongly affect a sample variance, while small samples may make \(s_1^2\) or \(s_0^2\) a noisy estimate of its population counterpart. If one arm has much smaller sample size and strongly skewed outcomes, a routine normal or Welch-\(t\) approximation deserves particular caution. Diagnostics do not prove that assumptions hold; they help reveal ways the approximation could be unreliable.
| Approach | Variance treatment | When it is useful |
|---|---|---|
| Welch standard error and \(t\) approximation | Uses each arm’s sample variance separately | A standard comparison when equal population variances are not justified |
| Pooled two-sample \(t\) procedure | Combines arm variances under a common-variance assumption | When a common population variance is defensible for the target populations |
| Equal allocation | Sets \(n_1=n_0\); it does not itself specify an inference procedure | A simple design choice, but not automatically the most precise under unequal variances |
A common mistake is to reason, “The arms have different variances, so equal allocation must be fairest and most precise.” Equal allocation can be attractive for reasons beyond precision, but the variance formula shows that it is not always the most precise allocation for estimating a difference in means. The opposite claim is also too strong: noticing unequal variances alone does not determine a single preferred allocation. Precision is only one design consideration, and estimates of the variances may themselves be uncertain.
Another pitfall is to use the pooled standard deviation simply because the two arms have similar sample sizes. Similar sample sizes do not establish equal population variances. The pooled procedure relies on a common-variance assumption; Welch’s procedure avoids that assumption by retaining separate variance contributions. Finally, if allocation counts are random rather than fixed, the displayed formula describes variance conditional on those counts under the model; additional variation associated with random counts may need consideration in an unconditional analysis.
Check Your Understanding
Try each question, then check the solutions below.
- Suppose \(\sigma_1^2=16\), \(\sigma_0^2=4\), \(n_1=40\), and \(n_0=60\). What are the variance and standard error of \(\hat{\Delta}\)?
- For the same variances and \(N=100\), compare the variance under allocations 50/50 and 70/30.
- Why does a pooled two-sample \(t\) procedure require an assumption that Welch’s procedure does not?
- Does the variance identity for \(\hat{\Delta}\) require normally distributed outcomes? What role can normality play in inference?
Solutions.
1. The variance is \(16/40+4/60=0.4000+0.06667=0.4667\) squared outcome units. The standard error is \(\sqrt{0.4667}=0.6831\) outcome units.
2. Under 50/50, the variance is \(16/50+4/50=0.4000\). Under 70/30, it is \(16/70+4/30=0.2286+0.1333=0.3619\). For these specified inputs, 70/30 has the smaller variance because more participants are allocated to the higher-variance arm. This comparison does not assess other design goals.
3. The pooled procedure assumes a common population variance across arms so that a combined variance estimate is appropriate. Welch’s procedure uses separate sample variances and does not require that equality assumption.
4. No. The variance identity follows from the variances of independent sample means and does not require normal outcomes. Normality can support exact small-sample distributional inference under the model; with nonnormal outcomes, approximate inference depends on sample sizes and distributional behavior.