Why Treatment Effects Inflate The Pooled Variance

Why a Masked Variance Can Be Larger

A sample-size re-estimation procedure may use outcomes observed so far to update a variance estimate while treatment labels remain masked. The introductory material on pooled variance estimation with treatment labels masked explains what is calculated from the combined observations. A key interpretive issue is that combining outcomes also combines the two treatment-group means. If those means differ, their separation contributes to the variability across all observed outcomes—even when the variability within each arm is unchanged.

This is a mathematical feature of a pooled, masked variance, not evidence that treatment has increased the within-arm outcome variance. The distinction matters because the variance used in a sample-size calculation is intended to represent the relevant outcome variability, while an overall masked sample variance can reflect both within-arm spread and between-arm separation.

Model and Definition

Definition: Consider two treatment arms with fixed sample sizes $n_1$ and $n_0$, and total sample size $N=n_1+n_0$. Let $Y_{1i}$ and $Y_{0j}$ denote outcomes in the two arms. The masked sample variance is the ordinary sample variance of all $N$ outcomes, calculated without using their treatment labels: $$ s^2_{\text{masked}}=\frac{1}{N-1}\left\{\sum_{i=1}^{n_1}(Y_{1i}-\bar{Y})^2+\sum_{j=1}^{n_0}(Y_{0j}-\bar{Y})^2\right\}, $$ where $\bar{Y}$ is the mean of all $N$ outcomes. Write the arm means as $\mu_1$ and $\mu_0$, the common within-arm variance as $\sigma^2$, and the mean difference as $\delta=\mu_1-\mu_0$.

For the expectation derived here, assume that the arm sizes are fixed and positive, outcomes are independent within and between arms, and each arm has a finite variance equal to $\sigma^2$. The arm means may differ. Normal outcomes are not required for this expectation result; normality would matter for some distributional results about a variance estimator, but is not needed here. In an interim analysis with fixed observed arm counts, the result applies conditional on those counts.

The denominator $N-1$ is important: $s^2_{\text{masked}}$ is the usual sample variance across all observations. It is not the within-arm pooled variance, which centers each arm separately and uses different degrees of freedom. Confusing these estimators changes the quantity being discussed.

Deriving the Between-Arm Contribution

Let $\bar{Y}_1$ and $\bar{Y}_0$ be the sample means within the two arms. The total sum of squares around the overall mean can be decomposed exactly into a within-arm part and a between-arm part:

\sum_{i=1}^{n_1}(Y_{1i}-\bar{Y})^2+ \sum_{j=1}^{n_0}(Y_{0j}-\bar{Y})^2 = \sum_{i=1}^{n_1}(Y_{1i}-\bar{Y}_1)^2+ \sum_{j=1}^{n_0}(Y_{0j}-\bar{Y}_0)^2 +\frac{n_1n_0}{N}(\bar{Y}_1-\bar{Y}_0)^2.

The identity follows by writing each outcome as its deviation from its own arm mean plus that arm mean's deviation from $\bar{Y}$. The cross-products sum to zero within each arm because deviations from an arm's sample mean sum to zero. The remaining between-arm terms simplify using $\bar{Y}=(n_1\bar{Y}_1+n_0\bar{Y}_0)/N$.

Under the stated assumptions, the expected within-arm sum of squares is $(n_1-1)\sigma^2+(n_0-1)\sigma^2=(N-2)\sigma^2$. Also, the arm sample means are independent, so their difference has mean $\delta$ and variance $\sigma^2/n_1+\sigma^2/n_0$. For any random variable $X$, $E(X^2)=\operatorname{Var}(X)+[E(X)]^2$; therefore:

E\{(\bar{Y}_1-\bar{Y}_0)^2\} =\delta^2+\frac{\sigma^2}{n_1}+\frac{\sigma^2}{n_0}.

Taking expectations in the sum-of-squares identity and dividing by $N-1$ gives the result. In particular, the variance terms contributed by the sample-mean difference add one more $\sigma^2$ to the $(N-2)\sigma^2$ from within-arm variation.

\begin{aligned} E(s^2_{\text{masked}}) &=\frac{(N-2)\sigma^2+\frac{n_1n_0}{N} \left(\delta^2+\frac{\sigma^2}{n_1}+\frac{\sigma^2}{n_0}\right)}{N-1}\\ &=\frac{(N-1)\sigma^2+\frac{n_1n_0}{N}\delta^2}{N-1}\\ &=\sigma^2+\frac{n_1n_0}{N(N-1)}\delta^2. \end{aligned}

Theorem: With fixed positive arm sizes, independent outcomes, and common within-arm variance $\sigma^2$, the expected masked sample variance is the within-arm variance plus the finite-sample between-arm contribution $n_1n_0\delta^2/[N(N-1)]$. The additional term is nonnegative and is zero when the arm means are equal.

If $\pi=n_1/N$ is the fraction assigned to arm 1, the contribution can also be written $N\pi(1-\pi)\delta^2/(N-1)$. For large $N$, the factor $N/(N-1)$ is close to one, so the contribution is approximately $\pi(1-\pi)\delta^2$. The exact finite-sample formula should be used when reporting an exact expectation.

Worked Example: Equal Allocation With a Six-Unit Mean Difference

Suppose a hypothetical interim data set has $n_1=n_0=50$, so $N=100$. Assume the common within-arm variance is $25$ units squared and the true arm mean difference is $6$ outcome units. Find the expected masked sample variance.

The model assumes independent outcomes and equal within-arm variances; the arm sizes are fixed for this calculation. The difference's sign does not affect the result, so only $\delta^2=6^2=36$ is needed. The exact between-arm contribution is:

\frac{n_1n_0}{N(N-1)}\delta^2 =\frac{50(50)}{100(99)}(36) =\frac{2500}{9900}(36) =9.0909\ \text{units}^2.

Thus $E(s^2_{\text{masked}})=25+9.0909=34.0909$ units squared, or about $34.09$ units squared. The result says that the masked variance averages about $9.09$ units squared above the common within-arm variance under this model; it does not say every interim sample will have that value.

How Allocation and Effect Size Matter

The between-arm component is proportional to $\delta^2$, so doubling the mean difference quadruples the contribution. Reversing which arm has the higher mean changes $\delta$'s sign but leaves $\delta^2$ unchanged. The allocation factor also matters: for a fixed total $N$, $n_1n_0$ is largest under equal allocation and decreases as allocation becomes more imbalanced. Thus, under this model, equal allocation gives the largest expected contamination for a given mean difference and total sample size.

Worked Example: Unequal Allocation and a Small Mean Difference

In a hypothetical trial, let $n_1=30$, $n_0=70$, $N=100$, $\sigma^2=25$ units squared, and $\delta=3$ units. Find the exact expected masked variance.

The independence and common-variance assumptions are as in the model above. The finite-sample coefficient is $30(70)/[100(99)]=2100/9900$, and the squared mean difference is $9$ units squared. Therefore:

\frac{30(70)}{100(99)}(3^2) =\frac{2100}{9900}(9) =\frac{18900}{9900} =1.9091\ \text{units}^2.

It follows that $E(s^2_{\text{masked}})=25+1.9091=26.9091$ units squared, or about $26.91$ units squared. The contribution is smaller than in the equal-allocation example, even though the total sample size is the same, because the allocation is more imbalanced and the mean difference is smaller.

Worked Example: Comparing Two Allocations at the Same Total Sample Size

Consider a hypothetical setting with $N=160$, $\sigma^2=16$ units squared, and $\delta=8$ units. Compare equal allocation ($80$ per arm) with allocation $40$ to arm 1 and $120$ to arm 0. In both cases the denominator in the exact coefficient is $N(N-1)=160(159)=25{,}440$.

With equal allocation, the contribution and expectation are:

\frac{80(80)}{160(159)}(8^2) =\frac{6400}{25440}(64) =16.1006\ \text{units}^2, \qquad E(s^2_{\text{masked}})=16+16.1006=32.1006\ \text{units}^2.

With allocation $40$ and $120$, they are:

\frac{40(120)}{160(159)}(8^2) =\frac{4800}{25440}(64) =12.0755\ \text{units}^2, \qquad E(s^2_{\text{masked}})=16+12.0755=28.0755\ \text{units}^2.

The imbalanced allocation produces a smaller expected between-arm contribution in this comparison. Both expectations include the same within-arm variance, $16$ units squared; the difference between them comes from the allocation-dependent contribution.

Interpretation, Diagnostics, and Alternatives

The extra term is between-arm variation, not an increase in either arm's own outcome variance. A combined sample can be more variable because it contains observations from two distributions with different means, even if each distribution has the same variance. In this sense, the masked observations need not behave like a single normal sample with variance $\sigma^2$; the expectation formula describes their overall sample variance, not the shape of their combined distribution.

Several checks help prevent misinterpretation during blinded re-estimation:

Check the variance definition. Confirm whether the calculation centers all observations at one overall mean or centers each arm separately. These are different estimators.
Check the allocation counts. Use $n_1$, $n_0$, and $N=n_1+n_0$ consistently. In particular, the denominator in the exact coefficient is $N(N-1)$, not an expression formed from one arm size alone.
Check dimensions. The coefficient is dimensionless, while $\delta^2$, $\sigma^2$, and the contribution all have outcome-units squared.
Check what an observed increase can establish. A high interim masked variance may be consistent with a nonzero treatment mean difference, but it can also occur through ordinary sampling variation. The masked variance alone does not identify the treatment effect or prove that it is the source of an increase.
Check the model assumptions. The displayed result uses a common within-arm variance and independent outcomes. If those assumptions are doubtful, the formula is not automatically an appropriate description of the variance estimator.

A principal alternative is to calculate a within-arm pooled variance with treatment labels available. Under the common-variance model, that estimator centers observations separately within each arm and targets $\sigma^2$, without the between-arm mean-separation term. Its drawback for a blinded procedure is that it requires treatment labels. The masked estimator preserves blinding but can include the additional contribution derived here. A variance specified from external information is another possible planning input, but it does not use the interim outcomes to reflect their observed variability.

The formula is an expectation, not a correction rule for every realized interim variance. A particular masked variance may fall below or above its expectation. The result also does not guarantee stable estimation at a small interim sample size. Its purpose here is to explain the source and size of the mean-separation component under explicit assumptions; deciding whether or how to adjust a variance estimate is a separate methodological step.

Common Pitfall

A common mistake is to interpret a larger masked variance as proof that treatment increased the within-arm variance, or to conclude that a masked sample variance must exceed $\sigma^2$ in every interim data set. The first statement confuses within-arm variability with between-arm mean separation; the second confuses an expectation with an observed value. The correct statement is that, under the stated model, the expected masked sample variance exceeds the common within-arm variance by $n_1n_0\delta^2/[N(N-1)]$, unless $\delta=0$. Sampling variation remains in any one interim estimate.

Key takeaway: Pooling without treatment labels combines within-arm spread and the separation of arm means. With fixed arm sizes and equal within-arm variances, the latter adds $n_1n_0\delta^2/[N(N-1)]$ to the expected masked sample variance.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For $n_1=n_0=40$ and $\delta=0$, what is the between-arm contribution to $E(s^2_{\text{masked}})$?
For $n_1=30$, $n_0=70$, and $\delta=3$ units, calculate the contribution in units squared.
With fixed $N$ and a fixed nonzero $\delta$, which allocation gives the largest between-arm contribution: equal or more imbalanced allocation?
Does the formula say that every observed masked sample variance exceeds $\sigma^2$? Explain.

Solutions. (1) Since $\delta^2=0$, the contribution $n_1n_0\delta^2/[N(N-1)]$ is zero. (2) Here $N=100$, so the contribution is $30(70)/[100(99)]\times 3^2=18900/9900=1.9091$ units squared. (3) Equal allocation maximizes $n_1n_0$ for fixed $N$, and therefore maximizes the contribution. (4) No. The formula gives the expectation under the model; any one observed sample variance is subject to sampling variation and can be below or above that expectation.