Precision Of Variance Estimates At Interim Looks

Why Interim Variance Precision Matters

A blinded sample size re-estimation uses interim outcome data without revealing which treatment each participant received. The pooled variance from those data is therefore affected by two things: the underlying within-group outcome variance and, when the treatment means differ, the separation between the groups. Even if the expected amount of treatment-effect inflation is understood, an estimate from a small interim sample can still be noisy. Precision matters because a noisy variance estimate can lead to an unstable sample size update.

The earlier material on correcting pooled variance for allocation and assumed effect addresses the expected contamination. Here the focus is different: how much does the masked pooled variance vary from one interim sample to another, and how does that variability depend on the number of participants observed?

Definition: At an interim look, let $n_1$ and $n_0$ be the numbers observed in the two treatment groups, and let $N=n_1+n_0$. The masked pooled sample variance is the ordinary sample variance of all $N$ outcomes, calculated without using treatment labels: $$ S^2=\frac{1}{N-1}\sum_{i=1}^{N}(Y_i-\bar{Y})^2. $$ Here $\bar{Y}$ is the mean of all $N$ outcomes. The notation $S^2$ is used for this variance estimate; its units are the square of the outcome’s units.

Model and Sampling Distribution

For the exact distribution below, assume that the interim group sizes are fixed, outcomes are independent, and each group has a normal outcome distribution with the same within-group variance. Specifically, suppose $Y_{1j}\sim N(\mu_1,\sigma^2)$ for $j=1,\ldots,n_1$ and $Y_{0j}\sim N(\mu_0,\sigma^2)$ for $j=1,\ldots,n_0$, with independence across participants. Define the true mean difference as $\Delta=\mu_1-\mu_0$. The groups may have different means: this is exactly what creates treatment-effect contamination in the masked variance.

The pooled variance is calculated around the overall mean, not around each group’s own mean. Consequently, its numerator contains both within-group deviations and the between-group separation. Under the model above, that decomposition gives the exact distribution

\frac{(N-1)S^2}{\sigma^2}\sim \chi'^2_{N-1}(\lambda), \qquad \lambda = \frac{\Delta^2}{\sigma^2(1/n_1+1/n_0)} = \frac{n_1n_0}{N}\frac{\Delta^2}{\sigma^2}.

The degrees of freedom are $N-1$, because one overall mean is estimated. The noncentrality parameter $\lambda$ measures how far the group means are separated relative to the within-group standard deviation, while accounting for allocation. The two expressions for $\lambda$ are equal because $1/n_1+1/n_0=N/(n_1n_0)$. In particular, the factor $n_1n_0/N$ multiplies $\Delta^2/\sigma^2$; it is not an extra factor in the denominator of the first expression.

Deriving the Mean and Precision

A noncentral chi-square random variable $Q\sim\chi'^2_\nu(\lambda)$ has $E(Q)=\nu+\lambda$ and $\operatorname{Var}(Q)=2(\nu+2\lambda)$. Applying these identities with $\nu=N-1$ and $Q=(N-1)S^2/\sigma^2$ gives the mean and variance of the masked estimate.

E(S^2)=\sigma^2\left(1+\frac{\lambda}{N-1}\right), \qquad \operatorname{Var}(S^2) = \frac{2\sigma^4\{(N-1)+2\lambda\}}{(N-1)^2}.

The expected excess over $\sigma^2$ is treatment-effect contamination. It is not sampling variability: the variance of $S^2$ describes how much the estimate fluctuates around its own expectation. These are distinct features of the estimator. Its standard deviation is the square root of $\operatorname{Var}(S^2)$, and its coefficient of variation (standard deviation divided by its mean) is

\operatorname{CV}(S^2) = \frac{\sqrt{2\{(N-1)+2\lambda\}}}{(N-1)+\lambda}.

The coefficient of variation is dimensionless. It describes relative sampling variability on the scale of the estimator’s expectation. It is not the relative bias: the latter is $E(S^2)/\sigma^2-1=\lambda/(N-1)$.

Justification: Write the vector of all outcomes as $Y$, and let $P=I-\mathbf{1}\mathbf{1}^{T}/N$, the projection that subtracts the overall mean. Then $(N-1)S^2=Y^{T}PY$. Under the stated normal model, $Y$ has covariance matrix $\sigma^2 I$, and $P$ is an idempotent projection of rank $N-1$. Therefore $Y^{T}PY/\sigma^2$ has a noncentral chi-square distribution with $N-1$ degrees of freedom. Its noncentrality is the squared projected mean vector divided by $\sigma^2$. Since the two group means differ by $\Delta$, that squared distance is $n_1(\mu_1-\bar{\mu})^2+n_0(\mu_0-\bar{\mu})^2=(n_1n_0/N)\Delta^2$, where $\bar{\mu}=(n_1\mu_1+n_0\mu_0)/N$. This yields the stated $\lambda$.

Worked Example: An Interim Look With Equal Allocation

Suppose a hypothetical trial has $n_1=24$ and $n_0=24$ participants at the interim look. Assume a common within-group standard deviation of $\sigma=10$ outcome units and a true mean difference of $\Delta=4$ outcome units. The assumed normal, independent, equal-variance model is taken to hold for this calculation.

Here $N=48$, so the degrees of freedom are $47$. The noncentrality parameter is $$ \lambda=\frac{24\cdot24}{48}\frac{4^2}{10^2} =12\cdot0.16=1.92. $$

The expected masked variance is $$ E(S^2)=100\left(1+\frac{1.92}{47}\right) =104.09\ \text{outcome-units}^2. $$ Thus its expected value is about $4.09\%$ above the within-group variance of $100\ \text{outcome-units}^2$, due to treatment-effect contamination.

Its sampling variance and standard deviation are $$ \operatorname{Var}(S^2) = \frac{2(100^2)(47+2\cdot1.92)}{47^2} = \frac{20{,}000(50.84)}{2{,}209} \approx460.3\ \text{outcome-units}^4, $$ $$ \operatorname{SD}(S^2)=\sqrt{460.3}\approx21.45\ \text{outcome-units}^2. $$ The coefficient of variation is $21.45/104.09\approx0.206$, or about $20.6\%$. The standard deviation is on the variance-estimate scale, so its units are squared outcome units.

Allocation and Sample Size

At a fixed total sample size, allocation affects contamination through $n_1n_0/N$. For a fixed $N$, that quantity is largest when allocation is balanced. If $r=n_1/N$, then $\lambda=Nr(1-r)\Delta^2/\sigma^2$. Thus, for the same total interim size, a balanced allocation has greater treatment-effect contamination in the raw pooled variance than a more unequal allocation. This is not a statement that balanced allocation is undesirable overall; it describes this particular feature of the masked variance.

Interim sample size also affects precision. When allocation proportions and the underlying $\Delta$ and $\sigma$ remain fixed, the degrees of freedom grow with $N$. For $\Delta\ne0$, $\lambda$ also grows proportionally with $N$; when $\Delta=0$, it remains zero. The coefficient of variation then generally decreases, approximately at a rate proportional to $1/\sqrt{N}$ for large $N$. A larger interim sample therefore tends to make the variance estimate more precise relative to its expectation. It does not eliminate contamination, and the expected contamination fraction $\lambda/(N-1)$ need not disappear as $N$ increases.

Worked Example: Correct Noncentrality Under Unequal Allocation

Consider a hypothetical interim data set with $n_1=30$, $n_0=20$, and $N=50$. Suppose $\sigma=5$ outcome units and $\Delta=3$ outcome units. Assume the independent normal, common-variance model. The correct noncentrality is $$ \lambda = \frac{3^2}{5^2(1/30+1/20)} = \frac{9}{25(1/12)} = 4.32. $$ Equivalently, $\lambda=(30\cdot20/50)(9/25)=12(0.36)=4.32$.

There are $49$ degrees of freedom. Therefore $$ E(S^2)=25\left(1+\frac{4.32}{49}\right) \approx27.20\ \text{outcome-units}^2. $$ The variance of this estimate is $$ \operatorname{Var}(S^2) = \frac{2(25^2)(49+2\cdot4.32)}{49^2} = \frac{1{,}250(57.64)}{2{,}401} \approx30.01\ \text{outcome-units}^4, $$ so its standard deviation is approximately $5.48\ \text{outcome-units}^2$. Its coefficient of variation is $5.48/27.20\approx0.201$.

The allocation-specific factor is $n_1n_0/N=12$, not $n_1n_0$ by itself. Keeping this denominator in the allocation factor is essential: using an incorrect $\lambda$ would distort both the estimated contamination and the precision calculations that depend on it.

Quantifying the Chance of a Noisy Estimate

The distribution above can be used to calculate exact probabilities with a noncentral chi-square distribution function. A simpler option is a conservative bound that requires only the mean and variance. For any $c>0$, Chebyshev’s inequality gives $$ P\left(\left|S^2-E(S^2)\right|\geq cE(S^2)\right) \leq \frac{\operatorname{Var}(S^2)}{c^2\{E(S^2)\}^2} = \frac{\operatorname{CV}(S^2)^2}{c^2}. $$

This bounds the probability that the estimate differs from its own expectation by at least a specified fraction. It does not bound the probability that it differs from $\sigma^2$ by that fraction: the expectation itself may be inflated. Also, the bound can be loose, so it is a guarantee rather than an estimate of the actual probability.

Worked Example: Comparing the Chance of a Large Relative Error

Compare hypothetical balanced interim looks with $N=24$ and $N=80$, under the same normal model, $\sigma=8$ outcome units, and $\Delta=2$ outcome units. The group sizes are $12$ per arm for $N=24$, and $40$ per arm for $N=80$.

For $N=24$, $\lambda=(24/4)(4/64)=0.375$, with $23$ degrees of freedom. Thus $$ \operatorname{CV}(S^2) = \frac{\sqrt{2(23+2\cdot0.375)}}{23+0.375} = \frac{\sqrt{47.5}}{23.375} \approx0.295. $$ For $N=80$, $\lambda=(80/4)(4/64)=1.25$, with $79$ degrees of freedom, and $$ \operatorname{CV}(S^2) = \frac{\sqrt{2(79+2\cdot1.25)}}{79+1.25} = \frac{\sqrt{163}}{80.25} \approx0.159. $$

For a deviation of at least $50\%$ from the estimator’s own expectation, Chebyshev’s bound is $\operatorname{CV}^2/0.5^2$. The bounds are approximately $0.295^2/0.25=0.348$ for $N=24$, and $0.159^2/0.25=0.101$ for $N=80$. Thus the larger interim sample gives a substantially tighter upper bound on the chance of such a large relative deviation. These are upper bounds, not calculated event probabilities; exact probabilities would require evaluating the relevant noncentral chi-square distribution.

Alternatives, Diagnostics, and Limitations

An unblinded estimate can calculate within-group variability around each treatment group’s own mean. Under the same common-variance normal model, the pooled within-group sum of squares divided by $\sigma^2$ has a central chi-square distribution with $N-2$ degrees of freedom. It avoids the between-group contribution in the masked raw variance, but requires treatment labels to be available to the people doing the calculation. A masked estimate preserves masking, while its raw form includes treatment-effect contamination. A correction based on an assumed treatment effect can target that contamination, but its performance depends on how well the assumed effect represents the true effect. These approaches trade off information, masking, and dependence on assumptions; none changes the sampling variability inherent in a finite interim sample.

Before applying the exact formulas, check that the inputs correspond to the look being analyzed: $N=n_1+n_0$, the allocation counts are correct, $\Delta$ is expressed in outcome units, and $\sigma$ is in the same units. The formula for $\lambda$ must be dimensionless. The reported variance estimate and its standard deviation must have units of outcome squared, while $\operatorname{Var}(S^2)$ has units of outcome to the fourth power.

The exact distribution relies on independent normal outcomes with a common variance and fixed group sizes. Strong departures, such as heavy tails, outliers, dependence between outcomes, or different group variances, can invalidate the stated distribution or its precision calculations. In that case the formulas should not be treated as exact. Moreover, a low coefficient of variation only describes precision relative to $E(S^2)$; it does not imply that the masked estimate is close to the within-group variance when treatment-effect contamination is substantial.

Common Pitfall

Wrong reasoning: “A larger interim sample makes the masked variance estimate less biased, so it will be close to $\sigma^2$.”
Correction: A larger sample generally improves relative precision, but does not by itself remove treatment-effect contamination. Under fixed allocation proportions, $\lambda/(N-1)$ can remain appreciable as $N$ grows. Assess expected inflation and sampling variability separately.
Wrong reasoning: “The chance bound from Chebyshev’s inequality is the actual chance of a noisy estimate.”
Correction: It is a conservative upper bound on deviation from $E(S^2)$. An exact probability requires the noncentral chi-square distribution and a clearly defined threshold.

Key takeaway: Under independent, equal-variance normal outcomes, the masked pooled variance has a scaled noncentral chi-square distribution. Its mean describes treatment-effect contamination, while its variance and coefficient of variation describe sampling precision. Both allocation and interim sample size affect these quantities.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For fixed $N$, which allocation maximizes $n_1n_0/N$, and what does that imply for $\lambda$ when $\Delta$ and $\sigma$ are fixed?
Suppose $\Delta=0$. What are the mean and variance of $S^2$ under the model?
In a study with $n_1=18$, $n_0=12$, $\Delta=2$, and $\sigma=4$, calculate $N$ and $\lambda$.
Explain why a smaller coefficient of variation does not necessarily mean that $S^2$ is close to the within-group variance $\sigma^2$.

Solutions. (1) The product $n_1n_0$ is largest when the allocation is balanced, so $n_1n_0/N$, and hence $\lambda=(n_1n_0/N)\Delta^2/\sigma^2$, is largest for balanced allocation. (2) With $\Delta=0$, $\lambda=0$, so $E(S^2)=\sigma^2$ and $\operatorname{Var}(S^2)=2\sigma^4/(N-1)$. (3) $N=18+12=30$, and $\lambda=(18\cdot12/30)(2^2/4^2)=7.2(0.25)=1.80$. (4) The coefficient of variation measures spread relative to $E(S^2)$, not distance from $\sigma^2$. If treatment-effect contamination raises the expectation, a precise estimate can still systematically exceed the within-group variance.