Estimating ANCOVA Residual Variance Under Masking

Why Estimate Variance From a Baseline Model?

In an ANCOVA trial, the planned treatment comparison adjusts for prespecified baseline covariates. Sample-size planning therefore depends on the outcome’s residual variance after that adjustment, not necessarily on the unadjusted outcome variance. At an interim review, however, treatment labels may remain masked. A regression of outcome on baseline covariates can still be fitted without revealing those labels, but its residual mean square does not automatically isolate the variance that the treatment-labelled ANCOVA will use.

The key distinction is between the variance of random error after baseline adjustment and the residual variation left after fitting only baseline terms to outcomes that may contain a treatment effect. The latter is observable under masking; the former may not be recoverable from it without assumptions. A usable re-estimation plan specifies the baseline model in advance, defines its variance estimator, and makes clear what that estimator targets.

The fit uses outcomes and baseline information, but not treatment labels. The model terms should match the planned ANCOVA adjustment: for example, a baseline measurement and any prespecified transformations or stratification terms. The model and its degrees of freedom must be fixed before examining the interim outcomes. Choosing terms because they make the residual variance look smaller or more stable can make the estimate difficult to interpret and can introduce selection bias.

Model and What the Masked Estimate Measures

For a simple derivation, suppose the planned outcome model is \(Y=Z\gamma+\tau T+\varepsilon\), where \(T\) is the treatment indicator, \(\tau\) is a constant treatment effect, and \(\varepsilon\) is the random error. Assume the design matrices are fixed for the calculation, \(Z\) has full column rank, and the errors have mean zero and covariance matrix \(\sigma^2 I\). Thus errors are uncorrelated, have common variance \(\sigma^2\), and have mean zero conditional on the design. Normal errors are not needed for the expectation below, although they are needed for the usual exact chi-square result for a correctly specified normal linear model.

The masked fit omits \(T\). Since \(M_ZZ=0\), its residual vector is \(M_ZY=\tau M_ZT+M_Z\varepsilon\). Expanding the residual sum of squares gives

Conditional on \(Z\) and \(T\), the middle term has expectation zero. Also, \(M_Z\) is a symmetric idempotent projection matrix with trace \(n-r_Z\), so \(E(\varepsilon'M_Z\varepsilon\mid Z,T)=\sigma^2(n-r_Z)\). Dividing by the residual degrees of freedom yields

The extra term is nonnegative. It is zero if \(\tau=0\) or if \(T\) is fully explained by the columns of \(Z\), so that \(M_ZT=0\). It is positive when \(\tau\ne 0\) and treatment has a component left after projection on the baseline terms. Thus, under a nonzero treatment effect, fitting only baseline terms can leave some treatment-related variation in the residuals. Masking avoids using treatment labels; it does not guarantee that this remaining variation is absent.

Worked Example: Fitting a Prespecified Baseline Model

Randomization and Expected Contamination

When treatment is assigned by complete randomization with a fixed treatment-group size, the treatment indicator is random rather than fixed. If the intercept is in \(Z\), the expected amount of treatment variation left after baseline projection can be stated under that randomization scheme. Let \(n_1\) participants be assigned to treatment, \(q=n_1/n\), and suppose all assignments with that group size are equally likely. Then, conditional on \(Z\),

To see why, the randomization gives \(E(T\mid Z)=q\mathbf{1}\) and \(\operatorname{Cov}(T\mid Z)=q(1-q)\frac{n}{n-1}(I-\mathbf{1}\mathbf{1}'/n)\). Because the intercept is in \(Z\), \(M_Z\mathbf{1}=0\). Taking the trace of \(M_Z\operatorname{Cov}(T\mid Z)\) therefore gives the expression above. Substitution into the conditional expectation of the masked mean square gives

This is an expectation over assignments as well as errors; it is not an exact correction for the observed interim estimate. It also relies on complete randomization and a constant treatment effect. Different randomization schemes, covariate-adaptive assignment, treatment-by-baseline interactions, unequal variances, or missing outcomes can change the relevant calculation. In particular, without the treatment labels, an observed masked residual mean square does not reveal both \(\sigma^2\) and \(\tau\).

Worked Example: Interpreting the Expected Inflation

Using the Estimate in ANCOVA Sample-Size Planning

For a two-group trial with equal allocation, a continuous outcome, a prespecified ANCOVA, a common residual variance \(\sigma^2_{\text{ANCOVA}}\), and target treatment difference \(\delta\), a common large-sample normal approximation for total sample size is

Here \(1-\beta\) is the target power, \(\alpha\) is the two-sided significance level, and \(z_u\) is the \(u\)-quantile of the standard normal distribution. This planning formula assumes the effect, allocation, analysis model, and variance refer to the same outcome scale and target population. It is an approximation; small-sample planning may require a \(t\)-based calculation, and the final design should account for anticipated non-evaluable participants separately.

A masked residual mean square can be inserted as a plug-in nuisance estimate only with a clear interpretation. If it may contain treatment-effect variation, using it directly can produce a larger planned sample size than would result from the true ANCOVA error variance. Conversely, an overfitted baseline model or a variance estimated from a selected subset can make the estimate too small. The planning decision should therefore be prespecified, with sensitivity calculations over plausible residual variances when uncertainty matters.

Diagnostics, Alternatives, and Limitations

Before relying on a masked estimate, check that the model was specified before interim outcomes were reviewed, that its design matrix has full rank, and that its residual degrees of freedom \(n-r_Z\) are positive. Inspect residuals for patterns against fitted values and baseline covariates, extreme observations, and evidence that the assumed common variance or functional form is implausible. These checks diagnose model problems; they do not establish that treatment has contributed no variation to the masked residuals.

The main alternatives answer different practical needs. A treatment-labelled ANCOVA can fit treatment and baseline terms together and estimate its residual variance directly under the model assumptions, but it reveals allocation. An external or historical variance estimate preserves interim masking but may not match the current population, measurement process, or analysis model. A masked baseline-only fit preserves labels but can include residualized treatment signal. No option removes the need to state the model and its assumptions.

The simple derivation assumes a constant treatment effect, homoscedastic independent errors, a correctly specified baseline mean structure, and complete outcome data for the analyzed sample. If treatment effects vary with baseline covariates, treatment-related variation can remain even when some baseline terms are included. If variances differ by treatment group, a single common \(\sigma^2\) may not describe the planned analysis. Missingness, site effects, or a randomization scheme that links assignment to baseline variables can also alter the behavior of the masked fit. A design-specific analysis or simulation may then be needed.

Finally, a variance update is part of a sequential design, not an isolated regression exercise. The adaptation rule, timing, minimum and maximum sample sizes, and treatment of interim information should be set in the protocol. Whether a particular sample-size update preserves the planned type I error depends on the full design and rule; masking alone is not a guarantee. Re-estimation should be evaluated under the actual trial operating characteristics when those features could affect inference.

• Common mistake: “The regression adjusts for baseline covariates, so its residual mean square must be the ANCOVA error variance.” This ignores that treatment can still have a component outside the baseline-model column space. The correction is to distinguish the observed masked residual mean square from \(\sigma^2\), and to assess the assumptions or sensitivity rule under which the former is used for planning.
• Another pitfall: dividing the residual sum of squares by \(n-1\) regardless of the model. The correct residual degrees of freedom for this linear fit are \(n-r_Z\), where \(r_Z\) is the rank of the prespecified baseline design matrix.