The Law of Total Variance in Regression Imputation

A deterministic conditional-mean prediction is \(m(X)\), not \(m(X)+\varepsilon\). Thus the spread of predictions across cases reflects the first component, while the predictions contain none of the second component as variation among imputed outcomes at the same predictor value. This does not mean the residual variation is unimportant or that it is always constant: it may differ across predictor values.

Deriving the Variance of a Deterministically Completed Outcome

Let \(R=1\) indicate that \(Y\) is observed and \(R=0\) indicate that it is missing. In this tutorial, imagine filling each missing outcome with the true conditional mean \(m(X)\), as if the model were known. The completed outcome is $$ Y_c=RY+(1-R)m(X). $$ Since \(Y=m(X)+\varepsilon\), this can also be written \(Y_c=m(X)+R\varepsilon\). The latter form shows that an observed outcome retains its residual, while a missing outcome receives no residual.

For the following variance calculation, assume that the conditional mean \(m(X)\) is correctly specified; \(X\) is available for cases with missing outcomes; and, conditional on \(X\), the observation indicator \(R\) is independent of the outcome residual. The last condition is a MAR-type condition for this outcome model: after conditioning on \(X\), observation status does not select cases according to their unobserved residual. Assume also finite second moments. Write \(r(X)=P(R=1\mid X)\).

Here the comparison is for the same target distribution of \(X\), with observed outcomes retaining their actual residuals and missing outcomes replaced by conditional means. It is not a claim that an estimated sample variance must equal these population expressions exactly.

A Short Derivation

Condition on \(X\). Because \(R\) is conditionally independent of \(\varepsilon\), and \(\operatorname{E}[\varepsilon\mid X]=0\), the completed outcome has conditional mean \(m(X)\). Its conditional variance is the variance of \(R\varepsilon\): $$ \operatorname{E}[Y_c\mid X]=m(X),\qquad \operatorname{Var}(Y_c\mid X)=\operatorname{Var}(R\varepsilon\mid X). $$

Since \(R\) is either zero or one, \(R^2=R\). Also, \(\operatorname{E}[R\varepsilon\mid X]=0\). Thus $$ \operatorname{Var}(R\varepsilon\mid X) = \operatorname{E}[R^2\varepsilon^2\mid X] = r(X)\operatorname{E}[\varepsilon^2\mid X] = r(X)\operatorname{Var}(\varepsilon\mid X). $$

Applying the law of total variance to \(Y_c\) gives the stated completed-data formula. Applying it to \(Y\) gives the full outcome variance. Subtracting the two expressions yields the omitted component: $$ \operatorname{Var}(Y)-\operatorname{Var}(Y_c) = \operatorname{E}[\operatorname{Var}(\varepsilon\mid X)] - \operatorname{E}[r(X)\operatorname{Var}(\varepsilon\mid X)] = \operatorname{E}[(1-r(X))\operatorname{Var}(\varepsilon\mid X)]. $$

The result is nonnegative because \(0\leq r(X)\leq 1\) and conditional variances are nonnegative. It is zero only when there is no missingness at predictor values with residual variation, or when the conditional residual variance is zero there. In the common special case of MCAR with missing fraction \(q\), \(r(X)=1-q\) for every \(X\). The reduction is then \(q\operatorname{E}[\operatorname{Var}(\varepsilon\mid X)]\). With predictor-dependent missingness, the amount lost depends on where missingness occurs as well as on the conditional residual variance there.

Worked Examples

The average conditional residual variance is 9 square units. Therefore the law of total variance gives $$ \operatorname{Var}(Y)=4+9=13\ \text{square units}. $$

A deterministic prediction takes only the values 10 and 14 in this setup, so its variance is 4 square units. The difference, \(13-4=9\) square units, is exactly the within-group residual component. The prediction captures the variation between the two group means, but not outcome variation among people in either group.

Thus $$ \operatorname{Var}(Y_c)=4+5.4=9.4\ \text{square units}. $$

The full variance was 13 square units, so the reduction is \(13-9.4=3.6\) square units. The same result follows from the omitted-component formula: \(q(9)=0.40(9)=3.6\) square units. The 40% missing fraction does not remove 40% of the total variance here; it removes 40% of the residual component, which is \(9/13\) of the total.

The average residual variance is \(\{4+16\}/2=10\) square units, so the full outcome variance is \(4+10=14\) square units. The observation probabilities are \(r_1=0.80\) and \(r_2=0.50\). The residual variance retained is $$ \operatorname{E}[r(X)\operatorname{Var}(\varepsilon\mid X)] = \frac{0.80(4)+0.50(16)}{2} = \frac{3.2+8}{2} = 5.6\ \text{square units}. $$

Consequently, \(\operatorname{Var}(Y_c)=4+5.6=9.6\) square units, and the reduction is \(14-9.6=4.4\) square units. Directly, the omitted residual component is $$ \frac{0.20(4)+0.50(16)}{2} = \frac{0.8+8}{2} = 4.4\ \text{square units}. $$

Although the groups are equally common, most of the reduction comes from group 2: it has both a larger residual variance and a larger missing fraction. A single overall missing percentage would not describe this pattern adequately.

Assumptions, Diagnostics, and Limits

The derivation identifies a specific variance component; it does not guarantee that a fitted regression model represents the data well. In practice, \(m(X)\) is estimated from observed cases. The formulas treat it as the correct conditional mean and set aside estimation error. They also presume that the predictor values needed for imputation are observed and that conditioning on the included predictors makes observation status independent of the outcome residual.

Several checks help assess whether the variance comparison is a useful description:

• Inspect residuals against fitted values and important predictors. Patterns or changing residual spread indicate that a constant-variance model may be inappropriate; the general result allows \(\operatorname{Var}(\varepsilon\mid X)\) to vary.
• Compare the spread of the observed residuals with the spread of the deterministic predictions. Predictions can vary substantially across \(X\) while still omitting residual variation at each \(X\).
• Check whether missingness plausibly depends on the outcome after conditioning on the predictors used. If it does, the conditional-independence assumption fails, and the displayed reduction formula need not describe the completed data.
• Look for predictor values among cases with missing outcomes that are poorly represented among observed cases. A correct conditional model cannot be verified in regions with little or no observed outcome information.

The law of total variance itself does not require a normal outcome distribution. Nor does it require homoscedastic residuals. It requires finite second moments, while the imputation-specific result additionally relies on the stated conditional mean and missingness assumptions. If these assumptions fail, deterministic predictions may omit residual variation and also misrepresent the conditional mean or the distribution of cases being imputed.

Mean imputation is a limiting comparison in which the same overall mean is assigned regardless of predictors, so it does not preserve between-predictor variation in the same way. Complete-case analysis avoids assigning predictions but changes which cases contribute to the analysis and can target a different distribution. Methods that add residual variation to predictions address the missing within-predictor component differently; their properties require separate assumptions and are not established merely by this variance identity.

Common Mistake

A frequent error is to say, “Regression imputation preserves the variance because the predictions differ across people.” That reasoning counts only variation in predicted means. The law of total variance shows that the outcome also varies around those means: \(\operatorname{E}[\operatorname{Var}(Y\mid X)]\). Deterministic predictions for missing outcomes have no such within-predictor residual deviations. The correction is to distinguish the variance of predictions from the full outcome variance, and to account for how many outcomes are missing at predictor values with residual spread.