Regression Imputation and Residual Variation

Why a Predicted Value Is Not the Whole Story

When a regression analysis has missing predictor values, a fitted model can use other recorded variables to estimate them. This can be more informative than filling every gap with the same overall mean: records with different observed characteristics can receive different predicted values. But a prediction is not the same as the unobserved value. Even records with identical predictors can differ from the regression’s predicted average.

That distinction matters in a completed data set. Replacing each missing covariate by its predicted conditional mean gives a smooth, reproducible fill, but it leaves out the residual variation around that mean. Adding a residual draw puts some of that variation back. Neither step, by itself, represents all the uncertainty involved in estimating the imputation model and completing missing data.

The Model Behind a Regression Imputation

Suppose \(X_i\) is a covariate with some missing values and \(Z_i\) denotes observed information used to predict it. A simple linear model for the observed values of \(X\) is

The fitted conditional mean for a record with observed \(Z_i\) is \(\hat{X}_i=\hat{\beta}_0+\hat{\beta}_1Z_i\). Deterministic regression imputation assigns this value to a missing \(X_i\). Residual-augmented imputation instead assigns \(\hat{X}_i+e_i^*\), where \(e_i^*\) is a residual draw intended to represent the remaining variation in \(X\) among records with similar \(Z\).

These procedures rely on a useful model for the conditional distribution of \(X\) given the included information. The fitted conditional mean should be reasonably specified, and the observed records used to fit it should provide relevant information about records with missing \(X\). For instance, if missingness depends on observed variables, those variables may need to be represented in the imputation model. If records with missing \(X\) have predictor values outside the range of the observed cases, the model is extrapolating, so its predictions deserve particular caution.

A residual-draw method has an additional requirement: the residuals used as draws should reasonably represent the residual variation for the records being imputed. A common simple approach assumes roughly constant residual spread across the predictor range. If that spread changes substantially with \(Z\), a single residual distribution may not be suitable. Normality is needed for a method that draws errors from a normal distribution, but not for the basic idea of adding resampled residuals.

What Deterministic Imputation Leaves Out

The reason for concern can be stated using the law of total variance. If the conditional mean model is appropriate, then the variation in \(X\) has two components: variation in the conditional mean across values of \(Z\), and residual variation among records with the same \(Z\).

A deterministic imputation sets a missing \(X_i\) to its estimated conditional mean. For the imputed records, it therefore treats the residual around that mean as zero. In effect, it omits the second component, \(E[\operatorname{Var}(X\mid Z)]\), from the variation represented by those filled-in values. This does not mean the overall variance in every particular completed sample must decrease by a fixed amount; it means the imputed cases do not carry the conditional spread that the model says exists.

The consequence can reach beyond a summary of \(X\). Regression coefficients in a later analysis depend on the covariate values and their relationships with other variables. Replacing missing values by conditional means can alter those relationships. The direction and size of the effect depend on the data and analysis; deterministic imputation does not guarantee that a later slope will move toward zero or in any other fixed direction.

Adding Residual Variation

A residual-augmented imputation starts from the same predicted mean and adds variation. One approach draws a residual from the fitted model’s residuals and adds it to the prediction. Another approach draws an error from a stated distribution, such as a normal distribution with a fitted residual standard deviation, when that distribution is justified. In either case, the result is a plausible draw under the chosen model, not a recovered observation.

For the method to represent residual variation appropriately, the residuals must be suitable for the records being imputed. Resampling residuals without regard to \(Z\) is questionable if residual spread or shape changes with \(Z\). A model that predicts only a conditional mean can also fail to represent important features of the conditional distribution, such as skewness or bounds. The imputed value should be checked for plausibility in the variable’s units and range.

Why Residual Noise Is Not All the Uncertainty

Adding residual variation addresses one source of uncertainty: individuals with the same recorded predictors can have different covariate values. But the regression model itself was estimated from a finite set of observed cases. Its fitted coefficients are uncertain, and the estimated residual distribution is uncertain too. A procedure that holds the fitted coefficients fixed and only draws residuals does not include those sources of uncertainty.

There are also uncertainties about whether the model form is adequate, whether the records used to fit it are informative about those with missing values, and whether the assumptions about missingness are reasonable. Residual noise cannot repair a poorly chosen predictor set, an unsuitable functional form, or a mismatch between observed and missing cases. These are distinct issues from whether a random residual was added.

Common Mistake: Treating Noise as a Complete Fix

A common mistake is to argue, “I added a random residual, so the imputed values now include all relevant uncertainty.” That conclusion is too strong. Residual draws can restore some within-predictor variation, provided the residual model is suitable. They do not, on their own, represent uncertainty in the fitted regression coefficients, uncertainty about the model, or uncertainty associated with the missing-data process.

The correction is to describe exactly what the imputation method does. A deterministic prediction fills a gap with an estimated conditional mean. A residual-augmented prediction adds a draw intended to represent residual variation. Neither should be presented as the known value or as a complete accounting of uncertainty. When the goal includes representing multiple sources of uncertainty in regression inference, repeated plausible completions and their analysis require a broader procedure than one residual-augmented fill.