Deriving Predictive Draws Under The Normal Linear Model

Why a Prediction Needs More Than a Fitted Mean

In a chained-equations update for a continuous variable, regression predictors are held at their current values while the variable is modeled using cases with observed outcomes. For a case whose outcome is missing, the fitted regression gives a conditional mean. That mean is not, by itself, a predictive draw: it does not account for uncertainty in the estimated regression coefficients or for the fact that individual outcomes vary around the regression mean.

This tutorial derives how those two sources of variation enter the predictive distribution. The focus is the normal linear model. It does not establish that this model is appropriate for every continuous variable; the model’s form, predictor set, and assumptions still require substantive and diagnostic consideration.

Model and Assumptions

Write the \(n\) observed outcomes as the vector \(\mathbf{y}\), and arrange their predictor vectors as the rows of the \(n\times p\) design matrix \(\mathbf{X}\). The model is

Thus, conditional on the predictors, errors have mean zero, common variance \(\sigma^2\), and are independent and normally distributed. The design matrix must have full column rank, so \(n>p\) is needed to estimate the residual variance using residual degrees of freedom. The model also assumes that the conditional mean is linear in the included predictors. These are assumptions about the outcome model, not guarantees that the model is suitable for a particular data set.

For a case with missing outcome, denote its predictor vector by \(\mathbf{x}_0\). The target is \(Y_0\), whose conditional mean at given coefficients is \(\mathbf{x}_0^T\boldsymbol{\beta}\). Under least squares, \(\hat{\boldsymbol{\beta}}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\), and the residual variance estimate is \(s^2=\mathrm{RSS}/(n-p)\), where \(\mathrm{RSS}=(\mathbf{y}-\mathbf{X}\hat{\boldsymbol{\beta}})^T(\mathbf{y}-\mathbf{X}\hat{\boldsymbol{\beta}})\).

Separating the Two Sources of Predictive Variation

First suppose \(\sigma^2\) is known and coefficients are uncertain under the normal linear model. The least-squares coefficient estimator has covariance matrix \(\sigma^2(\mathbf{X}^T\mathbf{X})^{-1}\). Consequently, the fitted conditional mean at \(\mathbf{x}_0\) has variance

The quantity \(h_0\) measures how uncertain the fitted mean is at this predictor vector. It depends on the design and the location of \(\mathbf{x}_0\), not on the outcome’s residual variance alone. Even if the conditional mean were known, a new outcome would vary around it with variance \(\sigma^2\). The two components therefore give

The residual term is not optional when predicting an individual outcome. Omitting it gives uncertainty for the estimated conditional mean, not for a new outcome around that mean.

Deriving the Predictive Distribution When Variance Is Unknown

In practice, \(\sigma^2\) is usually unknown. To obtain a predictive distribution, one standard derivation uses the normal linear model with the noninformative prior \(p(\boldsymbol{\beta},\sigma^2)\propto1/\sigma^2\). With a full-rank \(\mathbf{X}\), \(n>p\), and \(\mathrm{RSS}>0\), the posterior distributions are

Given \(\sigma^2\), a new outcome has residual variance \(\sigma^2\), while uncertainty in \(\mathbf{x}_0^T\boldsymbol{\beta}\) contributes \(\sigma^2h_0\). Integrating over the coefficient distribution first therefore gives

Now integrate over the posterior uncertainty in \(\sigma^2\). A normal distribution whose variance is scaled by an inverse-chi-square random variable has a Student t distribution. In this case, the degrees of freedom are \(\nu=n-p\), the location is \(\mathbf{x}_0^T\hat{\boldsymbol{\beta}}\), and the scale is \(s\sqrt{1+h_0}\):

The scale in this expression is not generally the standard deviation of the predictive distribution. When \(n-p>2\), its variance is \(s^2(1+h_0)(n-p)/(n-p-2)\); for \(n-p\leq2\), the t distribution has no finite variance. The t form also reflects uncertainty about \(\sigma^2\), in addition to the coefficient and residual components conditional on \(\sigma^2\).

Turning the Distribution Into a Predictive Draw

A draw from the predictive distribution can be generated in stages. Draw \(\sigma^2\) from its posterior distribution, draw coefficients from their conditional posterior given that variance, then draw a new residual and add it to the resulting conditional mean. This construction shows concretely where each source of uncertainty enters.

Diagnostics, Alternatives, and Limitations

The derivation depends on a reasonable conditional normal linear model. Inspect residual patterns for curvature, changing spread, and influential observations; examine whether predictions for missing cases are far beyond the predictor region represented by observed cases. A large \(h_0\) signals greater uncertainty in the fitted mean, and may indicate extrapolation. Such checks do not prove the model correct, but conspicuous departures can make its predictive draws unreliable.

The main alternative is a plug-in prediction, which uses \(\mathbf{x}_0^T\hat{\boldsymbol{\beta}}\) as the imputed value or adds residual noise while holding fitted parameters fixed. A deterministic plug-in omits both sources of uncertainty. Adding residual noise alone includes individual-level variation but still treats estimated coefficients as known. By contrast, the predictive construction above allows both coefficient and residual variation. In multiple imputation, omitting parameter uncertainty can understate uncertainty in subsequent analyses.

The normal linear model can also be unsuitable when the conditional mean is nonlinear, the residual spread changes substantially with predictors, or the outcome’s support is bounded or strongly skewed. Transformations, models for bounded values, and two-part models are distinct choices for such settings; a normal predictive draw should not be assumed to respect a variable’s natural limits. The predictive derivation also does not by itself settle whether the predictors included in an imputation model are adequate or whether missingness assumptions are appropriate.