Parameter Draws and Residual Variation

Why a Conditional Update Has Two Sources of Variation

In a MICE update for a variable with missing values, the conditioning variables are treated as known for that update: their observed values and current imputations form the design matrix. A regression fit to the observed responses estimates a conditional mean, but using that fitted mean alone does not represent all the uncertainty in an imputed response. There are two distinct sources to account for: uncertainty about the regression parameters and the remaining variation of individual responses around the regression mean.

The distinction matters even when the conditional model is a simple normal linear regression. Drawing a new coefficient vector allows the fitted relationship to vary from one update to another. Drawing a residual for each missing case allows cases with the same predictors to receive different values. These are not interchangeable forms of randomness: the parameter draw is shared across all missing cases in that variable update, whereas residual draws are case-specific.

Definition: In a normal linear-regression imputation update, a parameter draw is a draw of the regression coefficients and residual variance from their conditional posterior distribution given the observed responses and design matrix. A residual draw is a separate random error added to the drawn conditional mean for each missing response. Both contribute to stochastic imputations.

Model and Parameter Draws

Suppose \(Y\) is the variable being updated, \(X_o\) is the design matrix for the \(n_o\) cases with observed \(Y\), and \(y_o\) is their response vector. The design matrix includes an intercept if the specified conditional model includes one. Let \(p\) be the number of columns in \(X_o\), and let \(X_m\) contain the design rows for cases whose \(Y\) values are missing. For this update, use the normal linear model

y_o \mid \beta,\sigma^2 \sim N(X_o\beta,\sigma^2 I).

This model assumes a linear conditional mean, independent errors with a common variance, and normally distributed errors. The conditioning variables and their values are taken as given for the update. The model assumptions concern the conditional distribution being used; they do not by themselves establish assumptions about the missingness mechanism.

A standard parameter-draw scheme uses the diffuse prior \(p(\beta,\sigma^2)\propto 1/\sigma^2\). Define \(A=(X_o^\mathsf{T}X_o)^{-1}\), when that inverse exists, and let \(\hat{\beta}\) be the least-squares coefficient estimate. The residual sum of squares is \(SSE=(y_o-X_o\hat{\beta})^\mathsf{T}(y_o-X_o\hat{\beta})\), with \(df=n_o-p\). Under this model and prior, the posterior draws can be generated as follows.

Parameter-draw result: If \(X_o\) has full column rank, \(n_o>p\), and \(SSE>0\), the posterior is proper. Draw \(U\sim\chi^2_{df}\), set \(\sigma^2=SSE/U\), and then draw \(\beta\mid\sigma^2,y_o\sim N(\hat{\beta},\sigma^2 A)\). The condition \(SSE>0\) is necessary for propriety under this diffuse prior; full rank and \(n_o>p\) alone do not guarantee it.

In words, the first draw allows the residual variance to vary. The coefficient draw then varies around the least-squares fit, with covariance scaled by that drawn variance. For the missing cases, calculate the conditional means \(X_m\beta\) using this single drawn coefficient vector. Finally, add individual residual draws:

y_m^{\mathrm{imp}}=X_m\beta+\varepsilon_m, \qquad \varepsilon_m\mid\sigma^2\sim N(0,\sigma^2 I).

Conditional on the drawn parameters, residuals for different missing cases are independent in this model. A single \(\beta\) and \(\sigma^2\) draw is used for the entire update, not a separate parameter draw for every missing response.

Worked Regression Update

Worked Example: Draw Parameters and Impute One Response

Hypothetical data. Suppose four observed responses are \(y_o=(2,3,5,4)^\mathsf{T}\), at predictor values \(x=0,1,2,3\). We use a linear regression with an intercept, and one additional case has \(x=4\) but a missing response. The observed design matrix has full rank, there are \(n_o=4\) observed responses and \(p=2\) coefficients, and \(n_o>p\). We now calculate the fit and a stochastic update.

The observed predictor and response means are \(\bar{x}=1.5\) and \(\bar{y}=3.5\). The slope is \(\sum(x_i-\bar{x})(y_i-\bar{y})/\sum(x_i-\bar{x})^2=4/5=0.8\), and the intercept is \(3.5-0.8(1.5)=2.3\). Thus \(\hat{\beta}=(2.3,0.8)^\mathsf{T}\). The fitted values are \(2.3,3.1,3.9,4.7\), so the residuals are \(-0.3,-0.1,1.1,-0.7\), and

$$ SSE=(-0.3)^2+(-0.1)^2+(1.1)^2+(-0.7)^2 =0.09+0.01+1.21+0.49=1.8. $$

Here \(df=4-2=2\), and \(SSE>0\), so the stated conditions for a proper posterior hold. For illustration, suppose the random-number generator gives \(U=3\), a possible draw from \(\chi^2_2\). Then the variance draw is \(\sigma^2=1.8/3=0.6\), with residual standard deviation \(\sqrt{0.6}\approx0.7746\).

For predictor values \(0,1,2,3\), \(X_o^\mathsf{T}X_o=\begin{pmatrix}4&6\\6&14\end{pmatrix}\), and

A=(X_o^\mathsf{T}X_o)^{-1} = \begin{pmatrix} 0.7&-0.3\\ -0.3&0.2 \end{pmatrix}.

A Cholesky factor, rounded to six decimal places, is \(L\approx\begin{pmatrix}0.836660&0\\-0.358569&0.267261\end{pmatrix}\), so \(LL^\mathsf{T}\approx A\). If a standard normal coefficient draw is \(z=(0,1)^\mathsf{T}\), then \(\beta=\hat{\beta}+\sqrt{0.6}Lz\approx(2.300000,1.007020)^\mathsf{T}\).

Using the displayed rounded coefficients, the drawn conditional mean at \(x=4\) is \(2.300000+4(1.007020)\approx6.328080\). Suppose the individual residual draw is \(\varepsilon=-0.5\sqrt{0.6}\approx-0.387298\). The imputed response is therefore \(6.328080-0.387298\approx5.940782\). The conditional mean alone would be about \(6.328080\); the residual draw moves this particular imputation to about \(5.940782\).

How the Two Sources Contribute

For a fixed variance draw, the difference between parameter uncertainty and residual variation can be made explicit. For one missing case with design row \(x^\mathsf{T}\), the drawn conditional mean is \(x^\mathsf{T}\beta\). Given the observed data and \(\sigma^2\), its variance from drawing \(\beta\) is \(\sigma^2 x^\mathsf{T}Ax\). The residual has variance \(\sigma^2\). Since the residual draw is independent of the coefficient draw conditional on \(\sigma^2\), the variance of the imputed value is their sum:

\operatorname{Var}(Y^{\mathrm{imp}}\mid y_o,\sigma^2) = \underbrace{\sigma^2 x^\mathsf{T}Ax}_{\text{parameter uncertainty}} + \underbrace{\sigma^2}_{\text{residual variation}}.

This is a conditional variance decomposition for a fixed variance draw. It shows why residual variation cannot replace parameter uncertainty, or vice versa. The first term depends on how precisely the regression mean is estimated at \(x\); the second represents response variation around that mean.

Worked Example: Quantifying Both Contributions

Use the fitted design in the previous example and the illustrative variance draw \(\sigma^2=0.6\). At \(x=4\), the design row is \(x^\mathsf{T}=(1,4)\). The leverage contribution for the conditional mean is

x^\mathsf{T}Ax = (1,4) \begin{pmatrix}0.7&-0.3\\-0.3&0.2\end{pmatrix} \begin{pmatrix}1\\4\end{pmatrix} =0.7-2.4+3.2=1.5.

Thus the parameter-induced variance is \(0.6(1.5)=0.9\) response-units squared. The individual residual variance is \(0.6\) response-units squared, giving total conditional variance \(0.9+0.6=1.5\) response-units squared and standard deviation \(\sqrt{1.5}\approx1.225\) response units. The calculation is conditional on the chosen variance draw; across updates, the variance draw itself also changes.

If two responses are missing at \(x=4\) and \(x=5\), their conditional means share the same coefficient draw. The covariance due to that shared draw is \(\sigma^2(1,4)A(1,5)^\mathsf{T}\). Since \(A(1,5)^\mathsf{T}=(-0.8,0.7)^\mathsf{T}\), this covariance is \(0.6[1(-0.8)+4(0.7)]=0.6(2)=1.2\) response-units squared. Independent residual draws add no covariance between the two cases. Thus a coefficient draw can move their conditional means together, even though their residual draws are separate.

Why the Posterior Conditions Matter

Worked Example: A Perfect Fit Does Not Give a Proper Diffuse-Prior Draw

Consider three hypothetical observed responses \(y=(1,3,5)^\mathsf{T}\) at \(x=0,1,2\), again using an intercept and slope. The fitted line is exactly \(\hat{y}=1+2x\), so every residual is zero and \(SSE=0\). The design has rank \(p=2\), and \(n_o=3>2\), but those conditions do not suffice for a proper posterior.

Under the diffuse prior, after integrating over \(\beta\), the posterior density for \(\sigma^2\) is proportional to \((\sigma^2)^{-df/2-1}\exp[-SSE/(2\sigma^2)]\). Here \(df=1\) and \(SSE=0\), so its dependence on the variance is \((\sigma^2)^{-3/2}\). Its integral near zero diverges: \(\int_0^1(\sigma^2)^{-3/2}\,d\sigma^2=\infty\). Therefore the posterior is improper, and the stated inverse-chi-square parameter draw cannot be used with scale \(SSE=0\). A perfect fit is a failure condition to investigate, not a reason to generate a zero-variance imputation.

In a continuous normal-error model with \(n_o>p\), a perfect fit has probability zero under ordinary nondegenerate errors, but it is not ruled out by rank and sample size alone. It can occur in actual data, particularly with small samples, duplicated structure, or an overly flexible design. Check rank, residual degrees of freedom, and the residual sum of squares before attempting the draw.

Alternatives, Diagnostics, and Limitations

The main alternatives differ in which uncertainty they omit. Deterministically imputing the fitted conditional mean omits both forms of variation and tends to make completed data too concentrated. Drawing coefficients but setting each response equal to its conditional mean includes variation in the fitted relationship but omits individual residual variation. Adding residual draws around a fixed least-squares fit includes individual variation but omits uncertainty about the fitted coefficients. The parameter-and-residual approach represents both under the stated model.

Several checks help identify problems with this update:

Check the design and posterior conditions. A rank-deficient design prevents the inverse \(A\) from existing as written. Too few observed responses leave no positive residual degrees of freedom, while \(SSE=0\) invalidates the diffuse-prior draw described here.
Check whether the draw is genuinely stochastic. If repeated imputations at the same predictor values are always identical, the implementation may be substituting conditional means for residual draws, reusing fixed parameters, or otherwise suppressing variation.
Check the conditional model. Strong nonlinearity, unequal residual spread, dependence, or serious non-normality can make normal regression draws a poor representation of the conditional distribution. Random draws do not repair a misspecified model.
Check the scale and plausibility of generated values. A large residual draw can produce an implausible value, especially for outcomes with natural bounds. Normal regression is not automatically suitable for binary, categorical, or bounded outcomes.

The calculation above is specific to a normal linear conditional model with the stated prior. Other conditional models require parameter-drawing and response-drawing steps appropriate to their likelihoods. Even in the normal model, the two draws do not guarantee that the conditional model is well specified or that the broader imputation procedure is appropriate. They ensure that, within this model, the imputation includes both parameter uncertainty and individual response variation.

Key takeaway: In a stochastic normal-regression update, first draw the model parameters using the observed responses, then calculate means for missing cases, and finally draw a separate residual for each case. The shared parameter draw and case-specific residual draws represent distinct uncertainty.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A conditional model has \(n_o=12\) observed responses and \(p=3\) coefficients. What are its residual degrees of freedom, and what other conditions are needed for the diffuse-prior draw stated here to be proper?
For a missing case, suppose the variance draw is \(4\) and \(x^\mathsf{T}Ax=0.5\). What are the parameter-induced variance, residual variance, and total conditional variance of the imputed value?
Why does using a coefficient draw but no residual draw fail to represent the full uncertainty in an individual imputed response?
In the perfect-fit example, why do full rank and positive residual degrees of freedom fail to make the stated posterior proper?

Solutions. (1) \(df=12-3=9\). The design must have full column rank and \(SSE\) must be positive; the model also uses the stated normal linear likelihood and diffuse prior. (2) Parameter-induced variance is \(4(0.5)=2\); residual variance is \(4\); total conditional variance is \(2+4=6\) response-units squared. (3) The coefficient draw changes the conditional mean, but without a residual draw each imputation is exactly that mean. Individual variation around the mean, with variance \(\sigma^2\), is omitted. (4) In that example \(SSE=0\), so the exponential term in the variance posterior does not control its behavior near zero; the resulting density is nonintegrable there despite rank and positive residual degrees of freedom.