Proper Multiple Imputation from Predictive Distributions

Why Proper Imputation Requires Two Kinds of Draws

An imputed value is not an observed value. It is a plausible value generated under a model, given the information that was observed. That distinction matters for uncertainty: a missing measurement may vary even if the model parameters were known exactly, and the parameters themselves are usually uncertain because they were estimated from incomplete data.

Proper multiple imputation represents both sources of uncertainty. For each completed data set, it first draws model parameters from a distribution reflecting what the observed data say about them. It then draws the missing entries from their predictive distribution conditional on those parameters and the observed entries. Repeating this process produces multiple plausible completed data sets rather than several copies of one fixed prediction.

Definition: In a model-based multiple-imputation procedure, an imputation is generated by drawing a parameter value \(\theta^{(m)}\) from a distribution representing parameter uncertainty given the observed data, then drawing the missing data \(Y_{\mathrm{mis}}^{(m)}\) from \(p(Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}},\theta^{(m)})\). The observed entries \(Y_{\mathrm{obs}}\) are retained unchanged. The resulting procedure is proper when its imputations reflect the relevant uncertainty in the parameters and missing values under a suitable model and inference procedure; merely producing several completed data sets does not make a procedure proper.

A direct Bayesian version draws \(\theta^{(m)}\) from the posterior distribution \(p(\theta\mid Y_{\mathrm{obs}})\). Under an ignorable missingness mechanism, with the required prior conditions for Bayesian ignorability, that posterior is based on the observed-data likelihood. A frequentist procedure can instead use parameter draws calibrated to sampling uncertainty, such as draws based on an appropriate large-sample approximation. In either case, drawing only from a predictive distribution at a fixed estimate \(\hat{\theta}\) leaves out parameter uncertainty.

The imputation model must also represent the relevant joint structure. For a multivariate record, missing coordinates generally need to be drawn jointly conditional on observed coordinates, not filled one at a time from unrelated marginal distributions. When the missingness mechanism is not ignorable, a model for the mechanism may also be needed. The draws cannot repair a misspecified model or identify features that the observed-data design does not identify.

The Posterior Predictive Distribution

Let \(Y_{\mathrm{obs}}\) denote the observed entries and \(Y_{\mathrm{mis}}\) the missing entries. Suppose a model has likelihood \(p(Y_{\mathrm{obs}},Y_{\mathrm{mis}}\mid\theta)\), and suppose the posterior \(p(\theta\mid Y_{\mathrm{obs}})\) is well-defined. Integrating the conditional predictive distribution over parameter uncertainty gives

p(Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}}) = \int p(Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}},\theta) \,p(\theta\mid Y_{\mathrm{obs}})\,d\theta.

This integral is a mixture of predictive distributions: different parameter values can imply different distributions for the missing entries. It is usually not the same as the conditional distribution evaluated at a single estimated parameter. The following result shows exactly what the two-stage simulation produces.

Theorem: If \(\theta^{(m)}\) is drawn from \(p(\theta\mid Y_{\mathrm{obs}})\), and then \(Y_{\mathrm{mis}}^{(m)}\) is drawn from \(p(Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}},\theta^{(m)})\), the marginal distribution of \(Y_{\mathrm{mis}}^{(m)}\), conditional on \(Y_{\mathrm{obs}}\), is \(p(Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}})\).

Proof. For any measurable set \(A\) of possible missing-data values, condition first on the parameter draw. The probability that the generated missing data fall in \(A\), conditional on the observed data, is

\begin{aligned} P(Y_{\mathrm{mis}}^{(m)}\in A\mid Y_{\mathrm{obs}}) &= \int P(Y_{\mathrm{mis}}^{(m)}\in A \mid Y_{\mathrm{obs}},\theta)\, p(\theta\mid Y_{\mathrm{obs}})\,d\theta\\ &= \int \left\{\int_A p(y_{\mathrm{mis}}\mid Y_{\mathrm{obs}},\theta)\, dy_{\mathrm{mis}}\right\} p(\theta\mid Y_{\mathrm{obs}})\,d\theta\\ &= \int_A p(y_{\mathrm{mis}}\mid Y_{\mathrm{obs}})\,dy_{\mathrm{mis}}. \end{aligned}

The last equality follows from the definition of the posterior predictive distribution. Since this equality holds for every measurable \(A\), the two-stage draw has the claimed distribution. \(\square\)

Worked Example: Drawing a Missing Coordinate from a Gaussian Conditional Distribution

Suppose a two-variable observation \(X=(X_1,X_2)^\mathsf{T}\) follows a bivariate normal distribution with known mean vector \((0,0)^\mathsf{T}\) and covariance matrix

\Sigma= \begin{pmatrix} 1&0.6\\ 0.6&1 \end{pmatrix}.

For one individual, \(X_1=1\) is observed and \(X_2\) is missing. For a bivariate normal distribution, the conditional distribution of \(X_2\) given \(X_1=x_1\) is normal, with mean \(\Sigma_{21}\Sigma_{11}^{-1}x_1\) and variance \(\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}\). Substituting the values gives

E[X_2\mid X_1=1]=0.6(1)=0.6,\qquad \operatorname{Var}(X_2\mid X_1=1)=1-(0.6)(0.6)=0.64.

Thus the predictive distribution is \(X_2\mid X_1=1\sim N(0.6,0.64)\), and a stochastic imputation can be generated as \(0.6+0.8Z\), where \(Z\sim N(0,1)\). For example, if the random-number draw is \(Z=-0.5\), the imputed value is \(0.6+0.8(-0.5)=0.2\). The conditional mean, \(0.6\), is a useful summary, but it is not the imputed value in every draw. Here the covariance parameters are treated as known; a procedure with unknown parameters must also represent their uncertainty.

Drawing Parameters as Well as Missing Values

The next example isolates the role of parameter draws. It uses a Bayesian model so the distribution for those draws is explicit. The predictive distribution includes uncertainty about the unknown mean even though the observation model has a simple normal form.

Worked Example: A Normal Predictive Draw with an Unknown Mean

Suppose \(X_i\mid\mu\sim N(\mu,1)\) independently, and the prior distribution is \(\mu\sim N(0,1)\). Two observed values are \(x_1=1\) and \(x_2=2\), while \(X_3\) is missing. The posterior variance and mean of \(\mu\) are

\operatorname{Var}(\mu\mid x_1,x_2) = \left(1+\frac{2}{1}\right)^{-1} =\frac{1}{3}, \qquad E[\mu\mid x_1,x_2] = \frac{0+1+2}{1+2} =1.

Consequently, a parameter draw can be written \(\mu^{(m)}=1+Z_1/\sqrt{3}\), where \(Z_1\sim N(0,1)\). Given that draw, generate the missing value as \(X_3^{(m)}=\mu^{(m)}+Z_2\), with \(Z_2\sim N(0,1)\) independent of \(Z_1\). The parameter draw moves the center of the conditional predictive distribution; the second random draw represents variation of an individual observation around that center.

Because a sum of independent normal variables is normal, the resulting predictive distribution is

X_3\mid x_1,x_2 \sim N\left(1,\frac{1}{3}+1\right) =N\left(1,\frac{4}{3}\right).

For a numerical illustration, take \(Z_1=0.6\) and \(Z_2=-0.4\). Then \(\mu^{(m)}=1+0.6/\sqrt{3}\approx1.346\), and the missing value is \(X_3^{(m)}\approx1.346-0.4=0.946\). If instead one fixes \(\mu\) at its posterior mean, the predictive distribution is \(N(1,1)\): it omits the extra spread arising from uncertainty in \(\mu\). Fixing the parameter and then filling with the conditional mean gives \(1\) every time, omitting observation-level variation as well.

Why deterministic conditional means are not predictive draws: Suppose \(Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}}\) has a nondegenerate predictive distribution. Filling every missing entry with its conditional mean produces a single fixed value given \(Y_{\mathrm{obs}}\), so its conditional distribution is a point mass. A point mass cannot equal a nondegenerate predictive distribution.

Proof. Write \(c=E[Y_{\mathrm{mis}}\mid Y_{\mathrm{obs}}]\), assuming this expectation exists. Under deterministic conditional-mean filling, \(Y_{\mathrm{mis}}=c\) with conditional probability one. Therefore \(P(Y_{\mathrm{mis}}=c\mid Y_{\mathrm{obs}})=1\) for the imputed value. If the target predictive distribution is nondegenerate, it does not assign probability one to a single value. The two distributions differ, so deterministic filling does not generate predictive draws. \(\square\)

Multiple Imputations and the Role of Repetition

With \(M\) imputations, repeat the two-stage generation \(M\) times, producing completed data sets \(Y^{(1)},\ldots,Y^{(M)}\). Each contains the original observed entries and a new draw for each missing entry. In the Bayesian procedure, each completed data set is generated using a parameter draw from the posterior and a conditional draw from the model. If several entries are missing in one record, their joint conditional distribution should be used; if records are dependent under the model, their dependence should be preserved in the draws.

The imputations should generally differ, but they are not required to be numerically distinct in every data set. Discrete distributions can produce the same value more than once by chance. The relevant requirement is that the procedure samples from a distribution that represents the uncertainty, not that it manufactures visible variation. Likewise, increasing \(M\) gives a more stable numerical summary of the imputation procedure; it does not correct a poor imputation model.

Worked Example: Parameter Draws Affect Joint Predictions

Let \(p\) be the success probability for Bernoulli observations, with prior \(p\sim\operatorname{Beta}(1,1)\). Suppose the observed data contain three successes and one failure. The posterior is \(p\mid Y_{\mathrm{obs}}\sim\operatorname{Beta}(4,2)\), so its mean is \(4/(4+2)=2/3\). Consider two missing Bernoulli values from this same model.

A proper posterior predictive procedure draws one shared \(p^{(m)}\sim\operatorname{Beta}(4,2)\) for the parameter, then draws both missing values independently conditional on that \(p^{(m)}\). For either missing value, the predictive success probability is \(E[p\mid Y_{\mathrm{obs}}]=2/3\). For both to be successes, however, the probability is \(E[p^2\mid Y_{\mathrm{obs}}]\). The beta second moment is

E[p^2\mid Y_{\mathrm{obs}}] = \frac{4(4+1)}{(4+2)(4+2+1)} = \frac{20}{42} = \frac{10}{21}.

If instead the parameter is fixed at its posterior mean \(2/3\), the probability that both are successes is \((2/3)^2=4/9\). These are not equal: \(10/21=30/63\), whereas \(4/9=28/63\). Each missing value has the same marginal success probability under both procedures, but their joint predictive distributions differ. The shared parameter draw carries uncertainty about \(p\) into the joint predictions.

Common Mistakes and Practical Checks

Using the conditional mean as though it were a draw. The mean summarizes a predictive distribution; it does not represent its full range of plausible values. Use a random draw when the goal is predictive imputation.
Drawing missing values at a fixed fitted parameter and calling the result proper. This includes conditional variation but may omit uncertainty in estimated parameters. Use parameter draws calibrated to that uncertainty as well.
Drawing a separate parameter for every missing coordinate without justification. A shared model parameter generally governs multiple observations. Drawing it independently for each coordinate can discard dependence in the predictive distribution.
Assuming repeated imputations guarantee valid inference. Multiple data sets generated from a poorly specified or inappropriate model remain poorly specified imputations. The model and assumptions about missingness still matter.
Changing observed entries during imputation. Standard imputation fills missing entries while retaining the observed data. Replacing observed values is a different procedure and requires a separate justification.

Proper predictive imputation is useful because analyses of completed data should reflect uncertainty that would have remained had the missing values been observed. It is a simulation procedure, not a claim that any one imputed value is the truth. The parameter distribution, conditional predictive model, and assumptions about missingness determine which values are plausible. The ways to combine estimates and their variances across completed data sets are a separate step.

Check Your Understanding

For each question, distinguish the parameter draw from the missing-value draw where relevant.

In a two-stage Bayesian imputation procedure, what distribution is used for the parameter draw, and what distribution is used for the missing-data draw?
Why does drawing missing values at a fixed estimate \(\hat{\theta}\) generally fail to represent all uncertainty?
A missing scalar has a nondegenerate predictive distribution. Explain why replacing it by its conditional mean is not a predictive draw.
In the beta-Bernoulli example, why can two missing values have different joint predictions under a shared random parameter and a fixed posterior-mean parameter, even though their marginal success probabilities agree?
Name one assumption or modeling issue that repeated imputation alone cannot resolve.