Proper MI Variance for Reference-Based Imputations

Why Variance Requires More Than One Completed Data Set

A reference-based imputation is built from a specified model for outcomes after discontinuation. Under jump to reference (J2R), for example, post-discontinuation outcomes follow the reference-arm behavior specified by the imputation model; under copy reference (CR), the reference behavior is applied while retaining the copied history prescribed by that rule. These assumptions determine how missing outcomes are generated. They do not make the parameters of the imputation model known.

A completed data set contains both observed outcomes and simulated missing outcomes. An analysis of that data set gives an estimate, such as a treatment-minus-reference mean difference, and an estimated variance. But a single completed data set cannot show how much the estimate would change if the missing outcomes, or the model parameters used to generate them, were drawn differently. Proper multiple imputation represents that uncertainty by creating \(M\) completed data sets and combining two kinds of variation.

Here \(U_m\) describes sampling variation in the analysis estimate if completed data set \(m\) were treated as complete. The quantity \(B\) describes how the estimate changes across imputations. The factor \(1+1/M\) includes both the estimated missing-data contribution \(B\) and the extra Monte Carlo variance from using a finite number \(M\) of imputations. The corresponding standard error is \(\sqrt{T}\).

These quantities have the units implied by the target estimate: if \(Q_m\) is a mean difference in symptom-score points, then \(U_m\), \(B\), and \(T\) are in squared points, while \(\sqrt{T}\) is in points. The total variance is not obtained by averaging the \(U_m\) values alone; doing so ignores variation in the estimates across imputations.

Where Parameter Uncertainty Enters

In proper reference-based multiple imputation, missing values are not generated only from a predictive distribution with fixed, estimated parameters. The imputation procedure also draws uncertain parameters, or otherwise uses a procedure that adequately represents their sampling uncertainty. For example, the model may draw regression coefficients and covariance parameters and then draw missing outcomes conditional on the observed history and the reference-based rule. The precise draw must respect the chosen reference rule and the model used for the observed data.

When parameters are drawn, different imputations can have different predictive means, conditional variances, or both. The resulting \(Q_m\) values therefore vary because of uncertainty in the missing outcomes and uncertainty in the imputation parameters. That variation contributes to \(B\). Meanwhile, \(U_m\) still represents the complete-data analysis variance for imputation \(m\); it is not a substitute for the parameter uncertainty carried by the imputation draws.

This distinction matters especially when a reference-based rule borrows information from a reference arm. Uncertainty in estimated reference-arm means, regression relationships, and covariance parameters can affect the distribution assigned to missing outcomes. Treating those estimated quantities as fixed may make imputations too similar and can omit an important source of uncertainty. Proper variance estimation does not, however, establish that J2R, CR, or any other chosen assumption is scientifically correct. It quantifies uncertainty under the imputation model that was specified.

Why the Total Variance Has This Form

The formula can be understood by separating uncertainty in the target estimate from the simulation error in its finite-imputation average. In repeated imputations under a fixed observed data set, \(\bar{Q}\) estimates the estimate averaged over the imputation distribution. The spread of the \(Q_m\) values estimates the variation attributable to that distribution, represented by \(B\). The finite average \(\bar{Q}\) also has Monte Carlo variance approximately \(B/M\), since it averages \(M\) draws with estimated variance \(B\).

Under the standard large-sample multiple-imputation approximation, the uncertainty in the pooled estimate is estimated by adding the average complete-data variance, the missing-data contribution, and the finite-imputation simulation contribution:

The \(B/M\) term is not a second estimate of the uncertainty in the missing outcomes. It accounts for the fact that \(\bar{Q}\) is based on a finite Monte Carlo sample of imputations. As \(M\) increases, that term decreases, but \(B\) itself does not disappear: it represents uncertainty associated with the missing data under the imputation model. Increasing \(M\) improves the numerical stability of the pooled estimate; it does not remove uncertainty created by missing outcomes.

This argument uses the usual multiple-imputation conditions: the imputation procedure must represent the relevant uncertainty in the missing values and model parameters; the analysis must estimate the same target quantity across completed data sets; and the complete-data variance estimates \(U_m\) must be suitable for that analysis. The standard formula is an approximation, commonly justified when the analysis estimate is approximately normally distributed and its complete-data variance estimate is reliable. For reference-based analyses, these conditions apply relative to the specified reference-based imputation model. They do not turn a sensitivity assumption into an empirically verified fact.

What Goes Wrong With Fixed-Parameter Imputation

A common shortcut is to estimate the imputation-model parameters once and then hold them fixed in every imputation. Missing outcomes may still be randomly drawn, so the completed-data estimates can vary and \(B\) need not be zero. But that variation reflects only the random draws conditional on the fixed parameter estimates. It can fail to represent the additional variation that would arise if the model parameters themselves were uncertain.

This is not corrected merely by making many imputations. A larger \(M\) reduces Monte Carlo error in estimating the imputation distribution; it cannot restore parameter uncertainty that the imputation procedure excluded. If the missing values are strongly influenced by uncertain fitted quantities, the resulting \(B\), and consequently \(T\), may be too small. The direction and magnitude of any error depend on the model and analysis, so a fixed-parameter procedure should not be assumed to give a valid variance estimate.

Reference-Based Variation Across Imputations

The reference-based rule affects the distribution from which missing outcomes are generated, so it can affect both the completed-data estimates and their between-imputation variation. For example, with J2R, the post-discontinuation distribution is governed by the specified reference behavior, conditional on the information retained by the model. With CR, the copied reference behavior and the retained history determine the predictive distribution. In either case, the uncertainty in estimated means, covariance structures, and other model parameters should be represented as required by the chosen procedure.

The next hypothetical calculation illustrates the variance components for a treatment-minus-reference contrast after a reference-based imputation. Suppose five completed data sets produce estimates \(1,2,3,2,\) and \(2\) points, and complete-data variance estimates \(0.36,0.49,0.36,0.49,\) and \(0.30\) squared points. The estimate average is \(2\) points. The deviations are \(-1,0,1,0,0\), so:

Diagnostics, Alternatives, and Limitations

Variance calculations should be checked alongside the imputation procedure, rather than treated as a final mechanical step. Useful checks include:

• Check that parameters are uncertain in the procedure. Confirm that the imputation method draws or otherwise accounts for uncertainty in estimated model quantities, rather than merely substituting estimates into a predictive model.
• Inspect the \(Q_m\) values. Look for unusual outliers, strong skewness, or instability across imputations. These may signal a poorly behaved predictive distribution, influential observations, or a small reference group.
• Check the \(U_m\) values. Large discrepancies can arise when completed data sets differ greatly or when the analysis variance calculation is unstable. Confirm that every \(U_m\) estimates the variance of the same target estimate in the same units.
• Assess Monte Carlo stability. If results change materially when the number of imputations is increased, the finite simulation may be too noisy for reliable reporting. Increasing \(M\) addresses this numerical issue, not misspecification of the reference-based assumptions.

A single imputation discards the between-imputation variation and therefore does not provide the usual multiple-imputation variance estimate. Fixed-parameter multiple imputation can include residual variation but may omit parameter uncertainty. Proper reference-based multiple imputation aims to include both sources through parameter and outcome draws, then combines \(\bar{U}\) and \(B\) using \(T=\bar{U}+(1+1/M)B\). These approaches are not interchangeable simply because each produces completed data.

The variance formula has limitations. It relies on adequate imputation and analysis models, appropriate parameter draws, and the large-sample conditions behind the usual multiple-imputation approximation. It estimates uncertainty conditional on the selected reference-based assumptions; it does not account automatically for uncertainty about which sensitivity assumption is scientifically most credible. Nor does a plausible value of \(T\) diagnose whether the imputed distributions match the intended clinical scenario. Those questions require substantive justification and separate checks of the predictive behavior.

A frequent mistake is to report only \(\bar{U}\), reasoning that every completed data set already contains imputed outcomes. That reasoning overlooks the fact that different imputations can yield different analysis estimates. The correction is to calculate \(B\) from the \(Q_m\) values and include it, with the finite-\(M\) factor, in \(T\). A second mistake is to increase \(M\) while leaving model parameters fixed and assume that this repairs omitted parameter uncertainty. It does not: more imputations reduce Monte Carlo error, but proper parameter uncertainty must be represented by the imputation procedure itself.