Why Compare the Two Analyses?
When a regression dataset contains missing values, a complete-case analysis and a multiple-imputation analysis may use different information. A complete-case regression includes only records with observed values for every variable in the analysis. Multiple imputation instead creates several completed datasets, fits the same regression in each, and combines the estimates and their uncertainty. The resulting coefficients and standard errors can differ because the analyzed records, predictor distributions, and treatment of missing-data uncertainty differ.
The comparison is informative, but it is not a contest in which the method producing the smaller standard error or the more plausible coefficient automatically wins. Each result answers a question under assumptions about the regression model and the missing-data process. The purpose of comparison is to understand what changed, how much it changed, and which features of the data or analysis may explain the difference.
What the Comparison Can and Cannot Show
For a linear regression, suppose the target model relates an outcome \(Y\) to a predictor \(X\) and other covariates \(Z\). The coefficient of \(X\) describes the modelled change in the conditional mean of \(Y\) for a one-unit increase in \(X\), holding the included covariates fixed. A coefficient comparison is meaningful only if the two analyses fit the same model and interpret that coefficient in the same units and context.
The two methods can differ in at least four ways:
- Coefficient estimates: Complete cases may represent a selected subset, while imputation uses observed information from records that lack some analysis values.
- Standard errors: Complete-case uncertainty is calculated from the retained records under the fitted model. A properly pooled imputation standard error also reflects variation among the imputed-data estimates.
- Sample composition: Complete cases can over- or under-represent groups if completeness varies across them. Imputation retains records in the completed analyses, but does not make their missing values observed facts.
- Predictor support: The observed range and distribution of a predictor may differ in complete cases from the range represented in completed datasets. Imputed values can add information only to the extent justified by the imputation model and observed data.
Neither analysis is automatically valid. Complete-case results depend on whether restricting the analysis to complete records changes the relationship being estimated, as well as on the regression model and the data structure. Missing completely at random is one sufficient condition for complete cases to remain representative, but it is not the only setting in which a particular regression coefficient could be valid. Multiple imputation also requires suitable assumptions: for example, that the missingness can be explained by observed information included in an appropriate imputation model, and that the imputation procedure represents uncertainty adequately. Agreement between the results does not verify these assumptions.
Reading Pooled Estimates and Standard Errors
For a particular regression coefficient, let \(Q_i\) be its estimate in imputed dataset \(i\), and let \(U_i\) be the estimated variance of that coefficient in that dataset. The pooled estimate is the average of the estimates. The pooled variance combines the average within-imputation variance with the between-imputation variance:
Here \(\bar{U}\) summarizes the regression uncertainty within the completed datasets, while \(B\) measures how much the coefficient estimates vary across them. The factor \(1+1/m\) accounts for the fact that only a finite number of imputations were used. Inference after imputation also uses degrees of freedom that reflect the amount of missing information; a pooled standard error alone does not specify a complete confidence interval or test.
The structure of the pooled variance follows the general variance decomposition: total uncertainty includes uncertainty within a completed-data analysis and uncertainty about how the missing values could plausibly have been completed. The between-imputation sample variance \(B\) estimates the latter component. Because a finite set of imputations estimates that component with some imputation-sampling error, the pooled variance includes the additional \(B/m\), giving \(T=\bar{U}+B+B/m\).
Worked Example: Comparing Coefficients and Standard Errors
Suppose a hypothetical study examines the association between weekly exercise and a health score, adjusting for age and smoking status. The coefficient of exercise is measured in health-score points per additional hour of exercise per week. A complete-case analysis retains 62 records and estimates the coefficient as \(1.85\), with standard error \(0.300\). In five completed datasets, the coefficient estimates are \(2.10,\ 2.00,\ 2.20,\ 2.10,\ 2.10\), and each within-imputation variance is \(0.160\).
The pooled estimate is
The average within-imputation variance is \(\bar{U}=0.160\). The deviations from the pooled estimate are \(0,\ -0.10,\ 0.10,\ 0,\ 0\), so the between-imputation variance is
Thus the total pooled variance and standard error are
In this hypothetical example, the imputed estimate is \(2.10\) health-score points per additional weekly exercise hour, compared with \(1.85\) points in the complete cases. The pooled imputation standard error is about \(0.407\), compared with \(0.300\) for the complete-case estimate. The difference in coefficients is \(2.10-1.85=0.25\) points per hour. The larger imputation standard error is not a contradiction: it includes variation among the imputed estimates as well as within-dataset uncertainty. Neither estimate should be declared correct merely because it is larger, smaller, or more precise; the models, assumptions, and analyzed records must also be examined.
Compare Who Contributes to the Analysis
A complete-case analysis can silently change the sample being described. A useful comparison reports how many records were eligible, how many were complete, and whether completeness differs across observed groups or across important predictors. This does not by itself establish bias, but it reveals whether the complete-case estimate is being driven by a noticeably different subset.
Worked Example: Checking Sample Composition
A hypothetical dataset contains 100 people. Age is fully observed, but one or more other regression variables are missing for some people. Of 45 younger people, 20 are complete cases and 25 are incomplete. Of 55 older people, 40 are complete cases and 15 are incomplete.
The complete-case retention rates are
There are \(20+40=60\) complete cases. Younger people make up \(20/60=33.3\%\) of that subset, whereas they make up \(45/100=45.0\%\) of the full dataset. Thus the complete-case analysis includes a smaller proportion of younger people than the original dataset. A regression coefficient could change if the outcome-predictor relationship varies by age, if age is related to the predictors in a way that affects the fitted model, or if other variables linked to completeness matter.
Multiple imputation can retain all 100 records in each completed analysis, but this does not mean all 100 people supplied observed values for every variable. Their missing values are represented by model-based draws. The difference in sample composition is a reason to inspect the results and assumptions, not proof that the complete-case estimate is biased or that the imputed estimate is valid.
Compare Predictor Support, Not Just Coefficients
A regression coefficient summarizes a fitted relationship over the data that inform the fit. Therefore, compare the distributions and ranges of key predictors among complete cases with those in the observed data and in the completed datasets. Pay attention to sparse regions, influential extreme values, and values that are mostly or entirely supplied by imputation. A completed dataset may contain a wider predictor range, but a wider range does not guarantee stronger empirical support: those added values depend on the imputation model.
Worked Example: A Slope Difference and the Predictor Range
In a hypothetical analysis of a continuous outcome on a continuous exposure, complete cases have observed exposure values from 10 to 40 units. Some records with missing exposure have other observed variables that, under the specified imputation model, lead to imputed exposure values between 40 and 60 units. The complete-case slope is \(0.50\) outcome units per exposure unit, with standard error \(0.120\). Across five imputations, the slope estimates are \(0.28,\ 0.34,\ 0.31,\ 0.33,\ 0.34\); each within-imputation variance is \(0.0100\).
The mean imputed-data slope is
The deviations from \(0.320\) are \(-0.040,\ 0.020,\ -0.010,\ 0.010,\ 0.020\). Therefore,
The pooled variance and standard error are
The pooled slope is \(0.320\) outcome units per exposure unit, compared with \(0.50\) in complete cases. The difference is \(0.50-0.320=0.180\) outcome units per exposure unit. In this example, the imputed analysis also represents exposure values above the complete-case maximum of 40 units. That makes it especially important to inspect how the imputation model generated those values and how much the fitted slope depends on them. It does not, by itself, show that the imputed slope is wrong: the complete cases may also fail to represent the target population or its relevant exposure range.
Why Agreement Is Not a Validity Test
Similar estimates can be reassuring in a limited sense: the reported coefficient may not be highly sensitive to the choice between these two analysis procedures under their respective specifications. But similarity does not verify the assumptions needed for either procedure. Both estimates could be biased in the same direction—for example, if both analyses use a misspecified regression model, omit an important confounder, or rely on an inadequate description of the missing-data process. Conversely, different estimates may arise because the methods use different sample compositions or predictor support, not simply because one method has failed.
A useful comparison therefore combines numerical results with checks of what each result represents. At minimum, examine:
- Whether both analyses use the same outcome, predictors, units, transformations, and target coefficient.
- How many records are complete, and which observed groups or predictor values are less represented among them.
- Whether key predictor ranges and distributions differ between complete cases and completed datasets.
- Whether the imputation model contains appropriate information for the analysis, and whether its completed values are plausible in context.
- How the pooled coefficient and standard error compare with the complete-case estimate and standard error, without treating their difference as a formal test of validity.
If results differ, describe the difference and investigate which records, variables, or assumptions could explain it. If they agree, report that agreement without claiming that it proves the missing-data assumptions or establishes the true regression effect. The comparison is a diagnostic aid, not a substitute for a defensible analysis plan.
Common Mistake
Wrong reasoning: “The complete-case and imputed coefficients are nearly identical, so missing data cannot have biased the result.” Or, “The imputed standard error is smaller, so multiple imputation must be more reliable.”
Correction: Agreement and smaller standard errors do not validate a method. Both procedures may share bias, and standard errors describe uncertainty under the assumptions and model used; they do not measure all possible bias. Compare the estimates, uncertainty, sample composition, predictor support, and assumptions together. State what the comparison suggests, and avoid treating it as proof.
Check Your Understanding
Use the definitions and comparisons above to answer the following questions.
- What records are excluded from a complete-case regression, and what does multiple imputation do differently?
- Why can a pooled imputation standard error exceed the complete-case standard error even when the imputed analysis retains more records?
- In the sample-composition example, calculate the proportion of younger people among the 60 complete cases and compare it with their proportion in the full dataset.
- Why should an analyst inspect the predictor range represented by imputed values, rather than only compare coefficients?
- Give one reason two analyses could produce similar coefficients even if both are affected by bias.
- What can agreement between the two results reasonably suggest, and what can it not establish?