Comparing Missing-Data Strategies by Sensitivity

Why Compare Missing-Data Strategies?

A missing-value treatment is part of the analysis, not a clerical detail. Complete-case analysis can reduce the sample and change who is represented. Imputation can retain records, but it fills in values using assumptions that may affect summaries and model results. Comparing several treatments shows whether a conclusion depends on those choices; it does not, by itself, determine which choice is correct.

The comparison is most informative when the strategies answer the same question on the same underlying records. Keep the outcome definition, predictor set, and model specification fixed where possible. Then record what changes: the number and composition of cases, descriptive summaries, and downstream estimates or predictions.

The earlier material on MCAR, MAR, and MNAR distinguishes assumptions about why values are missing. In brief, MCAR means missingness is unrelated to observed and unobserved data; MAR allows dependence on observed information; MNAR allows dependence on unobserved values even after accounting for observed information. These are working assumptions, not labels that can usually be confirmed from the observed data alone. A strategy should be considered in light of the data-collection process and the analysis objective.

One Data Set, Three Strategies

Consider a hypothetical insurer studying the association between a property’s inspection score \(X\) and an annual maintenance-cost index \(Y\). The index is in arbitrary units, not dollars. All six outcomes are recorded, but two inspection scores are missing. Group indicates whether the property is in Zone A or Zone B.

We will compare three choices: retain only rows with observed \(X\) (complete-case analysis); fill both missing scores with the overall observed mean; and fill each missing score with the observed mean for its zone. The last two are single-imputation strategies: each missing value receives one assigned value, rather than a range that represents uncertainty.

How Imputation Changes a Regression

To make the downstream comparison explicit, fit a simple linear regression of \(Y\) on \(X\), with an intercept, separately under each strategy. For the point estimates, the model is \(Y_i=\beta_0+\beta_1X_i+\varepsilon_i\), and ordinary least squares chooses the line minimizing the sum of squared residuals. The slope is \(b_1=S_{XY}/S_{XX}\), where \(S_{XY}=\sum (x_i-\bar{x})(y_i-\bar{y})\) and \(S_{XX}=\sum (x_i-\bar{x})^2\); the intercept is \(b_0=\bar{y}-b_1\bar{x}\). These formulas require \(S_{XX}>0\). The usual interpretation of the fit as a conditional-mean model also requires that a linear mean is appropriate; standard errors and tests require additional assumptions. Here the calculations compare fitted lines, not evidence of causation or a claim that those assumptions hold.

Overall-mean imputation happens to preserve the slope here. This is a special algebraic result for a simple regression with an intercept when missing predictor values are all replaced by the observed predictor mean. The imputed scores equal \(\bar{x}\), so they add zero to the centered predictor sum of squares and to the centered cross-product; the mean of \(X\) also remains \(\bar{x}\). That does not make mean imputation generally harmless. It changes the intercept here, reduces the sample variance of \(X\), and can distort uncertainty estimates. With additional predictors, interactions, or a different model, the same invariance need not hold.

Interpreting and Reporting Sensitivity

A useful comparison reports both the result and its limits. In this example, the complete-case and overall-mean slopes match, while the within-zone slope is smaller. The threshold decision also changes under the within-zone strategy. These differences direct attention to the two records with missing scores, the relationship between zone and score, and the fact that their observed outcomes differ.

Do not select a strategy merely because it produces a preferred result. Consider why values are missing, which records are excluded, whether the imputation variables are substantively meaningful, and what the analysis is intended to estimate. A zone-specific mean may retain a group distinction, but it still assigns a single value and does not represent within-zone variation. A global mean is simple, but it may obscure meaningful subgroup differences. Complete cases preserve observed predictor values, but the retained sample may not represent the full target population.

In R, the comparison should be reproducible: define each treatment explicitly, fit the same outcome model, and record the number of rows used. Functions such as complete.cases can identify complete rows, and summaries should make any removal or imputation explicit. Avoid letting a model function silently drop different rows from different fits without checking its analysis sample. A comparison table can make the results auditable.

A stable conclusion across reasonable strategies supports the limited statement that the conclusion is not highly sensitive to the treatments compared. It does not prove that missingness is harmless or that every strategy is valid. A changed conclusion should be reported as a limitation and investigated in context, rather than hidden by presenting only one fit.

Common Mistake: Treating Imputation as New Information

A tempting but incorrect argument is: “The imputed data set has six records, so it contains six observed inspection scores and should be more informative than the four complete cases.” The arithmetic does produce six rows, but two scores were assigned rather than measured. For example, the overall-mean strategy produces six values and a sample variance of \(4.0000\), compared with \(6.6667\) among the four observed scores. That smaller variance is partly a mechanical consequence of inserting values at the mean; it is not evidence that the population has less score variability.

The correction is to describe both the row count and the source of the values: four scores were observed, two were imputed, and results are conditional on the imputation rule. In particular, single imputation does not carry the uncertainty about the missing values into ordinary model calculations. Do not treat a conventional standard error from a single-imputed data set as if the filled values had been observed.