Comparing Missing-Data Strategies Across Algorithms

Why Missing-Data Choices Affect Model Comparisons

A model comparison can appear to favor one algorithm when the actual difference is that the algorithms were trained or evaluated on different records. Complete-case analysis drops records with missing predictors; imputation retains them by filling in values; a missingness indicator adds a feature recording whether a value was absent. These choices can change the effective sample, the information available to a model, and the validation cases on which its score is calculated.

The issue applies across GLMs, penalized regression, and tree-based ensembles. A fair comparison therefore records the missing-data strategy for each candidate, fits any data-dependent preprocessing using training records only, and evaluates predictions on the same eligible validation records whenever possible. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Complete-case analysis fits or evaluates a model only on records with observed values for every variable required by that analysis. Imputation replaces a missing value with an estimated or specified value, such as a training-set median. A missingness indicator is a variable, often coded $M=1$ when a specified value is missing and $M=0$ otherwise, that allows the model to use the missingness pattern as a predictor.

Missingness mechanisms describe how absence relates to the data. Under MCAR (missing completely at random), the chance a value is missing is unrelated to observed or unobserved data. Under MAR (missing at random), missingness may depend on observed information but not on the missing value after conditioning on that information. Under MNAR (missing not at random), missingness still depends on the unobserved value, even after conditioning on observed information. These distinctions matter: a convenient imputation rule does not, by itself, remove bias caused by an unsuitable missingness assumption.

Three Strategies and Their Trade-offs

Strategy	What happens to a missing predictor?	Comparison concern
Complete-case analysis	Drop any record missing a required variable.	Smaller and potentially different training and validation samples.
Imputation	Replace the missing value using a rule estimated or specified from training data.	The imputation rule must be learned without using validation outcomes or predictor distributions.
Imputation plus indicator	Impute the value and add a flag for its absence.	The indicator can capture a pattern, but may reflect data collection processes and does not guarantee unbiased estimates.

The same strategy can behave differently across algorithms. A GLM can use an imputed predictor and an indicator as separate terms. A penalized model can also use them, although its penalty may shrink their coefficients. A tree or ensemble may accept missing values directly if its implementation has a defined missing-value routing rule; otherwise it may need imputation or record deletion. The analyst should document what the software actually does, rather than assuming that all implementations handle missing values identically.

A model that uses a missingness indicator can learn that records with an absent value have different outcomes, conditional on its other predictors. That may improve prediction when the pattern is stable. It does not mean the indicator recovers the missing value, establishes a causal explanation, or solves MNAR bias. It can also encode operational practices that later change.

Make Preprocessing Part of the Validation Procedure

Any quantity learned from data belongs inside the training portion of a split or fold. This includes a median used for imputation, a rule for creating an indicator, and any transformations whose parameters are estimated from the data. The rule is fitted on the training records and then applied unchanged to the corresponding validation records.

\text{training records} \longrightarrow \text{estimate preprocessing rule} \longrightarrow \text{fit model} \longrightarrow \text{apply rule and predict validation records}

The reason is direct. Let $T$ denote training data and $V$ validation data, and let $g(T)$ be an imputation statistic such as a median. A valid validation prediction uses $g(T)$. If the statistic is instead calculated from $T$ and $V$ together, the prediction depends on the validation feature distribution. The validation data have then influenced the fitted pipeline, so the score no longer measures the procedure that would be applied to genuinely new records.

Justification: Validation estimates performance for a procedure trained on $T$. If preprocessing is estimated from $T$, the fitted prediction rule is $f_T(x)=h(x;g(T),\widehat{\theta}_T)$, where $\widehat{\theta}_T$ denotes model parameters. Estimating the imputation rule from $T\cup V$ instead gives $h(x;g(T\cup V),\widehat{\theta}_T)$, a different rule that has used information from $V$. Therefore, preprocessing must be re-estimated within each training split or fold.

When algorithms are compared, use the same split assignments and the same validation records wherever feasible. If a complete-case candidate cannot produce predictions for some validation records, report its coverage and do not compare its score on a reduced subset with another candidate's score on the full set as if the metrics were directly comparable. A useful diagnostic is to calculate both candidates' metrics on their common eligible validation records, then separately report coverage and performance on records excluded by the complete-case candidate.

Worked Examples

Worked Example: Sample Changes and RMSE

A hypothetical insurer reserves 1,000 records for model development. After a random split made before preprocessing, 800 are in training and 200 in validation. A predictor is missing for 160 training records and 50 validation records. A complete-case GLM is fitted, while a second GLM uses median imputation. On the 150 validation records with an observed predictor, the complete-case model has a sum of squared errors of 300 and the imputed model has a sum of squared errors of 285. The imputed model's sum of squared errors on the 50 records with a missing predictor is 140. Compare the scores correctly.

Method. First calculate each training and validation sample size, then compute RMSE on a common validation subset and on each model's available evaluation set. RMSE is the square root of mean squared prediction error.

The complete-case GLM has $800-160=640$ training records. Its validation set contains only $200-50=150$ complete records. The imputed GLM retains all 800 training and all 200 validation records, assuming its fitted imputation rule can be applied to every record.

On the 150 shared validation records, the complete-case RMSE is $ \sqrt{300/150}=\sqrt{2}=1.4142 $ (rounded to four decimals). The imputed GLM's RMSE on those same 150 records is $ \sqrt{285/150}=\sqrt{1.9}=1.3784 $.

Across all 200 validation records, the imputed GLM's sum of squared errors is $285+140=425$. Its full-set RMSE is $ \sqrt{425/200}=\sqrt{2.125}=1.4577 $, rounded to four decimals. Thus, the imputed GLM is better on the shared 150 records, but its full-set RMSE is larger than the complete-case RMSE calculated on 150 records. Those two full-set figures are not a valid direct comparison because they use different validation populations. On the common subset, the imputed model's RMSE is lower by $1.4142-1.3784=0.0358$.

Check. The imputed model's total squared error is $425$, and $425/200=2.125$; its reported full-set RMSE is therefore consistent. The common-subset comparison uses 150 records for both models. The mistake to avoid is declaring a winner by comparing $1.4142$ on 150 records directly with $1.4577$ on 200 records.

Worked Example: Imputation and an Indicator in a GLM

A hypothetical renewal model uses a customer's annual mileage $x$, measured in thousands of miles, in a logistic GLM for a binary renewal outcome $Y$. A training-only median imputation rule replaces missing mileage with $4$. The fitted model is $ \log\{P(Y=1)/(1-P(Y=1))\}=-2+0.5x+0.8M $, where $M=1$ if mileage was missing and $M=0$ otherwise. Calculate the predicted probabilities for a record with observed mileage $4$ and a record whose mileage is missing.

Method. Apply the imputation rule first, set the indicator according to whether the original value was missing, then evaluate the linear predictor and transform it to a probability using $p=1/(1+e^{-\eta})$.

For observed mileage $4$, the imputed value is unnecessary and $M=0$. The linear predictor is $ \eta=-2+0.5(4)+0.8(0)=-2+2=0 $. Thus $p=1/(1+e^0)=0.5000$.

For missing mileage, the imputed value is $4$, but the indicator is $M=1$. The linear predictor is $ \eta=-2+0.5(4)+0.8(1)=-2+2+0.8=0.8 $. Therefore $p=1/(1+e^{-0.8})\approx 0.6900$, rounded to four decimals.

The indicator coefficient changes the log odds by $0.8$ for otherwise identical records; it does not change the imputed mileage value. The odds multiplier is $e^{0.8}\approx2.2255$. This is a conditional model association, not evidence that missingness itself causes renewal. The common mistake is to set $M=0$ merely because an imputed number is present. The flag must retain the fact that the original mileage was missing.

Check. For an observed value, the contribution from mileage is $0.5(4)=2$ and the indicator contributes zero, giving $\eta=0$. For a missing value, imputation still gives $x=4$, while the indicator adds $0.8$, giving $\eta=0.8$. Both probabilities follow from the logistic transformation.

Worked Example: Comparing Model Families on Shared Records

A development team compares a GLM, an elastic-net model, and a tree ensemble for a hypothetical claim-severity target measured in thousands of dollars. All three pipelines use the same training/validation split and produce predictions for the same four validation records. Actual values are $10,12,14,16$. The GLM predicts $11,11,13,17$; the elastic-net model predicts $10,13,13,16$; and the ensemble predicts $9,12,16,15$. Calculate and compare validation RMSE.

Method. For each model, subtract actual values from predictions, square each error, average the four squared errors, and take the square root. The target is in thousands of dollars, so RMSE is also in thousands of dollars.

For the GLM, prediction errors are $1,-1,-1,1$. The squared errors are $1,1,1,1$, with sum $4$. Its RMSE is $\sqrt{4/4}=1.0000$ thousand dollars.

For the elastic-net model, errors are $0,1,-1,0$. The squared errors are $0,1,1,0$, with sum $2$. Its RMSE is $\sqrt{2/4}=\sqrt{0.5}=0.7071$ thousand dollars.

For the ensemble, errors are $-1,0,2,-1$. The squared errors are $1,0,4,1$, with sum $6$. Its RMSE is $\sqrt{6/4}=\sqrt{1.5}=1.2247$ thousand dollars. To express these to the nearest cent in dollars, the values are $1,000.00, $707.11, and $1,224.74, respectively. On this validation sample the elastic-net model has the lowest RMSE.

Check. Each model was scored on the same four outcomes, so the comparison is on a common basis. These scores do not establish that the elastic-net model will always perform best; additional validation evidence and operational considerations may matter. The mistake to avoid is calculating a model's score only on cases where its preferred preprocessing happened to retain a record, then comparing that score with a competitor evaluated on a different set.

Interpreting Results Across Algorithms

A useful comparison table records more than a single metric. For each model and missing-data strategy, track the number of training records, the number of validation records receiving predictions, the preprocessing rule, and the validation metric on a clearly stated set of records. If complete-case analysis removes many records, the lost cases may differ systematically from those retained. A performance change can then reflect a change in the data distribution, not simply a change in algorithm.

For penalized models, include the indicator and imputed feature in the candidate predictor set if that is the intended strategy, and carry out any preprocessing consistently within each training fold. Do not assume that a coefficient will remain large: regularization can shrink it, and the fitted result depends on the penalty and scaling choices. For ensembles, identify whether missing values are handled by a native splitting rule or by preprocessing. A direct comparison is not meaningful if one implementation silently drops records or treats missingness differently from another.

Complete-case analysis can be a useful baseline, especially when few records are affected and its assumptions are defensible. Imputation can preserve sample size and make scoring possible when a value is absent. Adding an indicator may help when absence itself carries predictive information. None is universally superior. The appropriate choice depends on the data-collection process, the intended scoring environment, validation evidence, and the consequences of excluding records or using a missingness pattern.

Key takeaway: Compare missing-data strategies as complete pipelines, not as isolated algorithm labels. Fit preprocessing within training data, report sample counts and prediction coverage, and compare validation metrics on common records before interpreting a performance difference.

Check Your Understanding

For each problem, distinguish the preprocessing choice from the model-family choice and check which records support the comparison.

A training split has 500 records, 75 of which lack a predictor. What are the complete-case and imputed training sample sizes? (Answer: 425 and 500 records.)
A validation set has 80 records. A complete-case model predicts 68, with squared-error sum 68. An imputed model predicts all 80, with squared-error sum 88. Calculate both RMSE values and explain why the direct comparison is incomplete. (Answer: Complete-case RMSE $=\sqrt{68/68}=1.0000$; imputed RMSE $=\sqrt{88/80}=\sqrt{1.1}\approx1.0488$. They use different validation records, so compare on their shared 68 records as well.)
A training-only median for an exposure predictor is 6. A validation record has missing exposure and its missingness indicator is 1. What values should be supplied to a model using both fields? (Answer: Imputed exposure $=6$, indicator $=1$.)
A logistic model has linear predictor $\eta=-1+0.2x+0.5M$. For a record with missing $x$, the training median is 5 and $M=1$. Calculate its predicted probability. (Answer: $\eta=-1+0.2(5)+0.5=0.5$; $p=1/(1+e^{-0.5})\approx0.6225$.)
In a validation fold, should the imputation median be estimated from the training portion alone or from the training and validation portions together? State the reason. (Answer: Training portion alone; using validation predictors to estimate the median lets validation data influence the fitted pipeline.)

What You'll Learn