Artificial-Masking Calibration for Outcome Predictions

Why Artificially Mask Outcomes?

For a naturally missing outcome, its prediction cannot be checked against the true value in that case: the value is unavailable. Artificial-masking calibration offers a narrower check. It temporarily hides some outcomes that were observed, asks candidate models to predict them using the information that would have been available at the prediction time, and compares those predictions with the concealed values. The check can reveal predictive weaknesses under the artificial masking design. It does not, by itself, establish that predictions for naturally missing outcomes are accurate.

The distinction matters in dropout settings. A prediction for a patient's later outcome might be based on baseline covariates and earlier visits. To test that prediction, one can choose patients with observed later outcomes, conceal those outcomes in a dropout-like pattern, and compare predictions with the values that were concealed. The design of the concealment determines what this exercise can assess.

For a candidate model \(k\), write \(\hat F_k(\,\cdot\mid W)\) for its predictive distribution and \(\hat\mu_k(W)\) for its predicted conditional mean when that is the target. A loss function \(\ell\) measures the discrepancy between the true outcome and prediction. For example, squared-error loss for a mean prediction is \(\ell(Y,\hat\mu_k)=(Y-\hat\mu_k)^2\); absolute-error loss is \(|Y-\hat\mu_k|\). A probabilistic prediction can instead be evaluated with a proper scoring rule, such as the log score, provided the predictive distribution assigns a meaningful density or probability to the observed outcome.

The artificial-masking risk is the average loss among the outcomes concealed by the artificial design. It is a risk for that particular group, prediction procedure, loss, and masking design—not a universal measure of model quality. The model must not use the concealed value, or information derived from it, to fit the prediction being evaluated.

Designing a Meaningful Artificial Mask

A defensible exercise begins by specifying the prediction time and the information available then. In longitudinal data, if the target prediction is the next visit given baseline and earlier visits, the artificial mask must leave those predictors available and conceal the target visit. For a dropout-like assessment, it may conceal that visit and all later visits. Using a later outcome, a summary calculated from it, or a post-dropout variable to predict the concealed value would create information leakage and an artificially favorable result.

The artificial mask should also represent the pattern of interest as closely as possible. Relevant features might include the visit at which dropout begins, whether later outcomes are all unavailable, and the distribution of observed histories at that point. A mask based only on variables known before the concealed outcome can reproduce selected pattern features, but it cannot reproduce a dependence on the unseen outcome itself unless that dependence is somehow known from outside the observed data.

For a clean evaluation, predictions for artificially masked outcomes must be generated without fitting to those outcomes. One can reserve outcomes for evaluation or use a procedure that excludes each evaluated value from the fit used to predict it. The exact implementation depends on the data and candidate model; the essential condition is that the evaluated outcome is not allowed to influence its own prediction. When candidates use different MNAR assumptions, hold their shared observed-data components and prediction target as comparable as possible, while recognizing that their extrapolations can still differ.

What Calibration Risk Can Be Transported?

A central issue is whether the artificially masked cases represent the naturally missing cases. Let \(G=1\) denote membership in the artificial calibration group and \(G=0\) membership in the target group of naturally missing cases. For a fixed prediction rule and loss, define the conditional risks \(r_1(w)=E\{\ell(Y,\hat F(W))\mid W=w,G=1\}\) and \(r_0(w)=E\{\ell(Y,\hat F(W))\mid W=w,G=0\}\). These definitions separate outcome behavior at a fixed \(W\) from the distribution of \(W\) across groups.

If the conditional risks agree, \(r_1(w)=r_0(w)\), then calibration can inform target risk at that particular \(w\), provided the calibration group contains relevant cases there. But equality of conditional risks alone does not imply equal marginal risks. The two groups may have different distributions of \(W\), and risk may vary with \(W\).

The justification follows by conditioning on \(W\): the marginal expected loss in either group is the average of its conditional expected loss over that group's distribution of \(W\). Under conditional transport, the conditional-risk functions are the same, but their averages can differ when the averaging distributions differ. If \(F_0\) has a density relative to \(F_1\), a reweighted calibration average uses weights proportional to \(dF_0/dF_1\). In finite strata, this amounts to averaging each stratum's estimated conditional risk using the target stratum proportions. This requires adequate support: a target stratum with no calibration cases cannot be repaired by weighting.

Why This Does Not Validate MNAR Extrapolation

Artificial masking starts with outcomes that are naturally observed. Even if the artificial pattern resembles dropout, the cases available for masking have already passed through the natural observation process. When natural observation depends on the unseen outcome after conditioning on available information, the observed outcomes need not represent outcomes among naturally missing cases. Artificially hiding observed values cannot reveal the values that natural dropout concealed.

A useful way to state the limitation is that a calibration result supports transport only under assumptions about conditional outcome behavior, the prediction rule, and the covariate distribution. Under MNAR, the conditional distribution of the outcome among naturally missing cases can differ from that among naturally observed cases even at the same recorded history. Artificial masking may show that a candidate predicts well for selected observed cases under a chosen pattern; it does not identify the unobserved conditional distribution or prove the MNAR assumptions correct.

Diagnostics, Comparisons, and Limitations

Before interpreting a calibration result, check whether the masking experiment actually targets the intended prediction problem. Compare the artificial and natural patterns using variables known before dropout, including histories and the timing and extent of missingness. Check whether target cases have counterparts in the calibration data. If some histories or subgroups are absent, conditional transport for those regions cannot be assessed empirically. Very large reweighting factors, if used, signal that the target distribution is poorly represented and can make a weighted estimate unstable.

Also inspect the prediction process for leakage, and report the loss and its units or interpretation. Mean squared error emphasizes large errors and has squared outcome units; RMSE returns to the outcome's units. A proper score for predictive distributions assesses more than the predicted mean, but its value depends on the selected scoring rule. A small average loss can hide poor predictions in a subgroup or at a particular visit, so summaries by relevant histories and dropout patterns can be informative when sample sizes allow.

Artificial-masking calibration complements, rather than replaces, other checks. Observed-data residual checks assess discrepancies in outcomes that were actually observed. Posterior predictive checks assess whether model-generated data reproduce selected features of observed data under the fitted model. Artificial masking directly tests predictions against known values withheld from the prediction process, but only for the artificially selected cases. Sensitivity analysis across MNAR specifications addresses how conclusions change under alternative assumptions about unseen outcomes; artificial masking cannot select among those assumptions solely by revealing how well they predict outcomes from the observed group.

The main limitations are selection into the observed calibration sample, imperfect resemblance between artificial and natural dropout, limited support, and dependence of results on the chosen loss. Calibration also evaluates a prediction procedure fitted under the artificial exercise; that procedure may not be identical to the final analysis fit. Consequently, report the masking design and target clearly, and describe the result as evidence about prediction performance under that design—not as verification of the naturally missing outcomes.