Masking-Based Diagnostics for Block Imputation

Why Masking Can Diagnose Block Imputation

When an omics modality is observed for a subject, its measurements can be temporarily hidden and then predicted using the other information available for that subject. Comparing those predictions with the hidden measurements gives an empirical check on how well an imputation procedure predicts a whole missing modality block. This is useful because ordinary imputation output alone does not reveal how far predictions tend to be from values that could, in principle, have been observed.

The comparison is informative only if the validation design resembles the problem the imputation will face. Masking individual cells at random, for example, tests a different task from predicting an entire modality that is absent for a subject. And even a correctly masked block may not represent naturally missing blocks: the subjects whose modality was observed can differ systematically from those for whom it is missing.

Let \(Y_{im}\) denote the observed vector of measurements for subject \(i\) in target modality \(m\). A block may contain many features; masking the block means hiding all of \(Y_{im}\) together, not selecting a few entries while leaving the rest visible. Let \(\hat{Y}_{im}\) denote the predicted block produced without using \(Y_{im}\). For a feature \(j\) in modality \(m\), define the prediction error as \(e_{imj}=\hat{Y}_{imj}-Y_{imj}\). Positive errors indicate overprediction under this convention.

Two common summaries are the mean absolute error (MAE) and root mean squared error (RMSE). For \(N\) masked subject-feature measurements, MAE is the mean of the absolute errors; RMSE is the square root of the mean squared errors. Both are nonnegative and have the units of the feature being scored. Squaring gives RMSE more sensitivity to large errors than MAE. Neither metric by itself describes the full error distribution or establishes that an imputation model is appropriate for every scientific use.

Here \(\mathcal{V}\) is the set of subject-feature pairs in the validation blocks. When features have substantially different scales or units, pooling raw errors can cause large-scale features to dominate. Report feature-specific errors, or define a scaled score using scales estimated from training data and state clearly what that score means. Do not estimate scaling constants using held-out validation values.

Conditions for a Trustworthy Diagnostic

The calculation of an error metric is straightforward; interpreting it requires stronger conditions. First, the masked measurements must not influence the imputation procedure, directly or indirectly. This includes feature selection, normalization, tuning, dimension reduction, and any preprocessing used to construct the prediction. Such steps must be fitted using the training information available for the validation run. If a value is hidden only after a transformation that used it, the diagnostic is contaminated by leakage.

Second, the validation task should match the intended deployment task. If deployment requires predicting an entire modality for subjects with a particular pattern of other observed modalities, the validation should hide that entire modality and retain only the other modalities and covariates available for those subjects. The imputation model must be refitted or otherwise applied in a way that does not use the hidden values. A random split of individual entries generally does not satisfy this condition for block-missing data.

Third, masked observed blocks must be sufficiently informative about the blocks that are naturally missing. This is not guaranteed by random masking among observed subjects. It is more plausible when, within the groups used for evaluation, the prediction errors for observed blocks represent the errors that would occur for the naturally missing blocks. If missingness depends on unobserved block values even after conditioning on available information, masked validation among observed subjects cannot directly reveal the error for those missing blocks. This limitation is related to the issues addressed in the earlier material on Pattern-Mixture Sensitivity Analysis for Missing Blocks and Selection Models for Nonignorable Modality Missingness.

In practice, record each subject’s modality-availability pattern: which modalities are available and which are absent. For a target modality, stratify masked-block errors by the other-modality pattern and by relevant subject characteristics, such as a clinical subgroup or a measured covariate. The grouping variables should be chosen for a reason and specified without using validation errors. Very small strata yield unstable summaries, so report their sizes and avoid treating a noisy subgroup estimate as a precise comparison.

Worked Examples of Block-Level Error

Why Stratification and Weighting Matter

The logic behind pattern-specific validation can be stated precisely. Let \(S\) denote a subject’s availability pattern and let \(L\) be a loss, such as the squared error for a target feature or a block-level average squared error. The expected loss among naturally incomplete subjects can be decomposed by pattern:

Masked observed blocks estimate the second conditional expectation using observed subjects with pattern \(s\). Replacing that missing-subject expectation with the masked validation expectation is valid only when those errors are comparable within pattern, and when each relevant pattern has validation data. Multiplying each pattern’s estimate by the target proportion gives the weighted mean squared error; taking its square root gives the corresponding RMSE. This decomposition explains both the benefit and the limitation of stratification: it aligns the mix of patterns, but cannot repair differences in error within a pattern.

When some patterns are rare or absent among observed target blocks, their prediction error cannot be assessed directly by this design. Extrapolating from a different pattern should be identified as an assumption, not reported as a measured diagnostic result. If validation subjects were sampled with known unequal probabilities, appropriate design weights may be used to estimate a target average, but the weights and target population must be stated. This is distinct from changing the imputation model.

Diagnostics, Alternatives, and Limitations

A masking diagnostic should be treated as a procedure to scrutinize, not just a number to report. Check that each masked subject’s target block is entirely withheld; that preprocessing and tuning do not use the hidden values; and that the remaining modalities match what the intended imputation would actually receive. Report the number of masked subjects and blocks in each stratum, along with the metric and its units. Large differences across repeated splits or sparse strata indicate that the estimate is unstable. A few extreme errors can also make RMSE much larger than MAE, so reporting both can reveal whether the summary is especially sensitive to large misses.

Repeated masking can show how much a score varies with the selected validation subjects, but it does not create new independent subjects. If the same person appears in several splits, do not treat each appearance as an independent observation when describing uncertainty. Split or resample at the subject level when the subject is the independent sampling unit. For multiple-imputation procedures that produce several completed values, define whether the diagnostic scores each imputed draw or an aggregate prediction; those answer different questions about point prediction and imputation variability.

The main alternative is to compare imputation approaches using the same masked blocks and scoring rules. This supports a fair predictive comparison if the splits, available inputs, and tuning opportunities are comparable. Complete-case analyses do not measure prediction error for hidden blocks, and random entry-wise masking may be suitable for a different missingness task but is not a substitute for whole-block masking. No validation metric proves that an imputation is unbiased for a downstream association or that its uncertainty estimates are calibrated. Those questions require their own analysis.

Most importantly, masked observed blocks cannot directly validate predictions for values that are never observed under a nonignorable missingness process. If block availability depends on the hidden measurements, the observed blocks can be an unrepresentative subset even after stratifying on recorded characteristics. In that setting, report the validation result as performance among the masked observed subjects, examine how plausible that population is as a proxy for naturally missing subjects, and use sensitivity analysis where appropriate. Do not describe an observed-subject masking score as verified accuracy for unobserved blocks.

Common Mistake