Missing Predictors and Validation Coverage

Why Prediction Coverage Is Part of Validation

A prediction model can perform well for people whose predictors are available and still be unusable for a substantial part of its intended population. In genetic prediction, a score may not be calculable because genotypes fail quality checks, required variants are absent, or a participant lacks a usable genetic sample. A clinical predictor may also be missing, or the prediction procedure may fail for a combination of reasons. Reporting performance without reporting who received a prediction can therefore give an incomplete account of validation.

The key questions are distinct: How often did the specified prediction process produce a usable prediction? and How well did predictions perform for the people on whom performance was evaluated? The first is about coverage; the second is about performance. A performance metric calculated only among people with complete predictors does not, by itself, describe either performance or availability for everyone in the intended target population.

Availability is defined relative to a prediction procedure, not just a data column. For example, a score may require a minimum set of variants, a specified allele coding, and a successfully processed genotype file. A clinical model may require particular laboratory measurements. If the intended workflow permits a documented fallback model when one predictor is absent, that fallback is part of the procedure and should be evaluated as such. If no prediction or fallback is produced, the person is not covered by that procedure.

Define the Denominator and Report Availability

Let \(N\) be the number of eligible people in the intended validation population, and let \(R_i=1\) when person \(i\) has an available prediction under the specified procedure and \(R_i=0\) otherwise. The observed coverage is the number with an available prediction divided by the intended population size. It is not the proportion of complete records among a subset chosen after prediction failures have been removed.

The denominator should match the question. If the target is everyone eligible for prediction, use all eligible people as the denominator, including those with missing predictors. If outcome data are also required to calculate a performance metric, report separately how many people have both an available prediction and an observed outcome. Outcome missingness and predictor-related prediction failure are different reasons for exclusion and should not be combined into a single unexplained count.

When a procedure uses several inputs, report availability for important inputs as well as for the full prediction. For instance, genotypes may be available for 92% of eligible participants, the required clinical predictors for 88%, and both for 81%. The joint percentage is generally not the product of the separate percentages: missingness patterns may be associated. A clear report gives counts and denominators for each component and for the final prediction.

Complete-Case and Operational Evaluation Samples

A complete-case evaluation sample is formed by restricting performance calculations to people with all predictors needed for the model and an observed outcome. It is often straightforward to calculate a metric in this sample, but its interpretation is conditional: it describes performance among people who met those data-availability conditions. It does not automatically describe performance in the full intended population.

An operational evaluation assesses the prediction process as it is intended to be used. This requires spelling out what happens when inputs are absent. If the real procedure returns no prediction, report the proportion receiving no prediction and calculate conventional performance metrics among those with predictions and observed outcomes, clearly identifying that subset. If the procedure specifies a fallback, such as a clinical-only model when the genetic score is unavailable, evaluate the full rule, including its fallback predictions. Do not silently replace a failed prediction with zero or another value unless that is explicitly the model's prespecified procedure.

A practical report can separate three counts: eligible people, people for whom the procedure produced a prediction, and people with both a prediction and an observed outcome. Then report performance metrics on the relevant prediction-and-outcome sample, alongside coverage. For a binary outcome, for example, sensitivity and specificity use outcome-defined denominators among people with predictions: sensitivity is the proportion of cases classified positive, and specificity is the proportion of non-cases classified negative. These conditional metrics should not be presented as if they included people for whom no prediction was produced.

Availability Can Select a Different Group

Prediction failure is not necessarily random with respect to the people being studied. Genotype availability may differ by recruitment site, sample quality, age, or other measured characteristics. Clinical measurements may be more often missing among people with particular health-care histories. If availability is associated with the outcome, or with factors related to prediction error, performance among available people can differ from performance in the target population.

A simple way to see the issue is to compare the outcome distribution for people with and without predictions. Let \(Y\) be an outcome, let \(\pi=P(R=1)\) be the proportion with available predictions, and let \(\mu_1=E(Y\mid R=1)\) and \(\mu_0=E(Y\mid R=0)\). The overall mean outcome is a mixture of the two group means:

Subtracting this overall mean from the mean among people with predictions gives:

Thus, when coverage is incomplete, the available group has a different outcome mean from the target population if the available and unavailable groups have different outcome means. The difference is zero if coverage is complete or if the two group means are equal. This calculation concerns the outcome mean; it does not imply that every performance metric has the same bias formula. Measures such as AUC, calibration, and prediction error can respond differently to changes in case mix and predictor availability.

Diagnosing Missingness and Interpreting Metrics

A useful validation report examines who is and is not covered. Compare availability by site and by relevant demographic, clinical, and genetic-data characteristics that are appropriate to the study. Show the number eligible, the number with predictions, and the coverage percentage in each group. Also tabulate the reasons for failure where they can be identified, such as absent clinical measurement, genotype quality-control failure, or missing required variants. These summaries help distinguish a single general coverage problem from a failure concentrated in particular groups.

The comparisons are diagnostic, not proof of a missingness mechanism. Similar observed characteristics do not establish that availability is independent of unobserved outcomes or prediction errors. Conversely, an observed difference does not establish that missingness caused poor performance. The report should describe what is observed and avoid claiming that the available subset represents everyone unless that claim is justified by the design and assumptions.

Sometimes weighting can help estimate performance for the intended population. For example, if availability is conditionally independent of the outcome and prediction errors given measured characteristics \(Z\), and every relevant type of person has a positive chance of a prediction, one can use inverse-probability weights based on the estimated availability probability \(P(R=1\mid Z)\). Weighted outcome or error summaries may then transport from available people to the target population under those assumptions. For a ratio metric such as sensitivity, weighting should be applied to the relevant weighted counts before taking the ratio. Weighting does not create information for groups with no available predictions, repair an incorrect prediction procedure, or resolve dependence on unmeasured factors without further assumptions.

Performance metrics also need the correct interpretation. An AUC or Brier score calculated in the available-prediction group is a metric for that group and that outcome distribution. It is not a measure of coverage. A high AUC among the 70% who received predictions cannot compensate for failing to report that 30% had no score. Likewise, a complete-case calibration assessment does not establish calibration among people who could not be scored.

Common Mistakes

• Dropping failures before defining coverage. The wrong reasoning is that only records with a usable score are valid, so they define the denominator. The correction is to define the eligible target population first, then count prediction failures within it.
• Calling complete-case performance population performance. Metrics among people with all predictors and observed outcomes are conditional on those requirements. Report the condition and assess whether availability differs across groups or outcomes.
• Confusing outcome missingness with predictor failure. Both can reduce the sample used for a metric, but they have different causes and implications. Report separate counts for unavailable predictions and unobserved outcomes.
• Assigning an arbitrary prediction to a failure. A zero score or a default risk is not a neutral way to retain a participant. Use such a value only if it is an explicit part of the prediction rule, and validate that complete rule.
• Inferring a missingness mechanism from a table alone. Observed comparisons can reveal selection patterns in measured variables, but cannot prove independence from unmeasured outcomes or errors.