Why Predictive Checks Need a Decision Context
A predictive check is useful not simply because it measures how closely predictions match observations, but because it helps determine whether a mismatch could matter for the model’s intended use. A small average error may be unimportant when a decision is far from its threshold, yet consequential when the prediction is just on the decision boundary. Conversely, an error that looks large on a general scale may not affect the particular action the model is meant to support.
The earlier material on linking model outputs to decision rules established that a model output is interpreted through a specified rule. Here the practical question is how to read predictive evidence against that rule. For example, if a development decision is to proceed when the predicted probability of response is at least 0.60, validation results should be examined especially carefully for predictions near 0.60. The check does not replace the rule; it helps assess whether the evidence supports using the model to apply it.
Interpretation depends on conditions. Predictions must be generated without using the outcomes they are checked against; otherwise the comparison is not an honest check of predictive performance. The validation observations should also be relevant to the population, measurements, and decision setting of interest. Outcomes need a consistent definition, and dependence among observations (for example, multiple records from one participant) must be accounted for when summarizing uncertainty. These conditions do not guarantee that a model is adequate, but without them a discrepancy may reflect data leakage, a measurement mismatch, or an inappropriate comparison rather than the model’s predictive behavior.
Three Questions for Reading a Check
Begin by stating the decision rule and its threshold. Then ask three questions: what discrepancy is present, where does it occur, and how large is it relative to the distance between the model’s prediction and the decision boundary? The same numerical discrepancy can have different implications depending on the answers.
For continuous outcomes, let \(y_i\) be an observed outcome and \(\hat{y}_i\) its prediction. The signed prediction error is \(e_i=y_i-\hat{y}_i\): a negative value means the model overpredicted, and a positive value means it underpredicted. The mean signed error summarizes average direction, while the mean absolute error and root mean squared error summarize typical error magnitude in the outcome’s units. For binary outcomes, calibration compares predicted event probabilities with observed event frequencies. A model can be well calibrated on average while still making large errors for particular people or subgroups.
For a binary outcome, a calibration check can group predictions into ranges and compare the average predicted probability with the observed proportion of events in each range. The difference between those quantities is a calibration discrepancy. Grouping is a summary, not proof that each individual prediction is calibrated: different predictions within a group can behave differently, and small groups produce uncertain observed proportions.
Worked Examples
Worked Example: A Calibration Discrepancy Near the Threshold
Suppose a hypothetical development decision is to proceed with a regimen if its predicted probability of response is at least 0.60. In a separate validation set, 40 participants had predictions in a range near that threshold. Their average predicted response probability was 0.62, and 22 of the 40 responded. Does the check raise a decision-relevant concern?
The observed response proportion is \(22/40=0.55\). The calibration discrepancy, defined here as observed proportion minus average predicted probability, is \(0.55-0.62=-0.07\). Thus, in this range, the model’s average prediction exceeded the observed response proportion by 7 percentage points.
The predicted probability of 0.62 is only 2 percentage points above the 0.60 threshold. The observed discrepancy is larger than that margin and points downward. If this discrepancy were a reliable estimate of the model’s predictive error for the decision population, it could change the decision: \(0.62-0.07=0.55\), below the threshold. That arithmetic is a way to assess potential decision impact, not a justified recalibration by itself.
The group is small, so its observed proportion is imprecise. The check therefore flags a plausible decision concern but does not establish that the model’s true response probability is 0.55 or that the regimen should not proceed. The appropriate conclusion is that a prediction close to the threshold deserves closer assessment of the validation evidence and the consequence of a false proceed decision.
Worked Example: Prediction Error and a Continuous Decision Threshold
In a hypothetical setting, a model supports a decision to advance a regimen if its predicted mean improvement is at least 10 points. On four validation cases, predictions were 10, 12, 9, and 11 points, while observed improvements were 9, 11, 8, and 10 points. Calculate the prediction-error summaries and interpret them for a proposed regimen with a predicted improvement of 10.6 points.
The signed errors \(y_i-\hat{y}_i\) are \(-1,-1,-1,-1\) points. Therefore, mean signed error is \((-1-1-1-1)/4=-1\) point: predictions exceeded observations by 1 point on average in this small set.
Mean absolute error is \((1+1+1+1)/4=1\) point. Root mean squared error is \(\sqrt{(1^2+1^2+1^2+1^2)/4}=\sqrt{1}=1\) point. All three summaries use points as their units; the signed error indicates direction, while the other two indicate magnitude.
The proposed prediction is 0.6 points above the 10-point threshold. The observed mean signed error is larger in magnitude than that margin and is negative, so a similar overprediction could put the true mean below the threshold. This is a warning about decision sensitivity, not a validated correction of \(10.6-1=9.6\) points: four cases give limited evidence, and their errors may not represent the proposed regimen’s prediction context. An overall error summary should be considered alongside the relevance and precision of the validation evidence.
Worked Example: Opposing Subgroup Errors Hidden by Aggregate Calibration
A hypothetical model supports subgroup-specific decisions to proceed when predicted response probability is at least 0.60. In subgroup A, the average prediction is 0.70 and 11 of 20 participants respond. In subgroup B, the average prediction is 0.50 and 13 of 20 respond. Compare the subgroup checks with the aggregate check.
For subgroup A, the observed proportion is \(11/20=0.55\), so the discrepancy is \(0.55-0.70=-0.15\). The model overpredicts by 15 percentage points in this subgroup. Its average prediction is above the threshold, while its observed response proportion is below it.
For subgroup B, the observed proportion is \(13/20=0.65\), so the discrepancy is \(0.65-0.50=0.15\). The model underpredicts by 15 percentage points. Its average prediction is below the threshold, while its observed response proportion is above it.
With equal subgroup sizes, the aggregate average prediction is \((0.70+0.50)/2=0.60\). The total observed response proportion is \((11+13)/40=24/40=0.60\). Aggregate calibration looks exact, but the errors cancel. The subgroup checks reveal opposite discrepancies that could reverse the subgroup-specific decisions. Because each subgroup contains only 20 participants, these proportions are uncertain; the example demonstrates why aggregate agreement alone cannot rule out decision-relevant subgroup mismatch.
Why the Decision Margin Matters
A simple comparison makes the logic precise. Suppose a rule says to take an action when a predicted quantity \(q\) is at least a threshold \(c\). The model’s distance above the threshold is \(m=q-c\). If a relevant predictive discrepancy is \(\delta\), interpreted as an adjustment to the prediction, the adjusted quantity is \(q+\delta\). It falls below the threshold exactly when \(q+\delta<c\), or equivalently when \(\delta<-m\).
This is a decision-margin check, not a statistical test and not an instruction to adjust predictions using any observed discrepancy. Its value is conceptual: a discrepancy much smaller than the margin may be unlikely to change the action, whereas a discrepancy equal to or larger than the margin in the adverse direction may matter. If the action has substantial consequences, even a modest chance of crossing the threshold may warrant more evidence than a decision with limited consequences. The evidence standard should reflect the consequences of error, as established in the earlier material on decision consequences.
Name the model output, action, threshold, and direction of concern. A check cannot be called decision-relevant without a specified rule.
Check whether it occurs near the threshold, in a decision-relevant range, or in a subgroup whose action is determined separately.
Use outcome units or probability points, and ask whether an adverse discrepancy could cross the threshold.
Consider the amount and relevance of the validation evidence. Report a concern as a concern when uncertainty prevents a firm conclusion.
Diagnostics, Alternatives, and Limitations
Several patterns should prompt closer scrutiny: a consistent signed error in the direction that favors the wrong action; larger errors among predictions near the threshold; poor calibration in a subgroup whose decision is distinct; or a wide spread of individual errors despite a small average bias. Plotting predictions against observed outcomes or comparing calibration across decision-relevant ranges can help reveal these patterns. Any such display should be read with the amount of data in each range in mind. A sparse group can look extreme by chance, and a broad group can conceal important variation.
Different checks answer different questions. Calibration asks whether predicted probabilities agree with observed event frequencies at the level examined. Prediction-error summaries ask how far numerical predictions tend to be from outcomes and, with signed error, whether there is systematic over- or underprediction. A single overall goodness-of-fit or average-error summary can miss where errors occur. Subgroup checks can expose offsetting errors but may be noisy, particularly when subgroup samples are small. Decision-focused review brings these results together by asking whether the errors matter for the intended action; it does not make a weak or irrelevant validation dataset adequate.
A common mistake is to say, “The overall calibration error is zero, so the model supports every decision.” The reasoning fails when errors in different ranges or subgroups cancel, as in the worked example. The correction is to inspect calibration where the decision is made, including relevant subgroups, and compare discrepancies with the corresponding decision margins. The opposite overstatement is also a pitfall: observing a discrepancy larger than the margin in a small validation sample does not prove that the decision must change. Sampling variation, outcome measurement, and the match between validation observations and intended use all affect what can be concluded.
Predictive checks are evidence about performance in the setting represented by the validation data. They do not establish performance in every unobserved setting, nor do they by themselves select or justify a new decision threshold. Their role is to show whether observed predictive behavior is consistent with the model’s intended use and to identify where further evidence or more cautious interpretation may be needed.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A decision threshold is 0.70. A model predicts 0.74, and a relevant calibration check suggests an adverse discrepancy of 0.02. Could this discrepancy alone reverse the decision? Explain.
- For three continuous outcomes, predictions are 5, 8, and 10 units, and observations are 4, 10, and 9 units. Calculate the signed errors and mean absolute error.
- Two equally sized subgroups have predicted response rates of 0.80 and 0.40, and observed rates of 0.70 and 0.50. What is the aggregate predicted rate, the aggregate observed rate, and the main interpretation?
- Why is a decision-relevant discrepancy in a small validation subgroup not, by itself, proof that the model is unsuitable?
Solutions. (1) The margin is \(0.74-0.70=0.04\). An adverse discrepancy of 0.02 would leave \(0.74-0.02=0.72\), still above the threshold, so that discrepancy alone would not reverse the decision. (2) Signed errors \(y_i-\hat{y}_i\) are \(-1, 2, -1\) units. Mean absolute error is \((1+2+1)/3=1.33\) units. (3) The aggregate predicted rate is \((0.80+0.40)/2=0.60\); the aggregate observed rate is \((0.70+0.50)/2=0.60\). Aggregate agreement conceals overprediction in one subgroup and underprediction in the other. (4) A small subgroup proportion can be imprecise, and the observed discrepancy may not represent predictive behavior in the intended decision setting. It flags a concern to assess, but its evidential strength depends on the data and their relevance.