Why Diagnose Interim Outcome Models?
In covariate-adjusted response-adaptive allocation, an outcome model may help determine how likely a patient with a particular baseline profile is to benefit under each treatment. If its predictions are poorly calibrated or fail to distinguish outcomes, allocation decisions can be based on misleading estimates. A model can look adequate overall while making systematically wrong predictions for one treatment or one covariate profile.
The purpose of diagnostics is not to prove that a model is correct. It is to find evidence that its predictions are unreliable in the parts of the population that influence allocation. The checks below use outcomes already observed at an interim analysis and predictions that were available before those outcomes occurred.
What Must Be in Place for a Useful Check?
A diagnostic is interpretable only if the prediction and observed outcome refer to the same endpoint, treatment, and follow-up period. For example, a predicted 30-day event risk must be compared with 30-day event outcomes, not with events observed over varying follow-up times. The outcome must be recorded consistently, and the treatment assigned to each patient must be correctly identified.
Predictions used for checking should be generated without using the outcomes being checked to fit the model. In a sequential trial, this can mean storing each prediction at the time it is made and comparing it later with the patient's outcome, or evaluating predictions on a separate validation set. Evaluating fitted values on the same data used to estimate a flexible model can make performance look artificially good.
Adaptive assignment also matters. A treatment-specific diagnostic requires patients with observed outcomes on that treatment, and a profile-specific diagnostic requires enough patients in that profile. If some treatment-profile combinations are rare because their assignment probabilities are very small, the corresponding checks will be noisy or unavailable. A favorable overall summary cannot repair a lack of information in a sparse subgroup.
The group \(G\) should be examined separately by treatment and, where sample size permits, by important covariate profile. Calibration-in-the-large compares the overall observed event proportion with the overall mean predicted risk. More detailed calibration can be examined by grouping patients with similar predicted risks or by fitting a calibration regression. For binary outcomes, an ideal calibration intercept is 0 and an ideal calibration slope is 1: an intercept below or above 0 indicates systematic over- or underprediction on the log-odds scale, while a slope below 1 often signals predictions that are too extreme. These summaries should be estimated separately by treatment when each treatment has its own prediction model or when treatment-specific performance is important.
Calibration is different from discrimination. A model can rank higher-risk patients above lower-risk patients yet systematically overpredict everyone’s risk. Conversely, it can predict the overall event rate accurately while failing to rank patients meaningfully. For a binary outcome, the area under the receiver operating characteristic curve (AUC) measures ranking: it is the probability that a randomly selected patient with an event receives a higher predicted risk than a randomly selected patient without an event, with ties conventionally counted as one-half.
Worked Example: Overall Calibration Hides Treatment-Profile Errors
Suppose an interim binary-outcome model is checked in four treatment-profile groups. Each group contains 50 hypothetical patients. The table reports the group’s average predicted event risk and the number of observed events.
| Treatment | Profile | Mean predicted risk | Observed events | Observed event rate |
|---|---|---|---|---|
| A | Low-risk | 0.10 | 10 of 50 | 0.20 |
| A | High-risk | 0.50 | 20 of 50 | 0.40 |
| B | Low-risk | 0.20 | 5 of 50 | 0.10 |
| B | High-risk | 0.40 | 25 of 50 | 0.50 |
For treatment A, the mean predicted event count is \(50(0.10)+50(0.50)=5+25=30\). The observed count is \(10+20=30\), so the overall predicted and observed event rates are both \(30/100=0.30\). For treatment B, the mean predicted count is \(50(0.20)+50(0.40)=10+20=30\), and the observed count is \(5+25=30\), again giving an overall rate of 0.30.
Those arm-level averages look perfectly calibrated, but neither treatment is calibrated within both profiles. Under A, the low-risk group’s gap is \(0.20-0.10=0.10\), while the high-risk group’s gap is \(0.40-0.50=-0.10\). Under B, the corresponding gaps are \(0.10-0.20=-0.10\) and \(0.50-0.40=0.10\). Opposing errors cancel in the aggregate. If allocation depends on predicted outcomes by profile, the profile-specific errors—not just the arm-level average—are relevant.
Assessing Discrimination Without Confusing It With Calibration
For a binary outcome, the AUC summarizes ranking across all event–non-event pairs in the evaluated group. An AUC of 0.5 indicates no better ranking than chance in that group; an AUC of 1 indicates perfect ranking. It does not say whether predicted probabilities are numerically accurate. A treatment-specific AUC should be checked separately when the allocation model uses treatment-specific predictions. Profile-specific AUCs can also be informative, but small groups may make them highly uncertain.
Worked Example: AUC in One Treatment-Profile Group
In a hypothetical validation set for treatment A within one covariate profile, four patients experienced the event and four did not. Their predicted risks are:
Events: 0.90, 0.80, 0.40, 0.30. No event: 0.70, 0.60, 0.20, 0.10.
There are \(4\times4=16\) event–non-event pairs. The event patient with risk 0.90 outranks all four non-events, as does the event patient with risk 0.80. The patient with risk 0.40 outranks 0.20 and 0.10, giving two wins; the patient with risk 0.30 also outranks 0.20 and 0.10, giving two wins. Thus there are \(4+4+2+2=12\) correctly ordered pairs.
The AUC is \(12/16=0.75\). In this small hypothetical group, an event patient has a higher predicted risk than a non-event patient in 75% of randomly selected pairs. That is evidence of some ranking ability, not proof that the risks are well calibrated. For example, all eight risks could be shifted upward while leaving their order and AUC unchanged.
When profile composition differs across treatments, a pooled AUC may reflect differences between profiles rather than useful within-profile ranking. Therefore, inspect arm-specific results and, where practical, compare patients within clinically important profiles. The same caution applies to pooled calibration: results should not be assumed to hold in every treatment-profile subgroup simply because they look satisfactory overall.
How Diagnostic Failure Can Alter Allocation
An adaptive allocation rule translates estimated outcomes into assignment probabilities. A treatment-specific error can therefore affect the rule even if the model’s overall calibration gap is close to zero. If predicted outcomes overstate one treatment’s advantage in a profile, an allocation rule that favors predicted benefit may assign too many future patients from that profile to that treatment. If the error understates the advantage, the rule may adapt too little in that direction. The precise consequence depends on the allocation rule; diagnostics identify unreliable inputs, not a universal correction.
The basic reason is that the rule uses model predictions rather than unknown true conditional means. Suppose, for illustration, a rule for a particular profile assigns treatment B with probability \(p_B=0.5+0.5(\widehat{r}_B-\widehat{r}_A)\), where \(\widehat{r}_A\) and \(\widehat{r}_B\) are predicted success probabilities. This is only a simple illustrative rule, not a general recommendation. If the predicted treatment difference is wrong, the assignment probability inherits that error.
Worked Example: Miscalibration Changes an Illustrative Assignment Probability
For one hypothetical patient profile, suppose the model predicts success probabilities \(\widehat{r}_A=0.40\) and \(\widehat{r}_B=0.60\). Under the illustrative rule above,
Now suppose, purely for illustration, that the true success probabilities in this profile are \(r_A=0.55\) and \(r_B=0.60\). The true difference is only \(0.60-0.55=0.05\). If the same rule could use the true probabilities, it would give \(0.5+0.5(0.05)=0.525\). The model-driven probability of 0.60 is therefore 0.075, or 7.5 percentage points, higher than that benchmark.
The discrepancy occurs because the model underpredicts success under A by 0.15 while predicting B correctly, exaggerating B’s advantage. The calculation does not show that 0.525 is an optimal allocation probability; it shows how a treatment-specific prediction error can be transmitted by a specified allocation rule.
A Practical Diagnostic Sequence
A useful review keeps the diagnostics tied to the intended allocation decisions. It asks whether the model predicts the relevant outcome, for the relevant treatment, in the relevant profiles, using information available at the interim decision time.
Match each stored prediction to its treatment, baseline profile, outcome definition, and prediction time. Exclude predictions that used the outcome being evaluated.
Count observed outcomes by treatment and important profile. Flag cells with too few patients or events to support a stable comparison.
Compare mean predicted and observed outcomes by treatment, then inspect important profiles and risk ranges. Report the direction and size of gaps, not only an overall summary.
For binary outcomes, examine arm- and profile-specific ranking summaries such as AUC. Compare results across interim updates to see whether conclusions change sharply as a few outcomes arrive.
Identify which predicted quantities drive assignment probabilities and whether the observed errors could systematically increase or decrease allocation to a treatment in a profile.
The sequence is diagnostic rather than a mechanical pass–fail test. Calibration plots and subgroup summaries are often more revealing than a single statistic, while formal uncertainty intervals can be wide in small cells. In a sequential trial, each update is based on a changing and potentially selective set of observed outcomes. A stable assessment should therefore preserve prediction-time information and avoid treating repeated interim checks as if they were independent, fixed-sample confirmation.
Comparing the Main Checks
| Check | Question answered | What it cannot establish alone |
|---|---|---|
| Calibration-in-the-large | Does the average predicted outcome match the average observed outcome? | Whether predictions are accurate within profiles or across the risk range |
| Profile- and treatment-specific calibration | Are predictions systematically too high or too low in allocation-relevant groups? | Whether patients are ranked well within a group |
| AUC for a binary outcome | Does the model rank event patients above non-event patients? | Whether the numerical risks agree with observed event rates |
| Overall predictive error summaries | How close are predictions to outcomes under a chosen loss measure? | Which treatment-profile subgroup produced the error or whether the model is useful for the allocation rule |
Each check has limits. Sparse subgroups produce unstable observed rates and AUCs. Grouping predictions can conceal variation within groups or make results depend on the chosen cut points. A good AUC can coexist with poor calibration; good aggregate calibration can coexist with serious subgroup errors. And even a well-calibrated model at one interim time may become less reliable as patient mix, follow-up, or data quality changes.
Common Pitfall
Wrong reasoning: “The predicted and observed event rates match overall, so the model is safe for adaptive allocation.” This conclusion ignores cancellation: one treatment or profile can have overprediction while another has equal-sized underprediction. It also confuses average calibration with discrimination and with the accuracy of the particular treatment contrasts used by the allocation rule.
Correction: Inspect predictions by treatment and by allocation-relevant covariate profile, assess ranking separately from probability accuracy, and connect any observed failures to the quantities that drive assignment. Treat sparse or unstable cells as uncertainty, not as evidence that calibration holds.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a treatment-profile group of 40 patients, the mean predicted event risk is 0.25 and 12 events occur. Calculate the calibration gap and interpret its sign.
- A model has AUC 0.80 but overpredicts every patient’s event risk by a similar amount. Which property is supported, and which is not established?
- Why can an overall calibration gap of zero fail to show that predictions are reliable for adaptive allocation?
- What is one reason to avoid evaluating a flexible model using predictions fitted on the same outcomes used to estimate it?
- If the predicted success advantage of treatment B is exaggerated for one profile, what can happen under an allocation rule that increases B assignment when its predicted advantage rises?
Solutions. (1) The observed rate is \(12/40=0.30\), so the gap is \(0.30-0.25=0.05\): the group had 5 percentage points more events than predicted. (2) The AUC supports ranking ability in the evaluated data; it does not establish calibration, and the stated overprediction is a calibration failure. (3) Errors can cancel across treatments or profiles, as in the worked example, while the allocation rule uses predictions within those groups. (4) The fitted predictions can be overly tailored to those outcomes and make performance appear better than it is for new outcomes. (5) The rule may assign treatment B too often for patients in that profile; the size and direction of the change depend on the rule and the prediction error.