Checking Outcome-Model Calibration in the Control Cohort

Why Calibration Matters

An adjusted treatment contrast may depend on predictions from a model for outcomes under control. For example, an outcome-regression analysis can use that model to predict what trial participants would have experienced under control. If the model systematically predicts too few events for patients at high risk, the adjusted contrast may be distorted even if the model distinguishes higher-risk from lower-risk patients well.

Calibration checks compare predicted control outcomes with outcomes actually observed in controls. The comparison should be made not only overall, but also across risk groups and clinically relevant subgroups. A reassuring overall average can conceal errors that cancel one another—for example, overprediction in one subgroup and underprediction in another.

A calibration check requires comparable outcome definitions and follow-up: an event counted in the predictions must be the same event, over the same time horizon, as the event counted in the observed control data. The controls must also be relevant to the population for which the predictions will be used. If controls have been weighted to represent a target population, the observed outcomes and predictions must be summarized using the same target weights.

Where possible, assess predictions for control outcomes that were not used to fit the model, such as held-out observations or predictions generated by cross-validation. Checking predictions on the same observations used to fit a flexible model can make performance look better than it is for new individuals. This does not make a training-data comparison useless, but it makes it a limited diagnostic rather than an independent assessment.

Group-Level Comparisons

One practical approach is to divide controls into risk groups using their predicted control risks, then compare observed and expected event counts in each group. Group boundaries can be chosen to produce clinically interpretable ranges and adequate numbers of outcomes; equal-sized groups are convenient but not mandatory. For group \(g\), let \(n_g\) be the number of controls, \(O_g\) the number of observed events, and \(q_i\) the predicted risk for individual \(i\). The expected event count is the sum of the individual predicted risks.

The difference \(\widehat{r}_{\mathrm{obs},g}-\widehat{r}_{\mathrm{pred},g}\) is the observed-minus-predicted risk, in percentage points when multiplied by 100. The ratio \(O_g/E_g\), when \(E_g>0\), is the observed-to-expected ratio. A difference above zero or a ratio above one indicates more events were observed than predicted. Neither summary, by itself, explains why the discrepancy arose.

These comparisons are diagnostic summaries, not proof that a particular causal assumption holds. In particular, calibration of predicted outcomes among controls does not show that all confounders were measured, that treatment groups are exchangeable after adjustment, or that the treatment contrast is unbiased. It addresses one component: whether control-outcome predictions agree with observed control outcomes in the groups examined.

Why Grouping and Subgroups Both Matter

Risk-group checks ask whether predictions agree with outcomes across levels of predicted risk. Subgroup checks ask whether agreement holds for clinically important categories such as age band, disease stage, or baseline treatment history. These questions are different. A model can be calibrated across risk groups overall while consistently underpredicting for one subgroup and overpredicting for another.

A useful subgroup is chosen for clinical relevance or because there is a plausible reason the outcome model may behave differently there—not simply because many subgroups are available to search. Report subgroup sizes, observed events, and expected events alongside the comparison. Small groups can show large differences by chance, while groups with very few events provide little information about calibration.

Uncertainty and a Short Justification

Observed counts vary even when predicted risks are correct. A difference between observed and expected counts is therefore not automatically evidence of model failure. A simple reference calculation shows why sample size and predicted risk both matter. Suppose, for the moment, that the risks \(q_i\) are fixed and correct, and that the binary outcomes are independent conditional on those risks. Then \(O_g=\sum_{i\in g}Y_i\), with \(E(Y_i)=q_i\) and \(\operatorname{Var}(Y_i)=q_i(1-q_i)\).

The mean discrepancy is zero under the assumed model. Independence gives a variance for the discrepancy by adding the individual outcome variances; subtracting the fixed expected count does not change that variance.

This calculation justifies treating a modest discrepancy as potentially compatible with random outcome variation, especially in a small group. It is not a complete uncertainty analysis when the risks have been estimated. In a Bayesian analysis, model parameters and thus \(q_i\) are uncertain; their posterior uncertainty should be included when assessing the expected counts. A practical approach is to calculate \(E_g\) for each posterior draw of the control-outcome model, then examine the resulting distribution of group-level predicted risks or counts alongside the observed count and its sampling variability. Avoid interpreting a single posterior mean prediction as if it were known without error.

Diagnostics, Alternatives, and Limitations

A calibration plot can complement the tables: plot observed event proportions against average predicted risks across groups, with a reference line where observed and predicted risks agree. Show uncertainty intervals for observed proportions and, when feasible, uncertainty in the predictions. A systematic departure from the reference line is more concerning than an isolated noisy group, but the plot can be misleading if groups are very small or selected after looking at many possible partitions.

Check the data and analysis before interpreting a discrepancy as a model defect. Verify the endpoint coding, follow-up window, missing-outcome handling, and covariate definitions across sources. Check whether risk groups and subgroups contain enough controls and events to support the comparisons. If controls are weighted, calculate observed and predicted summaries using the same weights and use uncertainty estimates that account for weighting; an ordinary binomial standard error generally does not account for unequal weights. Revisit covariate overlap as well: predictions in regions with few comparable controls may be unreliable even if summaries look acceptable in the observed cohort.

Calibration has a different purpose from several other diagnostics. Discrimination measures how well predictions rank individuals by risk, not whether the predicted probabilities match event frequencies. Covariate balance evaluates differences in covariate distributions after adjustment, not agreement between predicted and observed outcomes. Calibration is also distinct from outcome-model fit assessed by broader checks of model-generated data: here the focus is specifically on observed-versus-predicted control outcomes in risk groups and clinically meaningful subgroups.

A calibration failure may signal a misspecified functional form, omitted interactions, population differences, or outcome measurement problems. This check alone cannot distinguish among them. Conversely, apparently good calibration cannot establish conditional exchangeability, rule out unmeasured confounding, or validate extrapolation to a population poorly represented by controls. In external-control analyses, calibration is most informative when assessed in controls relevant to the target population and interpreted together with the earlier material on overlap and covariate balance.