Why Examine Calibration Within Subgroups?
A transition prediction model can appear well calibrated overall while systematically overpredicting for one group and underpredicting for another. For example, the average predicted probability of entering a particular state may match the average observed probability in a validation cohort, even though the predictions are too high in one prespecified clinical group and too low in another. Subgroup calibration asks whether predicted transition probabilities agree with observed probabilities within groups that matter for clinical interpretation or model use.
The comparison must be made for a clearly defined transition-probability target, population, and time horizon. A subgroup contrast is not automatically evidence that a model behaves differently because of group membership: sampling variability, few transitions, incomplete follow-up, and differences in the distribution of predicted risks can all affect the estimates. The aim is to describe the size and uncertainty of subgroup-specific calibration errors, not to turn every observed difference into a definitive claim.
Calibration is a property of predictions in a specified validation population. A small average gap does not establish that every individual's prediction is accurate, and a subgroup gap does not by itself explain why the mismatch occurred. Groups may be defined by clinical or demographic variables, or by prespecified bands of predicted transition risk. If risk bands are used, the band definition and prediction time should be set without looking at validation outcomes; otherwise, apparent subgroup patterns can be selected from noise.
Estimating and Comparing Subgroup Gaps
For a fixed horizon with complete outcome ascertainment, \(\widehat O_{g,ij}(t)\) can be the observed proportion occupying state \(j\) at the horizon among subgroup members who began in state \(i\). With censoring before the horizon, a censoring-adjusted estimator is needed; for example, one may use a suitable state-occupation estimator or pseudo-observation approach. Its validity depends on the assumptions of that estimator, including appropriate handling of censoring. The prediction and observed estimate must refer to the same horizon, origin-state population, and outcome definition.
For a simple fixed-horizon binary outcome with \(n_g\) independent validation subjects and a complete outcome for each, suppose the observed proportion is \(\widehat O_g\). Treating the model predictions as fixed, an approximate standard error for the gap is \(SE(G_g)\approx\sqrt{\widehat O_g(1-\widehat O_g)/n_g}\). This is the binomial sampling uncertainty in the observed proportion. It does not include uncertainty from fitting the prediction model. If predictions are estimated as part of the validation process, or the aim is to account for model-fitting uncertainty, the uncertainty calculation must also reflect that estimation, for example through an appropriate resampling procedure.
A different question is whether two subgroup gaps differ. For disjoint groups with independent subjects, an approximate standard error for their difference is \(SE(G_A-G_B)\approx\sqrt{SE(G_A)^2+SE(G_B)^2}\). This contrast is not the same as asking whether either group’s gap differs from zero. Confidence intervals describe plausible values for the gap or gap contrast under the specified procedure; a wide interval signals limited precision, even when its point estimate looks large.
Worked Example: A Subgroup Gap With Complete Follow-Up
Suppose a hypothetical validation cohort includes 80 patients in a prespecified subgroup, all in the same origin state at the prediction time. For a particular destination state at a 12-month horizon, their mean predicted probability is 0.20. At 12 months, 20 of the 80 patients occupy that destination. Assume outcomes are completely observed and the predictions are treated as fixed.
The observed proportion is \(20/80=0.25\). Thus the subgroup calibration gap is \(0.25-0.20=0.05\), or 5 percentage points. The positive sign indicates that the observed destination probability is higher than predicted in this subgroup.
The estimated standard error, using the binomial approximation, is \(\sqrt{0.25(1-0.25)/80}=\sqrt{0.1875/80}=\sqrt{0.00234375}\approx0.0484\). An approximate 95% interval for the observed probability is \(0.25\pm1.96(0.0484)\), or approximately \((0.155,0.345)\). Subtracting the fixed mean prediction of 0.20 gives an approximate interval for the gap of \((-0.0449,0.145)\).
The point estimate suggests 5 percentage points of underprediction, but the interval is wide and includes zero. These data do not establish a nonzero subgroup gap; they are also compatible with appreciable underprediction or overprediction. The uncertainty is consequential and should be reported alongside the point estimate.
Contrasting Gaps Across Risk Groups
A gap comparison can identify whether the direction or magnitude of calibration differs across prespecified groups. For instance, define low- and high-predicted-risk groups before examining outcomes. Compute the observed-minus-predicted gap separately in each group, then estimate the contrast. Because the groups are disjoint, the sampling variation in their observed proportions can be combined using the square-root-of-sum-of-variances formula. A confidence interval for the contrast that includes zero means the data do not clearly distinguish the group gaps; it does not prove that the gaps are equal.
Worked Example: Comparing Two Prespecified Risk Bands
In a hypothetical validation sample, two prespecified risk bands each contain 100 subjects. For the same destination and horizon, the low-risk band has mean predicted probability 0.10 and observed proportion 0.12. The higher-risk band has mean predicted probability 0.10 and observed proportion 0.09. Assume complete outcome ascertainment, independent subjects, and fixed predictions.
The low-risk gap is \(0.12-0.10=0.02\); the higher-risk gap is \(0.09-0.10=-0.01\). The contrast is \(0.02-(-0.01)=0.03\), so the point estimate is a 3-percentage-point larger gap in the low-risk band.
The estimated standard error of the contrast is \(\sqrt{0.12(0.88)/100+0.09(0.91)/100}=\sqrt{0.001056+0.000819}=\sqrt{0.001875}\approx0.0433\). An approximate 95% interval for the difference is \(0.03\pm1.96(0.0433)\). Since \(1.96(0.0433)\approx0.0849\), the interval is approximately \((-0.0549,0.1149)\).
The estimated difference is small relative to its uncertainty, and the interval includes zero. The sample therefore does not provide precise evidence that calibration differs between these two bands. It would be incorrect to report that the groups are equally calibrated: differences of several percentage points in either direction remain compatible with the data. The interval also concerns the difference between the two gaps, not whether either individual gap is exactly zero.
Coherence Across Competing Destinations
For a given origin state and horizon, predicted probabilities for all possible destination states, including remaining in the origin state where appropriate, form a probability distribution and sum to one. Observed state-occupation probabilities have the same structure. Separate subgroup analyses can conceal this constraint: a model might overpredict one destination and underpredict another, while its total probability remains coherent. Conversely, adjusting one destination alone can make the full set of predictions incoherent.
When interpreting subgroup calibration, therefore, identify the target destination and check the full probability vector. If the target is a set of competing destinations, use a definition that matches that set, rather than treating a probability for one destination as though it were the probability of any event. Calibration of one destination does not automatically imply calibration of the others.
Worked Example: Checking Two Competing Destinations
Consider a hypothetical subgroup of 50 patients in an origin state at baseline, followed completely to a fixed horizon. The model's mean predicted probabilities are 0.60 for remaining in the origin state, 0.25 for entering state A, and 0.15 for entering state B. At the horizon, the observed counts are 30, 12, and 8, respectively.
The predicted probabilities sum to \(0.60+0.25+0.15=1.00\). The observed proportions are \(30/50=0.60\), \(12/50=0.24\), and \(8/50=0.16\), which also sum to \(1.00\). The subgroup gaps are 0.00 for remaining in the origin state, \(0.24-0.25=-0.01\) for state A, and \(0.16-0.15=0.01\) for state B.
This pattern is internally coherent: the one-percentage-point underprediction for state B is balanced by the one-percentage-point overprediction for state A, while the origin-state probability agrees. It does not establish that either destination-specific gap is precisely zero. For example, the approximate binomial standard error for the state A observed proportion is \(\sqrt{0.24(0.76)/50}\approx0.0604\), much larger than its 0.01 gap. The sample gives little precision for distinguishing such a small mismatch from sampling variation.
Assumptions, Diagnostics, and Common Pitfalls
Subgroup comparisons are interpretable only when the subgroup definitions, prediction horizon, and transition target are aligned. Predictions should be generated for each validation subject using information available at the prediction time, and the outcome estimate must correspond to the same starting population and horizon. With censoring, the chosen estimator must address it appropriately; an unadjusted proportion among subjects whose outcomes happen to be observed can be biased if follow-up is incomplete in a way related to outcome.
Diagnostics should begin with counts and uncertainty, not just point estimates. Tabulate the number of subjects in each subgroup, the number observed in each destination, and the amount of incomplete follow-up. Inspect whether a few influential subjects or very sparse transitions dominate the apparent gap. Look for wide intervals, unstable estimates under modest changes in grouping, and discrepancies between an overall gap and the pattern of subgroup gaps. Earlier material on influential subjects and rare-transition validation is directly relevant to these checks.
- Do not equate a nonsignificant contrast with equal calibration. A wide interval can reflect insufficient information. Report the interval and describe what differences it still permits.
- Do not infer a subgroup difference from separate significance tests. One group having an interval that excludes zero while another does not is not itself a test that the gaps differ. Estimate and report the gap contrast directly.
- Do not search many cut points and report only the most striking one. Post hoc grouping can exaggerate apparent differences. Prespecify clinically meaningful subgroups or risk bands, and acknowledge multiplicity when many comparisons are examined.
- Do not treat calibration as causation or fairness by itself. A subgroup gap describes agreement in the validation data; it does not identify why the gap occurs or establish the consequences of using the model.
The main alternative is to report only overall calibration, which is simpler but can conceal offsetting subgroup errors. A detailed calibration curve can show how predicted and observed risks relate across a continuum, but subgroup curves require enough data and can be noisy. A single subgroup gap is easier to summarize, though it can hide variation within the subgroup. These summaries answer different questions and should not be treated as interchangeable. The best choice depends on the prespecified validation question, event availability, and the precision the data can support.
Subgroup calibration is also limited by the validation cohort. Small or unrepresentative subgroups may yield unstable estimates, and a well-estimated gap in one cohort need not carry over to another. Comparisons across subgroups do not by themselves resolve differences in covariate distributions or other features of the validation setting. Such issues require separate assessment rather than being attributed automatically to subgroup-specific model behavior.
Name the origin state, destination, prediction time, horizon, and subgroup definitions before examining subgroup outcomes.
Calculate each subgroup's mean predicted probability and an observed probability estimate for the same population and horizon, using an appropriate approach to incomplete follow-up.
Report subgroup gaps and intervals. If comparing groups, estimate the gap contrast and its uncertainty directly.
Check sample sizes, event counts, coherence across destinations, and whether uncertainty permits meaningful subgroup differences.
Check Your Understanding
Answer each question using the specified transition target and subgroup context.
- A subgroup has mean predicted probability 0.18 and observed probability 0.14. What is the calibration gap, and what does its sign mean?
- Two disjoint subgroups have gaps 0.04 and 0.01. What is the gap contrast, and why is it different from testing each gap separately?
- Why can a wide interval that includes zero not establish equal calibration across groups?
- Predicted probabilities for three possible states are 0.70, 0.20, and 0.10. The observed proportions are 0.68, 0.22, and 0.10. What coherence check can be made, and what can this check not establish?
Solutions. (1) The gap is \(0.14-0.18=-0.04\), or −4 percentage points; observed probability is lower than predicted, so the model overpredicts on average in that subgroup. (2) The contrast is \(0.04-0.01=0.03\), or 3 percentage points. It directly estimates how far apart the subgroup gaps are; separate tests assess whether each gap individually differs from zero. (3) The interval may include zero because the estimate is imprecise, and it may also include nonzero differences that matter in context. It does not prove equality. (4) Both vectors sum to one: predicted \(0.70+0.20+0.10=1.00\), observed \(0.68+0.22+0.10=1.00\). This checks coherence of the probability distributions, but does not show that the individual gaps are estimated precisely or that they are exactly zero.