Subgroup Calibration of Joint-Model Predictions

Why Examine Calibration Within Subgroups?

A prediction model can appear well calibrated overall while systematically overpredicting risk in one clinical group and underpredicting it in another. For example, overprediction in a lower-risk subgroup can offset underprediction in a higher-risk subgroup when results are pooled. Subgroup calibration asks a focused question: at a specified prediction horizon, do predicted risks agree with observed risks among patients in a clinically defined group?

The horizon and subgroup must be specified before interpreting results. A risk prediction for the next 12 months is not the same quantity as a prediction for the next 5 years, and a subgroup defined by a baseline clinical characteristic is not interchangeable with one defined after observing future outcomes. Subgroup analyses are most interpretable when the groups are clinically motivated and defined using information available at prediction time.

The difference is an absolute risk difference, usually reported in percentage points. It summarizes calibration at one horizon and within one group; it does not describe discrimination or explain why a discrepancy occurs. Predictions should be evaluated in data not used to fit or select the model, or with a validation procedure that accounts for model development. Otherwise, apparent agreement can be optimistic.

Estimating Observed Risk With Censoring

If every subject's event status through \(\tau\) is known, the observed risk can be estimated by the event proportion. With right censoring before \(\tau\), however, subjects censored before the horizon have unknown event status at \(\tau\). Treating them as non-events biases the estimate when censoring occurs. For a single event type, a standard subgroup estimate is the complement of the Kaplan–Meier estimate.

This estimate requires appropriate handling of the risk set and, for the usual interpretation, censoring that is independent of event time within the subgroup (or conditionally independent given variables used to account for censoring). If follow-up differs substantially by subgroup, or censoring is informative, a simple Kaplan–Meier estimate may not be valid. The model's predictions and the observed-risk estimate must also target the same population, landmark, endpoint, and horizon.

Uncertainty When Events Are Few

A subgroup calibration result should report the number of subjects, the number of observed events by the horizon, mean predicted risk, observed risk, and an uncertainty interval. A subgroup with few events can have a similar point estimate to another subgroup but much less precise evidence. The event count matters alongside the subgroup sample size: a large group with very few events can still provide limited information about risk.

For Kaplan–Meier estimates, Greenwood's formula estimates the variance of \(\hat S_g(\tau)\). A common interval construction uses the complementary log-log transformation. Define \(G=\sum_{t_j\leq\tau}d_j/[r_j(r_j-d_j)]\). The estimated standard error of \(\log[-\log\{\hat S_g(\tau)\}]\), by the delta method, is approximately \(\sqrt{G}/[-\log\{\hat S_g(\tau)\}]\). An interval on this transformed scale can be converted back to an interval for event risk. This respects the risk range from 0 to 1 better than a simple normal interval for risk, though it does not make a sparse-data estimate precise.

Comparing Calibration Across Subgroups

To ask whether calibration differs between two groups, compare their calibration differences, not just their observed risks. Define the contrast as \(\hat D_A(\tau)-\hat D_B(\tau)\). Its uncertainty depends on both groups' observed outcomes and predicted risks. A visual or numerical difference between point estimates is not, on its own, evidence that calibration differs.

With complete follow-up and fixed predictions, a direct way to estimate uncertainty for a subgroup's mean residual is to use subject-level residuals \(Y_i-p_i(\tau)\), where \(Y_i=1\) if the event occurs by \(\tau\) and \(0\) otherwise. Their average equals the calibration difference when observed risk is the event proportion. The sample variance of these residuals divided by \(n_g\) estimates the variance of that average under independent subject sampling. For two independent subgroups, estimated variances of the subgroup differences add to estimate the variance of their contrast. Censoring requires an estimator that accounts for incomplete event status; the complete-follow-up calculation should not be applied unchanged.

Justification for the Uncertainty Calculation

For complete follow-up, write the residual for subject \(i\) as \(e_i=Y_i-p_i(\tau)\). Its sample mean is \(\bar e_g=n_g^{-1}\sum_{i\in g}Y_i-n_g^{-1}\sum_{i\in g}p_i(\tau)=\hat D_g(\tau)\). Under independent sampling of subjects, the variance of a sample mean is the variance of an individual residual divided by \(n_g\). Replacing that unknown variance with the residual sample variance \(s_{e,g}^2\) gives the estimated standard error \(s_{e,g}/\sqrt{n_g}\). For independent groups A and B, the variance of the difference of their sample means is the sum of their variances, yielding the standard error used in the example. This derivation relies on independent subjects and a sufficiently adequate approximation for the sampling distribution; sparse events can make that approximation poor.

Practical Choices, Diagnostics, and Limitations

There are several reasonable ways to summarize subgroup calibration, but they answer related rather than identical questions. A mean-risk difference is an absolute measure at the chosen horizon. A ratio of observed to mean predicted risk is relative, but can be unstable when predicted or observed risk is near zero. A calibration curve shows risk patterns across a range of predictions, but dividing a small subgroup into further prediction bins can leave very few events per bin. Subgroup-specific recalibration parameters provide another summary, but are often unstable with sparse data and should not replace reporting observed and predicted risks.

Check subgroup sample sizes and event counts before interpreting estimates. Also examine numbers at risk over time, censoring patterns, and whether a few subjects dominate the estimate. If groups differ in follow-up or censoring, apparent differences in observed risk may reflect estimation problems rather than calibration. If the interval is wide, report that uncertainty plainly instead of treating a nonsignificant comparison as proof of equivalence. Multiple subgroup comparisons increase the chance of finding an apparently large difference by chance; prespecification and cautious interpretation help.

A common mistake is to say, “The subgroup estimates differ, so the model is less calibrated in one group.” The correction is to quantify uncertainty in the difference itself, using a method appropriate to censoring and the data structure. Another mistake is to treat everyone censored before \(\tau\) as event-free. Their horizon status is unknown; use a suitable survival estimator and state its censoring assumptions. Finally, subgroup calibration is not subgroup discrimination: agreement between average predicted and observed risk does not tell whether the model ranks individuals well within that subgroup.

These estimates are limited by the validation sample, the selected horizon, subgroup definitions, and the assumptions used to handle censoring. Results from small or highly selected groups may not generalize. A subgroup gap can flag a question for further validation, but by itself it cannot establish why predictions differ from outcomes or justify causal claims.