Subgroup Calibration and Risk Heterogeneity

Why Examine Calibration Within Subgroups?

An overall calibration summary compares the total number of observed outcomes with the total number predicted. That comparison is useful, but it can hide important differences across people or settings. A model may predict too few events in one prespecified subgroup and too many in another, with the errors canceling in the aggregate. Subgroup calibration makes those patterns visible.

The calculations are straightforward; their interpretation requires care. A subgroup observed-to-expected ratio describes agreement in event totals for that group. It does not by itself show why a discrepancy occurred, establish that it is statistically meaningful, or guarantee that every person in the subgroup received an accurate prediction.

Observed and Expected Totals Within a Subgroup

For binary outcomes, person \(i\) has observed outcome \(Y_i\in\{0,1\}\) and predicted risk \(p_i\). If \(G_g\) is the set of people in subgroup \(g\), then

The expected total is a sum of individual predicted risks, not the subgroup size multiplied by an arbitrary representative risk. For Poisson counts with predicted mean \(\mu_i\), use \(E_g=\sum_{i\in G_g}\mu_i\) and \(O_g=\sum_{i\in G_g}Y_i\). If person \(i\) contributes exposure \(T_i\) and has predicted rate \(\lambda_i\), then \(\mu_i=T_i\lambda_i\). The ratio of observed to expected counts is also the ratio of observed to model-expected rates when both rates use the same total exposure.

Subgroups should be specified before inspecting the outcome discrepancies, using clinically or scientifically meaningful covariates where possible. The cleanest overall decomposition uses groups that are mutually exclusive and exhaustive: every observation belongs to exactly one group. Overlapping subgroups, such as separate age and sex summaries, can still be assessed, but their totals cannot be added together as if they formed a partition.

How Subgroup Ratios Combine

When groups form a partition and all their expected totals are positive, the overall ratio is an expected-total-weighted average of the subgroup ratios. The weights are determined by each group's share of the total expected outcomes, not by its share of the sample size.

Proof. Since \(R_g=O_g/E_g\), we have \(O_g=E_gR_g\) for each group. Summing over the partition gives

Each weight \(E_g/E\) is nonnegative, and the weights sum to \(\sum_gE_g/E=E/E=1\). Therefore the overall ratio can equal one even when some subgroup ratios are above one and others below one. In the binary example, the weights were \(10/40=0.25\) and \(30/40=0.75\), so \(0.25(1.40)+0.75(0.867)\approx1\), with the exact value \(0.25(1.40)+0.75(26/30)=1\). □

This identity also explains why an unweighted average of subgroup ratios is generally not the overall ratio. Groups with larger expected totals contribute more to the overall event count and therefore receive more weight.

Sampling Variation and Interpretation

A ratio differing from one is a descriptive discrepancy, not automatically evidence of systematic model failure. Even under a correctly specified model, observed counts vary. One useful reference calculation is available when predictions are fixed for an independent validation sample. Suppose the outcomes are conditionally independent given the covariates, and the model gives the correct conditional means.

Derivation. In the binary case, linearity of conditional expectation gives \(E[O_g\mid X]=\sum_{i\in G_g}p_i=E_g\). Conditional independence gives

For independent Poisson outcomes, each conditional mean and variance is \(\mu_i\), so summing gives \(E[O_g\mid X]=\sum_i\mu_i=E_g\) and \(\operatorname{Var}(O_g\mid X)=\sum_i\mu_i=E_g\). In both cases \(E_g\) is fixed after conditioning on the covariates and predictions. Dividing the conditional mean by \(E_g\) gives \(E[R_g\mid X]=1\); dividing the conditional variance by \(E_g^2\) gives the stated variance of the ratio. □

These results explain why ratios from groups with small expected totals can fluctuate substantially: dividing by a small \(E_g\) magnifies count variation. They are not, on their own, a complete uncertainty analysis for predictions estimated on the same outcomes, nor do they account for uncertainty from a complicated subgroup search. Treating model predictions as fixed in those situations can understate uncertainty.

Practical Reporting and Common Pitfalls

A useful subgroup summary reports the group definition, sample size or exposure, observed total, expected total, and \(O_g/E_g\). For a binary outcome, the observed and expected totals can be fractional only on the expected side; for counts, they are usually count totals and model means. Showing both totals prevents a ratio from obscuring whether it is based on substantial information.

• Do not average ratios without weights. The overall ratio uses expected-total weights, as derived above.
• Do not interpret an overall ratio of one as proof of subgroup calibration. Positive and negative subgroup discrepancies can cancel.
• Do not treat every observed subgroup difference as a confirmed interaction or model defect. Sampling variation, especially with small expected totals, can be considerable.
• Do not select many cutpoints after examining the outcomes and then interpret the largest discrepancy as if the subgroup had been prespecified. Such searching makes chance extremes more likely.
• Do not calculate a ratio when \(E_g=0\). The ratio is undefined. Report the observed and expected totals and inspect why the model assigns no expected events; if \(E_g\) is merely very small, flag the ratio as unstable.

Subgroup calibration is valuable because it asks where the aggregate agreement comes from. It can reveal that predictions systematically understate outcomes for one group and overstate them for another, while keeping the interpretation appropriately bounded: subgroup ratios describe event-total agreement at the chosen level of grouping, not exact calibration for every individual or a causal explanation for the errors.