Subgroup and Site-Specific Performance Diagnostics

Why Diagnose Performance by Subgroup?

A prediction model can perform differently across demographic groups or recruitment sites, even when its overall validation results look acceptable. Some groups may have weaker discrimination, systematically over- or underpredicted risks, or larger prediction errors. These differences matter because an overall average can obscure poor performance for a subgroup that is important to the intended use.

A subgroup diagnostic is a validation estimate within a group defined before examining its performance. The group might be a demographic category or recruitment site. The point is to evaluate how predictions behave in relevant parts of the stated validation population, not to infer that group membership caused a performance difference. Group definitions, prediction target, outcome definition, and metric should be specified consistently; otherwise the comparison may reflect changing definitions rather than changing performance.

Definition: For a prespecified subgroup \(g\), a subgroup-specific performance metric is the metric calculated using the validation participants in \(g\) and their corresponding predictions and outcomes. It describes performance for that subgroup in the validation population represented by those participants. It does not, by itself, establish performance in a different population.

Use predictions generated without using the validation outcomes to train or tune the model. The earlier material on independent holdout validation, cross-validation, information leakage, and validation splits for related individuals describes how to obtain suitable predictions and account for dependence in the split. This tutorial focuses on diagnosing those predictions across subgroups. Comparisons are most interpretable when all groups use the same outcome definition, prediction horizon, model version, and metric calculation.

Which Metrics Can Be Compared?

For a binary outcome, discrimination can be summarized by the area under the receiver operating characteristic curve, or AUC. Within subgroup \(g\), the AUC is the probability that a randomly selected case has a higher predicted risk than a randomly selected control from the same subgroup, with ties counted as one-half. It measures ranking, not whether the numerical risks are accurate.

\operatorname{AUC}_g = P(\hat r_{\text{case}}>\hat r_{\text{control}}\mid g) +\frac{1}{2}P(\hat r_{\text{case}}=\hat r_{\text{control}}\mid g).

Calibration asks whether predicted risks agree with observed outcome frequencies. One simple subgroup summary compares the number of observed cases with the total predicted number of cases. If \(Y_i\) is the binary outcome and \(\hat r_i\) is the predicted risk for participant \(i\), define:

O_g=\sum_{i\in g}Y_i,\qquad E_g=\sum_{i\in g}\hat r_i,\qquad \frac{O_g}{E_g}=\frac{O_g}{E_g}.

Here \(O_g\) is the observed case count and \(E_g\) is the expected case count implied by the predictions. A ratio above one indicates more observed cases than predicted in aggregate; a ratio below one indicates fewer. The ratio is an aggregate diagnostic and does not show whether calibration varies across the range of predicted risks.

Calibration intercepts and slopes give complementary diagnostics. With probabilities strictly between zero and one, the subgroup calibration model can be written:

\operatorname{logit}\{P(Y_i=1\mid g,\hat r_i)\} = a_g+b_g\operatorname{logit}(\hat r_i).

An ideal intercept and slope are \(a_g=0\) and \(b_g=1\). A positive intercept indicates a tendency toward higher observed risk than predicted after accounting for the fitted slope; a slope below one commonly indicates predictions that are too extreme, while a slope above one indicates predictions that are not extreme enough. These are fitted summaries, not substitutes for examining the distribution of predictions and outcomes. With the slope fixed at one, the intercept can instead be estimated by the offset model:

\operatorname{logit}\{P(Y_i=1\mid g,\hat r_i)\} = a_g+\operatorname{offset}\{\operatorname{logit}(\hat r_i)\}.

The Brier score measures squared error for binary predictions: the mean of \((Y_i-\hat r_i)^2\). Lower values indicate better average probability accuracy, but the score also depends on outcome prevalence and the mix of risks in the subgroup. For quantitative outcomes, useful summaries include mean squared error, root mean squared error (RMSE), and explained variance, calculated within each subgroup using the same outcome scale and definition.

Worked Examples

Worked Example: Comparing AUCs Within Two Groups

Suppose a hypothetical validation set contains two cases and two controls in each of two prespecified groups. In Group A, the case risks are 0.80 and 0.60, and the control risks are 0.40 and 0.20. Every case-control comparison is correctly ranked: each of the two case risks exceeds each of the two control risks. Thus there are four concordant pairs out of four, giving \(\operatorname{AUC}_A=4/4=1.00\).

In Group B, the case risks are 0.70 and 0.30, and the control risks are 0.60 and 0.10. The 0.70 case risk exceeds both control risks, contributing two concordant pairs. The 0.30 case risk is below 0.60 but above 0.10, contributing one concordant pair. Therefore \(\operatorname{AUC}_B=3/4=0.75\).

The observed AUC difference is \(1.00-0.75=0.25\). It is a descriptive difference from very small groups, not persuasive evidence that the model truly discriminates better in Group A. With only two cases and two controls in each group, a single changed ranking would substantially alter an estimate. An uncertainty interval and the numbers of cases and controls are essential context.

Worked Example: Aggregate Calibration and Brier Score

Consider six hypothetical participants in one subgroup, with predicted risks and binary outcomes as follows: \((0.10,0)\), \((0.20,0)\), \((0.20,1)\), \((0.40,0)\), \((0.40,1)\), and \((0.60,0)\). There are two observed cases, so \(O_g=2\). The total predicted number of cases is:

E_g=0.10+0.20+0.20+0.40+0.40+0.60=1.90,\qquad \frac{O_g}{E_g}=\frac{2}{1.90}=1.053.

Thus the observed case count is about 5.3% higher than the expected count in this small subgroup. This does not mean that each participant's risk is underpredicted by 5.3%; it is an aggregate comparison.

The Brier score is the mean of the six squared errors:

\operatorname{BS}_g = \frac{0.10^2+0.20^2+(1-0.20)^2+0.40^2+(1-0.40)^2+0.60^2}{6} = \frac{1.57}{6} = 0.2617.

The observed risk outcomes and predictions are the complete data used in this calculation. The ratio and score summarize different features: observed-to-expected compares aggregate case counts, whereas the Brier score penalizes individual probability errors.

Worked Example: Quantitative-Trait Error by Recruitment Site

Suppose two hypothetical sites each have four observed trait values, in the same units: 10, 12, 14, and 16. At Site A the predictions are 11, 11, 13, and 15. The prediction errors, defined as observed minus predicted, are \(-1,1,1,1\). The sum of squared errors is \(1+1+1+1=4\), so:

\operatorname{MSE}_A=\frac{4}{4}=1,\qquad \operatorname{RMSE}_A=\sqrt{1}=1\text{ unit}.

At Site B the predictions are 8, 10, 12, and 14. Each error is 2 units, so the sum of squared errors is \(4+4+4+4=16\), and:

\operatorname{MSE}_B=\frac{16}{4}=4,\qquad \operatorname{RMSE}_B=\sqrt{4}=2\text{ units}.

The Site B RMSE is one unit greater, and twice the Site A RMSE, for these observations. This is a clear descriptive contrast, but each site has only four participants. It should not be treated as a stable estimate of the site-level performance gap without considering uncertainty and the intended practical threshold.

Uncertainty, Comparisons, and Practical Importance

A subgroup estimate can be unstable because the subgroup contains few participants, few cases or controls, a narrow range of predicted risks, or highly variable outcomes. AUC especially requires both cases and controls; if either count is very small, its estimate can vary sharply. Calibration models may also be unstable when the event count is low or predictions are concentrated in a small range. Report subgroup sizes and, for binary outcomes, case and control counts alongside the metrics.

Uncertainty intervals should reflect the validation design. For independent participants, a participant-level bootstrap may be suitable for many metrics. If observations are related or clustered by site, resampling individuals independently can understate uncertainty; the resampling or variance method should preserve the relevant independent units. If the predictions came from cross-validation, the uncertainty calculation should also respect how predictions were generated. The earlier material on uncertainty in validation metrics provides the general framework.

A subgroup difference should be assessed as a contrast, not by checking whether two separate confidence intervals overlap. For a metric \(M\), the relevant quantity may be \(M_g-M_h\), with an uncertainty interval for that difference. A statistically detectable difference need not be practically important, and a practically concerning point estimate may remain uncertain. Define in advance what size of gap would matter for the intended use, and report both the estimate and its uncertainty.

The overall Brier score for a partition into nonoverlapping subgroups is a weighted average of subgroup Brier scores, with weights equal to subgroup sample proportions. This follows directly by separating the sum over all participants into subgroup sums:

\operatorname{BS}_{\mathrm{all}} = \frac{1}{n}\sum_g\sum_{i\in g}(Y_i-\hat r_i)^2 = \sum_g\frac{n_g}{n}\operatorname{BS}_g.

This identity explains why a large group can dominate an overall error score. AUC does not generally equal a weighted average of within-group AUCs: the pooled AUC also includes comparisons between cases and controls from different groups. Therefore, a pooled AUC and subgroup AUCs answer different questions and should not be interpreted as interchangeable summaries.

A Practical Diagnostic Sequence

Specify the comparison.
Define subgroups, outcome, prediction horizon, metrics, and practically important gaps before inspecting subgroup results.

Check the data support.
For each group, report sample size, outcome counts or outcome spread, and the range of predicted values. Note missingness or design features that limit interpretation.

Estimate and quantify uncertainty.
Calculate the same metrics in each subgroup and use an uncertainty method compatible with the validation design and dependence structure.

Interpret the gap in context.
Compare subgroup estimates and their uncertainty with the prespecified practical threshold. Avoid ranking groups based only on noisy point estimates.

Common Mistakes and Key Takeaway

Equating a small subgroup estimate with a stable difference. A visibly lower AUC in a group with very few cases may be sampling noise. Report counts and uncertainty before drawing conclusions.
Using only the pooled metric. A good overall score can conceal poor subgroup performance, especially when a subgroup is small. Present subgroup diagnostics as well as the overall summary when the question warrants them.
Calling a calibration ratio an individual-level correction. \(O_g/E_g\) compares totals; it does not describe the error for each predicted risk or establish calibration throughout the risk range.
Interpreting a group gap causally. A difference in predictive performance is an observed diagnostic, not evidence that demographic group or recruitment site caused the difference.
Confusing statistical and practical importance. A narrow interval around a small difference may indicate a precise but unimportant gap; a large estimated gap with a wide interval may warrant further evaluation without being conclusive.

Key takeaway: Subgroup and site-specific diagnostics make uneven prediction performance visible, but they are informative only when groups are prespecified, metrics are comparable, uncertainty is reported, and estimates are judged against practical needs rather than point estimates alone.

Check Your Understanding

Answer these questions using the definitions and examples above.

What does a subgroup AUC measure, and why does a subgroup with very few cases provide an unstable estimate?
If \(O_g/E_g\) is greater than one, what does that say about aggregate observed and predicted case counts? What does it not establish?
Why can the overall Brier score be dominated by a large subgroup even when a smaller subgroup has substantially worse error?
Why is a pooled AUC not generally the weighted average of subgroup AUCs?
What information should accompany a subgroup performance estimate before deciding whether a gap is important?