Subgroup Discrimination and Its Interpretation

Why Compare Discrimination Within Subgroups?

A prediction model can rank people who experience an event above people who do not, but its ranking performance need not be the same in every population subgroup. For a joint model, this question is usually tied to a prediction landmark: among people still event-free at time \(s\), how well does their predicted risk by a later horizon \(t\) separate those who experience the event from those who remain event-free? A subgroup comparison can reveal where ranking is weaker, but a difference in AUC does not by itself show that the model behaves differently in a fundamental or transportable sense. Subgroups may differ in the range of predicted risks, the mix of event severity or timing, and other characteristics that shape the case and control distributions.

This tutorial focuses on discrimination at a fixed horizon and on interpreting subgroup differences. The introductory material on subgroup calibration concerns whether predicted risks agree with observed event frequencies; discrimination instead concerns ranking. Good calibration does not guarantee good ranking, and a strong AUC does not guarantee calibrated risks.

Define the Subgroup-Specific Horizon AUC

Suppose prediction is made at landmark \(s\), and consider an event horizon \(t>s\). For each person \(i\) who is event-free at \(s\), let \(\hat R_i(t\mid s)\) be the joint model's predicted probability of experiencing the event by \(t\), conditional on the longitudinal history available at \(s\) and the person's information at that landmark. Let \(T_i\) be the event time and \(C_i\) the censoring time. Within subgroup \(g\), a case is a person with \(s<T_i\leq t\); a control is a person with \(T_i>t\). A person censored before their status at \(t\) is known cannot simply be labeled a control.

Definition: The horizon-specific AUC in subgroup \(g\), denoted \(\mathrm{AUC}_g(t\mid s)\), is the probability that a randomly selected case in subgroup \(g\) has a higher predicted risk by \(t\) than a randomly selected control from that subgroup, with half-credit for tied predictions: \[ \mathrm{AUC}_g(t\mid s) = P\{\hat R_{\mathrm{case}}(t\mid s)>\hat R_{\mathrm{control}}(t\mid s)\mid g\} +\tfrac12P\{\hat R_{\mathrm{case}}(t\mid s)=\hat R_{\mathrm{control}}(t\mid s)\mid g\}. \] A value of 0.5 indicates chance-level ranking for these case-control pairs; a value of 1 indicates perfect ranking. The AUC is a ranking measure, not an estimate of absolute-risk accuracy.

With complete follow-up through \(t\), the empirical estimator compares every case with every control. Write \(n_{1g}\) and \(n_{0g}\) for the numbers of cases and controls in subgroup \(g\). Let \(I(\cdot)\) equal 1 when its statement is true and 0 otherwise. Then

\widehat{\mathrm{AUC}}_g(t\mid s) = \frac{1}{n_{1g}n_{0g}} \sum_{i\in\mathcal{C}_g} \sum_{j\in\mathcal{N}_g} \left[ I\{\hat R_i(t\mid s)>\hat R_j(t\mid s)\} +\tfrac12 I\{\hat R_i(t\mid s)=\hat R_j(t\mid s)\} \right],

where \(\mathcal{C}_g\) and \(\mathcal{N}_g\) are the subgroup's case and control sets. This estimator is a sample version of the probability in the definition: each equally weighted case-control pair contributes 1 for a correctly ordered pair, 0 for a reversed pair, and 0.5 for a tie.

Worked Example: Counting Case-Control Pairs

Suppose a hypothetical validation sample has two cases and two controls in each of two subgroups. At a fixed horizon, subgroup A has case risks 0.80 and 0.60 and control risks 0.70 and 0.20. Subgroup B has case risks 0.90 and 0.50 and control risks 0.80 and 0.40. Assume everyone’s event status through the horizon is known.

In subgroup A, the case with risk 0.80 outranks both controls, giving 2 wins. The case with risk 0.60 outranks only the control with risk 0.20, giving 1 win. Thus \(\widehat{\mathrm{AUC}}_A=(2+1)/(2\times2)=3/4=0.750\). In subgroup B, the 0.90 case outranks both controls and the 0.50 case outranks only the 0.40 control. Therefore \(\widehat{\mathrm{AUC}}_B=(2+1)/4=0.750\).

Pooling the groups changes the comparison set. There are four cases and four controls, so 16 pairs. The cases with risks 0.80, 0.60, 0.90, and 0.50 contribute, respectively, 3.5, 2, 4, and 2 pair credits. The 0.80 case ties the 0.80 control and receives half credit for that pair. The total is \(3.5+2+4+2=11.5\), so the pooled AUC is \(11.5/16=0.71875\), or 0.719. It is not the average of the two subgroup AUCs: pooled comparisons include pairs across subgroups, whose score scales and distributions may differ.

Why Subgroup AUCs Can Differ

The AUC is not a fixed property of the model alone. It depends on the distribution of scores among cases and controls in the population being evaluated. If a subgroup contains cases and controls whose risks are well separated, its AUC may be high. If their score distributions overlap, the AUC may be lower. Differences in the mix of people who become cases or controls, or in the range of scores in the subgroup, can therefore change the AUC even when the prediction procedure has not changed.

A comparison should distinguish a descriptive difference between the observed subgroup AUCs from a claim that predictive performance differs after accounting for population composition. The former is directly measured by the subgroup AUCs. The latter requires a carefully defined comparison target. One option is to compare discrimination within prespecified strata of case mix and then average those stratum-specific estimates using the same weights in each subgroup. This is a standardized summary, not necessarily either subgroup's ordinary marginal AUC: the latter also includes comparisons between cases and controls from different strata.

Worked Example: Standardizing Over a Shared Case-Mix Distribution

Consider two hypothetical subgroups, each evaluated at the same landmark and horizon. Within a prespecified low-risk case-mix stratum, the estimated case-control pair AUC is 0.60 in both subgroups; within a high-risk stratum, it is 0.90 in both. Suppose 80% of the case-control pairs in subgroup A are from the low-risk stratum and 20% from the high-risk stratum. In subgroup B the proportions are reversed.

For illustration, the pair-weighted within-stratum summary using each subgroup's own pair proportions is \(0.80(0.60)+0.20(0.90)=0.66\) for A and \(0.20(0.60)+0.80(0.90)=0.84\) for B. These summaries differ because the stratum weights differ, despite identical AUCs within each stratum.

Now standardize both subgroup summaries to a common distribution that assigns 50% weight to each stratum. For either subgroup the result is \(0.50(0.60)+0.50(0.90)=0.75\). This removes the specified difference in stratum weights from this particular summary. It does not establish that every aspect of subgroup performance is equal, and it is not the ordinary marginal AUC: cross-stratum case-control pairs are not included in this calculation. The strata and weights should be chosen for a substantive reason and reported explicitly.

Censoring and Estimation

When follow-up is incomplete, excluding everyone censored before \(t\) can bias the comparison if censoring is related to risk or event status. A standard approach is inverse-probability-of-censoring weighting (IPCW). Let \(G_i(u)=P(C_i\geq u\mid Z_i)\) be the probability that person \(i\) remains uncensored through \(u\), conditional on measured variables \(Z_i\) used to model censoring. An observed case with event time \(T_i\leq t\) receives a weight proportional to \(1/G_i(T_i)\); an observed control known to remain event-free through \(t\) receives a weight proportional to \(1/G_i(t)\). The weighted AUC is the weighted average of pair credits, with each pair's weight equal to the product of its case and control weights.

This method relies on a sufficiently well-specified censoring model and conditional independent censoring: given the variables used in \(Z_i\), censoring must be independent of the event process relevant to the comparison. There must also be a positive probability of remaining uncensored at the event times and horizon being evaluated. Very small estimated \(G_i\) values produce very large weights and unstable estimates. IPCW cannot recover information reliably when a relevant subgroup has almost no observable follow-up through \(t\).

Worked Example: A Weighted AUC With Censoring

Suppose a hypothetical subgroup has two observed cases and two observed controls at a fixed horizon. The case risks are 0.80 and 0.60; the control risks are 0.70 and 0.40. For simplicity, assume the censoring model gives each case a weight of 1.25 and each control a weight of 1.25, except that the 0.60 case has weight 2.00. These correspond to inverse probabilities of remaining observable at the relevant times. Assume the conditional independent-censoring and positivity assumptions are reasonable for this example.

The total weight over all case-control pairs is \((1.25+2.00)(1.25+1.25)=3.25(2.50)=8.125\). The 0.80 case outranks both controls, contributing \(1.25(1.25)+1.25(1.25)=3.125\) weighted credits. The 0.60 case loses to 0.70 and outranks 0.40, contributing \(0+2.00(1.25)=2.50\). The weighted numerator is \(3.125+2.50=5.625\), so the weighted AUC is \(5.625/8.125=0.6923\).

This calculation illustrates the weighting arithmetic, not evidence that these weights are adequate in an actual validation study. The censoring model and its estimated probabilities must be examined, and the weighted estimate can be unreliable if those probabilities are near zero.

Comparisons, Diagnostics, and Limitations

A useful subgroup analysis reports, for every subgroup, the landmark and horizon, the number of eligible people, observed cases and controls, the AUC and its uncertainty, and the amount of censoring. Also examine the distributions of predicted risks among cases and controls. A low AUC based on a handful of cases may be highly uncertain; a numerical ranking of subgroup estimates is not persuasive evidence of a difference without uncertainty for the contrast. When subgroups come from the same validation cohort, paired resampling of individuals can preserve the dependence between subgroup estimates and provide an interval for their difference. A difference in two separately reported confidence intervals is not itself a confidence interval or test for the difference.

The main alternatives answer different questions. A pooled AUC describes ranking in the pooled case-control population, including cross-subgroup pairs. A subgroup-specific AUC describes ranking within each subgroup. A standardized within-stratum summary targets a common chosen composition but depends on the strata and weights. Calibration analysis asks whether predicted probabilities agree with observed frequencies, not whether rankings are correct. None of these quantities should be substituted for another without stating the target question.

Several limitations remain even after careful standardization. AUCs can be imprecise when subgroup event counts are small; censoring adjustment can add substantial variability; and chosen strata may leave important differences in composition uncontrolled. AUC also does not quantify whether predictions are useful at a particular decision threshold, nor does it establish that a model's risk estimates are accurate. If competing events are present, the definition of cases and controls must reflect the intended competing-risk estimand; the simple event-free-control definition used here is not automatically appropriate.

Set the target.
Specify the prediction landmark, horizon, subgroup definitions, and what counts as a case and a control.

Estimate within-subgroup ranking.
Compare case-control risks, handling ties and censoring according to the stated assumptions.

Interpret the contrast.
Report uncertainty and examine score ranges and case mix; use common-stratum standardization only when its target is clearly defined.

Common Mistakes

Calling a subgroup AUC a calibration measure. AUC measures ordering of cases relative to controls, not agreement between predicted and observed risks.
Comparing AUCs as if only the model changed. Subgroup case and control distributions may differ. Describe that dependence and, if useful, define a standardized comparison.
Counting a censored person as a control. If censoring occurs before the horizon, event-free status through the horizon is not known. Use an appropriate censoring method or clearly state a justified complete-follow-up restriction.
Interpreting a larger estimate as a proven performance difference. Quantify uncertainty in the subgroup contrast; sparse subgroups can produce unstable rankings.

Key takeaway: A subgroup AUC is a horizon-specific probability of correctly ranking a case above a control drawn from that subgroup. Its value depends on the case-control score distributions, so compare subgroup estimates with attention to uncertainty, risk range, case mix, and censoring.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

At a fixed horizon, one case has risk 0.70 and two controls have risks 0.70 and 0.40. What is the empirical AUC?
Two subgroups have different ordinary AUCs, but identical AUCs within each of two prespecified case-mix strata. Give one possible reason for this pattern.
Why is a person censored before the prediction horizon not automatically a control?
What does a common-stratum standardized AUC summary clarify, and what important comparison can it omit?

Solutions. 1. The case ties the first control, earning 0.5, and outranks the second, earning 1. The AUC is \((0.5+1)/2=0.750\). 2. The subgroups may have different proportions of case-control pairs in the two strata; their marginal comparisons also include cross-stratum pairs. 3. Their event status through the horizon is unknown, so labeling them event-free can misclassify cases and bias discrimination. 4. It averages within-stratum discrimination using the same specified weights, separating that part of the comparison from differences in stratum composition. It can omit cross-stratum case-control comparisons included in an ordinary marginal AUC.