Evaluating Score Performance Across Subgroups

Why Evaluate Performance Within Subgroups?

An overall performance estimate can conceal meaningful variation across the people for whom a polygenic risk score (PRS) may be used. A score may rank cases above controls well in one clinical subgroup and less well in another; predicted risks may also systematically differ from observed risks in a subgroup. These are distinct questions. Differences in the distribution of scores are not, by themselves, evidence that prediction performs differently.

The aim is to estimate performance within clearly defined demographic or clinical subgroups and to quantify uncertainty in comparisons. Such comparisons are descriptive unless a design and analysis support stronger conclusions. They do not establish that subgroup membership causes a performance difference, and they do not identify its mechanism.

Set Up the Comparison Carefully

Suppose a fixed PRS produces a predicted risk or score for each person, and the outcome \(Y\) is binary. Before comparing groups, specify the subgroup definitions, outcome definition, time horizon if relevant, score scale, and performance measures. Apply the same outcome and evaluation rules in every subgroup. A validation set not used to choose or tune the score is needed for an honest assessment; otherwise, performance can be optimistic.

For discrimination, the area under the receiver operating characteristic curve (AUC) is the probability that a randomly selected case has a higher score than a randomly selected control, with ties counted as one-half. It concerns ranking, not the accuracy of the predicted risk values. For calibration, compare predicted risks with observed outcome frequencies: among people assigned a given risk, the event frequency should correspond to that risk. When the model's linear predictor is the logit of its predicted risk, a calibration model can be written as

where \(\eta\) is the model's predicted log-odds and \(g\) indexes subgroup. Ideal calibration has \(a_g=0\) and \(b_g=1\). A nonzero intercept indicates systematic risk under- or overprediction on the log-odds scale; a slope below 1 often indicates predictions that are too extreme, while a slope above 1 indicates predictions that are insufficiently dispersed. These interpretations assume the stated logit-linear calibration model is adequate.

Score distributions answer a separate question: how are the score values themselves distributed in each group? A mean or standard deviation can describe location and spread, but neither directly measures ranking or calibration. If scores are standardized using the full validation sample, report that scale; standardizing each subgroup separately would force subgroup means and standard deviations to match by construction and hide distributional differences.

Quantifying Uncertainty in Subgroup Contrasts

Finite validation samples make subgroup estimates variable. For two independent subgroups and a fixed score, an approximate standard error for the difference between estimates is the square root of the sum of their variances. More generally, if estimates are dependent, the covariance must be included:

The independence formula is appropriate when estimates are based on disjoint validation individuals and the score is treated as fixed. If the same individuals contribute to both estimates, or uncertainty in a shared training and tuning process is part of the target, independence should not be assumed. A paired or full-pipeline resampling procedure may be needed.

A short justification follows from the variance identity. For random estimates \(U\) and \(V\),

Taking square roots gives the standard error of the contrast. Under approximate normality, an approximate 95% confidence interval is the estimate of the contrast plus or minus \(1.96\) standard errors. It describes uncertainty under the sampling model; it is not a probability statement about a fixed parameter.

For AUC, standard errors can be obtained with methods such as DeLong's estimator or bootstrap resampling. For calibration and score summaries, bootstrap resampling of individuals within each subgroup can provide uncertainty intervals. The resampling unit and scheme must reflect the design: for example, when estimating calibration in a subgroup, resample individuals rather than separately fixing the number of cases and controls, since the observed event fraction is part of calibration. For AUC uncertainty conditional on case and control counts, resampling cases and controls separately is a common option. If subgroups overlap or observations are clustered, resampling must preserve that structure.

Calibration Contrasts and Their Uncertainty

Calibration should be examined separately in each subgroup, since a good overall calibration summary can mask systematic errors within groups. Calibration plots are useful for revealing departures from the specified model, while estimated intercepts and slopes provide concise summaries. Estimates can be unstable when a subgroup has few events, few non-events, or a narrow range of predicted risks. In such settings, a slope may be especially imprecise.

Score Distributions Are Not Performance Metrics

Subgroup score distributions can differ because of how scores are scaled, differences in measured characteristics, or other features of the sampled groups. Such a difference may matter for interpretation and downstream decisions, but it does not establish that the PRS discriminates or calibrates differently. Conversely, similar score means do not guarantee similar discrimination: AUC depends on the relative ordering of cases and controls within a subgroup.

Diagnostics, Alternatives, and Limitations

A subgroup analysis is more credible when estimates are accompanied by uncertainty intervals and the underlying data support the chosen metric. Inspect subgroup sample sizes and event counts; a large total sample can still yield very small numbers of cases or controls in a particular group. For AUC, check that both cases and controls are present and that the score has useful variation. For calibration, inspect predicted-risk coverage and event counts rather than relying only on one intercept or slope. For score distributions, check for outliers, skew, and differences in scale or measurement.

Separate subgroup estimates are easy to interpret, but can be noisy. An alternative is a pooled model with subgroup indicators and interaction terms, such as a subgroup-by-score interaction for a calibration slope. The pooled approach can directly test a specified contrast and may use data more efficiently, but its interpretation depends on model form and adequate overlap in predictor values. It does not remove the need to report subgroup-specific quantities. For discrimination, direct comparison of subgroup AUCs or an appropriate paired/independent AUC method is generally more directly interpretable than inferring a difference from separate tests against chance.

When many subgroups or metrics are examined, some apparently large contrasts can occur by chance. Prespecify a limited set of comparisons where possible, report all examined contrasts, and avoid selecting only the most striking results. Confidence intervals quantify sampling uncertainty under a model; they do not account automatically for selective reporting, subgroup misclassification, or poor study design. Results from small or unrepresentative validation samples may not transport to other settings.

• Mistake: comparing significance labels. One subgroup's AUC may have a confidence interval excluding 0.5 while another's does not. This does not establish an AUC difference; estimate the contrast and its uncertainty directly.
• Mistake: treating different score means as different predictive performance. A mean contrast describes score location. Compare discrimination and calibration separately to assess prediction.
• Mistake: treating a wide interval that contains zero as proof of equality. It means the estimate is uncertain at the stated level; equivalence requires a prespecified margin and a suitable analysis.
• Mistake: interpreting calibration without event prevalence. Calibration depends on the relationship between predicted risks and observed outcomes in the evaluated population. Report the event count and prevalence alongside calibration summaries.