Why Calibration and Ranking Need Separate Checks
A genetic score can perform differently across ancestry groups in more than one way. It may rank people less well by their tendency to experience an outcome, or it may rank them reasonably well while assigning probabilities that are systematically too high or too low. These are distinct failures. A single summary statistic cannot generally diagnose both.
This tutorial considers a binary outcome \(Y\), where \(Y=1\) denotes an event and \(Y=0\) denotes no event during the specified follow-up period. For each person, a prediction model supplies a probability \(\hat p_i\) and, when needed, its logit \(\eta_i=\log\{\hat p_i/(1-\hat p_i)\}\). Diagnostics should be calculated separately in each ancestry group being compared, using groups defined consistently with the target population.
These checks require outcome and prediction data from a validation sample representative of the intended target group, with predictions generated without using those same outcomes to fit or tune the model. The outcome definition and follow-up must be consistent across groups. Otherwise, apparent group differences may reflect different outcome ascertainment or study design rather than score portability.
Calibration Plots and Residual Summaries
A calibration plot places predicted probability on the horizontal axis and observed event frequency on the vertical axis. The reference line is \(y=x\): if a group of people has mean predicted probability 0.10, for example, the corresponding observed event rate should be near 0.10. A smooth curve can show departures across the probability range; alternatively, predictions can be divided into bins and the mean prediction and event frequency computed in each bin.
Binning makes a plot easy to inspect but discards detail. Results depend on how bins are chosen, and small bins have noisy event rates. Show the number of people or events contributing to each point, and use uncertainty intervals or other uncertainty summaries when appropriate. A smooth curve is also an estimate, not a guarantee: its appearance depends on the smoothing method and data density. Avoid drawing strong conclusions in ranges with few people.
For an individual, the probability residual is \(r_i=Y_i-\hat p_i\). A positive residual means the observed outcome was higher than the prediction; a negative residual means it was lower. In a subgroup \(G\), the mean residual is \(\bar r_G=n_G^{-1}\sum_{i\in G}(Y_i-\hat p_i)\). It is the observed event proportion minus the mean predicted probability in that subgroup. A near-zero mean residual is evidence of average agreement in that subgroup, but it does not show that predictions are calibrated at every score level.
Worked Example: Comparing Calibration Across Two Ancestry Groups
In a hypothetical validation study, predictions are grouped into two ranges. Each group has 100 people in each range. The table gives the mean predicted probability and observed event count. The groups and outcome definitions are assumed comparable, and predictions were fixed before validation outcomes were examined.
| Group | Prediction range | Mean predicted probability | Events / people | Observed event rate |
|---|---|---|---|---|
| A | Lower | 0.05 | 6 / 100 | 0.06 |
| A | Higher | 0.20 | 18 / 100 | 0.18 |
| B | Lower | 0.05 | 12 / 100 | 0.12 |
| B | Higher | 0.20 | 32 / 100 | 0.32 |
For group A, the lower-range residual mean is \(6/100-0.05=0.01\), and the higher-range residual mean is \(18/100-0.20=-0.02\). The observed rates, 0.06 and 0.18, are fairly close to their respective predictions. For group B, the corresponding calculations are \(12/100-0.05=0.07\) and \(32/100-0.20=0.12\). Both observed rates exceed their predictions, with a larger gap in the higher range.
The plot would place group A’s points near the identity line and group B’s points above it. This is evidence of underprediction in group B over these ranges; it does not by itself establish why the gap arose or whether it persists elsewhere on the prediction scale. The comparison is a diagnostic, not a causal explanation.
Calibration Slopes
A calibration slope summarizes whether predictions are too spread out or too compressed on the log-odds scale. Within one ancestry group, let \(\eta_i\) be the logit of the model’s predicted probability. Fit the recalibration logistic model \(\operatorname{logit}\{P(Y_i=1\mid\eta_i)\}=a+b\eta_i\), with an intercept \(a\) and slope \(b\). This is a diagnostic model fitted in validation data; it does not change the original predictions.
The slope is estimated by logistic regression of the binary outcome on \(\eta_i\), allowing an intercept. This interpretation assumes the validation observations are appropriately sampled and the logistic recalibration relationship is a useful summary across the prediction range. A single slope can hide curvature or different behavior in parts of the range. It is therefore read alongside a calibration plot and subgroup residual summaries, not as a complete calibration assessment.
Worked Example: Interpreting a Calibration Slope
Suppose a hypothetical validation analysis in one ancestry group fits the recalibration model and reports \(\hat a=0.10\) and \(\hat b=0.62\). The model’s original logit predictions are \(\eta_i\). The fitted recalibration equation is
For a person with \(\eta_i=1\), the recalibrated log odds are \(0.10+0.62(1)=0.72\). For a person with \(\eta_i=-1\), they are \(0.10+0.62(-1)=-0.52\). The fitted difference between these two log odds is \(0.72-(-0.52)=1.24\), whereas the original predictions differ by \(1-(-1)=2\) log-odds units. The fitted slope, 0.62, is below the ideal value of 1, indicating a narrower spread in observed log odds than in the original predictions.
The intercept is not zero, so the departures are not described by slope alone. For example, at \(\eta=0\), the fitted recalibration log odds are 0.10 rather than 0; this is a shift in the fitted level. These estimates summarize the validation data and have sampling uncertainty. They do not prove that every prediction is wrong by the same amount.
Discrimination and Ranking
A common measure of discrimination for a binary outcome is the area under the receiver operating characteristic curve, or AUC. It has a pairwise interpretation: choose one person with the event and one without it at random from the group. The AUC is the probability that the event case has a higher score than the noncase, counting a tie as one-half. It ranges from 0 to 1; an AUC of 0.5 corresponds to chance-level pairwise ordering under this definition. AUC does not measure whether predicted probabilities are numerically accurate.
For a finite validation sample with \(n_1\) cases and \(n_0\) noncases, calculate the AUC by comparing all \(n_1n_0\) case–noncase pairs. If \(W\) pairs have the case’s score higher and \(T\) pairs are tied, then \(\mathrm{AUC}=(W+0.5T)/(n_1n_0)\). Scores must be oriented so that larger values indicate greater predicted risk.
Worked Example: Comparing Ancestry-Specific AUC Values
In a hypothetical validation sample, group A has three cases with scores 3, 4, and 5 and three noncases with scores 0, 1, and 2. Every case score exceeds every noncase score. Thus \(W=9\), \(T=0\), and \(n_1n_0=3\times3=9\):
In group B, the case scores are 1, 2, and 3 and the noncase scores are 0, 1, and 2. The case with score 1 has one win, one tie, and one loss; the case with score 2 has two wins and one tie; the case with score 3 has three wins. Therefore \(W=1+2+3=6\), \(T=1+1=2\), and there are still nine pairs:
The ranking is weaker in group B in this hypothetical sample. These small samples are for arithmetic illustration, not reliable estimates of population performance; uncertainty and sample size matter. AUC comparisons should be accompanied by uncertainty summaries and interpreted in light of the validation design.
Why the Two Diagnostics Are Distinct
A strictly increasing transformation of a score preserves all pairwise orderings: if \(S_i>S_j\), then \(f(S_i)>f(S_j)\). Consequently, it preserves the AUC, although it can alter the numerical scale of the predictions. This explains why recalibration can improve agreement between predicted and observed risks without improving ranking. Conversely, a score can have reasonable average calibration while ordering cases and noncases poorly.
In practice, compare the diagnostics within each ancestry group. A low or reduced AUC points to a ranking concern. A calibration curve away from the identity line, a slope far from 1, or systematic residual patterns point to a risk-scale concern. Both can occur together. A calibration slope alone does not measure discrimination, and an AUC alone does not establish that assigned probabilities are accurate.
Confirm comparable outcome definitions, follow-up, prediction procedures, and ancestry-group definitions. Keep model fitting and evaluation separate.
Plot observed against predicted probabilities, show group sizes, and examine residual summaries across the prediction range.
Fit the logistic recalibration model separately by group and interpret both slope and intercept, alongside the plot.
Calculate group-specific AUC values and their uncertainty. A difference in AUC addresses ranking, not risk-scale agreement.
State whether the evidence indicates calibration concerns, ranking concerns, both, or neither clearly, and note uncertainty and data limitations.
Worked Example: Separating a Risk-Scale Error From a Ranking Error
Suppose a hypothetical score has an AUC of 0.80 in each of two ancestry groups, but the calibration plot in group B lies consistently above the identity line. In group B, the mean predicted probability in a subgroup is 0.10, while 14 of 100 people experience the event. The subgroup residual mean is
The group’s observed event rate exceeds the mean prediction by 0.04, or 4 percentage points, for this subgroup. Similar departures across the plot would support underprediction on the probability scale. Equal AUC values do not contradict this result: AUC summarizes ordering, while the residual comparison evaluates average probability agreement.
If, instead, group B also had a clearly lower AUC than group A, the evidence would suggest both a risk-scale calibration concern and weaker ranking. Whether either difference is statistically persuasive requires uncertainty estimates; the point estimates alone do not establish that a population-level gap exists.
Common Mistakes and Practical Interpretation
- Treating AUC as calibration. An AUC of 0.80 does not mean predictions are accurate to within a particular probability error. It summarizes pairwise ordering only.
- Calling all prediction errors ranking failures. If observed rates exceed predictions while the ordering remains similar, the evidence may indicate underprediction rather than poor ranking.
- Reading a slope without the intercept or plot. The slope summarizes spread on the log-odds scale, but a shifted level or nonlinear departures can remain.
- Overinterpreting a subgroup average. A mean residual near zero can result from overprediction in one score range and underprediction in another. Inspect residuals across the range.
- Comparing unstable estimates as if exact. Small numbers of events, especially within ancestry-specific bins, yield noisy calibration points and AUC estimates. Report uncertainty and avoid strong conclusions from sparse data.
- Ignoring differences in validation conditions. Changes in outcome ascertainment, follow-up, or target-population definition can produce apparent portability differences that are not attributable solely to the score.
Check Your Understanding
Use these questions to check whether you can distinguish calibration from ranking.
- A group’s calibration plot lies above the identity line across most of the observed prediction range. What does that pattern indicate, and what would it not establish by itself?
- In a validation group, a recalibration model has slope 0.70. What does this suggest about the spread of predictions, and why should the intercept and plot also be checked?
- Explain the pairwise interpretation of an AUC of 0.75. Does it mean that predicted probabilities are accurate to within 25 percentage points?
- Why can a strictly increasing recalibration change predicted probabilities without changing AUC?
- A subgroup has mean predicted probability 0.08 and an observed event rate of 0.11. Calculate its mean probability residual and interpret the sign in context.
- What additional concern arises when calibration points are based on very small numbers of events?