Why Calibration Matters
A polygenic risk score can rank people well by their tendency to experience an outcome and still assign risks that are systematically too high or too low. Calibration addresses that second question: among people assigned a given risk, do outcomes occur at about that rate? This matters when predicted risks are used to communicate absolute risk, compare risk strata, or support decisions. A discrimination statistic, discussed in the earlier material on discrimination metrics, does not answer this question.
Calibration is evaluated in a target validation sample by comparing predictions made before observing its outcomes with the outcomes actually recorded. The predictions and the observed outcome must refer to the same population, outcome definition, follow-up period, and risk scale. For a binary outcome, write \(Y_i=1\) when individual \(i\) experiences the outcome and \(Y_i=0\) otherwise, and let \(\hat r_i\) be that individual's predicted probability, with \(0<\hat r_i<1\).
Risk-Stratum Plots
A risk-stratum plot groups individuals by predicted risk, often using quantiles of \(\hat r_i\). For each group \(g\), calculate the mean predicted risk and the observed event proportion:
Plot \(\hat p_g\) against \(\bar r_g\), with the equality line \(\hat p_g=\bar r_g\) as a reference. A group above that line had more events than predicted, so its average risk was underpredicted. A group below the line had fewer events than predicted, so its average risk was overpredicted. The plot is a grouped summary: differences within a group are concealed, and the choice and number of groups can change its appearance.
Worked Example: Comparing Predicted and Observed Risks Across Strata
Suppose a hypothetical validation cohort of 1,000 people is divided into five equal-sized groups by predicted risk. The table reports each group's mean prediction and event count.
| Stratum | People \(n_g\) | Mean predicted risk \(\bar r_g\) | Events | Observed risk \(\hat p_g\) |
|---|---|---|---|---|
| 1 | 200 | 0.050 | 8 | 0.040 |
| 2 | 200 | 0.080 | 18 | 0.090 |
| 3 | 200 | 0.100 | 26 | 0.130 |
| 4 | 200 | 0.120 | 38 | 0.190 |
| 5 | 200 | 0.150 | 54 | 0.270 |
For stratum 1, the observed risk is \(8/200=0.040\), compared with a mean prediction of \(0.050\); predictions are too high by 0.010, or 1 percentage point. For stratum 2, \(18/200=0.090\), which is 0.010 above the prediction of \(0.080\). The corresponding differences in strata 3, 4, and 5 are \(0.130-0.100=0.030\), \(0.190-0.120=0.070\), and \(0.270-0.150=0.120\). Thus the plot shows overprediction in stratum 1 and underprediction in strata 2–5, with underprediction increasingly pronounced at higher predicted risks.
Across all 1,000 people, the observed event proportion is \((8+18+26+38+54)/1000=144/1000=0.144\). The average predicted risk, using the equal-sized groups, is \((0.050+0.080+0.100+0.120+0.150)/5=0.100\). This substantial overall difference points to underprediction on average, while the stratum results show that the error is not a uniform shift across the risk range.
Calibration-in-the-Large
Calibration-in-the-large (CITL) asks whether predicted risks are systematically too low or too high overall. A standard definition fits a logistic model with the logit prediction as a fixed offset and estimates only an intercept:
The ideal value is \(a_{\mathrm{CITL}}=0\). A positive intercept means observed outcomes tend to exceed predictions after accounting for the prediction's individual risk pattern; a negative intercept indicates the reverse. This is not generally calculated by taking the logit of the overall observed event proportion minus the logit of the average predicted risk. That shortcut works in the special case where every individual has the same predicted risk, but not in general.
The intercept is estimated by maximum likelihood. Conditional on fixed predictions and independent Bernoulli outcomes, the likelihood score equation for the intercept is:
This equation says that the sum of fitted event probabilities in the offset model must equal the observed event count. When predictions vary, that equality usually requires numerical estimation; it does not reduce to a formula involving only the two overall averages.
Worked Example: Estimating Calibration-in-the-Large for Equal Predictions
In a hypothetical validation sample of 1,000 people, suppose every person was assigned predicted risk \(0.100\), and 120 people experienced the outcome. Here the observed event proportion is \(120/1000=0.120\). Because all predictions are equal, the offset model's fitted risk is \(\operatorname{expit}\{a_{\mathrm{CITL}}+\operatorname{logit}(0.100)\}\), and its maximum-likelihood value must equal \(0.120\).
Solving for the intercept gives:
The positive value indicates overall underprediction: observed risk is 12.0%, versus the average prediction of 10.0%, a difference of 2.0 percentage points. This intercept measures a shift on the log-odds scale, not a 20.48-percentage-point risk difference. Its risk-scale implication depends on the starting predicted risk.
Calibration Slope and the Logistic Calibration Model
A complementary diagnostic estimates an intercept and slope together by regressing the binary outcome on the logit of the predicted risk:
Perfect calibration corresponds to \(a=0\) and \(b=1\). The slope describes how strongly observed risk changes across the range of predicted log-odds. A slope below 1 commonly signals predictions that are too extreme: lower predictions tend to be too low and higher predictions too high, relative to the fitted calibration curve. A slope above 1 commonly signals predictions that are not extreme enough. These patterns must be interpreted together with the intercept, which shifts the fitted curve, and with the observed calibration plot. A slope is not a measure of discrimination.
The joint intercept \(a\) from this model is not the same quantity as CITL: CITL fixes the slope at 1, while the calibration model estimates both parameters. When predictions vary, allowing the slope to change can alter the fitted intercept as well. Report which model was used rather than calling any fitted logistic intercept a calibration-in-the-large estimate.
Worked Example: Interpreting a Calibration Slope Below One
Suppose a hypothetical validation analysis reports a fitted calibration slope \(\hat b=0.7763\). On the fitted calibration curve, the risk corresponding to a predicted risk of \(0.100\) is \(0.120\). Substituting these values into the logistic calibration model gives the intercept:
At predicted risk \(0.100\), the fitted linear predictor is \(-0.2867+(0.7763)(-2.1972)\approx-1.9924\), which transforms back to a fitted risk of \(0.120\). At predicted risk \(0.300\), \(\operatorname{logit}(0.300)\approx-0.8473\); the fitted linear predictor is \(-0.2867+(0.7763)(-0.8473)\approx-0.9444\), corresponding to fitted risk \(\operatorname{expit}(-0.9444)\approx0.2800\). Thus, in these examples, risk 0.100 is underpredicted and risk 0.300 is overpredicted. The slope below one is consistent with excessive spread in predictions, but the fitted curve and data should be examined rather than relying on the slope alone.
Assumptions, Diagnostics, and Limitations
These diagnostics are meaningful only when predictions are evaluated in data that were not used to fit the prediction model or select its hyperparameters. The validation outcomes must be observed consistently, and predicted and observed risks must refer to the same outcome, time horizon, and target population. The logistic calibration model treats each outcome as Bernoulli conditional on the prediction; its usual likelihood-based uncertainty calculations also require independent observations. Dependence, such as repeated observations on a person or clustering by site, calls for uncertainty methods that account for that structure.
A useful assessment combines the grouped plot with the numerical summaries. If risk strata depart from the equality line in a curved pattern, a single intercept shift cannot describe the problem. If there are few events, group proportions can be unstable; an apparently irregular plot may reflect sampling variation rather than a reproducible pattern. Uncertainty intervals can help show that imprecision. For a stratum with \(x_g\) events among \(n_g\) people, a simple approximate standard error is \(\sqrt{\hat p_g(1-\hat p_g)/n_g}\), though this approximation can perform poorly when event counts are small or proportions are near 0 or 1. Prefer an interval method suitable for the count and report group sizes and event counts.
Grouping is easy to interpret but discards within-group information, and results can depend on the chosen cut points. Logistic calibration summaries use individual predictions but impose a particular linear relationship between log-odds and predicted log-odds; they can miss other forms of miscalibration. A smooth calibration curve can reveal departures from that assumption, but its shape depends on the smoothing method and can be uncertain where data are sparse. None of these diagnostics alone establishes that a score will remain calibrated in a different population or follow-up period.
Confirm that predictions are fixed before validation outcomes are examined and that the outcome, time horizon, and population match the prediction target.
For each group, report its size, mean predicted risk, observed event count, and observed proportion; compare observed with predicted risk using the equality line.
Report CITL from the offset model and the intercept and slope from the joint logistic calibration model, with uncertainty where available.
Look for an overall shift, a slope departure, curvature, and sparse strata; avoid treating noisy group differences as definitive.
A common mistake is to see a high discrimination statistic and conclude that predicted risks are accurate. Ranking and calibration are different properties: a score can rank cases above non-cases while assigning risks that are too high or too low. The correction is to assess observed versus predicted risk directly, using calibration plots and appropriately defined calibration summaries. A second mistake is to equate the calibration slope with a simple overall risk difference. The slope describes change across predicted log-odds; the risk difference is measured in percentage points and varies by risk level.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A stratum has mean predicted risk 0.08 and observed risk 0.11. Is the average prediction too high or too low, and by how many percentage points?
- In an offset-model CITL analysis, the estimated intercept is negative. What does its sign indicate?
- Why is the difference between the logit of the overall observed proportion and the logit of average predicted risk not generally the CITL estimate?
- What does a calibration slope below one commonly suggest, and why should the intercept and plot also be examined?
Solutions. (1) The observed risk exceeds predicted risk, so predictions are too low by \(0.11-0.08=0.03\), or 3 percentage points. (2) A negative CITL indicates that outcomes are lower than predicted overall under the offset model. (3) With varying predictions, the offset-model likelihood equation sums fitted probabilities across individuals; the solution is not generally determined by the two overall averages. The shortcut applies when predictions are all equal. (4) A slope below one commonly indicates predictions that are too extreme, with low risks tending too low and high risks too high relative to the fitted curve. The intercept shifts that curve, and plots can reveal curvature or other patterns the slope alone misses.