Calibration of Polygenic Absolute-Risk Estimates

Why Calibration Matters

A polygenic model can rank people correctly yet give risks that are systematically too high or too low. For a person making a clinical decision, a ranking alone is not enough: a predicted 10% risk should correspond to an event frequency near 10% among comparable people. Calibration assesses that agreement between predicted absolute risks and observed outcomes.

Calibration must be evaluated for a defined outcome, population, and time horizon. In this tutorial, let \(\hat r_i\) be person \(i\)'s predicted probability of the target event by horizon \(t\), and let \(Y_i=1\) if that event occurs by \(t\), and \(Y_i=0\) otherwise. With competing risks, the target event is counted only if it occurs before or at \(t\); a competing event before \(t\) means the target event did not occur by the horizon. The predicted risk and the observed outcome must use this same definition.

Definition: A risk model is calibrated at horizon \(t\) if, among individuals assigned predicted risk \(r\), the conditional probability of the target event by \(t\) is \(r\): \(P(Y=1\mid\hat r=r)=r\). In practice, validation data provide estimates of this relationship rather than exact probabilities for every possible risk value.

A calibration plot places observed event frequency on the vertical axis and predicted risk on the horizontal axis. Perfect calibration corresponds to the 45-degree line: observed risk equals predicted risk. A point above that line indicates more events than predicted in that part of the risk range; a point below it indicates fewer.

Grouped Calibration and Overall Summaries

A common practical plot groups people into risk intervals, such as intervals chosen in advance or groups formed from quantiles of predicted risk. For each group \(g\), calculate the mean predicted risk, the observed event frequency, and the group size. Plot the first two quantities against one another. The group mean predicted risk is not generally the same as every individual's prediction within that group.

\bar r_g=\frac{1}{n_g}\sum_{i\in g}\hat r_i, \qquad \bar Y_g=\frac{1}{n_g}\sum_{i\in g}Y_i

Here, \(n_g\) is the number of people in group \(g\), \(\bar r_g\) is the group's mean predicted risk, and \(\bar Y_g\) is its observed event frequency. A group with 8 events among 100 people has observed frequency \(8/100=0.08\), or 8%. The plot is an empirical summary: grouping smooths a noisy individual-level comparison, but it can conceal variation within a group. Group boundaries and group sizes should therefore be reported.

An overall summary compares the total number of observed events, \(O\), with the total number expected from the individual predictions, \(E\). The expected count is the sum of predicted probabilities, not the number of people above a chosen risk threshold.

O=\sum_{i=1}^{n}Y_i, \qquad E=\sum_{i=1}^{n}\hat r_i, \qquad O/E=\frac{O}{E}

When \(O/E=1\), observed and expected event counts agree in aggregate. A ratio above 1 means more events occurred than the model predicted overall; a ratio below 1 means fewer occurred. The difference \(O-E\) gives the discrepancy in event counts, while \(O/n-E/n\) expresses the difference as an absolute risk difference. These summaries describe the validation sample as a whole and do not show where in the risk range any disagreement occurs.

Worked Example: Reading a Grouped Calibration Plot

Suppose a hypothetical validation sample of 300 people has complete outcome ascertainment through a specified horizon. Divide the people into three groups of 100. Their mean predicted risks are 4%, 10%, and 20%; the observed target-event counts are 3, 11, and 18. For this calculation, the endpoint is the same fixed-horizon event for all participants, including the stated competing-risk definition.

Group	People	Mean predicted risk	Observed events	Observed frequency
1	100	0.04	3	0.03
2	100	0.10	11	0.11
3	100	0.20	18	0.18

The observed frequencies are computed as \(3/100=0.03\), \(11/100=0.11\), and \(18/100=0.18\). Thus the plotted coordinates are \((0.04,0.03)\), \((0.10,0.11)\), and \((0.20,0.18)\). The first group lies below the equality line, the second slightly above it, and the third below it. These deviations alone do not establish systematic miscalibration; sampling variation can move group estimates away from the line.

The group expected counts are \(100(0.04)=4\), \(100(0.10)=10\), and \(100(0.20)=20\). Therefore \(E=4+10+20=34\), while \(O=3+11+18=32\). The overall ratio is \(O/E=32/34\approx0.941\), to three significant figures. The mean predicted risk is \(34/300\approx0.113\), or 11.3%; the observed frequency is \(32/300\approx0.107\), or 10.7%. Overall, the predictions exceed the observed event frequency by about 0.00667, or 0.667 percentage points. The aggregate agreement does not erase the differences between individual groups.

What the Summaries Can and Cannot Establish

A short justification relates the expected observed count to the predictions under marginal calibration. If \(E(Y_i\mid\hat r_i)=\hat r_i\) for each individual, then taking expectations and using linearity gives the following unconditional identity:

E(O) =\sum_{i=1}^{n}E(Y_i) =\sum_{i=1}^{n}E\left(E(Y_i\mid\hat r_i)\right) =\sum_{i=1}^{n}E(\hat r_i) =E\left(\sum_{i=1}^{n}\hat r_i\right)

This result motivates comparing observed and expected counts. It does not say that every validation sample will have \(O=E\): outcomes vary randomly, and calibration is a statement about probabilities, not a promise about the realized count. Nor does a ratio near 1 prove calibration across the risk range. Overprediction in one part of the range can offset underprediction in another.

Worked Example: A Good Overall Ratio Can Hide Group Differences

Consider a second hypothetical validation sample with 200 people. In a lower-risk group, 100 people have mean predicted risk 4%, giving \(E_1=100(0.04)=4\), but 8 target events are observed. In a higher-risk group, 100 people have mean predicted risk 16%, giving \(E_2=100(0.16)=16\), but 12 events are observed.

The observed frequencies are \(8/100=0.08\) in the lower-risk group and \(12/100=0.12\) in the higher-risk group. Compared with predictions of 0.04 and 0.16, the lower-risk group has more events than predicted and the higher-risk group has fewer. Yet the total observed count is \(O=8+12=20\), and the expected count is \(E=4+16=20\). Thus \(O/E=20/20=1.00\).

The overall ratio indicates agreement in event counts, but the grouped comparisons reveal opposing discrepancies across the risk range. A calibration plot is needed alongside the overall summary. This example also shows why a single summary should not be treated as a complete assessment of calibration.

Uncertainty, Grouping, and Incomplete Follow-Up

Observed frequencies and calibration-plot points are estimates based on a finite sample. Their uncertainty should be shown or otherwise assessed, especially when groups are small. A binomial confidence interval for a group frequency presumes a binomial sampling model: for example, independent Bernoulli outcomes with a common event probability within the group. If individual predicted probabilities differ, independent outcomes do not by themselves make the total event count binomial. Under the model, the count instead has a Poisson-binomial distribution, and uncertainty methods must reflect the differing probabilities or use an appropriate sampling or model-based approach.

Worked Example: Why Varying Risks Change the Count Model

Suppose a hypothetical group contains 30 independent individuals with predicted risks of 0.05 for 10 people, 0.15 for 10, and 0.25 for 10. If the risks are correct, the expected number of events is

$$ E=10(0.05)+10(0.15)+10(0.25)=0.5+1.5+2.5=4.5 $$

Under independent Bernoulli outcomes with those differing probabilities, the event-count variance is the sum of the individual variances:

\operatorname{Var}(O)=10(0.05)(0.95)+10(0.15)(0.85)+10(0.25)(0.75) =0.475+1.275+1.875=3.625

The standard deviation under this prediction model is \(\sqrt{3.625}\approx1.90\) events. In contrast, treating all 30 people as if they had a common event probability equal to the mean, 0.15, gives binomial variance \(30(0.15)(0.85)=3.825\), which is not the variance just calculated. The example illustrates why a binomial interval based on a pooled frequency is not automatically justified by independence. This model-based standard deviation describes event-count variation under the stated risks; it is not, by itself, a confidence interval for calibration.

If follow-up is incomplete before horizon \(t\), some individuals' event status by \(t\) is unknown. Simply dividing observed events by everyone enrolled, or excluding people censored before \(t\), can produce biased frequencies. Calibration assessment then needs an appropriate time-to-event method that accounts for censoring and, when relevant, estimates the target event's cumulative incidence in the presence of competing events. The risk predictions must target that same cumulative-incidence quantity. A competing event is not ordinary loss to follow-up: it prevents the target event from occurring first, and should not be treated as if the person remained at risk of the target event afterward.

Uncertainty procedures should also match the validation design. Resampling individuals may be appropriate for independent observations, while related participants or other clustered sampling may require resampling clusters or another method that accounts for dependence. The choice of risk groups affects plot appearance: many narrow groups give detail but can be noisy, while a few wide groups are more stable but may mask local departures. Reporting group sizes, boundaries, and uncertainty helps readers judge how much the plot supports.

Common Mistakes

Using different targets. Comparing predicted risk by a specified horizon with outcomes defined over another time period does not assess calibration for the stated prediction. Match the event definition, horizon, population, and competing-risk handling.
Calling a ratio of 1 proof of calibration. The ratio summarizes total events only. Inspect grouped comparisons because errors in different risk ranges can cancel.
Assuming independent outcomes imply a binomial count. That conclusion also requires a common event probability or an appropriate binomial sampling model. With differing probabilities, use a method that accounts for heterogeneity.
Treating censored individuals as event-free through the horizon. Their status by that horizon is unknown; use an appropriate method for incomplete follow-up rather than a naïve frequency.
Reading a small departure from the equality line as definitive. Calibration estimates are subject to sampling variability. Consider group size and uncertainty before drawing conclusions.

Key takeaway: Assess calibration across the risk range by comparing group mean predictions with observed event frequencies, and summarize aggregate agreement with observed and expected counts. Interpret both in the context of uncertainty, follow-up, competing risks, and the validation population.

Check Your Understanding

Try these questions without looking back.

For a group of 80 people with mean predicted risk 0.075 and 8 observed events, calculate the group's observed frequency and locate its calibration point relative to the equality line.
Explain why an overall observed-to-expected ratio of 1 can coexist with poor calibration across the risk range.
When can an event count be treated as binomial, and why is independence alone insufficient when individual risks differ?
How should a competing event before the prediction horizon be handled when defining the target outcome?
Why is a person censored before the horizon not automatically counted as event-free through that horizon?