Calibration-in-the-Large and Calibration Slope

Why Separate Calibration Level From Calibration Slope?

An absolute-risk prediction is useful only if its numerical values correspond adequately to observed event frequencies in the population and time period where it will be used. A model can rank people in a sensible order yet systematically give risks that are too high or too low. It can also give predictions whose spread is too wide or too narrow, even if the average predicted risk is close to the average observed risk. Calibration-in-the-large and the calibration slope summarize these two distinct features.

Consider predictions for an event by a specified time horizon. Let \(Y_i=1\) if person \(i\) experiences the event by that horizon and \(Y_i=0\) otherwise, and let \(p_i\) be that person’s predicted absolute risk. The logit of the predicted risk is \(z_i=\operatorname{logit}(p_i)=\log\{p_i/(1-p_i)\}\). The diagnostics below compare these predictions with outcomes in a validation sample; they do not establish that a prediction causes an outcome.

The offset in calibration-in-the-large is important: this diagnostic estimates a shift in overall level without also changing the spread of predictions. By contrast, the slope comes from a model in which both the intercept and slope are free. Thus, its estimate is not simply the coefficient from the calibration-in-the-large model, and the intercept estimated jointly with the slope need not equal calibration-in-the-large.

Reading the Intercept and Slope

In the calibration-in-the-large model, a positive \(a\) shifts predicted log-odds upward; the predictions, on average, are too low for the validation outcomes. A negative \(a\) shifts them downward; predictions are, on average, too high. These are overall tendencies, not guarantees that every risk group is miscalibrated in the same direction. A value near zero indicates little overall level shift, but does not demonstrate calibration across the entire range of predicted risks.

The slope describes how strongly observed log-odds vary with predicted log-odds after allowing a constant shift. If \(b<1\), observed differences are less pronounced than the predictions imply: high predictions tend to be too high and low predictions tend to be too low, after accounting for the intercept. This is often described as predictions being too extreme. If \(b>1\), observed differences are more pronounced: predictions tend to be too moderate, with high predictions too low and low predictions too high after accounting for the intercept. A slope of one means the fitted relationship has the same log-odds spread as the predictions, not that every subgroup is perfectly calibrated.

Estimating Calibration-in-the-Large

For independent individuals with fully observed binary outcomes by a common horizon, estimate \(a\) by fitting a logistic regression with \(z_i\) as an offset and \(a\) as the intercept. In a simple case where every individual has the same predicted risk \(p\), the estimate has a direct form. If the observed event proportion is \(\hat p\), the fitted risk is \(\operatorname{expit}(a+\operatorname{logit}(p))\), where \(\operatorname{expit}(u)=1/(1+\exp(-u))\). Setting this fitted risk equal to \(\hat p\) gives \(a=\operatorname{logit}(\hat p)-\operatorname{logit}(p)\).

Estimating and Interpreting the Slope

To estimate the calibration slope, fit the logistic model \(\operatorname{logit}\{P(Y_i=1)\}=a+bz_i\) with both coefficients free. The intercept permits an overall shift while the slope estimates the relationship between predicted and observed log-odds. In typical validation data, estimation uses the individual outcomes and predictions, rather than treating a few groups as the entire data set. The following grouped example makes the arithmetic transparent because the fitted model reproduces each group’s observed proportion exactly.

A Calibration Model as a Risk Adjustment

The fitted recalibration model can also be used to illustrate how a predicted risk changes under the estimated intercept and slope. For a given original risk \(p\), calculate \(z=\operatorname{logit}(p)\), then map it to \(\operatorname{expit}(\hat a+\hat b z)\). This calculation is an adjustment implied by the fitted model, not a guarantee that the adjusted risk will be calibrated in a new population. Its usefulness depends on the validation data being relevant to the population and outcome horizon of interest.

Justification: Why the Offset Intercept Tracks Overall Level

For the calibration-in-the-large model, write \(q_i(a)=\operatorname{expit}(a+z_i)\). Under independent Bernoulli outcomes, the log-likelihood for \(a\), apart from constants, is \(\ell(a)=\sum_i [Y_i\log q_i(a)+(1-Y_i)\log\{1-q_i(a)\}]\). Since \(dq_i/da=q_i(1-q_i)\), differentiating gives the score

At a finite maximum-likelihood estimate, the score is zero, so \(\sum_i q_i(\hat a)=\sum_i Y_i\). The total fitted number of events therefore equals the observed number of events. When every \(z_i\) is the same, all fitted probabilities are equal, and this equality says that the fitted risk equals the observed event proportion, as used in the first example. With varying predictions, the equality concerns the overall totals; it does not require each risk stratum to match its observed event rate.

The slope model has a different purpose: it asks how observed log-odds change as predicted log-odds change, while estimating a free intercept. Both parameters are obtained by maximizing the Bernoulli likelihood for that model. This distinction is why an estimated joint intercept and calibration slope should be reported as a pair when describing the recalibration relationship, while calibration-in-the-large is specifically assessed with slope fixed at one.

Assumptions, Limits, and a Common Pitfall

These logistic formulations directly apply when the target is a binary event by a fixed horizon and outcome status is adequately observed for the validation individuals. Predictions and observed outcomes must refer to the same event definition, horizon, and target population. If follow-up is censored before the horizon, treating an individual without a recorded event as a known non-event can bias calibration estimates; appropriate survival or competing-risk calibration methods are needed instead. Dependence between observations, such as clustering, also affects standard errors and should be handled in the analysis.

Neither statistic summarizes every form of miscalibration. A model can have \(a\) near zero and \(b\) near one yet show curved or subgroup-specific discrepancies that cancel in the aggregate. Calibration plots and other checks can reveal patterns that a single intercept and slope cannot. Estimates are also subject to sampling variability, especially when events are few or predictions occupy a narrow range.

• Common mistake: “Calibration-in-the-large is the intercept from a model that estimates both intercept and slope.” Correction: Calibration-in-the-large fixes the coefficient of predicted logit at one; the jointly estimated intercept belongs to the separate slope model.
• Common mistake: “A slope below one means all risks are too high.” Correction: It usually indicates predictions are too extreme: high risks tend to be too high and low risks too low after accounting for the intercept.
• Common mistake: “An intercept near zero and slope near one prove perfect calibration.” Correction: They indicate little global shift and an approximately correct log-odds spread, but can miss nonlinear or subgroup-specific errors.