Calibration Slopes for Binary Risk Models

Why Estimate a Calibration Slope?

A binary risk model assigns a predicted probability to each subject. Calibration asks whether those probabilities agree with event frequencies. An overall shift in risk and predictions that are too extreme are different problems: a calibration intercept describes the first, while a calibration slope helps assess the second. Estimating both together makes that distinction explicit.

For subject $i$, let $Y_i\in\{0,1\}$ be the observed outcome and let $\hat p_i$ be the model's predicted risk, with $0<\hat p_i<1$. Write $z_i=\operatorname{logit}(\hat p_i)=\log\{\hat p_i/(1-\hat p_i)\}$. The logistic calibration model regresses outcomes on these predicted-risk logits.

Definition: The logistic calibration model is $$ P(Y_i=1\mid z_i)=\operatorname{logit}^{-1}(\alpha+\beta z_i), $$ where $\operatorname{logit}^{-1}(u)=e^u/(1+e^u)$. The calibration intercept is $\alpha$, and the calibration slope is $\beta$. Perfect calibration in this model corresponds to $\alpha=0$ and $\beta=1$. The slope is interpreted with the intercept estimated at the same time unless a procedure explicitly fixes the intercept.

With independent binary outcomes, the parameters are estimated by logistic regression with $z_i$ as the sole predictor and an intercept. For grouped data, if group $j$ has $y_j$ events among $n_j$ subjects sharing predicted logit $z_j$, the corresponding likelihood is binomial. Both forms estimate the same relationship; grouped counts simply collect subjects with the same predictor value.

Likelihood, Estimation, and a Useful Result

For independent Bernoulli outcomes, the log-likelihood, omitting no parameter-dependent terms, is

\ell(\alpha,\beta)=\sum_{i=1}^n \left[Y_i(\alpha+\beta z_i)-\log\{1+\exp(\alpha+\beta z_i)\}\right].

For grouped binomial outcomes, replace each term by $y_j(\alpha+\beta z_j)-n_j\log\{1+\exp(\alpha+\beta z_j)\}$, plus the binomial coefficient term, which does not depend on $\alpha$ or $\beta$. In either case, let $m_i=\operatorname{logit}^{-1}(\alpha+\beta z_i)$ for individual data, or use $m_j$ for grouped data. Differentiating gives the score equations

\sum_i(Y_i-m_i)=0,\qquad \sum_i z_i(Y_i-m_i)=0.

These equations say that the fitted model matches the observed total number of events and the event total weighted by predicted-risk logit. In grouped data, each term is weighted by its group size: the equations become $\sum_j(y_j-n_jm_j)=0$ and $\sum_j z_j(y_j-n_jm_j)=0$.

Result: If the fitted values are finite and the calibration design has at least two distinct $z_i$ values with positive weight, the log-likelihood is strictly concave in $(\alpha,\beta)$. Consequently, any finite solution to the score equations is the unique maximum-likelihood estimate.

To verify this result, write $x_i=(1,z_i)^\mathsf{T}$ and $\gamma=(\alpha,\beta)^\mathsf{T}$. The second derivative matrix is

\nabla^2\ell(\gamma)=-\sum_i m_i(1-m_i)x_ix_i^\mathsf{T}.

For any nonzero vector $v=(v_0,v_1)^\mathsf{T}$, the associated quadratic form is

v^\mathsf{T}\nabla^2\ell(\gamma)v =-\sum_i m_i(1-m_i)(v_0+v_1z_i)^2.

At finite $\gamma$, every $m_i(1-m_i)>0$. The sum can equal zero only if $v_0+v_1z_i=0$ for every positively weighted observation. Two distinct $z_i$ values then force $v_0=v_1=0$, contradicting the choice of $v$. Thus the Hessian is negative definite and the log-likelihood is strictly concave. Strict concavity makes any finite stationary point the unique global maximum. It does not, by itself, guarantee that a finite maximum exists; complete or quasi-complete separation can prevent a finite logistic estimate. $\square$

Worked Example: A Slope Below One

Suppose three groups each contain four subjects. Their predicted logits are $z=-2,0,2$, and their event counts are $1,2,3$, respectively. The predicted risks are $\operatorname{logit}^{-1}(-2)\approx0.1192$, $0.5$, and $\operatorname{logit}^{-1}(2)\approx0.8808$. We check the candidate estimates $\hat\alpha=0$ and $\hat\beta=\log(3)/2\approx0.5493$.

At $z=-2$, the fitted event probability is $\operatorname{logit}^{-1}(-\log 3)=1/4$, giving $4(1/4)=1$ fitted event. At $z=0$, it is $1/2$, giving $2$ fitted events. At $z=2$, it is $\operatorname{logit}^{-1}(\log 3)=3/4$, giving $3$ fitted events. Each fitted count equals the observed count, so every residual $y_j-n_jm_j$ is zero. Both score equations therefore hold. Since the logits vary and the fit is finite, strict concavity confirms these are the unique maximum-likelihood estimates.

The slope is below one: on the log-odds scale, the observed event pattern is less steep than the original predicted-risk logits. In this example the intercept is zero, so the fitted risks are less extreme than the predictions, particularly toward the ends of the risk range.

Worked Example: A Slope Above One

Now let three groups of four have predicted logits $z=-1,0,1$, with event counts $1,2,3$. Their predicted risks are approximately $0.2689$, $0.5$, and $0.7311$. Consider $\hat\alpha=0$ and $\hat\beta=\log 3\approx1.0986$. The fitted probabilities are $\operatorname{logit}^{-1}(-\log3)=1/4$, $1/2$, and $\operatorname{logit}^{-1}(\log3)=3/4$.

The corresponding fitted event counts are $4(1/4)=1$, $4(1/2)=2$, and $4(3/4)=3$. Thus each residual is zero, both score equations are satisfied, and strict concavity gives the unique finite maximum. Because $\hat\beta>1$, the observed relationship is steeper than the predictions' logits: in this example, predicted risks are too compressed toward the middle.

Separating an Overall Shift from a Slope Change

When $\beta=1$, the calibration model is $\operatorname{logit}P(Y_i=1\mid z_i)=\alpha+z_i$. The odds of the event are then multiplied by $e^\alpha$ relative to the predicted odds, uniformly across subjects. This is an overall odds shift, not a change in the spacing of the predictions on the logit scale. The intercept is estimated alongside the slope in the calibration model; calibration-in-the-large instead assesses an overall shift with the slope fixed at one.

Worked Example: An Overall Odds Shift with Slope One

Consider three groups with predicted logits $z=-\log3,0,\log3$, corresponding to predicted risks $1/4,1/2,3/4$. Suppose their group sizes are $5,3,7$, and their event counts are $2,2,6$. The observed event proportions are $2/5,2/3,6/7$.

Set $\alpha=\log2$ and $\beta=1$. The fitted event probabilities are $\operatorname{logit}^{-1}(\log(2/3))=2/5$ at $z=-\log3$, $\operatorname{logit}^{-1}(\log2)=2/3$ at $z=0$, and $\operatorname{logit}^{-1}(\log6)=6/7$ at $z=\log3$. The fitted counts are consequently $2,2,6$, exactly the observed counts. The residuals are all zero, so both score equations hold; strict concavity establishes the unique estimates $\hat\alpha=\log2$ and $\hat\beta=1$.

Here the event odds are twice the predicted odds at every risk level, because $e^{\hat\alpha}=2$, but the slope indicates no change in their logit-scale spacing. This illustrates why an intercept shift and a slope departure answer different questions.

The slope is also unchanged by shifting the origin of the predictor logits. If every $z_i$ is replaced by $z_i+c$, then

\alpha+\beta z_i=(\alpha-\beta c)+\beta(z_i+c).

The same fitted linear predictors result with intercept $\alpha-\beta c$ and slope $\beta$. This identity explains why the slope describes steepness, while the numerical intercept depends on where zero is placed on the logit scale.

Interpretation and Important Limitations

A slope below one is often seen when predictions are too extreme, as can occur when a model is overfit in the data used to develop it. A slope above one indicates that predictions are relatively too compressed. These are useful diagnostic interpretations, not proof of a particular cause. Data shift, a different target population, or a poorly specified relationship can also produce a slope away from one.

A fitted slope summarizes a particular logistic recalibration model. If the true relationship between outcome risk and predicted logit is not linear on the logit scale, one slope cannot describe all departures from calibration. Its value can also be imprecise when the sample is small, outcomes are rare, predicted logits cover a narrow range, or a few observations have extreme logits. A finite estimate is not automatically a stable one.

The data used to evaluate calibration matter. Estimating the slope on the same observations used to fit a flexible prediction model can make the predictions appear too tailored to those outcomes; this often produces an apparently low slope. Evaluation on new observations or a suitable resampling procedure is needed to assess performance beyond the development sample. Even on independent evaluation data, a low slope is evidence of a pattern, not a definitive diagnosis of overfitting.

Common Mistakes

Reading the slope as the intercept. A slope below one describes a flatter relationship between observed outcome log-odds and predicted logits; it does not simply mean that all risks are too high or all are too low.
Fixing the intercept without saying so. The joint calibration model estimates both parameters. A slope from a model constrained to have $\alpha=0$ is a different fit and can confound an overall risk shift with slope.
Calling every low slope proof of overfitting. Overfitting is one possible explanation, especially for in-sample assessments, but changes in population or model misspecification can also matter.
Ignoring boundary predictions or separation. The logit is not finite for predicted probabilities of exactly zero or one. Such predictions require careful handling, and separation in the calibration regression can also make ordinary maximum-likelihood estimates infinite.

Key takeaway: Estimate the intercept and slope together in a logistic regression of outcomes on predicted-risk logits. A slope below one indicates a flatter observed risk relationship than predicted, a slope above one a steeper one, and an intercept departure can represent an overall shift in odds. Interpret the estimate in light of how and where it was evaluated.

Check Your Understanding

Use the definitions and results above to answer these questions.

In the joint calibration model, what does $\beta=1$ imply about the relationship between observed and predicted odds?
Why does estimating an intercept alongside the slope matter when assessing whether predictions are too extreme?
For a fitted slope of $0.7$, describe the logit-scale relationship indicated by the estimate, and name one possible—but not certain—explanation.
What feature of the predictor values makes the two calibration parameters identifiable in the stated strict-concavity result?
Why can a finite calibration slope estimated on the development data still give an optimistic view of performance on new observations?