Calibration Slopes For Transition Predictions

Why Examine the Calibration Slope?

A transition prediction can rank individuals well and still give risks that are too extreme or not extreme enough. For example, people assigned a high probability of a specified transition may experience that transition less often than predicted, while people assigned a low probability may experience it more often. A calibration slope summarizes this pattern across predicted risks: it assesses the spread of predictions, rather than only whether their average level is too high or too low.

The target must be defined before fitting the slope. In this tutorial, consider a cohort of individuals who are in state $r$ at a specified prediction origin. For each individual, a model supplies a probability $p_i$ that a designated transition from $r$ to state $s$ occurs within a specified horizon. The binary outcome $Y_i$ is 1 if that transition occurs within the horizon and 0 otherwise. If another transition occurs first, that individual has $Y_i=0$ for this endpoint. Thus the endpoint is a transition-specific risk, not simply the probability of occupying state $s$ at the horizon.

Definition: For predictions $p_i$ strictly between 0 and 1, the logistic calibration model is $$ \operatorname{logit}\{P(Y_i=1\mid p_i)\}=a+b\,\operatorname{logit}(p_i), $$ where $\operatorname{logit}(u)=\log\{u/(1-u)\}$, $a$ is the calibration intercept, and $b$ is the calibration slope. Ideal calibration under this model has $a=0$ and $b=1$.

The intercept and slope have related but different roles. The intercept shifts the calibration curve up or down on the log-odds scale; the slope changes how rapidly the fitted event risk rises as predicted log-odds increase. In practice, estimate both together using validation data. A slope-only fit with the intercept fixed at zero answers a different, more restrictive question and can misrepresent the slope when predictions are also systematically too high or too low.

Estimation and Interpretation

For fully observed binary outcomes and independent validation subjects, fit a logistic regression with $Y_i$ as the response and $\operatorname{logit}(p_i)$ as the sole predictor, including an intercept. The likelihood for parameters $a,b$ is

L(a,b)=\prod_{i=1}^{n} \left[\operatorname{expit}\{a+b\operatorname{logit}(p_i)\}\right]^{Y_i} \left[1-\operatorname{expit}\{a+b\operatorname{logit}(p_i)\}\right]^{1-Y_i},

where $\operatorname{expit}(v)=1/(1+e^{-v})$. The maximum-likelihood estimates generally require numerical optimization. The fitted slope $\hat b$ is interpreted as the change in log odds of the observed transition outcome associated with a one-unit increase in predicted log odds, under this logistic calibration relationship. It is not a causal effect.

The key spread interpretation follows directly from the model. With $a$ held fixed, a predicted logit $x=\operatorname{logit}(p_i)$ maps to the fitted outcome logit $a+bx$. If $0<b<1$, departures of $x$ from zero are reduced: predictions are too spread out, or too extreme, relative to the observed outcome pattern. If $b>1$, those departures are enlarged: predictions are too compressed toward the middle. A negative slope means the observed risk tends to run in the opposite direction from the predictions, a serious warning rather than ordinary miscalibration.

This interpretation concerns spread and is distinct from average-level error. The fitted intercept and slope should be read together. A nonzero intercept can shift the curve even when the slope is one; conversely, a slope different from one describes a spread problem even if the calibration curve happens to cross the ideal curve at a particular risk.

Worked Calculations

Worked Example: A Slope Below One From Two Risk Groups

Suppose a hypothetical, fully observed validation cohort has two prediction groups. In the group assigned $p=0.10$, 8 of 100 individuals experience the designated transition by the horizon. In the group assigned $p=0.30$, 20 of 100 experience it. Treat each group as having a common prediction and observed event proportion. With exactly two distinct predictor values and two observed proportions strictly between 0 and 1, the logistic calibration curve can pass through both group proportions.

The group logits are $\operatorname{logit}(0.10)=-2.1972$, $\operatorname{logit}(0.30)=-0.8473$, $\operatorname{logit}(0.08)=-2.4423$, and $\operatorname{logit}(0.20)=-1.3863$. The fitted slope is the change in observed logits divided by the change in predicted logits:

\hat b= \frac{\operatorname{logit}(0.20)-\operatorname{logit}(0.08)} {\operatorname{logit}(0.30)-\operatorname{logit}(0.10)} = \frac{-1.3863-(-2.4423)}{-0.8473-(-2.1972)} = \frac{1.0560}{1.3499} \approx 0.7823.

The intercept is $\hat a=\operatorname{logit}(0.08)-\hat b\operatorname{logit}(0.10)\approx -2.4423-(0.7823)(-2.1972)\approx -0.7235$. The slope below one indicates that predictions are too spread out for this observed pattern. This is a deliberately small, grouped illustration; with only two prediction values, it provides no meaningful check that the logistic relationship holds between or beyond those values.

Worked Example: A Slope Above One

In a second hypothetical, fully observed validation set, one group has predicted risk $0.10$ and observed transition proportion $0.05$; another has predicted risk $0.20$ and observed proportion $0.25$. The prediction logits are $-2.1972$ and $-1.3863$; the observed logits are $-2.9444$ and $-1.0986$.

\hat b= \frac{-1.0986-(-2.9444)}{-1.3863-(-2.1972)} = \frac{1.8458}{0.8109} \approx 2.2762.

Using the first group, $\hat a=-2.9444-(2.2762)(-2.1972)\approx 2.0568$. The fitted line also gives $2.0568+(2.2762)(-1.3863)\approx -1.0986$ for the second group's observed logit. Since the slope exceeds one, the observed risk contrast is steeper than the predicted contrast: here the predictions are too compressed. The intercept is large and positive, so the example also shows why slope should not be interpreted as a complete calibration assessment on its own.

Worked Example: Calculating an Intercept at a Reference Risk

Suppose a calibration analysis has estimated a slope of $0.4952$. At a reference predicted risk of $0.10$, the observed transition proportion is $0.15$. The calibration curve constrained to pass through this reference point has intercept $a=\operatorname{logit}(0.15)-b\operatorname{logit}(0.10)$. This calculation illustrates the intercept implied by that anchor; it is not a substitute for estimating the intercept and slope jointly from all validation subjects.

\hat a=-1.7346-(0.4952)(-2.1972) \approx -1.7346+1.0881 \approx -0.6465.

The slope $0.4952$ indicates substantially over-spread predictions under the logistic calibration model. At the reference predicted risk of 10%, the curve is anchored to a 15% observed proportion. Both the slope and intercept are needed to describe those two features.

Assumptions, Censoring, and Justification

The model and its interpretation rely on several conditions. First, the validation endpoint must correspond to the event whose risk was predicted: the same origin state, target transition, prediction horizon, and handling of competing transitions must be used. Second, $p_i$ must be a probability on the relevant scale, and must be strictly between zero and one to use its logit. Predictions rounded to exactly zero or one should not be treated as genuine certainty without examining how they arose.

Third, the logistic relationship between observed log odds and predicted log odds should be a reasonable summary over the prediction range. The fitted slope is a one-number summary; if calibration bends or changes direction, it may conceal important departures. Fourth, standard likelihood-based uncertainty calculations presume independent subjects and an adequately specified outcome model. Repeated records from the same person require methods that account for within-person dependence; treating them as independent can make uncertainty estimates too small.

The simple likelihood above also presumes that each binary outcome is observed. If a subject is lost to follow-up or censored before the horizon, it is not valid to automatically code that subject as having no target transition. The endpoint is then unknown. The analyst needs an appropriate method for incomplete transition outcomes, with its censoring assumptions stated and checked. In the simplest setting, complete follow-up through the horizon avoids that complication. If censoring is handled by weighting or another procedure, the resulting estimation and uncertainty calculation must reflect that procedure.

Fix the prediction target.
Specify the state at prediction origin, target transition, horizon, and treatment of other transitions.

Check outcome ascertainment.
Establish whether every subject's transition status is known through the horizon; do not label early censoring as absence of transition.

Fit the calibration model.
Regress the binary outcome on the logit of predicted risk with a freely estimated intercept.

Interpret with context.
Report slope and uncertainty, describe spread, and consider the intercept and prediction range before drawing an overall calibration conclusion.

Diagnostics, Alternatives, and Limitations

Potential warning signs include very few target transitions, predictions concentrated at only a few values, extreme fitted coefficients, or failure of the fitting procedure to converge. Sparse transitions can produce a highly uncertain slope; if outcomes are completely separated by predicted risk, a finite ordinary maximum-likelihood estimate may not exist. A narrow range of predicted risks also gives little information about spread. Confidence intervals should therefore accompany the point estimate, and their width should be interpreted in light of event counts and prediction range.

A useful diagnostic is to compare observed event proportions with predictions across suitably sized risk groups or against a smooth calibration curve. Such summaries can reveal curvature that a single slope misses, but grouping loses information and can be unstable when events are rare. A slope near one is not proof of perfect calibration: a nonzero intercept, local departures, or limited precision can remain. Conversely, a slope far from one is evidence about the validation sample and model form, not proof that the prediction model will fail identically in every population.

Calibration slope is complementary to the baseline-risk recalibration discussed in the earlier material on recalibrating baseline transition risks. A baseline adjustment addresses a level shift; a slope evaluates spread on the logit scale. Neither replaces discrimination measures, which concern ranking. Moreover, one transition-specific binary endpoint is assessed at a time here. Fitting separate calibration relationships for multiple possible destinations does not by itself ensure that the resulting probabilities are mutually coherent; that is a separate issue.

A practical limitation is that the slope depends on the validation population's range of predicted risks and event information. Estimates can differ across populations even when the underlying model is unchanged. Report the transition, horizon, event count, slope estimate, uncertainty interval, and any censoring approach so readers can judge the evidence and its scope.

Key takeaway: A calibration slope below one indicates predictions that are too spread out, while a slope above one indicates predictions that are too compressed, under the fitted logistic calibration model. Interpret the estimate alongside the intercept, uncertainty, endpoint definition, and data quality.

Check Your Understanding

Answer each question, then compare with the solutions.

What is the calibration slope when observed transition log odds change by 0.8 for a one-unit increase in predicted log odds? What does it suggest?
Why should a subject censored before the prediction horizon not automatically be coded as having no target transition?
If the fitted slope is one but the intercept is not zero, is the calibration model ideal? Explain.
Why can a calibration slope near one fail to establish good calibration throughout the risk range?

Solutions. (1) The slope is $0.8$. It is below one, suggesting predictions are too spread out relative to observed outcomes under the logistic calibration model. (2) The target-transition status by the horizon is unknown for that subject; coding it as zero falsely asserts that the transition did not occur and can bias estimation. (3) No. Ideal calibration requires both $a=0$ and $b=1$; a nonzero intercept shifts the calibration curve away from the ideal even when its slope is one. (4) A single slope summarizes the overall logit-scale relationship and can hide curvature or localized miscalibration; limited events or a narrow prediction range can also make the estimate imprecise.