Tutorials › Statistical Genetics › Calibration Intercepts and Slopes

Home  /  Tutorials  /  Prediction Models and Validation

Prediction Models and Validation · Tutorial 724 of 989

Calibration Intercepts and Slopes

Estimate calibration-in-the-large and the calibration slope for binary predictions, and interpret what their values reveal about systematic risk errors.

Advanced 10 min read

What You'll Learn

  • Define calibration-in-the-large using a logistic model with the predicted log odds as an offset
  • Estimate the calibration intercept and interpret its direction as overall underprediction or overprediction
  • Define the calibration slope using a logistic regression with an intercept and predicted log odds
  • Interpret slopes below and above one as overly extreme and overly moderate predictions, respectively
  • Distinguish overall calibration from the joint intercept and slope assessment
  • Check the assumptions and limitations behind these validation estimates

Why Calibration Intercepts and Slopes Matter

A genetic prediction model can rank people well while assigning risks that are systematically too high or too low. It can also get the average risk about right but make its predictions too extreme, or not extreme enough. Calibration intercepts and slopes summarize these two distinct features of binary risk predictions. They complement discrimination measures such as the area under the ROC curve: ranking and agreement between predicted and observed risks are different properties.

Suppose a model gives a predicted probability \(p_i\) for each person \(i\) in a validation sample, and \(Y_i=1\) indicates that the binary outcome occurred. Calibration analysis compares these probabilities with the outcomes in the validation population. The predictions should have been generated without using the validation outcomes; otherwise, the apparent agreement may be optimistic.

Definition: Write \(\eta_i=\operatorname{logit}(p_i)=\log\{p_i/(1-p_i)\}\) for the predicted log odds. Calibration-in-the-large is estimated by fitting a logistic model with \(\eta_i\) as a fixed offset and estimating an intercept \(\alpha_{\mathrm{CIL}}\): \(\operatorname{logit}\{P(Y_i=1\mid\eta_i)\}=\alpha_{\mathrm{CIL}}+\eta_i\). The calibration slope and its accompanying intercept are estimated from \(\operatorname{logit}\{P(Y_i=1\mid\eta_i)\}=\alpha+\beta\eta_i\). Ideal values are \(\alpha_{\mathrm{CIL}}=0\), and, in the joint model, \(\alpha=0\) and \(\beta=1\).

What the Estimates Mean

In the calibration-in-the-large model, the slope is held at one. The intercept adjusts all predicted log odds by the same amount. A positive \(\alpha_{\mathrm{CIL}}\) means that outcomes occurred more often than predicted overall: the predictions underpredicted risk. A negative value indicates overall overprediction. More precisely, adding \(\alpha_{\mathrm{CIL}}\) multiplies predicted odds by \(\exp(\alpha_{\mathrm{CIL}})\); it does not add a constant number of percentage points to each predicted risk.

The joint intercept-and-slope model asks a different question. The intercept \(\alpha\) is the fitted log odds when the predicted log odds are zero—that is, at a predicted risk of 0.5. The slope \(\beta\) describes how observed log odds change as predicted log odds change. A slope below one means that the observed risks vary less across the prediction scale than the model predicts: its predictions are too extreme. A slope above one means they vary more: its predictions are too moderate. The intercept and slope must be read together; a joint-model intercept is not automatically the same as the calibration-in-the-large intercept.

These estimates are calibration summaries on the log-odds scale, not measures of discrimination. A slope of one does not by itself guarantee a zero intercept, and a zero calibration-in-the-large intercept does not guarantee a slope of one. Each can be correct while the other reveals a problem.

Estimating the Calibration-in-the-Large Intercept

For a binary outcome, the calibration-in-the-large estimate is the value of \(\alpha\) that makes the total number of fitted events equal the total number observed, when the predicted log odds are used as an offset. If \(\operatorname{expit}(z)=1/(1+\exp(-z))\), the fitted risk for person \(i\) is \(\operatorname{expit}(\alpha+\eta_i)\). The logistic likelihood gives the following score equation:

$$ \sum_{i=1}^{n}\left[Y_i-\operatorname{expit}(\alpha+\eta_i)\right]=0. $$

Thus, the estimate is generally obtained by solving this equation numerically. At \(\alpha=0\), the fitted risks are the original predictions. If the observed event count exceeds the sum of the predicted risks, the score at zero is positive and the fitted intercept must increase to raise the fitted risks. This explains why a positive estimate indicates overall underprediction.

Worked Example: Overall Underprediction in a Small Validation Sample

Suppose, hypothetically, that 20 unrelated people each receive a predicted risk of \(p_i=0.20\), and the outcome occurs in 6 of them. The predicted odds for each person are \(0.20/(1-0.20)=0.25\); the observed event proportion is \(6/20=0.30\). Because all predictions are identical, the offset model reduces to an intercept that changes the common predicted risk to the observed proportion.

The estimate is the difference between the observed and predicted log odds:

$$ \widehat{\alpha}_{\mathrm{CIL}} =\operatorname{logit}(0.30)-\operatorname{logit}(0.20) =\log\left(\frac{0.30}{0.70}\right)-\log\left(\frac{0.20}{0.80}\right) =\log(0.4286/0.25) =0.5390. $$

The positive estimate indicates underprediction overall. Specifically, the adjusted odds are multiplied by \(\exp(0.5390)=1.714\), changing the predicted risk from 0.20 to 0.30 in this constant-prediction example. The difference is 10 percentage points here, but that is a consequence of these particular risks—not the general interpretation of the intercept.

A rough Wald standard error for this intercept is \(\sqrt{1/6+1/14}=0.4880\), using the binomial information for 6 events and 14 non-events. A 95% Wald confidence interval is \(0.5390\pm1.96(0.4880)\), or approximately \((-0.417,1.495)\). This interval is wide: the sample provides limited precision about the overall calibration shift. It describes a procedure with 95% coverage under its assumptions; it does not mean there is a 95% probability that this fixed parameter lies in this particular interval.

Estimating and Reading the Calibration Slope

To estimate the slope, fit a logistic regression of \(Y_i\) on \(\eta_i\), including a freely estimated intercept. The predicted log odds are a single predictor; this procedure does not refit the original genetic model or alter its weights. The calibration slope describes the relationship between the original predictions and outcomes in the validation sample.

A slope below one is commonly seen when predictions are too spread out—for example, when a model performs less well in new people than in the data used to develop it. The validation outcomes then show a weaker contrast between low- and high-prediction groups than the original predictions imply. A slope above one indicates a stronger contrast in observed risks than the predictions imply. Neither direction identifies the cause by itself.

Worked Example: Predictions That Are Too Extreme

Consider a hypothetical validation sample summarized in three groups of 100 people. Within each group, everyone has the same predicted risk. The low-, middle-, and high-risk groups have predictions of 0.10, 0.50, and 0.90, with 25, 50, and 75 observed events, respectively. Their predicted log odds are \(-\log(9)\), \(0\), and \(\log(9)\); their observed event proportions are 0.25, 0.50, and 0.75.

The joint logistic calibration model is \(\operatorname{logit}(q_i)=\alpha+\beta\eta_i\), where \(q_i\) is the fitted outcome probability. At \(\alpha=0\) and \(\beta=0.5\), the fitted probabilities in the three groups are:

$$ \operatorname{expit}\{-0.5\log(9)\}=0.25,\qquad \operatorname{expit}(0)=0.50,\qquad \operatorname{expit}\{0.5\log(9)\}=0.75. $$

These fitted probabilities equal the observed proportions in every group, so the logistic likelihood is maximized at \(\widehat{\alpha}=0\) and \(\widehat{\beta}=0.5\). For example, in the high-risk group the calculation is \(\operatorname{expit}\{0.5\log(9)\}=\operatorname{expit}\{\log(3)\}=3/(1+3)=0.75\). The slope below one indicates predictions that are too extreme: a prediction of 0.90 corresponds to an observed proportion of 0.75, while a prediction of 0.10 corresponds to 0.25. The zero intercept does not cancel this slope problem.

Worked Example: Correct Average Risk but Predictions That Are Too Moderate

In a second hypothetical validation sample, 100 people have predicted risk 0.25 and 100 have predicted risk 0.75. There are 10 events in the lower-prediction group and 90 in the higher-prediction group. The observed proportions, 0.10 and 0.90, are farther apart than the predictions suggest.

The corresponding predicted log odds are \(-\log(3)\) and \(\log(3)\). At \(\alpha=0\) and \(\beta=2\), the fitted risks are:

$$ \operatorname{expit}\{-2\log(3)\}=\operatorname{expit}\{-\log(9)\}=0.10,\qquad \operatorname{expit}\{2\log(3)\}=\operatorname{expit}\{\log(9)\}=0.90. $$

The fitted probabilities match both observed proportions, giving \(\widehat{\alpha}=0\) and \(\widehat{\beta}=2\). The slope above one indicates that the predictions are too moderate: the observed contrast is stronger. The overall observed event proportion is \(100/200=0.50\), and the mean predicted risk is \((0.25+0.75)/2=0.50\), so there is no overall shift in this example. Correct average risk therefore does not rule out a calibration-slope problem.

Model Conditions and a Short Justification

These logistic estimates require care in how the validation data are formed and analyzed. The outcome must be binary, and each included person must have an outcome and a corresponding prediction for the same target and prediction horizon. Predictions must be strictly between zero and one to have finite logits; exact zero or one predictions require attention before fitting. For the usual likelihood and standard errors, outcomes are assumed independent conditional on the predictions. Related people or other clustered observations can violate this assumption, so uncertainty calculations may need to account for clustering. The slope requires variation in the predicted log odds; if all predictions are identical, the slope cannot be estimated.

A short likelihood argument explains the fitting rule. For independent Bernoulli outcomes with fitted risks \(q_i=\operatorname{expit}(\alpha+\beta\eta_i)\), the log-likelihood, apart from constants, is \(\ell=\sum_i[Y_i\log(q_i)+(1-Y_i)\log(1-q_i)]\). Differentiating with respect to the intercept gives:

$$ \frac{\partial\ell}{\partial\alpha} =\sum_{i=1}^{n}(Y_i-q_i). $$

At a finite maximum-likelihood estimate this derivative is zero, so \(\sum_iY_i=\sum_iq_i\): fitted and observed event totals agree. For the joint model, differentiating with respect to the slope gives \(\partial\ell/\partial\beta=\sum_i\eta_i(Y_i-q_i)=0\) as well. These equations explain the estimation conditions; they do not imply that each person's outcome can be predicted correctly.

Common Mistakes and Practical Interpretation

  • Calling the intercept a percentage-point correction. It shifts log odds, multiplying odds by \(\exp(\alpha)\); the corresponding change in risk depends on the starting risk.
  • Using the joint-model intercept as though it were calibration-in-the-large. The joint intercept is estimated while the slope is free and refers to predicted risk 0.5. Calibration-in-the-large fixes the slope at one.
  • Concluding that a zero intercept means full calibration. It says nothing by itself about whether the slope equals one. The third worked example has no overall shift but a slope of two.
  • Reversing the slope interpretation. A slope below one indicates predictions that are too extreme; a slope above one indicates predictions that are too moderate.
  • Treating a calibration estimate as causal evidence. These are descriptions of agreement in a validation population, not estimates of the effect of changing a genetic score.
  • Ignoring how validation was done. Reusing outcomes to develop or tune the prediction model can make calibration look better than it will be in new individuals.

Calibration intercepts and slopes are useful because they separate an overall shift in risk from a mismatch in the spread of predictions. They do not show every feature of calibration: different patterns of error can produce similar intercept and slope estimates. Interpret them as concise summaries for the specified validation population, alongside the model's target, validation design, and uncertainty.

Check Your Understanding

Try each question by identifying the model quantity involved and stating its interpretation in context.

  1. A calibration-in-the-large estimate is \(-0.25\). What does its sign indicate, and by what factor are predicted odds shifted under this correction?
  2. Why does calibration-in-the-large hold the coefficient of predicted log odds at one?
  3. A joint calibration model has slope 0.7. What does this say about the spread of risks predicted by the model?
  4. Can a model have a calibration-in-the-large intercept of zero and a slope different from one? Explain.
  5. What prevents a calibration slope from being estimated when all individuals have the same predicted risk?

Previous

Next

Back to Tutorials
Clinical Biostats