Calibration Intercepts and Slopes in Validation

Why Check Calibration Intercept and Slope?

A validation model can rank risks sensibly and still predict the wrong level of loss. For example, predictions may be consistently too low, or the difference between low- and high-risk predictions may be too pronounced. Calibration-in-the-large measures an overall level correction; the calibration slope measures whether the spread of predictions is too strong or too weak on the model’s link scale.

These are checks on predicted values and observed outcomes in a validation sample. They do not, by themselves, establish why the model is miscalibrated, whether a correction will persist, or whether the model ranks risks better than a competitor. The earlier material on calibration checks for predicted loss costs considers observed and predicted totals and group-level comparisons; intercept-and-slope analysis adds a model-based way to describe the level and scale of the discrepancy.

Definition: Let $y_i$ be the validation response for observation $i$, $e_i$ its exposure, and $p_i>0$ the model’s predicted response rate per unit exposure. For a log-link calibration model, write $$ E[Y_i\mid p_i,e_i]=e_i\exp(\alpha)p_i^\beta. $$ The calibration-in-the-large intercept is estimated with the slope fixed at $\beta=1$. The calibration intercept and slope are estimated together when both $\alpha$ and $\beta$ are free. Ideal calibration corresponds to $(\alpha,\beta)=(0,1)$. When the slope is fixed at one, the ideal intercept is $\alpha=0$.

Here, $p_i$ must be on the scale of the expected response rate: for example, predicted claim frequency, or predicted loss cost per unit exposure. The response distribution and fitting method must be appropriate to the outcome. For claim counts, a common specification is $Y_i\sim\operatorname{Poisson}(\lambda_i)$, with $\lambda_i=e_i\exp(\alpha)p_i^\beta$. For aggregate losses, the mean structure can use the same form, but a suitable loss distribution or quasi-likelihood method is needed; claim counts should not be treated as dollar losses. The Poisson examples below are hypothetical and assess frequency calibration.

What the Intercept and Slope Say

With $\beta=1$, the calibration model multiplies every prediction by the same factor, $\exp(\alpha)$. A positive $\alpha$ means observed outcomes are higher overall than predicted; a negative $\alpha$ means they are lower. In a Poisson model, the intercept-only maximum-likelihood estimate has a simple form:

\widehat{\alpha}_{\text{CITL}} =\log\left(\frac{\sum_i y_i}{\sum_i e_i p_i}\right).

The slope asks a different question. In the joint model, $\beta=1$ means the log predictions have the right spread, after allowing an overall intercept adjustment. A slope below one indicates that observed outcomes vary less sharply with predicted risk than the model’s predictions do; a slope above one indicates a steeper relationship. The interpretation is conditional on the chosen link, response model, and validation sample.

Interpretation guide: Under the log-link model, $\exp(\widehat{\alpha})$ is the multiplicative level adjustment at $p_i=1$. A slope of one preserves the predicted log-scale spread. A positive slope other than one preserves the ordering of strictly positive predictions, because $p^\beta$ is increasing in $p$ when $\beta>0$. Neither an intercept shift nor a positive slope change alone proves that ranking performance has improved or deteriorated.

Why the Intercept-Only Formula Works

Assume independent Poisson validation counts with fixed predictions and exposures, so that $Y_i\sim\operatorname{Poisson}(e_i p_i\exp(\alpha))$. When the slope is fixed at one, the log-likelihood, apart from terms that do not depend on $\alpha$, is

\ell(\alpha) =\sum_i\left[y_i\{\log(e_i p_i)+\alpha\} -e_i p_i\exp(\alpha)\right]+\text{constant}.

Differentiate with respect to $\alpha$, set the score to zero, and solve:

\frac{\partial\ell}{\partial\alpha} =\sum_i y_i-\exp(\alpha)\sum_i e_i p_i=0 \quad\Longrightarrow\quad \exp(\widehat{\alpha}_{\text{CITL}}) =\frac{\sum_i y_i}{\sum_i e_i p_i}.

Thus, in this Poisson model, the intercept-only fit matches the total fitted count to the total observed count. With other response distributions or estimation criteria, the intercept estimate need not equal the log of this ratio. Also, if both intercept and slope are estimated, the simple total-ratio formula generally does not give the joint-model intercept: changing the slope changes how predictions contribute to the fitted total.

Fit and Interpret the Calibration Models

Align the validation data.
Use outcomes, exposures, and predictions covering the same risks and period. For counts, use predicted rates with exposure, or predicted expected counts directly with the corresponding offset.

Fit the level check.
Fix $\beta=1$ and estimate the intercept. Under the Poisson assumptions above, compare total observed counts with total predicted counts.

Fit the slope check.
Estimate $\alpha$ and $\beta$ together with an appropriate model. The joint intercept is the level adjustment conditional on the estimated slope, not the intercept-only calibration-in-the-large estimate.

Interpret cautiously.
Describe level and spread separately. Check estimation uncertainty and data alignment before treating a departure from the ideal values as evidence for a pricing action.

Worked Example: Estimate an Overall Frequency Correction

A hypothetical validation set contains three groups of risks. Each group has one unit of exposure. The model predicted expected claim counts of 12, 18, and 30; the observed counts were 15, 18, and 33. Assume independent Poisson counts and use a log-link calibration model with slope fixed at one. Estimate and interpret the calibration-in-the-large intercept.

Method and assumptions. This is an intercept-only Poisson calibration fit. The exposure is one for each group, so the predicted rates and predicted expected counts have the same numerical values. The total predicted count is $12+18+30=60$, and the total observed count is $15+18+33=66$.

The fitted multiplicative factor is $66/60=1.10$. Therefore, $\widehat{\alpha}_{\text{CITL}}=\log(1.10)=0.09531$. The calibrated expected counts are $13.2$, $19.8$, and $33.0$; their sum is $66.0$, matching the observed total.

Check and interpretation. Since the intercept is positive, the validation total is 10% above the prediction total. The calculation avoids the common mistake of averaging group-level percentage differences without regard to their predicted volumes. It does not show whether high-risk predictions are too dispersed; that requires estimating the slope as well.

Worked Example: Estimate a Calibration Slope

In a hypothetical validation exercise, three exposure-one groups have predicted claim frequencies $p_i=1,2,4$ and observed claim counts $y_i=2,8,32$. Assume independent Poisson counts and fit $\lambda_i=\exp(\alpha)p_i^\beta$. Estimate both parameters, then compare the result with the intercept-only fit.

Method and assumptions. The joint calibration model uses the log of predicted frequency as a covariate. At $\alpha=\log 2$ and $\beta=2$, the fitted means are:

For $p=1$, $\lambda=2(1^2)=2$. For $p=2$, $\lambda=2(2^2)=8$. For $p=4$, $\lambda=2(4^2)=32$. These equal the observed counts $2,8,32$, respectively. Because a Poisson likelihood for each observation is maximized when its mean equals its observed count, this parameter pair maximizes the joint likelihood. Thus $\widehat{\alpha}=\log 2=0.6931$ and $\widehat{\beta}=2$.

Now fix the slope at one. Total observed counts are $2+8+32=42$; total predicted counts are $1+2+4=7$. The intercept-only estimate is $\log(42/7)=\log 6=1.7918$, a factor of six. This differs from the joint intercept $\log 2$, because the joint model also changes the slope.

Check and interpretation. The joint fitted counts sum to $42$, as they must here because each fitted mean equals its observed count. The slope of two indicates a steeper relationship between log predicted frequency and log expected count than the ideal slope of one. Since the slope is positive, the predictions retain their ordering in this example. The trap is to report $\log 6$ as the intercept from the joint fit; it belongs to a different model, where the slope is fixed at one.

Worked Example: Keep Exposure in the Calibration

A hypothetical portfolio has three groups with predicted claim frequencies $0.02$, $0.05$, and $0.10$ per exposure unit. Their exposures are 100, 200, and 100 units, and their observed claim counts are 3, 8, and 12. Assume independent Poisson counts and fix the calibration slope at one. Calculate the intercept and explain why an unweighted average of the predicted frequencies is not the appropriate total comparison.

Method and assumptions. The predicted expected counts are exposure times predicted frequency: $100(0.02)=2$, $200(0.05)=10$, and $100(0.10)=10$. The total prediction is $2+10+10=22$. The total observed count is $3+8+12=23$. Therefore, $\widehat{\alpha}_{\text{CITL}}=\log(23/22)=0.04445$, and the multiplicative factor is $23/22=1.04545$.

The calibrated expected counts are $2(23/22)=2.0909$, $10(23/22)=10.4545$, and $10(23/22)=10.4545$, which sum to $23.0000$ before rounding. The unweighted mean of the three predicted frequencies would give each group equal influence despite the middle group having twice the exposure of either other group. For this Poisson total check, the relevant comparison is between exposure-adjusted totals.

Check and interpretation. The positive intercept indicates an overall shortfall in predicted frequency of about 4.55% relative to the observed total. This is an intercept-only result, not a slope estimate. Rounding the displayed group counts before checking the total can produce a small apparent discrepancy; the unrounded multiplier gives an exact total of 23.

Calibration Is Not Ranking

A calibration intercept shifts predictions on the log scale without changing their order. A positive calibration slope also preserves ordering, although it changes the spacing between predictions. Consequently, departures from $(0,1)$ can signal a level or scale problem without showing that one model distinguishes higher-risk from lower-risk observations better than another. Conversely, a model may rank risks well while systematically predicting too high or too low.

A slope near zero means the validation response has little estimated association with the log predictions under this calibration model. A negative slope means the estimated association is reversed. Neither conclusion should be made from a point estimate alone: sampling variation, sparse outcomes, influential observations, and changes between training and validation populations can all matter. Estimation uncertainty and formal comparisons are separate questions.

Common Traps and Practical Limits

Confusing the two intercepts. The intercept-only calibration-in-the-large estimate fixes $\beta=1$. The joint calibration intercept is estimated with $\beta$, and generally differs from it.
Ignoring exposure. When predictions are rates, compare observed counts with exposure times predicted rates, not with the sum of rates alone.
Using the wrong response model. Poisson likelihood is a model for counts, not dollar losses. For loss-cost validation, use a response distribution or estimating approach suitable for the loss data and preserve the exposure structure.
Taking logarithms of zero predictions. The log-link slope formulation requires positive $p_i$. Investigate zero predictions and choose a defensible model or representation; silently adding an arbitrary constant changes the calibration question.
Calling a slope departure a ranking failure. A positive nonunit slope changes calibration spread but preserves the order of positive predictions. Ranking discrimination requires its own assessment.
Over-interpreting one validation sample. Calibration describes performance for the data and period assessed. It does not guarantee that a multiplicative correction will hold in another period or population.

Key takeaway: Read the calibration-in-the-large intercept as an overall level adjustment and the joint slope as a link-scale spread adjustment. State the response model and exposure treatment, distinguish the two intercept estimates, and do not infer ranking performance from calibration alone. This treatment is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Check Your Understanding

For each question, use the stated model and distinguish an intercept-only estimate from a joint slope fit where relevant.

A hypothetical validation set has Poisson observed counts totaling 88 and predicted expected counts totaling 80. With the calibration slope fixed at one, find the intercept and multiplicative adjustment. (Answer: $\widehat{\alpha}=\log(1.10)=0.0953$; factor $1.10$.)
Two exposure-one groups have predicted frequencies $p=1$ and $p=3$, with observed Poisson counts 2 and 18. Under $\lambda=\exp(\alpha)p^\beta$, find the joint fit. (Answer: $\widehat{\alpha}=\log 2=0.6931$, $\widehat{\beta}=2$.)
A portfolio has exposures 50 and 150, predicted frequencies 0.04 and 0.02, and observed counts 3 and 4. Under a Poisson intercept-only calibration model, calculate the intercept. (Answer: predicted total $50(0.04)+150(0.02)=5$; observed total $7$; $\widehat{\alpha}=\log(7/5)=0.3365$.)
A fitted calibration model has $\widehat{\alpha}=-0.10$ and $\widehat{\beta}=0.80$. State the direction of the level adjustment and whether the positive slope changes the ordering of positive predictions. (Answer: the level factor is $\exp(-0.10)\approx0.9048$, a downward adjustment; the positive slope preserves ordering.)
In an intercept-only Poisson check, an analyst reports $\log(23/22)$ as the intercept from a joint model in which both $\alpha$ and $\beta$ were fitted. What is the issue? (Answer: $\log(23/22)$ is the intercept-only estimate with $\beta=1$; it is not generally the joint-model intercept.)

What You'll Learn