Why Check Calibration?
A tuned binomial glmnet model can assign each observation a predicted event probability, but a useful-looking score is not automatically a trustworthy risk estimate. If a model assigns probabilities near 0.20 to a group of comparable cases, calibration asks whether about 20% of that group experience the event. This is different from asking whether cases that experience the event tend to receive higher scores than cases that do not.
Calibration should be assessed using outcomes that were not used to select the penalty or fit the reported predictions. In the setting of the earlier material on using a final test set after penalty tuning, calculate predictions on that reserved set and compare them with its observed outcomes. Reusing the tuning data can make apparent agreement optimistic. Calibration diagnostics describe predictive agreement on the evaluated population; they do not establish that the relationship will remain unchanged in a different population or time period.
The target is agreement between predicted and observed probabilities, not equality of every individual outcome and prediction. A prediction of 0.20 does not mean the event must occur for one case in every five consecutive cases. It means that, among sufficiently many comparable cases assigned that risk, the event frequency should be near 0.20.
Grouped Calibration Summaries
A basic diagnostic groups test-set predictions into ranges or bins. For each bin, report its number of observations, mean predicted probability, and observed event rate. If the bin contains \(n_b\) cases, the mean prediction is \(\bar p_b\), and \(e_b\) events occur, its observed event rate is \(e_b/n_b\). A well-calibrated model should have these two quantities reasonably close across bins.
A table makes the comparison explicit; a calibration plot displays the same information, usually with mean predicted probability on the horizontal axis and observed event rate on the vertical axis. The reference line is \(y=x\). Points above that line indicate that observed rates exceed predictions in those bins; points below it indicate that predictions exceed observed rates. The plot should include the number of cases in each bin, or another indication of sample size, because rates from small bins are noisy.
Worked Example: Reading a Grouped Calibration Table
A hypothetical insurer evaluates a tuned binomial glmnet model on 10 independent test cases. Treat each outcome as Bernoulli, with \(Y_i=1\) for a claim and \(Y_i=0\) otherwise. The first five cases have predictions \(0.10, 0.10, 0.10, 0.10, 0.10\), and one has a claim. The other five have predictions \(0.30, 0.30, 0.30, 0.30, 0.30\), and two have claims. Compare predicted and observed rates by group, and calculate the weighted mean absolute calibration gap across the two groups.
Method. For each group, average its probabilities and divide the number of observed events by the group size. The weighted mean absolute gap weights each group’s absolute difference by its share of all test cases.
For the first group, \(n_1=5\), the predicted mean is \(5(0.10)/5=0.10\), and the observed rate is \(1/5=0.20\). The absolute gap is \(|0.20-0.10|=0.10\).
For the second group, \(n_2=5\), the predicted mean is \(5(0.30)/5=0.30\), and the observed rate is \(2/5=0.40\). Its absolute gap is \(|0.40-0.30|=0.10\).
Each group represents \(5/10=0.50\) of the test cases. Therefore, the weighted mean absolute gap is \(0.50(0.10)+0.50(0.10)=0.10\), or 10 percentage points. Across all cases, the mean prediction is \((5(0.10)+5(0.30))/10=0.20\), while the observed event rate is \((1+2)/10=0.30\). The model underpredicts the overall event rate by 0.10 in this test set.
Check. The total predicted events, in expectation, are \(5(0.10)+5(0.30)=2\); the observed number is 3. Dividing each by 10 gives 0.20 and 0.30, consistent with the overall comparison. The common mistake is to average the two group rates without weights: that would give a group-level average of 0.30, but it is only valid for an overall case-level rate when the groups have equal sizes.
Grouped summaries are useful, but they depend on how bins are formed. Equal-width bins can leave some ranges nearly empty, especially when predictions cluster near zero. Equal-frequency bins tend to have more cases in each bin, but their probability ranges can vary greatly. Either approach can hide variation within a bin, and neither turns a noisy observed rate into a precise estimate. Report the binning choice and use the individual prediction distribution to understand where the test data lie.
Calibration-in-the-Large and Calibration Slope
Two logistic regression summaries provide a compact view of systematic calibration patterns. Define the logit of a probability \(p\), for \(0<p<1\), as \(\operatorname{logit}(p)=\log\{p/(1-p)\}\). Calibration-in-the-large estimates an intercept while holding the model’s log-odds predictions fixed as an offset. The ideal intercept is zero. A positive intercept indicates that the outcomes occur more often overall than the predictions imply; a negative intercept indicates overprediction overall.
The calibration slope is estimated by fitting an intercept and a coefficient to the original predicted log-odds. The ideal intercept is zero and the ideal slope is one. A slope below one commonly signals predictions that are too spread out: high predicted risks tend to be too high and low predicted risks tend to be too low. A slope above one indicates a weaker spread in predictions than the observed pattern supports. These are diagnostic interpretations, not guarantees about every subgroup.
The two summaries answer related but distinct questions. Calibration-in-the-large measures an overall shift with the original log-odds scale held fixed. The joint intercept and slope assess both a shift and a change in spread. Consequently, their intercept estimates need not be the same. State which model was fitted when reporting an intercept.
Why the Calibration Intercept Has This Form
The offset form follows from the binomial likelihood. Suppose the test outcomes are independent Bernoulli observations and the calibration-in-the-large model assigns probability \(q_i\), where \(\operatorname{logit}(q_i)=\alpha+\operatorname{logit}(p_i)\). The log-likelihood, apart from constants, is \(\sum_i[Y_i\log(q_i)+(1-Y_i)\log(1-q_i)]\). Since the derivative of a Bernoulli log-likelihood with respect to its linear predictor is \(Y_i-q_i\), differentiating with respect to the shared intercept gives:
At a finite maximum-likelihood estimate, this score is zero, so \(\sum_iY_i=\sum_iq_i\). Thus the fitted probabilities in the intercept-only recalibration model sum to the number of observed events. The intercept shifts all predicted logits by the same amount until their fitted total matches the observed total. This justification assumes the stated logistic model and a finite estimate; complete separation or boundary cases can prevent an ordinary finite maximum-likelihood estimate.
Worked Example: Calculating Calibration-in-the-Large
In a hypothetical independent test set, a model assigns probability \(0.20\) to each of 10 cases. The outcomes are Bernoulli and four of the 10 cases experience the event. Calculate the calibration-in-the-large intercept and interpret its sign. Use natural logarithms.
Method. With all predictions equal, the fitted probabilities after adding intercept \(\alpha_{\mathrm{CIL}}\) are all \(\operatorname{logit}^{-1}\{\alpha_{\mathrm{CIL}}+\operatorname{logit}(0.20)\}\). The score equation from the derivation requires their mean to equal the observed rate, \(4/10=0.40\).
The logit of the original prediction is \(\log(0.20/0.80)=\log(0.25)=-1.3863\). The observed rate has logit \(\log(0.40/0.60)=\log(2/3)=-0.4055\). Therefore:
The positive estimate indicates overall underprediction in this test set. As a check, adding the intercept gives a fitted logit of \(-1.3863+0.9808=-0.4055\), whose inverse logit is \(0.40\); the recalibrated mean is therefore \(0.40\), matching the observed rate. The common mistake is to report \(0.40-0.20=0.20\) as the intercept. That difference is a probability-scale gap, whereas the calibration intercept is a log-odds-scale shift.
Calibration Is Not Ranking Performance
Ranking asks whether cases with events tend to receive higher scores than cases without events. A common ranking summary is the area under the receiver operating characteristic curve, or AUC. Calibration asks whether the numerical probabilities match event frequencies. A model can rank cases well and still systematically overstate or understate their risks; it can also have reasonable average calibration while ordering cases poorly.
Worked Example: Good Ranking With Miscalibrated Probabilities
A hypothetical test set contains 20 independent Bernoulli outcomes. Ten cases receive prediction \(0.10\), and two of those ten have events. Ten receive prediction \(0.80\), and six of those ten have events. Calculate the calibration slope and the AUC, and explain what each says.
Calibration slope. The observed rates are \(2/10=0.20\) in the low-prediction group and \(6/10=0.60\) in the high-prediction group. Their prediction logits are \(\operatorname{logit}(0.10)=-2.1972\) and \(\operatorname{logit}(0.80)=1.3863\). The observed-rate logits are \(\operatorname{logit}(0.20)=-1.3863\) and \(\operatorname{logit}(0.60)=0.4055\). With two distinct prediction values and two fitted parameters, the logistic calibration model matches these two group rates exactly. Its slope is:
The fitted intercept is \(-1.3863-0.5000(-2.1972)=-0.2877\). The slope of 0.50, rather than the ideal 1, indicates that the predictions are too extreme relative to these observed rates. In fact, the fitted probabilities at the two prediction values are 0.20 and 0.60.
AUC. There are 8 events and 12 non-events, giving \(8(12)=96\) event–non-event pairs. Each of the 6 high-score events outranks all 8 low-score non-events, contributing \(6(8)=48\) correctly ordered pairs. Within the high-score group, 6 events and 4 non-events are tied, contributing \(6(4)/2=12\) equivalent correctly ordered pairs. Within the low-score group, 2 events and 8 non-events are tied, contributing \(2(8)/2=8\). Thus the AUC is \((48+12+8)/96=68/96=0.7083\).
The AUC indicates better-than-chance ranking in this example, while the calibration slope indicates a substantial spread problem. Neither result contradicts the other: they measure different properties. The common mistake is to treat a respectable AUC as evidence that a predicted probability such as 0.80 is accurate. Ranking alone does not establish that.
Practical Interpretation and Common Traps
For a tuned binomial glmnet fit in R, obtain test-set probabilities on the response scale, for example with a prediction call using type = "response". Compare those probabilities with the test outcomes, not with fitted values from the data used to estimate the model. Calibration intercept and slope can be estimated with binomial logistic regressions: an intercept-only fit with the prediction logit as an offset for calibration-in-the-large, and a fit with the prediction logit as a predictor for the calibration slope. If probabilities are exactly zero or one, their logits are infinite; any numerical clipping used before taking logits should be stated and applied consistently. Clipping changes the diagnostic inputs slightly.
A calibration plot is most informative when it shows the reference line, the observed points, and the amount of data supporting each point. A smooth curve can show broad patterns, but its appearance depends on the smoothing method and sample size. Do not interpret small deviations from the diagonal as proof of a systematic defect, particularly with few events. Conversely, an overall event rate that matches the overall mean prediction can conceal offsetting errors among low- and high-risk groups.
- Do not confuse calibration with discrimination. AUC concerns ordering; calibration concerns probability accuracy.
- Do not ignore group sizes. A rate based on a few cases can move sharply when one outcome changes.
- Do not omit the evaluation population. Calibration is assessed for the population represented by the evaluated data and may differ elsewhere.
- Do not present one summary as a complete diagnosis. A global intercept or slope can miss local departures visible in a plot.
In an actuarial analysis, calibrated risks matter when predicted probabilities feed into expected costs, pricing, or operational decisions. But a calibration result is evidence about agreement on an evaluation sample, not a guarantee of future performance. Report the evaluation data, the summary or plot construction, and material limitations. This diagnostic practice is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each hypothetical problem, treat outcomes as independent Bernoulli variables and use only the stated test-set results. Answers use probabilities to four decimals where applicable.
- A test set has 20 cases with mean predicted probability 0.15 and 5 observed events. Find the observed event rate and the observed-minus-predicted gap. (Answer: observed rate \(=5/20=0.2500\); gap \(=0.2500-0.1500=0.1000\), indicating underprediction overall.)
- A calibration table has two bins: 20 cases with mean prediction 0.10 and 4 events, and 30 cases with mean prediction 0.40 and 9 events. Find each observed rate and the case-weighted mean absolute calibration gap. (Answer: observed rates \(0.2000\) and \(0.3000\); weighted gap \(=(20/50)(0.10)+(30/50)(0.10)=0.1000\).)
- Each of 40 test cases has prediction 0.25, and 14 experience the event. Calculate the calibration-in-the-large intercept using natural logarithms. (Answer: \(\operatorname{logit}(0.35)-\operatorname{logit}(0.25)=\log(21/13)=0.4796\), a positive intercept indicating underprediction.)
- Two equally sized prediction groups have predictions 0.20 and 0.80, with observed event rates 0.30 and 0.70. Calculate the logistic calibration slope. (Answer: \([\operatorname{logit}(0.70)-\operatorname{logit}(0.30)]/[\operatorname{logit}(0.80)-\operatorname{logit}(0.20)]=1.6946/2.7726=0.6111\); the probabilities are too spread out relative to the observed rates.)
- A model has a high AUC on a held-out set, but its calibration plot lies consistently below the diagonal. Interpret the two findings together. (Answer: the model can rank cases well, while its predictions exceed observed event rates across the displayed risk range; high AUC does not imply calibration.)