Why Calibration Needs Monitoring
A probability prediction is useful only if its scale continues to mean what it appears to mean. If a model assigns many cases a predicted event probability near 10%, then, across comparable cases with mature outcomes, an observed event rate substantially above 10% may indicate systematic underprediction. An observed rate below 10% may indicate overprediction. The discrepancy is a monitoring concern, not by itself proof of a particular cause.
Calibration is assessed by comparing predictions with outcomes, often within ranges of predicted probability. The comparison should be made on cases whose outcomes are available and defined consistently. When monitoring across time, compare like score ranges and interpret changes in light of changes in the population or operating setting.
The prediction \(p_i\) for case \(i\) is not automatically the actual event probability for that case. It is the model's forecast. If the actual event probability is different, then \(E[Y_i]\) differs from \(p_i\). Treating predictions as true probabilities is precisely the calibration assumption that monitoring is intended to assess.
Comparing Predictions and Outcomes by Score Range
A score range, or bin, groups cases with similar predicted probabilities. For a bin \(B\) containing \(n_B\) cases, calculate the mean predicted probability and the observed event rate:
Here \(n_B\bar p_B\) is the model-implied expected event count for the bin, while \(n_B\bar y_B\) is the actual number of observed events. The difference \(\bar y_B-\bar p_B\) is the observed-minus-predicted rate gap. A positive gap indicates that the observed rate exceeds the mean prediction (underprediction); a negative gap indicates that it falls below the mean prediction (overprediction).
The observed-to-expected ratio is another way to express the comparison:
provided the predicted total is positive. A ratio above 1 indicates more observed events than predicted; a ratio below 1 indicates fewer. A ratio of 1 does not establish that every score range is calibrated, or that predictions are calibrated for each individual case.
Worked Example: Reading Calibration Across Three Score Ranges
A hypothetical warranty service uses predicted probabilities of a claim during the next year. Once outcomes are mature, it groups 300 contracts into three score ranges. The mean predictions and event counts are: low range, 100 contracts with mean prediction 0.0200 and 4 claims; middle range, 100 contracts with mean prediction 0.1000 and 10 claims; high range, 100 contracts with mean prediction 0.3000 and 24 claims. Compare predicted and observed rates.
Method. For each range, divide the observed claim count by the number of contracts to find the observed rate. Compare it with the supplied mean prediction, then calculate the O/E ratio.
Low range. Predicted count \(=100(0.0200)=2\). Observed rate \(=4/100=0.0400\). The gap is \(0.0400-0.0200=0.0200\), or 2.00 percentage points. The O/E ratio is \(4/2=2.000\). There were twice as many claims as predicted in this range.
Middle range. Predicted count \(=100(0.1000)=10\). Observed rate \(=10/100=0.1000\). The gap is \(0.1000-0.1000=0.0000\), and O/E \(=10/10=1.000\). The aggregate rates match in this range.
High range. Predicted count \(=100(0.3000)=30\). Observed rate \(=24/100=0.2400\). The gap is \(0.2400-0.3000=-0.0600\), or -6.00 percentage points. The O/E ratio is \(24/30=0.800\). The model overpredicted claims in this range.
Check. The predicted total is \(2+10+30=42\); the observed total is \(4+10+24=38\). Across all contracts, the predicted rate is \(42/300=0.1400\), and the observed rate is \(38/300=0.1267\) to four decimal places. The overall shortfall does not describe the opposing patterns in the low and high ranges. The common mistake is to report only the overall comparison and miss those range-specific gaps.
Why the Expected Count Is Not Automatically the Actual Expected Count
For each case, \(Y_i\) is an indicator: it equals 1 if the event occurs and 0 otherwise. Under the model's forecast distribution, case \(i\) has event probability \(p_i\). The model-implied expected count is therefore the sum of its forecast probabilities. This statement describes the forecast, not necessarily the real data-generating process.
The justification is linearity of expectation: the expectation of a sum is the sum of the expectations. Independence is not required. For the actual outcome process, however, the expected event count is \(\sum_i E[Y_i]\). It equals \(\sum_i p_i\) only if the actual conditional event probability for each case equals its prediction, or if some weaker aggregate equality happens to hold.
In particular, the individual calibration condition \(E[Y_i]=p_i\) for every case is sufficient for equality of the totals, but is stronger than necessary for aggregate equality. Even when the totals agree, errors in different score ranges may cancel.
Worked Example: Calculating the Mean Prediction When Scores Differ
A hypothetical equipment-protection program reviews 50 service agreements in one score range. Twenty agreements have predicted failure probabilities of 0.0400 each, and the other 30 have predicted probabilities of 0.1000 each. There are 6 observed failures. Calculate the mean prediction, observed rate, rate gap, and O/E ratio.
Method. Sum the individual predictions first; do not treat one of the two prediction levels as representative of all 50 agreements.
The forecast expected count is \(20(0.0400)+30(0.1000)=0.8000+3.0000=3.8000\). Therefore, the mean prediction is \(3.8000/50=0.0760\). The observed rate is \(6/50=0.1200\).
The observed-minus-predicted gap is \(0.1200-0.0760=0.0440\), or 4.40 percentage points. The O/E ratio is \(6/3.8000=1.5789\) to four decimal places. In this range, the observed failure rate exceeds the forecast rate.
Check. Multiplying the mean prediction by 50 gives \(50(0.0760)=3.8000\), the same forecast expected count obtained from the two groups. The common mistake is to average 0.0400 and 0.1000 without weighting by their group sizes; that would give 0.0700 and an incorrect expected count of 3.5.
Calibration Drift Is a Change Over Time
A model can be well calibrated in an earlier period and become miscalibrated in a later one. Calibration drift refers to a change in the relationship between predicted probabilities and observed event rates. To investigate it, calculate the same comparisons by score range in multiple periods, using consistent outcome definitions and sufficiently mature labels.
Whenever practical, retain the same score-range boundaries across periods. If cutpoints change, a reported change in a range's observed rate may reflect a different mix of predictions rather than a change in calibration. Even with fixed boundaries, a broad bin can conceal variation within it. Binned comparisons are diagnostic summaries, not proof that every case with a given prediction has the same actual probability.
Worked Example: Opposing Range Errors Can Hide a Period Change
In a hypothetical portfolio, a prior period had 80 low-score cases with mean predicted event probability 0.0500 and 4 events, and 20 high-score cases with mean prediction 0.4000 and 8 events. In a later period, the same score ranges contain 80 low-score cases with 8 events and 20 high-score cases with 4 events; the mean predictions remain 0.0500 and 0.4000. Compare the two periods.
Prior period. In the low range, predicted count \(=80(0.0500)=4\); observed rate \(=4/80=0.0500\), so the gap is 0.0000 and O/E \(=4/4=1.000\). In the high range, predicted count \(=20(0.4000)=8\); observed rate \(=8/20=0.4000\), so the gap is 0.0000 and O/E \(=8/8=1.000\). The total predicted and observed counts are both \(4+8=12\).
Later period. The predictions remain 4 and 8 by range, for a total of 12. The low-range observed rate is \(8/80=0.1000\), giving a gap of \(0.1000-0.0500=0.0500\) and O/E \(=8/4=2.000\). The high-range observed rate is \(4/20=0.2000\), giving a gap of \(0.2000-0.4000=-0.2000\) and O/E \(=4/8=0.500\).
Check. The later observed total is still \(8+4=12\), so the overall observed rate remains \(12/100=0.1200\), equal to the overall predicted rate \((4+8)/100=0.1200\). Yet the low range has underprediction and the high range has overprediction. The common mistake is to conclude that calibration is unchanged merely because the portfolio totals match.
Interpreting Evidence Carefully
Observed rates fluctuate, especially when a range contains few cases or events. A gap between observed and predicted rates may reflect sampling variation, a real change in calibration, or both. Under the specific null assumption that outcomes are independent Bernoulli variables with actual event probabilities equal to their predictions, the forecast count has variance:
This variance formula requires independence; without it, covariance terms may contribute to the variance. More fundamentally, using \(p_i\) in the formula as the actual event probability assumes the calibration condition being examined. A small observed gap is not proof of calibration, and a large gap should be interpreted in context, including sample size, label maturity, and changes in data or operations.
A practical reading of a calibration comparison follows a simple sequence:
Check that the outcome definition, observation window, and label maturity are consistent across the cases and periods being compared.
Calculate the mean predicted probability, observed event rate, and preferably the predicted and observed counts for each range.
State whether observed rates exceed or fall below predictions, and whether the pattern repeats across ranges or periods. Treat this as evidence to investigate, not an automatic explanation.
Calibration monitoring complements measures of realized performance, but answers a distinct question: do predicted probability levels correspond to observed event frequencies? A model can rank cases usefully while its probability scale is off, or match an overall event rate while being wrong in important score ranges. Calibration comparisons help make those distinctions visible.
Check Your Understanding
For each problem, calculate or interpret the range-level comparison before drawing a conclusion.
- A hypothetical travel-assistance model assigns 200 trips a mean claim probability of 0.0300. There are 9 claims. Find the observed rate, rate gap, and O/E ratio. (Answer: observed rate \(=0.0450\); gap \(=0.0150\), indicating underprediction; O/E \(=1.500\).)
- A score range contains 40 accounts predicted at 0.0500 and 60 accounts predicted at 0.1500. There are 13 observed events. Find the mean prediction and observed rate. (Answer: expected count \(=2+9=11\); mean prediction \(=0.1100\); observed rate \(=0.1300\); gap \(=0.0200\), indicating underprediction.)
- A low range has 50 cases, mean prediction 0.0400, and 1 event. A high range has 50 cases, mean prediction 0.2000, and 13 events. Calculate the range-specific O/E ratios and state the direction of error. (Answer: low predicted count \(=2\), O/E \(=1/2=0.500\), so overprediction; high predicted count \(=10\), O/E \(=13/10=1.300\), so underprediction.)
- In an earlier period, a range had 100 cases, mean prediction 0.0800, and 8 events. In a later period it has 100 cases, the same mean prediction, and 14 events. Find both observed rates and describe the change. (Answer: earlier rate \(=0.0800\), later rate \(=0.1400\); the later gap is \(0.0600\), indicating underprediction in that period and a deterioration in range-level calibration relative to the earlier match.)
- For two score ranges, the forecast expected counts are 6 and 14, and the observed counts are 10 and 10. What do the aggregate and range-level comparisons show? (Answer: both aggregate totals equal 20, so the overall O/E is 1.000; range 1 has O/E \(=10/6=1.6667\), indicating underprediction, while range 2 has O/E \(=10/14=0.7143\), indicating overprediction. The aggregate match conceals opposing range errors.)