Why Calibration Matters
A predictive interval is useful only if its stated uncertainty is credible in repeated use. If a model reports 90% intervals for future claim counts, then, across many comparable forecasts made under the same conditions, approximately 90% of the realized counts should fall inside their respective intervals. A single miss does not establish a problem; a persistent pattern of misses may indicate that uncertainty is being understated or the forecasts are systematically misplaced.
Calibration is evaluated by comparing intervals with outcomes that occur after the forecasts are made. The forecast information and interval must be recorded before observing the future outcome. Otherwise, the assessment can be contaminated by information that was not available when the prediction was produced.
For counts, interval endpoints are usually integers, and an outcome at either endpoint is included when the interval is written with inclusive bounds. For losses, endpoints and outcomes must be expressed in the same units. The comparison is case by case: each actual future outcome is checked against the interval made for that case, not against one pooled interval.
A nominal target such as 90% is a probability statement about the prediction procedure over repeated outcomes under its assumptions. It does not guarantee that exactly 9 of every 10 outcomes in a small sample will be covered. In addition, discrete outcomes can make it impossible for an interval to attain exactly its nominal coverage. If the interval procedure has a known actual predictive coverage different from its nominal label because of discreteness, that fact should be considered when choosing the benchmark.
Measuring Coverage and Its Uncertainty
The basic calibration statistic is the proportion of observed outcomes that fall inside their forecast intervals. A useful first comparison is the difference between empirical coverage and the target, measured in percentage points. Also count outcomes below the lower endpoint and above the upper endpoint separately. Those two miss types can reveal directional problems that the overall coverage rate conceals.
A simple uncertainty calculation is available when the \(m\) cases are independent and comparable, and each interval has the same true coverage probability \(q\). Under those assumptions, the number \(K\) of covered outcomes has a binomial distribution with parameters \(m\) and \(q\). The standard error of empirical coverage under the target is then \(\sqrt{q(1-q)/m}\). This measures the ordinary sampling variation expected even if the target coverage is correct.
The standard error is a diagnostic, not a universal pass-or-fail rule. If forecasts concern dependent observations, or their true coverage probabilities differ substantially, the simple binomial calculation need not apply. For independent cases with different probabilities \(p_j=P(I_j=1)\), the expected empirical coverage is \(\frac{1}{m}\sum_j p_j\), not necessarily one common \(q\). In practice, dependence and differences in case mix may also make a pooled result less informative.
Why the Binomial Calibration Check Works
The binomial check follows directly from the coverage indicators. If each case has coverage probability \(q\), then \(E[I_j]=q\). Because an indicator is Bernoulli, its variance is \(q(1-q)\). With independent cases, variances add when the indicators are summed. Dividing the sum by \(m\) to form the empirical proportion divides its variance by \(m^2\).
Taking the square root of the variance gives the stated standard error. This derivation depends on equal coverage probability and independence. If cases share features that create dependence, or their interval procedures have different coverage probabilities, the variance expression must be reconsidered rather than applied mechanically.
Worked Example: Checking Nominal 90% Claim-Count Intervals
A hypothetical insurer evaluates 20 independent, comparable claim-count forecasts. Each forecast is stated to be a 90% predictive interval. Sixteen realized counts are inside their intervals, two are below the lower endpoints, and two are above the upper endpoints. Assess coverage and its sampling uncertainty.
Method. Count covered outcomes and divide by the number of evaluated forecasts. Then compare with the 90% target and calculate the binomial standard error under the stated independence and comparability assumptions.
Step 1: Empirical coverage. There are 16 covered outcomes out of 20, so \(C_{20}=16/20=0.80\), or 80%.
Step 2: Difference from target. The observed coverage is \(80\%-90\%=-10\) percentage points. There are four misses: two below and two above, so neither tail accounts for more misses in this small sample.
Step 3: Sampling uncertainty. Under a correct 90% target, the standard error is \(\sqrt{0.90(0.10)/20}=\sqrt{0.0045}\approx0.0671\), or 6.71 percentage points. The standardized difference is \((-0.10)/0.0671\approx-1.49\).
The observed rate is below target, but the difference is about 1.49 standard errors from the target under this approximation. This small set of forecasts is not, by itself, compelling evidence of miscalibration. The calculation checks the arithmetic: 16 covered plus 2 below plus 2 above equals all 20 cases.
Common mistake avoided: Calling one or more misses proof of undercoverage. A 90% interval procedure is expected to miss some outcomes; calibration must be assessed across repeated cases and interpreted with sampling uncertainty.
Tail Direction and Case Mix
Coverage summarizes whether outcomes fell inside intervals, but it does not show how misses occurred. If losses repeatedly exceed upper endpoints, the forecast may be missing high outcomes or understating upper-tail risk. If outcomes frequently fall below lower endpoints, the forecasts may be centered too high or have an inappropriate lower range. These patterns are diagnostic clues, not automatic explanations: investigate the model and the evaluated cases before assigning a cause.
A pooled coverage rate can also hide differences between groups. For instance, intervals could perform well for one class of risks and poorly for another while producing an apparently acceptable overall rate. Assessing coverage by relevant, pre-specified groups can help identify this pattern. Avoid creating many small groups after seeing results: their rates are noisy, and repeated searching can produce chance extremes.
Worked Example: Diagnosing an 80% Interval for Claim Counts
A hypothetical portfolio team reviews 30 independent claim-count forecasts, each with an 80% predictive interval. The realized count is inside 21 intervals, below the lower endpoint for 5, and above the upper endpoint for 4. Calculate empirical coverage, compare it with target, and interpret the misses.
Step 1: Calculate coverage. \(C_{30}=21/30=0.70\), or 70%.
Step 2: Compare with target. The difference is \(70\%-80\%=-10\) percentage points. There are \(5+4=9\) misses, and \(21+9=30\), confirming that all cases are accounted for.
Step 3: Check expected sampling variation. Under the independent, comparable, common-coverage assumptions, the standard error is \(\sqrt{0.80(0.20)/30}=\sqrt{0.005333\ldots}\approx0.0730\), or 7.30 percentage points. The standardized difference is \((-0.10)/0.0730\approx-1.37\).
Coverage is below target, but the observed difference is only about 1.37 standard errors under this approximation. The five lower misses and four upper misses are fairly balanced, so this sample does not show a strong directional pattern. A larger set of comparable forecasts, together with checks for case mix and dependence, would provide more evidence.
Common mistake avoided: Reporting \(9/30=30\%\) as coverage. That is the miss rate. Coverage is \(1-9/30=21/30=70\%\).
Worked Example: Finding a Subgroup Problem in Loss Intervals
A hypothetical insurer evaluates 50 independent forecasts of future losses, all stated to be 90% predictive intervals. For a lower-risk group of 25 risks, 24 losses are covered, 1 is below its interval, and none is above. For a higher-risk group of 25, 20 losses are covered, 3 are below, and 2 are above. Assess pooled and subgroup coverage.
Step 1: Pool the results. There are \(24+20=44\) covered losses among 50, so pooled coverage is \(44/50=0.88\), or 88%. The pooled miss count is \(1+3=4\) below and \(0+2=2\) above; \(44+4+2=50\).
Step 2: Compare pooled coverage with target. The difference is \(88\%-90\%=-2\) percentage points. The standard error under a common 90% target is \(\sqrt{0.90(0.10)/50}=\sqrt{0.0018}\approx0.0424\), or 4.24 percentage points. The standardized difference is \((-0.02)/0.0424\approx-0.47\).
Step 3: Inspect the groups. Lower-risk group coverage is \(24/25=96\%\). Higher-risk group coverage is \(20/25=80\%\). Thus, the pooled 88% result conceals a 16-percentage-point difference between groups. Each subgroup rate is based on only 25 cases, so neither should be treated as a precise estimate on its own.
The pooled result is close to 90%, but that does not establish that forecasts are equally calibrated across groups. The higher-risk group warrants further review; the observed contrast is a diagnostic signal, not proof of its cause.
Common mistake avoided: Concluding that every group is calibrated because the pooled rate is near 90%. Aggregation can conceal meaningful differences in forecast performance.
Interpreting Calibration Without Overclaiming
A calibration assessment supports a conclusion about the observed forecasts and the evaluation design, not a guarantee about every future case. Before interpreting results, verify that outcomes were evaluated against intervals made for the correct future period, that the interval endpoints and observed outcomes use consistent units, and that cases are sufficiently comparable for the chosen benchmark.
Coverage alone is not a complete measure of forecast quality. Very wide intervals can achieve high coverage while providing limited information. Conversely, narrow intervals may be useful but miss too often. Calibration should therefore be interpreted alongside interval width and the purpose of the forecast; high coverage does not, by itself, establish that intervals are informative.
A disciplined review proceeds from the overall rate to more detailed diagnostics. First calculate empirical coverage and its difference from the target. Next examine lower- and upper-tail misses. Then consider whether relevant groups behave differently and whether dependence or unequal case mix undermines a simple binomial calculation. Only after these checks should a pattern be described as evidence of undercoverage or overcoverage.
Use each interval with the future outcome it was designed to predict, and confirm that the forecast was fixed before that outcome was observed.
Count outcomes inside, below, and above intervals; confirm that the three counts sum to the number of cases.
Use the nominal target when appropriate, while recognizing that discrete interval procedures may have actual coverage different from the label.
Use a binomial standard error only when its assumptions are reasonable, and inspect important groups before drawing an overall conclusion.
Check Your Understanding
For each problem, calculate the requested coverage diagnostic and interpret it in context.
- A hypothetical insurer evaluates 18 independent, comparable 90% claim-count intervals. Sixteen observed counts fall inside their intervals. Calculate empirical coverage and its difference from target. (Answer: 88.9% coverage, 1.1 percentage points below target.)
- Of 50 independent, comparable 80% predictive intervals for future claim counts, 37 contain the observed counts. Calculate empirical coverage and the percentage-point difference from target. (Answer: 74.0% coverage, 6.0 percentage points below target.)
- For 40 independent, comparable 95% loss intervals, 2 losses are below their intervals and 1 is above. Find coverage and the lower- and upper-tail miss rates. (Answer: 37/40 = 92.5% coverage; lower-tail miss rate 5.0%; upper-tail miss rate 2.5%.)
- There are 60 independent forecasts with a common 90% target; 52 outcomes are covered. Calculate empirical coverage, the standard error under the target, and the standardized difference. (Answer: 86.7% coverage; standard error \(\sqrt{0.9(0.1)/60}\approx3.87\) percentage points; standardized difference approximately \(-0.86\).)
- A hypothetical book has two groups of 20 independent risks each, with 90% intervals. Group A has 19 covered outcomes; Group B has 15. Calculate pooled coverage and each group’s coverage. What feature of the results should be investigated? (Answer: pooled coverage is 34/40 = 85%; Group A is 95% and Group B is 75%. The 20-percentage-point subgroup difference merits investigation, while recognizing the small group sizes.)