Why Performance Measures Need Uncertainty Estimates
An experience-rating plan can appear accurate, well calibrated, or stable in one observed portfolio and perform differently in another. A reported value alone does not show how much of the result might be due to which risks happened to be observed. For example, a mean prediction error of a particular size may change materially if a few policies with unusually large losses are replaced by other policies from the same population.
A bootstrap uses repeated samples from the observed data to estimate the sampling variability of a statistic. It can be applied to an accuracy measure, a calibration ratio, or a renewal stability measure, provided the resampling unit reflects how the data were generated. The method estimates uncertainty; it does not establish that the plan is adequate, fair, or causally responsible for an observed result. This approach is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
The resampling unit is central. If each observation is a policy, sample policies with replacement and retain all fields for each selected policy. If one policy contributes several years, claims, or paired renewal records, those records are not automatically independent observations. Resampling individual rows can break the within-policy relationship and understate uncertainty. In that case, resample whole policies or other independent clusters, keeping their linked records together.
Bootstrap Procedure and Interpretation
Let the observed independent units be \(Z_1,\ldots,Z_n\), and let the performance statistic be \(T(Z_1,\ldots,Z_n)\). The observed estimate is \(\hat{\theta}=T(Z_1,\ldots,Z_n)\). For each bootstrap replicate, draw \(n\) units with replacement from the observed units and calculate the same statistic. Denote the result in replicate \(b\) by \(\hat{\theta}^{*(b)}\), for \(b=1,\ldots,B\).
State what the measure means, such as a difference in mean squared prediction error or an observed-to-predicted loss ratio.
Keep paired predictions and outcomes, or repeated records within a policy, together when their dependence is part of the data structure.
Draw \(n\) units with replacement and recompute the complete statistic. Repeat many times, using the same definition and any required fitting steps each time.
Use its spread to assess uncertainty and, when appropriate, its quantiles to form an interval. Report the method and the unit resampled.
A common percentile interval with confidence level \(1-\alpha\) uses the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the bootstrap replicates. For instance, a 90% percentile interval uses the 5th and 95th percentiles. With very few replicates, quantiles are coarse; the examples below use five deliberately small sets of replicates to make the arithmetic visible, not to suggest that five is adequate for operational analysis.
The rationale is a plug-in approximation. The unknown population distribution \(F\) generates the observed sample, and the statistic's sampling variability depends on \(F\). The empirical distribution \(F_n\), which assigns probability \(1/n\) to each observed unit, estimates \(F\). Drawing bootstrap samples from \(F_n\) approximates taking new samples from \(F\); recalculating the statistic therefore approximates its sampling distribution. This justification is strongest when the observed independent units represent the target population and the statistic behaves regularly as sample size increases. It does not repair selection bias, incorrect outcome definitions, or an unrepresentative sample.
Worked Examples
Worked Example: Uncertainty in Comparative Prediction Accuracy
A hypothetical insurer compares an experience-rating plan with a baseline predictor on three independent held-out policies. Losses and predictions are in thousands of dollars. Observed losses are 100, 140, and 80. Plan predictions are 110, 120, and 100; baseline predictions are 100, 150, and 90. Define the accuracy statistic as the plan's mean squared error minus the baseline's mean squared error. Calculate the estimate and an 80% percentile interval from the five specified policy bootstrap samples below. A positive statistic means the plan has greater squared error.
Method and observed estimate. Keep each policy's observed loss and both predictions together. For policy 1, plan squared error is \((110-100)^2=100\), while baseline squared error is \((100-100)^2=0\); the difference is 100. For policy 2, the errors are \((120-140)^2=400\) and \((150-140)^2=100\), giving a difference of 300. For policy 3, the errors are \((100-80)^2=400\) and \((90-80)^2=100\), also giving a difference of 300. The observed mean difference is
Thus, the plan's mean squared error exceeds the baseline's by 233.33 thousand-dollars-squared on these three policies.
Bootstrap replicates. Resample three policy records with replacement. The specified index samples and calculations are:
- \((1,1,2)\): \((100+100+300)/3=166.67\).
- \((1,2,3)\): \((100+300+300)/3=233.33\).
- \((2,2,3)\): \((300+300+300)/3=300.00\).
- \((1,3,3)\): \((100+300+300)/3=233.33\).
- \((1,1,1)\): \((100+100+100)/3=100.00\).
Sorted, the replicates are 100.00, 166.67, 233.33, 233.33, and 300.00. Using the nearest-rank empirical quantile convention, the 10th percentile is the first value and the 90th percentile is the fifth value, so the illustrative 80% percentile interval is [100.00, 300.00]. It lies above zero, consistent with larger plan squared error in this small sample. The interval is very coarse because it is based on only five replicates and three policies; it is not a reliable production interval.
Check and common mistake. The sign is plan minus baseline, so positive values indicate worse plan accuracy under this measure. A common error is to compare the plan's predictions with the baseline predictions and omit the observed losses. That does not calculate predictive error. The calculation above checks by first obtaining each prediction's squared error against the same observed loss, then comparing those errors policy by policy.
Worked Example: Bootstrap Uncertainty in Calibration
A hypothetical plan has four independent policies with observed losses of 90, 110, 80, and 120 and corresponding predicted losses of 100 for each policy. Define the calibration ratio as total observed loss divided by total predicted loss; a ratio of 1 is the calibration target. Calculate the estimate and an 80% percentile interval from the five stated bootstrap samples, each containing four policies.
Observed estimate. Total observed loss is \(90+110+80+120=400\). Total predicted loss is \(100+100+100+100=400\). Therefore, the observed calibration ratio is \(400/400=1.00\).
Bootstrap calculations. Resample whole observed-predicted policy pairs. For sample \((1,1,2,2)\), observed total is \(90+90+110+110=400\), predicted total is 400, and the ratio is 1.00. For \((1,2,3,3)\), totals are 360 and 400, giving 0.90. For \((2,2,4,4)\), totals are 460 and 400, giving 1.15. For \((1,3,4,4)\), totals are 410 and 400, giving 1.025. For \((3,3,3,4)\), totals are 360 and 400, giving 0.90.
The sorted ratios are 0.90, 0.90, 1.00, 1.025, and 1.15. With the nearest-rank convention, the 10th and 90th percentiles give an 80% interval of [0.90, 1.15]. It includes the target of 1.00. That is consistent with calibration being plausible given this limited sample, but it does not prove exact calibration or establish that calibration holds for every policy.
Check and common mistake. The denominator is the sum of predicted losses in each resample, not the original total carried into every replicate. In this example the denominator happens to remain 400 because every policy prediction is 100 and every sample contains four policies. With varying predictions, both totals must be recomputed for each replicate. Resampling observed losses separately from their predictions would destroy their pairing and produce a different, generally inappropriate uncertainty estimate.
Worked Example: Uncertainty in Renewal Stability
For four hypothetical policies, the experience modifications in consecutive renewals change by \(-2\), \(0\), \(4\), and \(6\) percentage points. Define a stability measure as the mean absolute change in percentage points; lower values mean greater stability. Calculate the estimate, the bootstrap standard error, and an 80% percentile interval using these five resamples of four policies: \((-2,-2,0,0)\), \((-2,0,4,6)\), \((0,0,4,4)\), \((-2,4,6,6)\), and \((6,6,6,6)\).
Observed estimate. The absolute changes are 2, 0, 4, and 6 percentage points. Their mean is \((2+0+4+6)/4=12/4=3\) percentage points.
Replicate statistics. For the first sample, the mean absolute change is \((2+2+0+0)/4=1\). For the second it is \((2+0+4+6)/4=3\). For the third it is \((0+0+4+4)/4=2\). For the fourth it is \((2+4+6+6)/4=18/4=4.5\). For the fifth it is \((6+6+6+6)/4=6\). Thus the sorted bootstrap statistics are 1, 2, 3, 4.5, and 6 percentage points.
The bootstrap mean is \((1+2+3+4.5+6)/5=16.5/5=3.3\). The sample standard deviation of the replicates is the bootstrap standard error. The squared deviations from 3.3 are \(5.29, 1.69, 0.09, 1.44,\) and \(7.29\); their sum is 15.80. The sample variance is \(15.80/(5-1)=3.95\), so the standard error is \(\sqrt{3.95}=1.99\) percentage points, rounded to two decimals. The 10th and 90th nearest-rank percentiles give an 80% interval of [1, 6] percentage points.
Check and common mistake. All three summaries are in percentage points: the observed mean absolute change is 3, the replicate standard error is about 1.99, and the interval is [1, 6]. A stability measure based on absolute change must not be calculated by averaging the signed changes, which here gives \((-2+0+4+6)/4=2\). That signed average allows increases and decreases to cancel and answers a different question. In an actual paired renewal analysis, resample policies with both renewal records intact before calculating their changes.
Practical Choices, Limitations, and Exam Traps
The three examples use statistics with different interpretations. Accuracy compares prediction errors against observed outcomes. Calibration compares aggregate observed and predicted amounts. Stability measures how much a result changes across renewals. A confidence interval must therefore be interpreted in the units and direction of its own statistic. An interval for an observed-to-predicted ratio that contains 1 is not the same claim as an interval for a prediction-error difference that contains 0.
There is also a distinction between uncertainty conditional on a fitted plan and uncertainty that includes fitting the plan. If the purpose is to assess the plan as already specified, it can be appropriate to hold its parameters fixed while resampling evaluation policies. If the purpose is to estimate the variability of the complete development process, each replicate must repeat the relevant fitting and selection steps as well as the evaluation. State which target is being estimated. Otherwise, an interval can appear more precise simply because it omits a source of variation relevant to the question.
Bootstrap intervals can be unreliable when the observed sample is very small, has little representation of rare but influential outcomes, or has dependence that the resampling scheme ignores. For example, resampling individual claims as if they were independent when several claims belong to the same policyholder may produce an interval that is too narrow. A percentile interval can also be affected by bias or a strongly skewed bootstrap distribution. The method and its limitations should be reported rather than treating any interval as automatically valid.
- Do not resample predictions independently of outcomes. Their pairing is needed to evaluate prediction performance.
- Do not mix the target and the statistic. State whether zero error difference, a ratio of 1, or a small renewal change is the relevant benchmark.
- Do not interpret an interval as a probability statement about a fixed parameter. A 90% confidence procedure is designed to cover the target parameter in 90% of repeated samples under its assumptions; the calculated interval is not a 90% probability that the fixed parameter lies inside it.
- Do not infer causation from a bootstrap interval. Resampling quantifies sampling uncertainty under the data design; it does not remove confounding or selection effects.
Check Your Understanding
For each problem, identify the relevant uncertainty calculation and check the units and resampling structure.
- A plan and a benchmark have squared errors on four independent held-out policies of 8, 5, 3, and 4 thousand-dollars-squared for the plan, and 6, 7, 2, and 5 for the benchmark. Define the paired mean error difference as plan minus benchmark. What is its observed value, and what does its sign indicate? (Answer: \((8-6+5-7+3-2+4-5)/4=0\); the observed mean difference is zero, so neither has lower mean squared error on these four policies.)
- Five bootstrap calibration ratios, sorted in increasing order, are 0.94, 0.98, 1.01, 1.04, and 1.10. Using the nearest-rank convention, what is the 80% interval, and does it include the calibration target? (Answer: [0.94, 1.10]; yes, it includes 1.)
- Four policies have absolute modification changes of 1, 3, 3, and 7 percentage points. A bootstrap replicate resamples the second policy twice, the first policy once, and the fourth policy once. What is that replicate's mean absolute change? (Answer: \((3+3+1+7)/4=3.5\) percentage points.)
- Each of 40 policies has three annual records, and the performance statistic uses all three records for each policy. What should be resampled if the target is uncertainty across policies, and what must remain together? (Answer: Resample 40 policies with replacement, retaining all three annual records for each selected policy; do not resample the 120 policy-year rows independently.)
- A plan's observed-to-predicted ratio is 1.06, with a bootstrap 90% interval of [0.99, 1.13]. State the target and a cautious conclusion. (Answer: The calibration target is 1; the interval includes 1, so exact aggregate calibration is compatible with the estimated uncertainty, but the result does not prove calibration or establish calibration for each risk.)