Purpose and Structure of an Evaluation Report
An experience-rating plan should be evaluated on more than whether its modifications move in the intended direction. A concise report should show how well the plan performs on outcomes not used to set or tune it, whether predictions are calibrated across modification bands, how much modifications change at renewal, and whether results differ across relevant segments. It should also examine whether observed responses are consistent with the plan’s intended incentives.
The report is a decision aid, not a claim that the plan caused every observed outcome. Comparisons require a consistent basis: for example, losses and predictions adjusted to comparable exposure, trend, and coverage terms. The analyst should identify the evaluation period, the population, the loss measure, and any exclusions. Earlier material on standardizing exposure mix and evaluating plan changes supplies useful design tools; this report brings the resulting evidence together.
A useful report begins with a short statement of scope, followed by results, interpretation, limitations, and recommended actions. Keep units explicit, such as dollars or thousands of dollars. Explain whether a metric is account-weighted, exposure-weighted, or based on aggregate dollars; a number without its weighting convention can be misleading.
Out-of-Time Accuracy and Calibration
Two simple aggregate measures provide a readable starting point. The weighted absolute percentage error (WAPE) expresses the total absolute prediction error relative to total observed loss. The signed aggregate error shows whether predictions are high or low overall:
A positive signed error means aggregate predictions exceed observed losses; a negative value means they fall short. WAPE measures error magnitude and cannot reveal that direction by itself. Neither measure is sufficient alone: offsetting overprediction and underprediction can make signed error small even when individual predictions are poor. Use them on an untouched later-period set and report the number of risks and the total exposure alongside the results.
Calibration by modification band asks whether observed losses align with predicted losses among risks receiving similar modifications. For a band \(b\), define its calibration ratio as actual standardized loss divided by predicted standardized loss. A ratio above 1 indicates underprediction in that band; below 1 indicates overprediction. The ratio is not an account-level accuracy score and should be interpreted with the band’s volume and uncertainty in mind.
Worked Example: Testing Accuracy on a Later Period
A hypothetical insurer evaluates predictions on a later period that was not used for fitting. For four risks, standardized predicted and observed losses, in thousands of dollars, are respectively 8 and 10, 10 and 8, 12 and 15, and 20 and 17. Calculate WAPE and signed aggregate error. A training-period WAPE was 12%.
Method. Sum predictions: \(8+10+12+20=50\) thousand dollars. Sum observed losses: \(10+8+15+17=50\) thousand dollars. Absolute errors are \(2,2,3,3\), totaling \(10\) thousand dollars. Thus \(\operatorname{WAPE}=10/50=0.20\), or 20%.
The signed error is \((50-50)/50=0\), or 0%. Predictions balance in aggregate, but this does not mean the four predictions are individually accurate: the absolute error totals $10,000. The later-period WAPE is 8 percentage points higher than the training-period WAPE: \(20\%-12\%=8\%\). That gap is a reason to investigate generalization; by itself it does not identify the cause.
Check. The signed error is zero because predicted and observed totals are both $50,000, while WAPE remains positive because the absolute account-level errors sum to $10,000. A common mistake is to report the zero signed error as evidence of no prediction error.
Worked Example: Reading Calibration by Modification Band
For a hypothetical held-out portfolio, a report groups risks by their experience modification. Predicted and actual standardized losses, in thousands of dollars, are: below 0.80, 100 and 120; 0.80 to below 1.00, 240 and 228; 1.00 to below 1.25, 300 and 330; and 1.25 or above, 160 and 144. Calculate each band’s calibration ratio and the total ratio.
Method. Divide actual by predicted in each band. The ratios are \(120/100=1.20\), \(228/240=0.95\), \(330/300=1.10\), and \(144/160=0.90\). Thus observed loss is 20% above prediction in the lowest-modification band and 10% below prediction in the highest-modification band; the two middle bands have the indicated 5% overprediction and 10% underprediction.
Total predicted loss is \(100+240+300+160=800\) thousand dollars. Total actual loss is \(120+228+330+144=822\) thousand dollars. The overall ratio is \(822/800=1.0275\), or 1.03 when rounded to two decimals. This is consistent with the weighted band ratios: \((100(1.20)+240(0.95)+300(1.10)+160(0.90))/800=822/800\).
Interpretation and check. Overall calibration is close to 1, but it masks differences by band. A common mistake is to average the four ratios equally: \((1.20+0.95+1.10+0.90)/4=1.0375\). That is not the portfolio’s aggregate ratio because the bands have different predicted losses. Use the predicted-loss-weighted calculation for the aggregate result.
Renewal Stability and Segment Results
Predictive fit and renewal stability are distinct. For risks observed at two successive renewals, one simple stability measure is the mean absolute change in modification, \(\sum_j |M_{j,2}-M_{j,1}|/n\), where \(M_{j,1}\) and \(M_{j,2}\) are the two modifications and \(n\) is the number of matched risks. A report can also state the share whose absolute change meets or exceeds a specified threshold. Specify the threshold before interpreting the result and show how many risks are matched; unmatched accounts should not silently disappear from the analysis.
Modification changes should be read in context. A large change may reflect new credible experience or a change in exposure mix, not necessarily plan instability. Conversely, stable modifications do not establish that predictions are accurate. When useful, describe how many risks move between bands and whether the direction of change is consistent with the plan’s intended response.
Segment results help show whether aggregate performance conceals a material weakness. Choose segments relevant to the rating context and ensure comparisons use a comparable basis, including exposure mix where appropriate. Report each segment’s volume, predicted and actual totals, and calibration ratio. Avoid treating a segment ratio as proof that individual risks within it are treated equitably: aggregate summaries cannot establish that.
Worked Example: Combining Renewal, Segment, and Incentive Checks
A hypothetical plan is reviewed for five risks seen at two consecutive renewals. Their modifications change as follows: 0.70 to 0.75, 0.85 to 0.80, 1.00 to 1.10, 1.20 to 1.15, and 1.40 to 1.30. The report also has standardized segment totals: urban risks, predicted $400,000 and actual $440,000; rural risks, predicted $300,000 and actual $270,000. Separately, among hypothetical policyholders, those completing a recommended inspection had observed claim frequency 0.08, compared with 0.12 for those not completing it. Calculate the stated diagnostics and explain what they do and do not establish.
Renewal stability. The absolute modification changes are \(0.05, 0.05, 0.10, 0.05,\) and \(0.10\). Their sum is 0.35, so mean absolute change is \(0.35/5=0.07\). Two of five risks have a change of at least 0.10, giving \(2/5=40\%\). These results describe this matched group; they do not establish a general stability standard.
Segment calibration. For urban risks, actual divided by predicted is \(440/400=1.10\), indicating actual loss 10% above prediction. For rural risks, \(270/300=0.90\), indicating actual loss 10% below prediction. Across both segments, predicted loss is $700,000 and actual loss is $710,000, for an aggregate ratio of \(710/700=1.0143\), or 1.01 to two decimals. The near-1 aggregate masks the opposite segment deviations.
Incentive check. The observed frequency difference is \(0.12-0.08=0.04\), or 4 percentage points. Relative to the 0.12 frequency, the observed reduction is \(0.04/0.12=0.3333\), or 33.3% to one decimal place. This direction is consistent with the intended incentive if the inspection is meant to improve risk management. It does not prove that inspection caused lower frequency: the groups may differ in other ways, and selection into inspection may matter.
Check and common mistake. The overall segment ratio is \(710/700\), not the unweighted mean \((1.10+0.90)/2=1.00\), because predicted segment totals differ. Likewise, the frequency association is a diagnostic, not a causal estimate. A responsible report would retain both segment results and describe the incentive finding as suggestive unless the evaluation design supports a causal conclusion.
Incentive Evidence, Limitations, and Recommended Actions
An incentive check asks whether observed behavior and outcomes are directionally consistent with the plan’s intended response. Depending on the plan, useful evidence may include whether claim-management actions change, whether claims appear to be shifted across experience-period boundaries, or whether behavior differs after a modification change. Interpret such patterns alongside the plan’s design and the period over which behavior could respond. An observed change is not automatically caused by the plan; selection, exposure changes, random claim variation, and other concurrent changes may also explain it.
A concise report should state material limitations rather than imply more certainty than the data allow. Common limitations include a short or unusually volatile evaluation period, small segment counts, incomplete claim development, unmatched renewals, changes in coverage or exposure, and results that depend on the chosen standardization basis. A band with little predicted loss can show a volatile ratio, and a single portfolio-wide metric can hide local miscalibration.
Recommendations should follow from the evidence and be proportionate to it. For example, a persistent out-of-time accuracy deterioration supports checking the model’s inputs, implementation, and period comparability before recalibration. Repeated band-level deviations support targeted review of those bands and their volume. Large renewal movements support checking the drivers and the effect on matched risks. A suggestive incentive pattern supports further monitoring or a stronger evaluation design, not an unsupported claim of causation.
Identify the population, time period, loss definition, exposure and coverage basis, and any adjustments or exclusions.
Show out-of-time accuracy and signed bias, with counts and dollar totals. Keep error magnitude separate from aggregate direction.
Show calibration by modification band and relevant segment, then summarize matched-risk renewal changes and incentive diagnostics.
State what the design cannot establish, recommend a specific review or monitoring action, and identify what evidence would trigger a plan change.
The practical standard is not a single score that declares a plan successful. It is a transparent chain from comparable data to distinct diagnostics, qualified interpretation, and a follow-up action. That discipline makes an evaluation useful to decision-makers while keeping its conclusions within the evidence. This integrated reporting approach is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each hypothetical evaluation, calculate the requested measure and interpret it in context.
- On a held-out set, predictions total $90,000 and observed losses total $100,000. The sum of absolute prediction errors is $18,000. Calculate WAPE and signed aggregate error, using the definition in this tutorial. (Answer: WAPE = 18%; signed error = -10%, so predictions understate observed loss in aggregate.)
- A modification band has predicted standardized loss of $250,000 and actual standardized loss of $275,000. Calculate its calibration ratio and state the direction of deviation. (Answer: \(275/250=1.10\); actual loss is 10% above prediction.)
- For four matched risks, modifications change from 0.80 to 0.85, 0.95 to 0.85, 1.10 to 1.20, and 1.30 to 1.25. Find mean absolute change and the percentage with an absolute change of at least 0.10. (Answer: absolute changes are 0.05, 0.10, 0.10, and 0.05; mean = 0.30/4 = 0.075; 2/4 = 50% meet the threshold.)
- A segment has predicted loss $160,000 and actual loss $144,000. A second segment has predicted loss $240,000 and actual loss $264,000. Calculate each calibration ratio and the combined ratio; explain why an equal-weight average of the segment ratios is not the combined ratio. (Answer: ratios are 0.90 and 1.10; combined ratio is \(408/400=1.02\). The equal-weight average is 1.00, but predicted losses differ, so the aggregate ratio is predicted-loss weighted.)
- Observed claim frequencies are 0.06 among policyholders completing a safety review and 0.09 among those not completing it. Calculate the relative observed reduction using the latter group as the reference, then give one limitation on causal interpretation. (Answer: \((0.09-0.06)/0.09=0.3333\), or 33.3%; the association does not establish causation because the groups may differ in risk characteristics or selection into review.)