Calibration Diagnostics by Modification Band

Why Check Calibration by Modification Band?

An experience modification is intended to distinguish risks with different expected losses. A useful diagnostic is therefore to group risks by their indicated modifications and ask whether each group’s later observed losses are consistent with its predicted losses. If a band repeatedly produces more loss than the plan predicted, its predictions may be too low for that band; if it produces less, they may be too high.

This is a calibration check: it compares predicted amounts with observed outcomes at a chosen grouping level. It does not, by itself, tell whether the plan ranks individual risks correctly, whether the overall rate level is adequate, or whether a difference is statistically persuasive. Those are distinct questions. In particular, calibration by band is most informative when the modification bands are set before the later outcomes are examined.

Definition: A modification band is a specified interval of indicated experience modifications, such as $0.70 \leq M < 0.80$. A risk is assigned to a band using its indicated modification $M$, before its later observed loss is used in the diagnostic. For a band, predicted loss $P$ is the sum of the risks’ predicted losses on a stated basis, and observed loss $O$ is the sum of their later losses on the same basis.

Band-Level Calibration Measures

For each band, calculate both the dollar difference and the observed-to-predicted ratio. The dollar difference describes the size and direction of the aggregate deviation. The ratio puts the deviation in relation to the amount predicted.

\text{Dollar difference}=O-P,\qquad \text{O/P ratio}=\frac{O}{P},\qquad P>0.

An O/P ratio greater than 1 means observed loss exceeded predicted loss, so the band was underpredicted in this comparison. A ratio below 1 means observed loss was lower than predicted, so the band was overpredicted. A ratio of 1 means the aggregate amounts agree. For example, an O/P ratio of 1.08 represents observed loss 8% above predicted loss; it does not mean the modification itself should automatically be increased by 8%.

If a risk’s prediction is calculated from an expected loss before modification, one possible form is $\text{predicted loss}=\text{unmodified expected loss}\times M$. The appropriate prediction depends on the plan and the evaluation target. Whatever method is used, apply it consistently, and define whether amounts are per risk, per exposure unit, or totals. Compare losses for the same coverage, time basis, and loss definition. For example, do not compare a prediction at one cost level with unadjusted observed losses from a different cost level.

Interpretation: In a band, $O/P>1$ indicates aggregate underprediction and $O/P<1$ indicates aggregate overprediction for the comparison being made. These are descriptive findings. Random variation, changes in exposure, claim development, and differences between the predicted and observed loss bases can affect the result; the ratio alone does not establish why a deviation occurred.

Why the Ratio Is a Calibration Diagnostic

The justification follows directly from the meaning of a prediction. Suppose the predicted amounts $P_i$ for risks in a band are their expected losses for the later observation period, measured on the same basis as observed losses $O_i$. If those predictions are calibrated, then, in expectation, the total observed loss equals the total predicted loss. The ratio compares these totals.

E\left[\sum_i O_i\right]=\sum_i E[O_i] =\sum_i P_i \quad\Longrightarrow\quad E[O]=P.

Thus the ratio $O/P$ is centered around 1 when predictions are calibrated in this aggregate sense. One observed ratio need not equal 1, because actual losses vary around their expectations. The ratio is a diagnostic for systematic patterns across bands or evaluation periods, not a guarantee that any single band will realize its predicted amount.

When combining bands, use total observed loss divided by total predicted loss. This is equivalent to averaging band ratios with weights proportional to predicted losses, not to taking a simple average of the ratios. A band with a small predicted amount should not automatically count as much as a band with a large predicted amount in the portfolio-wide result.

\frac{\sum_b O_b}{\sum_b P_b} = \sum_b \left(\frac{P_b}{\sum_j P_j}\right)\frac{O_b}{P_b}.

The equality holds because each weighted ratio is $(P_b/\sum_jP_j)(O_b/P_b)=O_b/\sum_jP_j$, and summing gives $\sum_b O_b/\sum_jP_j$. The weights are nonnegative and sum to 1 when the predicted amounts are positive. This explains why a simple average of band ratios generally answers a different question.

Worked Band Comparisons

Worked Example: Calculating Calibration for Two Bands

A hypothetical insurer assigns experience modifications to risks before a later observation period. Each exposure unit has an unmodified predicted loss of $1,000. For three risks in the $0.70 \leq M < 0.80$ band, the modifications are 0.70, 0.75, and 0.78; their exposure units are 2, 1, and 3. Their later observed losses are $1,600, $900, and $2,700. In a second band, $0.80 \leq M < 1.00$, the modifications are 0.85, 0.92, and 0.98; exposure units are 2, 1, and 2, and observed losses are $1,500, $1,100, and $2,200. Calculate each band’s calibration measures and the combined ratio.

Method and prediction basis. For each risk, predicted loss equals exposure units multiplied by $1,000 and by the indicated modification. The later observed losses are assumed to be on the same coverage and cost basis.

First band predicted losses. They are $2\times \$1{,}000\times0.70=\$1{,}400$, $1\times \$1{,}000\times0.75=\$750$, and $3\times \$1{,}000\times0.78=\$2{,}340$. Therefore $P_1=\$1{,}400+\$750+\$2{,}340=\$4{,}490$. Observed loss is $O_1=\$1{,}600+\$900+\$2{,}700=\$5{,}200$.

The difference is $O_1-P_1=\$5{,}200-\$4{,}490=\$710$, and the ratio is $O_1/P_1=\$5{,}200/\$4{,}490=1.1581$. This band’s observed loss is about 15.81% above its prediction, indicating underprediction in this comparison.

Second band predicted losses. They are $2\times \$1{,}000\times0.85=\$1{,}700$, $1\times \$1{,}000\times0.92=\$920$, and $2\times \$1{,}000\times0.98=\$1{,}960$. Thus $P_2=\$4{,}580$, while $O_2=\$1{,}500+\$1{,}100+\$2{,}200=\$4{,}800$. The difference is $\$220$, and the ratio is $4{,}800/4{,}580=1.0480$, also indicating underprediction.

Combined result. Total predicted loss is $\$4{,}490+\$4{,}580=\$9{,}070$; total observed loss is $\$5{,}200+\$4{,}800=\$10{,}000$. The combined ratio is $10{,}000/9{,}070=1.1025$, with an aggregate excess of $\$930$. The arithmetic checks: the two band differences, $\$710+\$220$, equal $\$930$.

Common mistake avoided: Do not calculate the combined ratio by giving the two bands equal weight. The correct calculation uses total observed divided by total predicted loss.

Worked Example: Why Predicted-Loss Weights Matter

A hypothetical equipment insurer reviews two modification bands. In the lower band, three risks have predicted losses of $4,000, $2,000, and $6,000, with later observed losses of $3,600, $2,500, and $7,200. In the higher band, predicted losses are $3,000, $9,000, and $3,000, while observed losses are $3,300, $8,100, and $4,200. Calculate the band ratios and the portfolio ratio, and compare the correct aggregate result with simple averaging.

Lower band. Predicted total is $P_L=\$4{,}000+\$2{,}000+\$6{,}000=\$12{,}000$. Observed total is $O_L=\$3{,}600+\$2{,}500+\$7{,}200=\$13{,}300$. Therefore $O_L/P_L=13{,}300/12{,}000=1.1083$, an aggregate underprediction of $\$1{,}300$.

Higher band. Predicted total is $P_H=\$3{,}000+\$9{,}000+\$3{,}000=\$15{,}000$. Observed total is $O_H=\$3{,}300+\$8{,}100+\$4{,}200=\$15{,}600$. Thus $O_H/P_H=15{,}600/15{,}000=1.0400$, an aggregate underprediction of $\$600$.

The combined predicted total is $\$12{,}000+\$15{,}000=\$27{,}000$, and the combined observed total is $\$13{,}300+\$15{,}600=\$28{,}900$. The correct combined ratio is $28{,}900/27{,}000=1.0704$, with observed loss $\$1{,}900$ above predicted loss.

The simple average of the two band ratios is $(1.1083+1.0400)/2=1.0742$, which differs from the combined ratio. The correct predicted-loss weights are $12{,}000/27{,}000=0.4444$ for the lower band and $15{,}000/27{,}000=0.5556$ for the higher band. Applying these weights to the unrounded band ratios gives $1.0704$, as required.

Common mistake avoided: Averaging ratios equally gives each band equal influence even though their predicted loss totals differ. Use total observed divided by total predicted when the target is aggregate portfolio calibration.

Worked Example: Adjusting Observed Losses to a Common Cost Level

A hypothetical insurer evaluates one modification band. The predicted losses are $50,000, $30,000, and $20,000. Later observed losses are recorded at an earlier cost level as $44,000, $29,000, and $19,000. Assume a stated trend factor of 1.10 applies to every observed amount to bring it to the prediction’s cost level. Calculate the adjusted O/P ratio and explain why the unadjusted comparison is misleading.

Method. Apply the stated factor to each observed loss before summing. The adjusted amounts are $\$44{,}000\times1.10=\$48{,}400$, $\$29{,}000\times1.10=\$31{,}900$, and $\$19{,}000\times1.10=\$20{,}900$.

The adjusted observed total is $\$48{,}400+\$31{,}900+\$20{,}900=\$101{,}200$. The predicted total is $\$50{,}000+\$30{,}000+\$20{,}000=\$100{,}000$. Therefore the adjusted ratio is $101{,}200/100{,}000=1.0120$, and the adjusted difference is $\$1{,}200$. The band is close to aggregate calibration on this adjusted basis, with observed loss 1.20% above prediction.

Without adjustment, observed loss totals $\$44{,}000+\$29{,}000+\$19{,}000=\$92{,}000$, producing a ratio of $0.9200$. That comparison would suggest substantial overprediction, but it mismatches the observed cost level and the prediction basis. The calculation checks because multiplying the raw total, $\$92{,}000$, by 1.10 gives $\$101{,}200$.

Common mistake avoided: Comparing observed losses at one cost level directly with predictions at another can create an apparent calibration error. Apply the stated adjustment consistently, and do not introduce an adjustment unless the evaluation basis calls for it.

Design and Interpretation Traps

The diagnostic is only as clear as its setup. Define the bands and the loss basis before looking at later outcomes. Use the modification indicated at the rating or evaluation date, rather than assigning risks using information from the observation period. Otherwise, outcomes can influence group membership and make the comparison difficult to interpret.

Also make the denominators and time periods consistent. If predictions are totals for earned exposure over a period, the observed amounts should correspond to that exposure and period. State whether observed loss means paid, incurred, or another defined measure, and treat development consistently across the risks. Where a trend or other adjustment is specified, apply it before comparing totals.

A band ratio near 1 is not proof that the modification plan is sound in every respect. A band may contain risks with differing characteristics, and a single period can produce unusual losses. Conversely, a ratio far from 1 is a reason to investigate the prediction basis, exposure records, claims treatment, and persistence of the pattern; it is not, by itself, a quantified confidence statement or an automatic factor change. Calibration describes agreement of aggregate predictions and observations, not the ordering of risks within a band.

Wrong calculation: average the individual or band O/P ratios equally when the intended measure is aggregate calibration.
Correction: divide total observed loss by total predicted loss, or use predicted-loss weights.
Wrong comparison: compare totals on different exposure, time, or cost bases.
Correction: align the loss definitions and stated adjustments before interpreting the ratio.
Wrong conclusion: treat one ratio above 1 as proof that every risk in the band was underpriced.
Correction: describe it as aggregate underprediction for the evaluated band and investigate whether the pattern persists.

Key takeaway: Assign risks to modification bands using indicated modifications determined before later outcomes. Sum predictions and comparable observed losses within each band, then calculate $O/P$ and $O-P$. Interpret ratios above or below 1 as aggregate under- or overprediction for that comparison, and weight bands by predicted loss when calculating an overall ratio.

Check Your Understanding

For each problem, calculate on the stated basis and interpret the result.

A modification band has predicted loss of $8,000 and later observed loss of $9,200. Calculate the O/P ratio and state the direction and percentage of the deviation. (Answer: O/P $=1.1500$; underprediction of 15.00%.)
Band A has predicted and observed losses of $5,000 and $4,500. Band B has predicted and observed losses of $10,000 and $11,000. Calculate each band ratio and the combined ratio. (Answer: Band A $=0.9000$, Band B $=1.1000$, combined $=15{,}500/15{,}000=1.0333$.)
Two risks have predicted losses of $2,000 and $6,000 and observed losses of $2,600 and $5,700. Calculate the aggregate O/P ratio and compare it with the simple average of the individual ratios. Which is the appropriate aggregate calibration measure? (Answer: Aggregate $=8{,}300/8{,}000=1.0375$; simple average $=(1.3000+0.9500)/2=1.1250$; use the aggregate ratio $1.0375$.)
A band has predicted loss of $40,000. Its raw observed loss is $37,000 at an earlier cost level, and the stated factor to the prediction level is 1.08. Calculate adjusted observed loss and O/P. (Answer: Adjusted observed loss $=\$39{,}960$; O/P $=39{,}960/40{,}000=0.9990$.)
Band A has predicted loss of $12,000 and observed loss of $10,800. Band B has predicted loss of $18,000 and observed loss of $20,700. Calculate the band ratios and combined ratio, then describe each band’s direction. (Answer: Band A $=0.9000$, overprediction; Band B $=1.1500$, underprediction; combined $=31{,}500/30{,}000=1.0500$, aggregate underprediction.)

What You'll Learn