Out-of-Time Backtesting of Experience Modifications

What an Out-of-Time Backtest Measures

An experience modification is calculated from earlier experience and used to adjust a prospective rating measure. To assess its predictive value, compare the modification known at the rating date with losses observed in a later period. The time separation matters: the outcome-period losses must not have been used to calculate or revise the modification being tested. Otherwise, the test partly measures how well the modification reflects information that was already included in it, rather than how well it predicts a new outcome.

A backtest is meaningful only when the forecast and outcome refer to a comparable risk, exposure, and loss basis. For example, a forecast for one year of earned exposure should not be compared directly with losses from a different amount of exposure. Likewise, losses subject to one coverage limit or valuation basis should not be compared with forecasts developed for another. The appropriate alignment depends on the purpose and design of the rating plan; any adjustment should be stated and applied consistently.

Definition: An out-of-time backtest evaluates predictions made using information available before an outcome period by comparing those predictions with aligned outcomes from that later period. For a risk $j$, let $F_j$ be its forecast loss amount and $A_j$ its actual loss amount, both expressed on the same exposure and loss basis. Define signed forecast error as $e_j=A_j-F_j$. Thus, $e_j>0$ means actual losses exceeded the forecast, and $e_j<0$ means actual losses were below it.

The convention $e=A-F$ is worth keeping explicit. It implies $F+e=A$, not $A+e=F$. A positive error indicates underprediction; a negative error indicates overprediction. Relative error, when the forecast is positive, is $e/F$. A ratio $A/F$ gives the same directional information: a value above 1 indicates actual losses exceeded forecast losses.

Specify the Forecast and Align the Outcome

For a backtest of a modification used as a loss-cost relativity, a simple prospective forecast for risk $j$ is

$$ F_j=E_jB_jM_j, $$

where $E_j$ is the appropriate prospective exposure, $B_j$ is the baseline expected loss cost per exposure unit, and $M_j$ is the modification fixed before the outcome period. This formula assumes that the modification is intended to adjust expected losses, the baseline is appropriate for the risk and period, and the units of exposure and loss cost match. If a modification adjusts total premium, which includes expenses or other components, it cannot automatically be interpreted as a multiplier of expected losses. A loss forecast requires a defensible mapping to the loss component.

The actual outcome $A_j$ should be developed and adjusted as needed to match the forecast definition. Relevant decisions can include the valuation date, claim development, trend or price level, coverage scope, limits, deductibles, and exposure units. Do not make an adjustment merely because it makes results look closer to forecast; document its purpose and apply it consistently to the comparison population.

Freeze the prediction.
Record each risk’s modification and forecast inputs as of the rating decision, before observing the later-period outcome.

Define the comparison basis.
Specify the exposure unit, loss definition, valuation basis, and any trend, development, limit, or coverage adjustments.

Align actual experience.
Convert outcome-period losses and exposures to the defined basis using a stated and appropriate method.

Calculate and interpret error.
Compare aligned actual losses with forecasts overall and by relevant modification levels, retaining the sign convention $e=A-F$.

Worked Example: Forecasting a Later Period for Three Risks

A hypothetical portfolio uses modifications as loss-cost relativities. The baseline expected loss cost is $8,000 per exposure unit for each risk. Modifications were fixed before the outcome period. Calculate each forecast, the aggregate signed error, and the relative aggregate error. Outcome losses are already developed and on the same basis as the forecasts.

Method. For each risk, use $F=EBM$; then calculate $e=A-F$. The common mistake to avoid is reversing the sign convention.

Risk 1. Exposure is 2 units and the modification is 0.75. Its forecast is $2\times \$8{,}000\times0.75=\$12{,}000$. Actual losses are $13,200, so $e_1=\$13{,}200-\$12{,}000=+\$1{,}200$.

Risk 2. Exposure is 1.5 units and the modification is 1.10. Its forecast is $1.5\times \$8{,}000\times1.10=\$13{,}200$. Actual losses are $12,000, so $e_2=\$12{,}000-\$13{,}200=-\$1{,}200$.

Risk 3. Exposure is 2.5 units and the modification is 1.20. Its forecast is $2.5\times \$8{,}000\times1.20=\$24{,}000$. Actual losses are $25,800, so $e_3=\$25{,}800-\$24{,}000=+\$1{,}800$.

The aggregate forecast is $\$12{,}000+\$13{,}200+\$24{,}000=\$49{,}200$. The aggregate actual is $\$13{,}200+\$12{,}000+\$25{,}800=\$51{,}000$. Therefore, aggregate signed error is $\$51{,}000-\$49{,}200=+\$1{,}800$, and relative aggregate error is $\$1{,}800/\$49{,}200=0.036585$, or 3.66% rounded to two decimal places.

The check is $\$49{,}200+\$1{,}800=\$51{,}000$, so forecast plus signed error equals actual. The positive aggregate error indicates underprediction in total, although Risk 2’s losses were below its forecast.

Why Aggregate Error Adds Across Risks

For a portfolio with aligned forecasts $F_j$ and actuals $A_j$, define aggregate forecast $F_{\text{tot}}=\sum_j F_j$, aggregate actual $A_{\text{tot}}=\sum_j A_j$, and individual errors $e_j=A_j-F_j$. Then

A_{\text{tot}}-F_{\text{tot}} =\sum_j A_j-\sum_j F_j =\sum_j(A_j-F_j) =\sum_j e_j.

This identity justifies calculating the aggregate error either from the portfolio totals or by summing individual signed errors. It also explains why positive and negative errors can offset. A small total error alone does not establish that forecasts were accurate for every type of risk.

When exposure differs, the comparison must still use actual and forecast amounts for corresponding exposure. Dividing both by the same total exposure gives a per-unit comparison, but averaging individual percentage errors without weights can answer a different question: it gives each risk equal influence regardless of exposure or forecast size. State whether the objective is total dollars, per-exposure performance, or an unweighted summary before choosing a measure.

Worked Example: Standardizing Actual Losses to Forecast Exposure

A hypothetical group has a forecast of $27,000 for 25 standard exposure units. Its outcome-period losses are $27,000 for 26 earned exposure units. Assume the exposure units are comparable and losses scale proportionally with exposure for this standardization. Convert actual losses to 25 units, calculate signed and relative error, and check the sign.

Method. First calculate the observed loss amount per exposure unit, then multiply by the target exposure. This avoids comparing a 25-unit forecast with an unadjusted 26-unit outcome.

The observed loss per unit is $\$27{,}000/26=\$1{,}038.461538\ldots$. Standardized actual losses for 25 units are $\$27{,}000\times(25/26)=\$25{,}961.538461\ldots$, or $25,961.54 rounded to cents.

Using the unrounded standardized amount, the signed error is $\$27{,}000\times(25/26)-\$27{,}000=-\$1{,}038.461538\ldots$, or $-\$1{,}038.46$ to cents. Relative error is $-\$1{,}038.461538\ldots/\$27{,}000=-0.0384615\ldots$, or -3.85% rounded to two decimal places.

The sign check is forecast plus signed error: $\$27{,}000-\$1{,}038.461538\ldots=\$25{,}961.538462\ldots$, which is the aligned actual. The forecast exceeds aligned actual, so the error must be negative. A common mistake is to multiply by $26/25$, which would increase actual losses to $28,080 and standardize in the wrong direction. Do not use proportional exposure scaling when exposure units differ materially in risk characteristics; standardization then requires a more appropriate adjustment.

Read Errors Across Modification Levels

A portfolio total can conceal different forecast behavior at different modification levels. Grouping risks into modification ranges and comparing group actuals with group forecasts can show whether errors tend to be positive or negative at particular levels. This is a descriptive backtest: it does not by itself establish why the pattern occurred or prove that the modification caused it. Differences may reflect random claim variation, changes in mix, an unsuitable baseline, or other factors.

For a group $g$, calculate $F_g=\sum_{j\in g}F_j$, $A_g=\sum_{j\in g}A_j$, and $e_g=A_g-F_g$. If $F_g>0$, the ratio $A_g/F_g$ or relative error $e_g/F_g$ helps compare groups of different sizes. Dollar errors show the financial amount; relative errors show error in relation to forecast size. Neither measure should be interpreted without checking exposure and loss-basis alignment.

Worked Example: Offset by Modification Level

A hypothetical backtest groups risks into lower- and higher-modification ranges. Forecasts and actual losses are aggregated within each range on the same basis. Calculate the error and relative error in each range and overall, then interpret how the overall result relates to the two group results.

Lower-modification range. Forecast losses are $48,000 and actual losses are $45,600. The error is $\$45{,}600-\$48{,}000=-\$2{,}400$. Relative error is $-\$2{,}400/\$48{,}000=-5\%$; actual is 95% of forecast.

Higher-modification range. Forecast losses are $32,000 and actual losses are $36,800. The error is $\$36{,}800-\$32{,}000=+\$4{,}800$. Relative error is $\$4{,}800/\$32{,}000=15\%$; actual is 115% of forecast.

Overall, forecast losses are $\$48{,}000+\$32{,}000=\$80{,}000$, while actual losses are $\$45{,}600+\$36{,}800=\$82{,}400$. The aggregate error is $+\$2{,}400$, and relative aggregate error is $\$2{,}400/\$80{,}000=3\%$. The check is $\$80{,}000+\$2{,}400=\$82{,}400$.

The overall 3% underprediction coexists with a 5% overprediction in the lower range and a 15% underprediction in the higher range. The lower-range error offsets part of the higher-range error in dollars. Reporting only the overall result would hide that contrast; reporting only percentages would also omit the differing forecast sizes. The conclusion is about observed forecast errors in these groups, not a definitive explanation for them.

Common Backtesting Traps

Reversing signed error. With $e=A-F$, positive means underprediction. The identity is $F+e=A$.
Using information from the outcome period. A modification recalculated with later losses is not the original out-of-time prediction.
Comparing unlike exposure amounts. Standardize exposure only using a justified method, and compare amounts on a common exposure basis.
Mixing loss bases. Align the forecast and actual for coverage, limits, valuation, trend, and development as appropriate to the test.
Treating a premium factor as a loss factor without justification. A total-premium modification need not scale expected losses in the same proportion.
Equating a low total error with uniform accuracy. Errors across groups can cancel, so inspect results at relevant modification levels as well as in aggregate.

Key takeaway: A sound out-of-time backtest preserves the prediction as it was known before the outcome, aligns forecast and actual exposure and loss bases, and interprets $A-F$ consistently. Aggregate error describes portfolio-level prediction; error by modification level can reveal patterns that the total conceals.

Check Your Understanding

For each problem, use signed error $e=A-F$. Amounts are hypothetical; assume all stated losses are on the specified basis.

A risk has 3 exposure units, a baseline loss cost of $6,000 per unit, and a modification of 0.90. Actual losses are $17,010. Calculate forecast, signed error, and relative error. (Answer: Forecast $16,200; error +$810; relative error +5.00%.)
A forecast is $25,000 for 20 standard exposure units. Actual losses are $31,200 for 24 comparable units. Standardize actual to 20 units and calculate error. (Answer: Aligned actual $26,000; error +$1,000; relative error +4.00%.)
Two risks have the same baseline loss cost of $3,500 per exposure unit. Risk A has 4 units, modification 1.10, and actual losses $16,170. Risk B has 2 units, modification 0.80, and actual losses $5,200. Find aggregate forecast, aggregate actual, and signed aggregate error. (Answer: Forecast $21,000; actual $21,370; error +$370.)
A lower-modification group has forecast losses of $48,000 and actual losses of $45,600. A higher-modification group has forecast losses of $32,000 and actual losses of $36,800. Calculate each group’s relative error and the aggregate relative error. (Answer: Lower group -5%; higher group +15%; aggregate +3%.)
A forecast is $50,000 for 50 standard exposure units. Actual losses are $48,000 for 40 comparable units. Standardize actual to 50 units, then report signed and relative error. Explain the direction. (Answer: Aligned actual $60,000; error +$10,000; relative error +20%; actual exceeds forecast.)

What You'll Learn