Backtesting Projections Against Mature Accident Periods

Why Backtest Chain-Ladder Projections?

A chain-ladder ultimate is a projection made using information available at a particular valuation date. Backtesting asks a later question: how close was that archived projection to the loss amount eventually observed for the same accident period? The comparison provides evidence about historical performance; it does not establish that a projection method will be accurate in every future period.

A valid test preserves the information boundary. Reconstruct or retrieve each projection using only the triangle, development factors and tail selection available at its original valuation date. Compare it with a later observed ultimate on a consistent basis. Substituting factors estimated with later data would introduce information that was unavailable when the projection was made and would make the backtest misleading.

Set Up a Comparable Test

For a standard cumulative chain-ladder projection, write \(C_{i,j}\) for cumulative losses for accident period \(i\) at development age \(j\), and \(F_j\) for the selected cumulative development factor from that age to ultimate, including any selected tail factor. The projection at the original valuation is

This form assumes that the cumulative amount and development factors are defined on a consistent basis and that the selected factors are appropriate for the data being projected. In a backtest, use the factors selected at valuation date \(t\), not factors recalculated using later development. The later observed \(U_i^*\) should be sufficiently mature for the purpose of the test. If it is still developing materially, it is an interim comparison amount, not a settled measure of the projection’s accuracy.

The error identity explains what the comparison measures. If \(U_i^*=C_{i,j}+R_i\), where \(R_i\) is the subsequently observed development after age \(j\), then

Thus, the signed error compares the future development anticipated by the factor with the future development later observed. It is not an error in the cumulative amount already recorded at the valuation date.

Measures of Projection Accuracy

For \(n\) tested accident periods, the mean signed error measures average direction, while the mean absolute error measures average error size in dollars. Mean absolute percentage error expresses each absolute error relative to that period’s observed ultimate. These measures answer different questions.

The mean signed error can be close to zero even when individual errors are large, because overprojections and underprojections can offset. MAE cannot cancel in that way. MAPE gives equal weight to each period’s percentage error, so a small-dollar accident period can have as much influence on MAPE as a large one. When all \(U_i^*\) are positive, a size-weighted alternative is weighted absolute percentage error, \(\sum_i |e_i^{(t)}|/\sum_i U_i^*\times100\%\). It gives more weight to periods with larger observed ultimates.

MAPE is undefined for a period with \(U_i^*=0\), and can be unstable when the observed ultimate is very small. Do not silently divide by zero or remove such periods without explaining the treatment. In those cases, report dollar errors or a suitable aggregate measure and describe the limitation.

Comparing Projections at Different Valuation Dates

A later valuation can produce a revised ultimate for the same accident period because more development is available. Testing both archived projections against one common later mature amount can show how the projection changed as information emerged. However, the two errors are repeated measurements for one accident period, not two independent accident periods. Keep the valuation date attached to each projection and avoid combining repeated snapshots as though they were separate years of experience.

Interpretation, Comparability and Common Traps

Backtest results are most useful when their scope is explicit. State the valuation date or development age, the accident periods included, the selected projection method and the maturity standard for the comparison amounts. If you summarize different valuation dates together, label them and explain how repeated accident periods are treated. A change in the mix of tested periods can change the summary even when the method itself has not changed.

The comparison also depends on a stable meaning for “loss.” For example, paid and incurred amounts are different measures; changing the loss definition between the archived projection and later amount breaks comparability. Confirm that the currency, gross or net basis, expense treatment and other material definitions agree. If amounts are adjusted to a common basis, specify the adjustment and apply it consistently. Identifying such changes is part of sound interpretation, not a reason to revise the archived projection after the fact.

• Using hindsight in the projection. Re-estimating the original factors with later data does not test what the original projection anticipated.
• Reversing the error sign. With \(e=\hat U-U^*\), positive means overprojection and negative means underprojection. State the convention before interpreting bias.
• Letting errors cancel in an accuracy measure. Use absolute errors for MAE; signed errors measure direction, not typical error size.
• Ignoring the MAPE denominator. MAPE is undefined at a zero actual and can be dominated by periods with very small actuals.
• Comparing unlike amounts. An immature later value or a changed loss basis can make apparent accuracy misleading.

Backtesting is a diagnostic, not a guarantee. A pattern of errors may motivate closer examination of the data, assumptions or development behavior, but the summary alone does not identify the cause. A small average signed error can coexist with substantial period-level errors, and strong historical performance does not eliminate future uncertainty.