Back-Test Selected Factors Against Subsequent Emergence

Why Back-Test Selected Factors?

A selected link ratio is a judgment about how losses will develop from one age to the next. A back-test asks how that judgment would have performed when applied to historical data: using losses available at an earlier age, what would the selected factor have projected, and how did that projection compare with losses observed later? This provides evidence about a selection’s performance, though it cannot guarantee that future development will follow the same pattern.

The comparison must use a consistent starting age, ending age, and loss basis. For example, a projection from 12 to 24 months should be compared with the same cohort’s actual cumulative losses at 24 months—not with a value at a different age or on a different basis. Back-testing selected development factors is aligned with the CAS Exam 5 (Basic Ratemaking and Estimated Claim Liabilities) syllabus.

Definition: Let $C_{i,j}$ denote cumulative losses for origin period $i$, observed at development age $j$. Let $\hat f_j$ be the selected link ratio from age $j$ to the next age. A back-test starts with losses known at age $k$, projects them to a later age $m$ using selected factors, and compares the projection with the subsequently observed $C_{i,m}$.

The selected factors are treated as fixed for the test. If a back-test is meant to represent what could have been known at a historical valuation date, the factors must be the selections that were available then. Using information that emerged afterward to select factors and then calling the result a historical forecast would introduce hindsight.

Project, Compare, and Summarize

For a projection from age $k$ through age $m$, multiply the selected link ratios for each intervening age. Define the projected value, signed dollar error, and signed percentage error as follows:

\widehat C_{i,m\mid k} = C_{i,k}\prod_{j=k}^{m-1}\hat f_j, \qquad E_i = \widehat C_{i,m\mid k}-C_{i,m}, \qquad e_i = \frac{E_i}{C_{i,m}}\times 100\%.

Here, $\widehat C_{i,m\mid k}$ is the projection made from age $k$, while $C_{i,m}$ is the value that later emerged. Under this sign convention, a positive error means overprediction and a negative error means underprediction. The percentage error uses the subsequently observed value as its denominator. If that observed value is zero, the percentage error is undefined; report the dollar error instead.

The multiplication in the projection follows directly from the definition of a link ratio. At each age, the projected next value equals the current value multiplied by the factor for that transition. Applying this rule first at age $k$, then at $k+1$, gives:

C_{i,k}\hat f_k\hat f_{k+1} = C_{i,k}\left(\prod_{j=k}^{k+1}\hat f_j\right).

Repeating the same step through age $m-1$ yields the product in the projection formula. Factors are therefore multiplied, not added. The same logic applies whether the back-test covers one development interval or several.

Fix the test point.
Specify the origin period, starting age, ending age, loss basis, and the selected factors being tested.

Make the projection.
Start with the value known at the starting age and multiply by every selected factor needed to reach the ending age.

Compare with emergence.
Subtract the subsequently observed value from the projection. Keep the sign convention consistent across cohorts.

Interpret the pattern.
Review cohort-level errors, age-specific differences, and aggregate results. An aggregate near zero can conceal substantial errors that offset one another.

Worked Back-Tests

Worked Example: Testing a Single 12-to-24-Month Factor

A hypothetical insurer selected a 12-to-24-month factor of $1.20$. For three historical origin periods, the losses at 12 months were \$800,000, \$1,000,000, and \$1,200,000. The subsequently observed 24-month values were \$960,000, \$1,260,000, and \$1,380,000. Calculate each projection and error, then assess the aggregate result.

Method. Apply the selected factor to each 12-month value. For each origin period, calculate projected less observed, then divide by observed losses for the percentage error.

For the first period, the projection is $800{,}000\times1.20=\$960{,}000$. The error is $960{,}000-960{,}000=\$0$, or $0/960{,}000=0.00\%$.

For the second period, the projection is $1{,}000{,}000\times1.20=\$1{,}200{,}000$. The error is $1{,}200{,}000-1{,}260{,}000=-\$60{,}000$. The percentage error is $-60{,}000/1{,}260{,}000=-4.7619\%$, so the projection underpredicts.

For the third period, the projection is $1{,}200{,}000\times1.20=\$1{,}440{,}000$. The error is $1{,}440{,}000-1{,}380{,}000=\$60{,}000$. The percentage error is $60{,}000/1{,}380{,}000=4.3478\%$, so the projection overpredicts.

The projected total is $960{,}000+1{,}200{,}000+1{,}440{,}000=\$3{,}600{,}000$. The observed total is $960{,}000+1{,}260{,}000+1{,}380{,}000=\$3{,}600{,}000$. The aggregate dollar error is zero and the aggregate percentage error is $0/3{,}600{,}000=0.00\%$. The aggregate matches exactly, but two individual periods have errors of \$60,000 in opposite directions. The total alone does not show that variation.

Common mistake avoided. Do not conclude that every cohort was projected accurately just because the aggregate error is zero. Check the individual errors as well as the aggregate.

Worked Example: Locating Divergence Across Two Ages

A hypothetical book has selected factors of $1.30$ from 12 to 24 months and $1.10$ from 24 to 36 months. For one origin period, cumulative losses were \$500,000 at 12 months, \$620,000 at 24 months, and \$682,000 at 36 months. Back-test the 12-to-36-month projection and identify which interval explains the difference.

Method. First project from 12 to 36 months using both selected factors. Then compare actual and selected development at each interval.

The selected cumulative factor is $1.30\times1.10=1.43$. The projection from 12 months is $500{,}000\times1.43=\$715{,}000$. Compared with the observed 36-month value, the dollar error is $715{,}000-682{,}000=\$33{,}000$. The percentage error is $33{,}000/682{,}000=4.8387\%$, an overprediction.

At the first interval, the selected projection is $500{,}000\times1.30=\$650{,}000$, compared with the actual 24-month value of \$620,000. The dollar error at 24 months is $650{,}000-620{,}000=\$30{,}000$. The actual 12-to-24-month ratio is $620{,}000/500{,}000=1.24$, below the selected $1.30$.

At the second interval, the actual ratio is $682{,}000/620{,}000=1.10$, equal to the selected $1.10$. This indicates that the first interval accounts for the discrepancy in this example; the second interval’s selected factor matches the observed ratio.

Check. Projecting sequentially gives $650{,}000\times1.10=\$715{,}000$, the same as multiplying the original value by $1.43$. The projection and cumulative calculation agree. This test identifies where the historical difference arose; it does not establish that the same pattern will hold for future origin periods.

Common mistake avoided. Adding the factors gives $1.30+1.10=2.40$, which is not a cumulative factor. The correct cumulative factor is their product, $1.30\times1.10=1.43$.

Worked Example: Interpreting a Small Aggregate Error

A hypothetical insurer tests a selected 12-to-24-month factor of $1.10$ on four origin periods. The 12-month values are \$400,000, \$600,000, \$800,000, and \$1,000,000. The subsequent 24-month values are \$440,000, \$690,000, \$920,000, and \$1,050,000. Calculate the aggregate error and explain what the cohort results indicate.

Method. Project each 12-month value using $1.10$, calculate each signed error, and then compare total projected losses with total observed losses.

The projections are $400{,}000\times1.10=\$440{,}000$, $600{,}000\times1.10=\$660{,}000$, $800{,}000\times1.10=\$880{,}000$, and $1{,}000{,}000\times1.10=\$1{,}100{,}000$.

The corresponding errors are $440{,}000-440{,}000=\$0$, $660{,}000-690{,}000=-\$30{,}000$, $880{,}000-920{,}000=-\$40{,}000$, and $1{,}100{,}000-1{,}050{,}000=\$50{,}000$. The selected factor underpredicts two periods and overpredicts one, with one exact result.

The projected total is $440{,}000+660{,}000+880{,}000+1{,}100{,}000=\$3{,}080{,}000$. The observed total is $440{,}000+690{,}000+920{,}000+1{,}050{,}000=\$3{,}100{,}000$. The aggregate error is $3{,}080{,}000-3{,}100{,}000=-\$20{,}000$, and the percentage error is $-20{,}000/3{,}100{,}000=-0.6452\%$. Thus, the aggregate projection underpredicts by about $0.65\%$.

The aggregate percentage error is small relative to the total, but it results from offsetting period-level errors. This is evidence of a modest aggregate underprediction in this test, not proof that the factor is unbiased in general. The test includes only four periods, and the results vary by period.

Check. The total of the four dollar errors is $0-30{,}000-40{,}000+50{,}000=-\$20{,}000$, which agrees with the difference between projected and observed totals.

Interpreting Results and Avoiding Traps

A back-test is useful because it gives a selected factor a concrete historical performance check. A recurring pattern of positive errors suggests that the tested projections tended to exceed subsequent emergence; recurring negative errors suggest they tended to fall short. One or two errors, however, are not enough to establish a stable bias. Claims can emerge unevenly, and a small number of historical periods may not represent the conditions expected for the projection being made.

Keep the test comparable and interpret its scope carefully:

Compare like with like. Use the same loss measure, development ages, and cohort definitions for projection and observation. A comparison across different bases does not isolate factor performance.
Preserve the historical information set. For a genuine historical forecast test, use factors selected with information available at the historical valuation date. Later information can be used for an illustrative test of a current selection, but that is not evidence of what would have been forecast then.
Review errors at more than one level. Dollar and percentage errors describe different aspects of performance. A large cohort can dominate the aggregate dollar error, while percentage errors reveal relative differences across cohorts.
Locate the age of divergence. When a multi-age projection misses, compare actual and selected development at each interval. This can show whether the difference arose early, late, or across several ages.
Do not overinterpret a back-test. Historical accuracy does not guarantee future accuracy. Changes in the book or claims environment may limit how comparable earlier experience is to the projection being evaluated.

The key is to treat the back-test as evidence for evaluating a selection, not as an automatic rule for changing it. The arithmetic identifies the direction and size of historical differences; judgment is still needed to decide how relevant those differences are.

Check Your Understanding

For each problem, use projected less observed as the signed dollar error. A positive result indicates overprediction; a negative result indicates underprediction.

A selected 12-to-24-month factor is $1.08$. Losses were \$750,000 at 12 months and \$825,000 at 24 months. Find the projection and percentage error. (Answer: Projection \$810,000; error $-\$15,000$, or $-1.8182\%$, an underprediction.)
Selected factors are $1.20$ from 12 to 24 months and $1.05$ from 24 to 36 months. A cohort had \$900,000 at 12 months and subsequently had \$1,050,000 at 24 months and \$1,092,000 at 36 months. Find the 12-to-36-month projection and percentage error. (Answer: Cumulative factor $1.26$; projection \$1,134,000; error $+\$42,000$, or $3.8462\%$, an overprediction.)
A selected 12-to-24-month factor is $1.12$. Two cohorts had 12-month values of \$500,000 and \$1,000,000, and later observed 24-month values of \$540,000 and \$1,120,000. Find the aggregate projection and percentage error. (Answer: Projection \$1,680,000; observed total \$1,660,000; error $+\$20,000$, or $1.2048\%$, an overprediction.)
A historical validation is described as a test of what could have been projected at a past valuation date. The factor was selected using loss information that emerged after that date. What is the issue? (Answer: The test uses hindsight and does not represent a projection made with information available at the stated historical valuation date.)
A selected 12-to-24-month factor of $1.15$ is applied to two cohorts with 12-month values of \$400,000 and \$600,000. Their later observed values are \$460,000 and \$660,000. Find the aggregate percentage error and state whether the two cohort results have the same direction. (Answer: Projection \$1,150,000; observed total \$1,120,000; error $+\$30,000$, or $2.6786\%$, an aggregate overprediction. The first cohort is exact and the second is overpredicted.)

What You'll Learn