Back-Testing Reserve Predictions With Emergence Data

What a Reserve Back-Test Compares

A reserve prediction can be assessed only after the claims it was intended to predict have had time to emerge. The central comparison is between a forecast made using information available at a valuation date and the corresponding claims observed over a later, specified horizon. The comparison is meaningful only when the forecast and observation refer to the same quantity, cohort, time period, and accounting or payment basis.

For example, a forecast of payments during the next 12 months should be tested against payments made during those 12 months, not against an ending unpaid reserve balance. A forecast made for one accident-year cohort should not be compared with actuals that combine it with other cohorts unless the forecasts are combined on the same basis. These alignment decisions precede any calculation of error.

Definition: Let $U$ be a specified future claims-emergence quantity, such as paid claims for a cohort over a defined horizon. Let $\mathcal{I}$ denote the information available at the valuation date. A reserve back-test compares the realized value $U_{\mathrm{obs}}$ with the prior predictive distribution of $U$ conditional on $\mathcal{I}$. Write its predictive mean as $m=E[U\mid\mathcal{I}]$ and its predictive variance as $s^2=\operatorname{Var}(U\mid\mathcal{I})$. The signed prediction error is $e=U_{\mathrm{obs}}-m$, and, when $s>0$, its standardized prediction error is $z=e/s$.

The sign convention here is actual minus predicted. Thus $e>0$ means emergence was greater than forecast, while $e<0$ means it was less. State this convention when reporting results: reversing the subtraction reverses the interpretation of the sign.

The predictive variance should represent uncertainty in the future quantity, conditional on the information that was available when the forecast was made. Depending on the forecasting method, it may include process variability, uncertainty in estimated parameters, and other uncertainty represented by the model. It is not automatically the same as a process variance alone. Use the variance appropriate to the forecast being tested, and do not add a component a second time if it is already included.

Why Standardized Errors Help

A raw error is measured in claim dollars, so its size cannot be interpreted independently of the scale of the forecast. An error of $50,000 may be substantial for a small cohort but modest for a large one. Dividing by the predictive standard deviation expresses the error in units of forecast uncertainty. A positive $z$ indicates higher-than-expected emergence; a negative $z$ indicates lower-than-expected emergence.

If the predictive distribution is approximately normal and its mean and variance are well specified, a single standardized error is approximately standard normal. In that setting, unusually large absolute values may be surprising under the model. Standardization alone, however, does not make a non-normal predictive distribution normal. It is a scale comparison; probability statements require an appropriate predictive distribution.

Worked Example: Testing One Cohort’s One-Year Emergence

At a valuation date, a hypothetical insurer forecasts that a particular accident-year cohort will generate $600,000 of paid claims during the next 12 months. The forecast predictive variance is $10{,}000$ in squared thousands of dollars, so the forecast is expressed in thousands of dollars. Actual payments over exactly that horizon are $660,000. Calculate the signed and standardized prediction errors.

Method. Compare the observed and forecast payments over the same cohort and 12-month period, then divide their difference by the predictive standard deviation.

First convert the amounts to thousands of dollars: the predictive mean is $m=600$, and observed emergence is $U_{\mathrm{obs}}=660$. The signed error is

e=U_{\mathrm{obs}}-m=660-600=60

The predictive standard deviation is

s=\sqrt{10{,}000}=100

Therefore,

z=\frac{e}{s}=\frac{60}{100}=0.6000

Emergence exceeded the forecast by $60,000, or 0.6000 predictive standard deviations. The units are consistent: both the error and standard deviation are in thousands of dollars. The calculation is checked by multiplying the standardized error by the standard deviation: $0.6(100)=60$.

Common mistake avoided: Dividing by the variance would give $60/10{,}000$, which is not a standardized error and has incompatible units. The denominator is the standard deviation, the square root of the variance.

Combining Cohorts and Preserving Dependence

A back-test may examine one cohort at a time or combine several forecast quantities. For a combined test, add the forecast means and the observed amounts. The variance of the combined quantity must include covariance terms whenever the cohort outcomes are dependent.

For two quantities $U_1$ and $U_2$, with predictive means $m_1,m_2$, predictive variances $s_1^2,s_2^2$, and predictive covariance $c=\operatorname{Cov}(U_1,U_2\mid\mathcal{I})$, the aggregate error is the sum of the two individual errors. Its variance follows from the variance-of-a-sum identity:

\operatorname{Var}(U_1+U_2\mid\mathcal{I}) =s_1^2+s_2^2+2c

The factor of 2 appears because the covariance occurs twice when expanding the variance of a sum. Omitting it is justified only if the predictive covariance is zero. For more than two quantities, add all individual variances and twice each distinct pairwise covariance.

Worked Example: Aggregating Two Dependent Cohort Errors

A hypothetical reserve model forecasts emergence for two cohorts, in thousands of dollars. The predictive means are 30 and 45; the predictive variances are 36 and 49; and the predictive covariance is 14. Subsequent observed emergence is 36 and 52, respectively. Calculate the aggregate standardized error, using the model’s covariance.

Method. Sum the actuals and means to obtain the aggregate error, then calculate the predictive variance of the sum, including twice the covariance.

The individual errors are

e_1=36-30=6,\qquad e_2=52-45=7

The aggregate observed emergence is $36+52=88$, and the aggregate predictive mean is $30+45=75$. Thus the aggregate error is $88-75=13$, also equal to $e_1+e_2=6+7=13$.

The aggregate predictive variance is

$$ 36+49+2(14)=36+49+28=113 $$

The aggregate standard deviation is $\sqrt{113}\approx10.6301$. Therefore the aggregate standardized error is

z_{\mathrm{agg}}=\frac{13}{\sqrt{113}}\approx1.2229

The check is that the aggregate error is 13 and its predictive standard deviation is about 10.6301, giving $13/10.6301\approx1.2229$. The combined emergence was about 1.2229 standard deviations above its predictive mean.

Common mistake avoided: Ignoring dependence gives the incorrect variance $36+49=85$, and hence $13/\sqrt{85}\approx1.4096$. The supplied covariance is not zero, so the aggregate variance must include $2(14)=28$. Using the covariance once rather than twice is also incorrect.

Justifying the Aggregate Error Formula

The aggregation rule follows directly from the definition of variance. Let $U_1$ and $U_2$ have conditional means $m_1$ and $m_2$. The aggregate prediction error is

$$ (U_1+U_2)-(m_1+m_2) =(U_1-m_1)+(U_2-m_2) $$

For any random variables $A$ and $B$, expanding the squared centered sum gives

$$ E[(A+B)^2]=E[A^2]+E[B^2]+2E[AB] $$

When $A$ and $B$ are centered errors, $E[A^2]$ and $E[B^2]$ are their variances, and $E[AB]$ is their covariance. Consequently, the variance of the aggregate error is the sum of the two variances plus twice their covariance. This is why an aggregate standardized error can differ materially from a calculation that treats cohorts as independent.

Reading a Set of Back-Test Results

A single standardized error is not, by itself, proof that a reserve model is biased or miscalibrated. It is one realization of a future quantity and may be large by chance. More informative evidence can come from a series of genuinely out-of-sample predictions, provided their horizons and definitions are comparable. Look for repeated signed errors in one direction, errors whose scale is consistently too large or too small relative to predictive uncertainty, and differences between cohorts or periods that have a plausible actuarial interpretation.

For separate outcomes with zero predictive covariance, the variance of their summed error is the sum of their predictive variances. With nonzero covariance, the covariance terms must be included. Do not assume independence merely because the outcomes belong to different accident years: a shared calendar-year effect, common parameter estimates, or other modeled dependence can link them. Conversely, do not insert covariance without a basis in the predictive model.

A useful review records, for each back-tested forecast, the valuation date, the information set used, the cohort, the emergence horizon, the forecast mean, the predictive variance, the observed quantity, and the signed and standardized errors. Keeping these items together helps reveal a common source of misleading results: a forecast can appear wrong simply because actuals and predictions were constructed on different bases.

Worked Example: Reviewing Errors Across Four Independent Cohorts

A hypothetical model makes one-year emergence predictions for four cohorts. All amounts are in thousands of dollars. The forecast means are 50, 60, 40, and 80; the predictive variances are 25, 36, 16, and 64. Actual emergence is 55, 54, 44, and 88. Assume the four forecast errors have zero pairwise covariance. Calculate each standardized error and the standardized error for aggregate emergence.

Method. For each cohort, subtract the forecast mean from actual emergence and divide by the square root of that cohort’s predictive variance. For the aggregate, sum errors and variances under the stated zero-covariance assumption.

The predictive standard deviations are $\sqrt{25}=5$, $\sqrt{36}=6$, $\sqrt{16}=4$, and $\sqrt{64}=8$. The cohort errors and standardized errors are

e_1=55-50=5,\quad z_1=\frac{5}{5}=1.0000

e_2=54-60=-6,\quad z_2=\frac{-6}{6}=-1.0000

e_3=44-40=4,\quad z_3=\frac{4}{4}=1.0000

e_4=88-80=8,\quad z_4=\frac{8}{8}=1.0000

The total observed emergence is $55+54+44+88=241$, and the total forecast mean is $50+60+40+80=230$, so the aggregate error is $241-230=11$. Equivalently, the sum of individual errors is $5-6+4+8=11$. Under the stated zero-covariance assumption, the aggregate predictive variance is

$$ 25+36+16+64=141 $$

Thus the aggregate standardized error is $11/\sqrt{141}\approx0.9264$, since $\sqrt{141}\approx11.8743$. The individual errors include three positive values and one negative value, while the aggregate error is about 0.9264 standard deviations above its forecast. These are descriptive results, not proof of systematic bias from only four observations.

Check. The aggregate means sum to 230, the actuals sum to 241, and their difference is 11. Dividing 11 by the aggregate standard deviation of approximately 11.8743 gives approximately 0.9264.

Common mistake avoided: Adding the four individual standardized errors does not produce the aggregate standardized error. Standardize the summed error using the aggregate predictive standard deviation.

Common Back-Testing Traps

Look-ahead information. A forecast used in a back-test must be reconstructed from information available at its original valuation date. Fitting the forecast using the later emergence being tested can make the apparent performance misleading.
Comparing different quantities. Match the predicted and observed cohort, horizon, payment or incurred basis, and claim definition. A future payment forecast is not an ending reserve forecast.
Using variance instead of standard deviation. The standardized error is $e/\sqrt{s^2}$, not $e/s^2$.
Dropping dependence. For an aggregate, include covariance terms implied by the predictive model. Zero covariance is an assumption, not a consequence of different cohort labels.
Over-interpreting one result. A positive error means actual emergence exceeded the forecast in that test. It does not, by itself, establish a persistent bias.
Mixing scales. If errors are in thousands of dollars, variances must be in squared thousands of dollars. Convert consistently before calculating a standardized error.

Key takeaway: A defensible back-test preserves the forecast’s original information set, compares like with like, and standardizes actual-minus-forecast emergence using the correct predictive standard deviation. When combining cohorts, use the variance of the sum, including all relevant covariances. This approach is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Check Your Understanding

For each problem, use actual minus predicted as the signed error and use the predictive standard deviation for standardization.

A hypothetical cohort has predicted 12-month emergence of $420,000, predictive variance $6{,}400$ in squared thousands of dollars, and actual emergence of $440,000. Find the signed and standardized errors. (Answer: $20,000; $0.2500$)
Two cohorts have predictive means 18 and 22, variances 9 and 16, and covariance $-3$, all in consistent thousand-dollar units. Actual emergence is 21 and 20. Find the aggregate signed error and standardized error. (Answer: $1$ thousand dollars; $1/\sqrt{19}\approx0.2294$)
Three independent cohorts have errors of 4, $-3$, and 5 thousand dollars and predictive variances of 16, 9, and 25. Find the aggregate standardized error. (Answer: $6/\sqrt{50}\approx0.8485$)
For a cohort, the predicted mean is $900,000, the predictive variance is $22{,}500$ in squared thousands of dollars, and actual emergence is $840,000. Calculate the standardized error and state its direction. (Answer: $-0.4000$; actual emergence is below the forecast)
Two dependent cohorts have errors of 8 and 4 thousand dollars, variances of 25 and 9, and covariance 5. Find the aggregate variance and standardized error. (Answer: $25+9+2(5)=44$; $12/\sqrt{44}\approx1.8091$)

What You'll Learn