Backtesting and Outcome Analysis for Capital Models

Why Compare Forecasts With Outcomes?

A capital model describes a distribution of possible losses, not just a single capital number. Backtesting compares forecasts made before outcomes were known with the losses that subsequently emerged. The comparison can reveal patterns worth investigating: too many threshold breaches, losses repeatedly near the forecast tail, or outcomes that do not look consistent with the predicted distribution. It cannot, by itself, prove that a model is valid or invalid. In particular, a short run of annual outcomes gives limited evidence about rare losses.

A useful comparison first makes the forecast and the outcome commensurable. The forecast and observed loss should cover the same time horizon, business scope, valuation basis, and treatment of expenses, recoveries, and risk mitigation. If a forecast is for annual net losses, comparing it with quarterly gross losses does not test the forecast that was made. Backtesting is most informative when the predictions are recorded before the outcomes occur and the comparison rules are set in advance.

Exceedance Frequency and Its Assumptions

For a continuous forecast distribution that is correctly calibrated, \(P(X_t>q_t)=1-p\). Thus a 99th-percentile threshold has a 1% strict-exceedance probability, and a 95th-percentile threshold has a 5% probability. This probability statement depends on both calibration and the distributional assumptions. It is not a consequence of the quantile label alone.

If the forecast has point masses, the strict-exceedance probability need not equal \(1-p\). With the quantile convention above, \(F_t(q_t)\ge p\), so \(P(X_t>q_t)=1-F_t(q_t)\le 1-p\). A tie at the threshold is not a strict exceedance. Consequently, before comparing an observed count with a nominal exceedance rate, specify how ties and discrete forecasts are handled.

The expectation follows by adding indicator expectations: \(E[N]=\sum_t E[I_t]=nr\). Independence gives \(\operatorname{Var}(N)=\sum_t\operatorname{Var}(I_t)=nr(1-r)\), because each indicator is Bernoulli with probability \(r\) and all covariance terms are zero. This explains why a count near its expected value is not a requirement in a short sample. The count is random and may differ materially from its mean even when the assumptions hold.

If the periods are independent but have different strict-exceedance probabilities \(r_t\), the count is generally Poisson-binomial rather than binomial. Its expectation is \(\sum_t r_t\), and its variance is \(\sum_t r_t(1-r_t)\). If outcomes are dependent over time, even these variance calculations need covariance terms: \(\operatorname{Var}(N)=\sum_t\operatorname{Var}(I_t)+2\sum_{s<t}\operatorname{Cov}(I_s,I_t)\). Clustering of exceedances can therefore make a simple binomial comparison misleading.

Complementary Calibration Diagnostics

A threshold count compresses each period's outcome to a zero or one. It discards how far an exceedance lies beyond the threshold and what happened in the rest of the distribution. Other diagnostics can therefore add useful information, provided their assumptions and limitations are also made clear.

For a continuous predictive CDF, the probability integral transform is \(U_t=F_t(X_t)\). If the full forecast distribution is correctly calibrated for the period's outcome, \(U_t\) is uniformly distributed on \([0,1]\). This follows because, for \(0\le u\le1\), \(P(F_t(X_t)\le u)=u\) when the predictive CDF is continuous and correctly describes the outcome. A collection of PIT values can be examined for concentration near 0 or 1, which may indicate tail errors, or for other departures from uniformity. If the forecasts are conditional on past information, the interpretation also depends on using the appropriate conditional CDFs; serial dependence can undermine tests that assume independent PIT values.

In discrete distributions, the ordinary PIT is not generally uniform because outcomes can produce repeated CDF values. A randomized PIT can address this: for observed \(X_t\), set \(U_t=F_t(X_t^-)+V_t\{F_t(X_t)-F_t(X_t^-)\}\), where \(F_t(X_t^-)\) is the left limit of the CDF and \(V_t\) is an independent uniform random variable on \([0,1]\). This distributes the probability mass at each outcome across its CDF jump. The diagnostic must be interpreted with the randomization and the discrete model in mind.

Coverage of forecast intervals is another straightforward check. For example, if a model produces a central 90% predictive interval each year, the observed losses should fall inside such intervals about 90% of the time over repeated comparable periods, under correct calibration. Record both overall coverage and which tail contains any misses: equal overall coverage can hide too many low-tail misses and too many high-tail misses. Interval coverage is still a sample statistic, not a verdict from a single observation.

Interpreting Evidence From Limited Observations

A backtest asks whether an observed pattern is consistent with a forecast under stated assumptions. It does not show that the assumptions are true, and a non-unusual result does not confirm the model. For rare thresholds, the expected number of breaches may be far below one over the available record. For example, even a correctly calibrated 99th-percentile forecast produces only one expected strict exceedance in 100 independent years. The absence of a breach in a much shorter record is weak evidence about the far tail.

A surprising count is a signal to examine, not a standalone conclusion. Consider whether periods are independent, forecasts were formed using only information available at the time, exposure and loss definitions remained consistent, and any model or portfolio changes affected comparability. Also inspect the sizes and direction of misses and diagnostics beyond counts. The earlier material on sparse data, parameter uncertainty, scenarios, stress tests, and sensitivity analysis provides relevant context: a model's apparent fit to a short record may reflect chance, while uncertainty in assumptions can remain substantial.

Exam trap: writing \(E[N]=n(1-p)\) without specifying what the indicator counts and why its probability is \(1-p\). Under a strict rule \(I_t=\mathbf{1}\{X_t>q_t\}\), this formula is justified when \(P(X_t>q_t)=1-p\) for every period, as with continuous calibrated forecasts at that quantile. It is not generally justified for discrete forecasts with mass at the threshold. Backtesting is strongest when the event definition, assumptions, calculation, and limits of inference are all stated explicitly.