Why Credit Models Need Backtesting
A credit model is useful only if its forecasts provide a credible description of outcomes over the horizon for which they are used. A model may estimate default probabilities, assign relative risk scores, predict rating migrations, or produce a distribution of portfolio losses. Each output calls for a validation method that matches the forecast and the available observations.
Backtesting compares forecasts made using information available at a particular date with outcomes observed later. The comparison must use a consistent horizon, exposure definition, and outcome definition. For example, a one-year probability of default should be compared with defaults over the following year, not with a mixture of shorter observation periods. Testing forecasts against data used to fit the model can make performance look better than it would on genuinely subsequent observations.
Validation is not a single pass-or-fail calculation. Credit defaults may be infrequent, observations may be dependent, and the economic environment may change during the testing period. A statistical test should therefore be reported with its assumptions and interpreted alongside the forecast horizon, data quality, and practical materiality of any discrepancy.
Testing Default-Probability Calibration
Suppose a set of exposures has the same forecast probability of default \(p\), and each exposure has a binary outcome: default or no default during the stated horizon. If the outcomes are independent and each has probability \(p\), the number of defaults \(D\) has a binomial distribution with parameters \(n\) and \(p\). Its expected value is \(np\), and its variance is \(np(1-p)\).
For a sufficiently large expected number of defaults and nondefaults, a useful approximate standardized residual is
Under the calibration assumptions, this residual is approximately standard normal. A two-sided 5% test flags a discrepancy when \(|z|>1.96\). This approximation can be unreliable when the expected number of defaults is small; an exact binomial test is then preferable.
When exposures have different probabilities \(p_i\), the total default count is generally not binomial with one common probability. If outcomes are independent, its expectation and variance are instead
This follows by writing \(D=\sum_i I_i\), where \(I_i\) equals 1 for a default and 0 otherwise. Each indicator has mean \(p_i\) and variance \(p_i(1-p_i)\). Independence makes the covariances between distinct indicators zero, so the variances add. If defaults are dependent, the variance also contains covariance terms; ignoring them can make a calibration test appear more decisive than it should.
Worked Example: Testing a Common Default Probability
A hypothetical lender forecasts a one-year probability of default of 2% for each of 1,000 comparable loans. The loans are assumed independent, and the model and forecast are fixed before the year begins. There are 30 defaults. Use a two-sided 5% normal-approximation test of the common probability.
Method and assumptions. Under the null hypothesis of calibration, \(D\sim\operatorname{Binomial}(1000,0.02)\). The expected count is \(1000(0.02)=20\), and the expected nondefault count is 980, so the normal approximation is reasonable for this calculation.
Calculate the standard deviation. The variance is \(1000(0.02)(0.98)=19.6\), giving standard deviation \(\sqrt{19.6}=4.4272\).
Calculate the standardized residual. The observed count exceeds expectation by \(30-20=10\), so \(z=10/4.4272=2.2588\). Since \(2.2588>1.96\), reject the common 2% calibration hypothesis at the approximate 5% level.
The observed default rate is \(30/1000=3.00\%\), compared with the forecast 2.00%. The test identifies evidence of underprediction in this sample; it does not by itself establish why the difference occurred or prove that the model will underpredict in future periods. The calculation avoids the common mistake of dividing the count difference by the expected count instead of by its sampling standard deviation.
Probability Scores and Discrimination
A calibration table groups forecasts into ranges and compares the average forecast in each range with the observed default rate. Such grouping is easy to explain, but results can depend on how the ranges are chosen. A score that uses individual forecasts can complement a grouped assessment.
A lower Brier score is better for the observations being assessed. It reflects both probability accuracy and the ability to distinguish outcomes, so it should not be treated as a pure measure of calibration. Comparisons are most informative when models are assessed on the same observations and horizon. A low score on a small sample is not, by itself, evidence of reliable future performance.
Worked Example: Calculating and Interpreting a Brier Score
A hypothetical four-loan validation sample has forecasts \(0.10, 0.20, 0.40, 0.80\) and outcomes \(0,0,1,1\), in that order. Calculate the Brier score and compare it with a constant forecast equal to the sample default rate.
Model score. The four squared errors are \((0.10-0)^2=0.01\), \((0.20-0)^2=0.04\), \((0.40-1)^2=0.36\), and \((0.80-1)^2=0.04\). Their sum is \(0.45\), so the Brier score is \(0.45/4=0.1125\).
Constant-forecast comparison. There are two defaults among four loans, so the sample default rate is \(2/4=0.50\). Forecasting 0.50 for every loan gives four squared errors of \(0.25\), for a score of \((4(0.25))/4=0.25\).
The model’s score is lower on this sample: \(0.1125<0.25\). This is a descriptive comparison, not a conclusive statistical test: four observations provide little evidence about performance on a broader portfolio. The method avoids comparing raw forecast probabilities with outcomes without first applying a scoring rule.
A common discrimination measure is the area under the receiver operating characteristic curve, or AUC. It can be interpreted as the probability that a randomly selected defaulter receives a higher risk score than a randomly selected nondefaulter, with a tied pair contributing one-half. This interpretation assumes that larger scores mean higher risk; if the score convention is reversed, reverse the ranking first.
The pairwise definition explains the tie adjustment: a tie provides neither a correct nor an incorrect ordering, so it receives half credit. An AUC near 0.5 indicates little ranking power in the sample; a higher AUC indicates stronger ranking. AUC does not establish whether the numerical probabilities are calibrated.
Worked Example: Computing AUC With Tied Scores
A hypothetical validation sample has three defaulters with risk scores \(0.9,0.6,0.6\) and three nondefaulters with scores \(0.8,0.6,0.2\). Higher scores mean higher risk. Calculate AUC, counting a tie as one-half.
Method. Compare each of the three defaulters with each of the three nondefaulters, giving \(3(3)=9\) pairs.
Pair comparisons. The defaulter scored 0.9 beats all three nondefaulters, for 3 wins. Each defaulter scored 0.6 loses to the nondefaulter scored 0.8, ties the one scored 0.6, and beats the one scored 0.2. Each therefore contributes 1 win, 1 tie, and 1 loss. Across both such defaulters, that is 2 wins and 2 ties in addition to the first defaulter’s 3 wins.
Calculate. There are 5 wins and 2 ties, so the AUC numerator is \(5+\tfrac12(2)=6\). Thus \(\operatorname{AUC}=6/9=0.6667\), to four decimal places.
The result is checked by noting that the total is 5 wins, 2 ties, and 2 losses, which accounts for all 9 pairs. A frequent exam trap is to count a tied pair as a full win or to miscount the pair comparisons. Neither treatment is correct: each tie contributes one-half.
Comparing Rating Migration Forecasts
A rating transition model forecasts probabilities of moving from an initial rating to each possible destination rating over a specified horizon. For one initial rating with \(n\) observed accounts, multiply each forecast transition probability by \(n\) to obtain expected destination counts. A Pearson chi-square statistic summarizes the difference between observed counts \(O_j\) and expected counts \(E_j\):
If the category probabilities are specified in advance and the expected counts are sufficiently large, this statistic is approximately chi-square with \(k-1\) degrees of freedom. The approximation is not dependable when expected cells are very small. Categories may need to be combined on a defensible basis, or a method suited to sparse counts should be used. Combining categories after seeing the outcomes just to obtain a preferred result undermines the test.
Worked Example: Comparing One Rating Row With Observed Migrations
A hypothetical model forecasts that accounts currently rated A will, over one year, remain A with probability 0.80, move to B with probability 0.15, or default with probability 0.05. Among 100 A-rated accounts, the observed counts are 75 remaining A, 20 moving to B, and 5 defaulting. Assess the row using a 5% chi-square test. Assume independent outcomes and that the probabilities were specified in advance.
Expected counts. Multiplying each probability by 100 gives 80 remaining A, 15 moving to B, and 5 defaults. The observed and expected counts each sum to 100.
Test statistic. The contributions are \((75-80)^2/80=25/80=0.3125\), \((20-15)^2/15=25/15=1.6667\), and \((5-5)^2/5=0\). Therefore, \(X^2=0.3125+1.6667+0=1.9792\).
There are three categories and no fitted probabilities, so the degrees of freedom are \(3-1=2\). The 5% chi-square critical value for 2 degrees of freedom is 5.991. Since \(1.9792<5.991\), do not reject the transition probabilities at the 5% level. This does not prove the model is correct; this sample simply does not provide sufficient evidence to reject this row. The calculation avoids comparing percentage-point differences alone, which would ignore the differing expected counts.
A full migration review can examine several initial-rating rows and multiple periods, but the analyst must account for how many comparisons are being made and whether the same accounts or periods contribute repeatedly. Transition tests assess migration frequencies; they do not by themselves establish that default probabilities are well calibrated within every subgroup or that portfolio loss tails are accurate.
Backtesting Portfolio Loss Forecasts
For a portfolio model that forecasts a loss quantile, a direct coverage backtest records whether realized loss exceeded the forecast. For example, a 99% one-day Value-at-Risk forecast should be exceeded on about 1% of days if it is correctly calibrated for that horizon and market-credit environment. Define an exception indicator as 1 when the realized loss exceeds the forecast and 0 otherwise. Under independent, stable observations and correct 99% coverage, the exception count in \(n\) days has a binomial distribution with probability 0.01.
This test measures unconditional coverage: how often exceptions occur. It does not measure how severe losses are on exception days, and it does not show whether exceptions cluster in time. A model can have a plausible total exception count while failing during particular stress periods. Backtests should use forecasts produced before the corresponding outcomes, with consistent loss definitions and horizons. Overlapping horizons or serially dependent outcomes weaken the simple binomial assumption.
Worked Example: Testing 99% Portfolio VaR Coverage
A hypothetical portfolio model produces a one-day 99% VaR forecast for each of 1,000 days. Assume the forecasts were recorded before each day’s outcome and that daily loss observations are independent. Loss exceeds the forecast on 18 days. Use a two-sided 5% normal-approximation coverage test.
Null expectation. Under correct coverage, the exception probability is \(p=0.01\). The expected number of exceptions is \(1000(0.01)=10\), and the variance is \(1000(0.01)(0.99)=9.9\).
Standardize. The standard deviation is \(\sqrt{9.9}=3.1464\). The standardized residual is \(z=(18-10)/3.1464=2.5426\).
Decision and interpretation. Since \(2.5426>1.96\), the observed exception count is inconsistent with nominal 99% coverage at the approximate 5% level. The observed rate is \(18/1000=1.80\%\), above the forecast exception rate of 1%. This flags a coverage issue under the stated assumptions; it does not identify whether the cause is an incorrect loss distribution, a change in conditions, or dependence between observations. The common mistake is to compare 18 exceptions directly with 1% without translating the rate difference into sampling variability.
Practical Interpretation and Common Traps
The strength of a validation conclusion depends on the question asked and on the data supporting it. Calibration tests compare predicted and observed frequencies; AUC evaluates ordering; migration tests compare destination counts; portfolio exception tests evaluate quantile coverage. A favorable result for one property does not establish the others.
- Match horizon and outcome. Compare a one-year default forecast with one-year default outcomes, using a consistent definition of default and exposure.
- Keep the forecast genuinely out of sample. Record forecasts before outcomes are known and avoid evaluating only on the data used to fit the model.
- Check test assumptions. Common-probability binomial tests require a common probability and independent outcomes. Portfolio exceptions can be dependent, particularly in stressed periods.
- Do not overstate non-rejection. Failure to reject a model is not proof of calibration. A limited sample may have little power to detect meaningful errors.
- Do not confuse metrics. AUC can be high even when probabilities are systematically too large or too small; calibration and ranking answer different questions.
- Investigate the size and pattern of differences. A test statistic is a diagnostic, not an explanation. Review subgroups, time periods, and the financial significance of errors.
Check Your Understanding
For each problem, state the relevant validation method and show the key calculation before checking the final answer.
- A hypothetical pool has 400 independent accounts, each forecast at a one-year default probability of 1.5%. There are 11 defaults. Using the normal approximation and a two-sided 5% threshold of 1.96, calculate \(z\) and decide whether to flag calibration. (Answer: Expected defaults \(=6\), variance \(=400(0.015)(0.985)=5.91\), \(z=5/\sqrt{5.91}=2.057\); flag at the approximate 5% level.)
- Two defaulters have risk scores 0.7 and 0.5; two nondefaulters have scores 0.6 and 0.5. Higher scores mean higher risk. Calculate AUC, counting ties as one-half. (Answer: 2 wins and 1 tie among 4 pairs, so AUC \(=(2+0.5)/4=0.625\).)
- For three accounts, forecasts are 0.10, 0.30, and 0.70, while outcomes are 0, 1, and 1. Calculate the Brier score to four decimal places. (Answer: \((0.01+0.49+0.09)/3=0.1967\).)
- A model forecasts migration probabilities 0.70, 0.20, and 0.10 for remaining in rating C, moving to D, and default. Of 100 C-rated accounts, observed counts are 65, 25, and 10. Compute the chi-square statistic and, using 2 degrees of freedom and a 5% critical value of 5.991, decide whether to reject. (Answer: Expected counts are 70, 20, and 10; \(X^2=25/70+25/20+0=1.6071\), so do not reject at 5%.)
- A hypothetical model issues 500 independent one-day 99% VaR forecasts, with 9 realized exceptions. Using a two-sided 5% normal-approximation threshold of 1.96, calculate the standardized residual and interpret it. (Answer: Expected exceptions \(=5\), variance \(=500(0.01)(0.99)=4.95\), \(z=4/\sqrt{4.95}=1.798\); do not flag at the approximate 5% level.)