Why Compare VaR Forecasts Through Backtesting?
A VaR model turns available information into a loss threshold for a specified horizon and confidence level. Backtesting compares that threshold with the loss that subsequently occurs. When two models are under consideration, the comparison should ask not only which model had fewer exceptions, but also whether the models were tested on the same dates, how large the exception-rate differences are relative to the target, and whether exceptions arrived in a concerning pattern.
Backtesting evaluates a particular aspect of a model: whether realized losses exceed its forecast threshold at a rate and with a timing pattern consistent with the forecast's assumptions. It is evidence about performance, not proof that the full loss model is correct. In particular, a finite record may contain too few exceptions to distinguish reliably between competing forecasts.
The continuity condition matters. For a discrete loss distribution, a quantile can have positive probability mass at the threshold, so the probability of a strict exceedance need not equal \(1-\alpha\). A backtest must use a consistent quantile and tie convention. The examples below assume continuous losses, with exceptions defined by strict exceedance.
What a Fair Comparison Requires
For a meaningful head-to-head comparison, evaluate both models using the same realized losses, dates, horizon, confidence level, and exception definition. Use forecasts made before the corresponding losses were observed; forecasts fitted using future data can make a model look better than it would have looked in real use. If the test periods or confidence levels differ, raw exception counts are not directly comparable.
Under correct calibration, with the same target probability \(p\) for each period, the expected exception count over \(n\) periods is \(np\). If exceptions are also independent across periods, the count \(N=\sum_{t=1}^n I_t\) has a binomial distribution. These assumptions explain the basic count benchmark; they are not automatic features of actual financial data.
The expectation follows because each indicator has mean \(p\), so the expected sum is the sum of the expectations. For the variance, independence makes the covariances between different indicators zero. Each indicator has variance \(p(1-p)\), giving \(np(1-p)\). If exception probabilities differ by period, the expected count is instead \(\sum_t p_t\). If the indicators are dependent, the binomial distribution and its variance formula no longer apply as stated.
Worked Example: Comparing Exception Frequencies
Two hypothetical models issue one-day VaR forecasts at the 95% confidence level for the same 100 trading days. Their losses are treated as continuous, and the forecasts are evaluated out of sample. Model A has 4 exceptions and Model B has 11. Compare their exception frequencies with the target. Use the independent, equal-probability benchmark for a descriptive standardized comparison; do not treat that approximation as a formal test.
Step 1: Find the target count. The target exception probability is \(p=1-0.95=0.05\). Over 100 forecasts, the expected count is \(np=100(0.05)=5\).
Step 2: Calculate the observed rates. Model A's rate is \(4/100=0.04\), or 4%. Model B's rate is \(11/100=0.11\), or 11%. Model A is one exception below the target count; Model B is six above it.
Step 3: Scale the differences by the benchmark standard deviation. Under independent Bernoulli exceptions with \(p=0.05\), the standard deviation of the count is \(\sqrt{100(0.05)(0.95)}=\sqrt{4.75}\approx2.179\). The standardized differences are \((4-5)/2.179\approx-0.459\) for Model A and \((11-5)/2.179\approx2.753\) for Model B.
Model B's observed frequency is more elevated relative to the target in this sample. That is a reason to investigate its calibration, not by itself a definitive rejection: this calculation is a descriptive normal-scale comparison, not an exact binomial test, and the assumptions may not hold. Model A's lower count does not establish that it is superior; a low count can also occur by chance or reflect an overly conservative threshold.
Method and trap avoided. Compare each observed rate with its own target rate before comparing the models. The common mistake is to call the model with fewer exceptions “better” without checking whether its forecasts were at the same confidence level and tested on the same dates.
Use Paired Outcomes When Models Share the Same Dates
When two models are evaluated on identical dates, each date supplies a pair of outcomes: either both forecasts have an exception, neither does, or exactly one does. The dates on which exactly one model has an exception are especially informative about the difference between the models. A paired comparison preserves that date-by-date structure; comparing two counts as if they came from unrelated samples discards it.
For a simple paired diagnostic, let \(b\) be the number of dates when Model A has an exception and Model B does not, and let \(c\) be the number when Model B has an exception and Model A does not. Under the null hypothesis that the models have equal exception probabilities, and under assumptions supporting independent paired observations, the direction of each discordant pair is equally likely to favor either model. Conditional on \(b+c\), the number favoring A can therefore be compared with a binomial distribution with parameters \(b+c\) and \(0.5\). This is an exact paired test for equality of exception probabilities under those assumptions; it is not a test that either model's probability equals its target.
Worked Example: A Paired Exception Comparison
Two hypothetical 99% one-day VaR models are evaluated on the same 50 out-of-sample days. Assume continuous losses and, for the paired test, independent date-pairs. Both models have the same target exception probability, 1%. The records show 3 dates with exceptions for both models, 8 dates with an exception for A only, 2 dates with an exception for B only, and 37 dates with neither model in exception. Compare the exception counts and perform a two-sided exact paired test at the 5% level.
Step 1: Check the table and counts. The four outcomes sum to \(3+8+2+37=50\). Model A has \(3+8=11\) exceptions; Model B has \(3+2=5\). Their observed rates are \(11/50=22\%\) and \(5/50=10\%\), respectively, compared with the 1% target. The differences are \(b=8\) and \(c=2\), so there are \(b+c=10\) discordant dates.
Step 2: Calculate the exact two-sided probability under equal exception probabilities. Conditional on 10 discordant dates, the count favoring A has a \(\operatorname{Binomial}(10,0.5)\) distribution under the null. The observed imbalance is at least as extreme as 8 in either direction, so \[ P(K\geq8)+P(K\leq2) =\frac{\binom{10}{8}+\binom{10}{9}+\binom{10}{10}}{2^{10}} +\frac{\binom{10}{0}+\binom{10}{1}+\binom{10}{2}}{2^{10}} =\frac{45+10+1+1+10+45}{1024} =\frac{112}{1024} =0.1094. \]
At the 5% level, this test does not reject equal exception probabilities because \(0.1094>0.05\). That result does not make either model acceptable: both observed rates are far above the 1% target in this sample. The paired test addresses whether the models differ from each other, not whether they are calibrated.
Method and trap avoided. Use discordant pairs for the exact paired comparison, and assess calibration against the target separately. A common error is to interpret a failure to reject equality as evidence that both models pass; it only means this paired test did not find a statistically significant difference at the chosen level.
Exception Timing Adds a Different Diagnostic
Exception frequency summarizes how many thresholds were exceeded; timing shows when those exceedances occurred. Two models can have identical exception counts but very different sequences. Exceptions grouped together may indicate that failures are not behaving like isolated, independent events. The earlier material on diagnosing exception clustering develops tests for such patterns; the comparison here is to recognize that the sequence matters in addition to the total.
Worked Example: Same Count, Different Timing
Two hypothetical models issue 95% one-day VaR forecasts for the same 80 days. Assume continuous losses, an equal 5% target exception probability, and one forecast per nonoverlapping day. Model A has exceptions on days 10, 30, 50, and 70. Model B has exceptions on days 10, 11, 12, and 13. Compare their count and timing diagnostics.
Step 1: Compare frequencies. Each model has 4 exceptions in 80 days, so each observed rate is \(4/80=0.05\), or 5%. The expected count under the target is \(80(0.05)=4\). On frequency alone, both match the target count.
Step 2: Compare gaps. Model A's successive exception-day gaps are \(30-10=20\), \(50-30=20\), and \(70-50=20\) days. Model B's gaps are \(11-10=1\), \(12-11=1\), and \(13-12=1\) day. A's exceptions are separated; B's occur in a consecutive run.
Step 3: Interpret the distinction. The count does not distinguish the models, but the timing suggests different behavior. Model B's run is a warning to investigate whether exceptions are clustered rather than independent. Four exceptions alone are not enough to establish the cause or to prove that B is miscalibrated; a formal timing test and attention to its assumptions would be needed.
Method and trap avoided. Inspect both count and sequence. The common mistake is to conclude that the models behaved identically because their exception totals match.
What Backtesting Can—and Cannot—Establish
Backtesting can flag a threshold that is exceeded too often, a possible difference in exception rates between competing forecasts, or a sequence that merits a clustering diagnostic. A useful comparison keeps these questions distinct:
- Calibration: Is each model's exception frequency consistent with its target probability?
- Relative performance: When tested on the same dates, does one model have more exceptions than the other, and is the difference supported by a paired comparison?
- Timing: Are exceptions spread through the evaluation period or concentrated in runs?
A model can pass a chosen test and still be inaccurate. A backtest covers a finite sample, so it may have limited ability to detect moderate misspecification, particularly when the target exception probability is small and only a few exceptions are expected. In addition, the exception indicator records whether a loss crossed the VaR threshold, not how far beyond that threshold it went. A passing exception test therefore cannot establish that every feature of the loss distribution is modeled correctly.
Results also depend on the evaluation design. Forecasts must be genuinely out of sample, and the horizon, confidence level, loss definition, and treatment of ties must be consistent. Serial dependence or changing exception probabilities can invalidate a simple independent-binomial benchmark. A test result should be reported with the assumptions and the sample size, rather than as a general certificate of model quality.
Check Your Understanding
For each problem, use the stated exception convention and show how the result informs a model comparison.
- Two continuous-loss models make 99% one-day VaR forecasts for the same 250 out-of-sample days. Model A has 4 exceptions and Model B has 1. Find the target count and each observed rate. What can the counts alone establish? (Answer: Target count \(250(0.01)=2.5\); rates are 1.6% and 0.4%. The counts alone do not establish which model is more accurate or whether either is calibrated.)
- A continuous-loss model produces 2 exceptions in 60 one-day forecasts at the 97.5% confidence level. Calculate the target exception count and observed rate, then compare the rate with the target. (Answer: Target count \(60(0.025)=1.5\); observed rate \(2/60=3.333\%\), compared with the 2.5% target.)
- Two continuous-loss models are tested on the same 40 days. There are 6 A-only exception dates and 1 B-only exception date. Under the independent-pairs null of equal exception probabilities, calculate the two-sided exact paired probability. (Answer: There are 7 discordant pairs; \(P(K\leq1\text{ or }K\geq6)=2(1+7)/2^7=0.1250\).)
- Two continuous-loss models produce 6 exceptions each in 120 days at 95% confidence. In Model A, all six exceptions are isolated; in Model B, all six occur consecutively. Find the expected count and identify the diagnostic distinction. (Answer: Expected count \(120(0.05)=6\); both counts match the target, but Model B's run raises a clustering concern not visible in the total.)
- A model produces exactly 5 exceptions in 1,000 continuous-loss, one-day forecasts at 99.5% confidence. State the expected count and explain why matching it does not prove accuracy. (Answer: Expected count \(1{,}000(0.005)=5\); matching the count is limited finite-sample evidence about the exception rate and does not prove the model is accurate.)