What a VaR Backtest Counts
A VaR forecast is a threshold for losses over a specified horizon. Backtesting asks whether realized losses exceed that threshold with a frequency consistent with the forecast’s confidence level. The count is easy to calculate, but a sound interpretation depends on matching the VaR forecast to the realized loss, using the correct tail probability, and keeping the testing period clear.
Let \(L_t\) denote the realized loss for period \(t\), and let \(v_t\) be the VaR threshold forecast for that same period. For a confidence level \(\alpha\), a VaR threshold is commonly defined as the \(\alpha\)-quantile of the loss distribution. For a continuous loss distribution, the probability of exceeding a correctly calibrated threshold is \(1-\alpha\).
This tutorial uses the strict exceedance convention \(L_t>v_t\). A loss equal to VaR is therefore not counted as an exception. The convention must be applied consistently to both the observed losses and the model probability. This matters particularly for discrete distributions, where probability may be concentrated at the VaR threshold.
Expected Exception Counts
For a continuous loss model with a calibrated VaR forecast at confidence level \(\alpha\), the model assigns probability \(p=1-\alpha\) to an exception in each period. Over \(n\) periods at the same confidence level, the expected number of exceptions is \(np\). For example, a 99% VaR forecast has a 1% exceedance probability; over 250 comparable periods, the expected count is 2.5.
The expected count is an average implied by the model, not a promise that exactly that many exceptions will occur. Counts are integers, while an expectation need not be. A count above expectation is not, by itself, proof that a model is inadequate; interpreting how surprising a count is requires a statistical test, which is a separate step.
A useful justification follows directly from the definition of the count. If \(P(I_t=1)=p\), then \(E[I_t]=p\), because each indicator is 1 with probability \(p\) and 0 otherwise. By linearity of expectation,
This result for the expectation does not require the exceptions to be independent. It does require the stated exception probability for each period. Independence is relevant to the distribution of the count and to formal inference, not to this expectation calculation. If period-specific exception probabilities differ, use their sum instead:
Worked Example: A 99% Normal VaR Over 250 Days
A hypothetical risk team forecasts a one-day loss \(L_t\) in thousands of dollars using a normal model \(L_t\sim N(100,30^2)\). Assume this distribution is appropriate for each of 250 testing days, and use a 99% VaR with the strict exception rule \(L_t>v\). The standard normal 99th percentile is \(z_{0.99}\approx2.326348\). Calculate the VaR, expected exception count, and observed frequency if four exceptions occur.
Method. For a normal loss with mean \(\mu\) and standard deviation \(\sigma\), the \(\alpha\)-quantile is \(\mu+\sigma z_\alpha\). Here, the stated variance is \(30^2\), so the standard deviation is 30, not \(30^2\).
Thus, the forecast threshold is 169.79044 thousand dollars, or $169,790.44 to the nearest cent. Under the continuous normal model, the exceedance probability is \(1-0.99=0.01\). The expected count is
With four observed exceptions, the observed frequency is
The observed frequency is 0.6 percentage points above the model’s 1% tail probability, and the count is 1.5 above its expected value of 2.5. This is a descriptive comparison, not a formal conclusion about model performance. The method avoids the common mistake of treating 99% as the exception probability; 99% is the confidence level, while 1% is the upper-tail probability.
When Forecast Confidence Levels Differ
Not every backtest uses the same confidence level on every day. If confidence levels vary, the expected count is generally not \(n\) multiplied by one arbitrarily chosen tail probability. Instead, calculate each period’s model exception probability and sum those probabilities. For continuous loss models, a forecast at level \(\alpha_t\) has exception probability \(1-\alpha_t\).
Worked Example: Combining Two Confidence Levels
A hypothetical insurer evaluates 100 daily loss forecasts. On 60 days, it issued 99% VaR; on the other 40 days, it issued 95% VaR. Assume continuous loss models and correctly calibrated forecasts for their stated levels. There are three observed exceptions on the 99% days and one on the 95% days. Find the expected count, total observed frequency, and the observed frequencies within each group.
Method. Work by confidence-level group, because the periods have different tail probabilities. For the 99% group, \(p=1-0.99=0.01\), giving an expected count of \(60(0.01)=0.6\). For the 95% group, \(p=1-0.95=0.05\), giving \(40(0.05)=2.0\).
There were \(3+1=4\) observed exceptions in 100 days, so the total observed frequency is \(4/100=0.04=4\%\). Within the 99% group, the frequency is \(3/60=0.05=5\%\). Within the 95% group, it is \(1/40=0.025=2.5\%\).
The overall model-implied average exception probability is \(2.6/100=2.6\%\), not 1% or 5%. The observed total frequency is higher, but the two group-specific frequencies point in different directions. Comparing only the overall frequency can hide that difference. The method avoids applying one confidence level to all days when the forecasts actually used two.
Discrete Losses and the Actual Tail Probability
For a discrete loss distribution, the rule \(p=1-\alpha\) needs care. Define the VaR quantile by \(v_\alpha=\inf\{x:F_X(x)\geq\alpha\}\). When the distribution has an atom at \(v_\alpha\), the probability strictly above that threshold, \(P(X>v_\alpha)\), can be less than \(1-\alpha\). Under the strict exception convention in this tutorial, the relevant probability is the model’s actual probability of a loss strictly greater than VaR.
Worked Example: A Discrete VaR Threshold
A hypothetical portfolio has a one-period loss distribution with probability 0.94 of a loss of $0, probability 0.04 of a loss of $10,000, and probability 0.02 of a loss of $50,000. The portfolio is backtested at the 95% VaR level over 200 periods. Assume this distribution applies in every period. Seven losses exceed the VaR threshold. Find the VaR, the appropriate expected exception count, and the observed frequency.
Method. First build the cumulative probabilities: \(P(X\leq0)=0.94\), \(P(X\leq10{,}000)=0.94+0.04=0.98\), and \(P(X\leq50{,}000)=1\). The first loss amount whose cumulative probability is at least 0.95 is $10,000. Therefore, the 95% VaR is $10,000.
With the strict rule, only losses greater than $10,000 are exceptions. That occurs only at $50,000, with probability 0.02. Thus, the expected count is \(200(0.02)=4\), not \(200(0.05)=10\). The observed frequency is \(7/200=0.035=3.5\%\).
The observed frequency is 1.5 percentage points above the actual model exceedance probability of 2%. The distinction arises because 4% probability sits exactly at the VaR amount and does not meet the strict \(X>v_\alpha\) exception rule. The method avoids assuming that a discrete quantile necessarily leaves exactly \(1-\alpha\) probability strictly above it.
Interpreting Exception Counts Carefully
An exception count is meaningful only when the forecast and realized outcome refer to the same loss definition, portfolio, and time horizon. A one-day VaR should be compared with a corresponding one-day loss, not a weekly or cumulative loss. The observation period must also be specified: a count of four has a different interpretation over 100 periods than over 500.
For forecasts with a common continuous-model confidence level, compare \(K\) with \(n(1-\alpha)\), and compare \(K/n\) with \(1-\alpha\). For varying levels or a discrete model, use each period’s actual modeled exception probability. A compact reporting table can make these pieces explicit.
| Quantity | Calculation | Interpretation |
|---|---|---|
| Expected count | \(\sum_{t=1}^{n}p_t\) | Model-implied average number of exceptions |
| Observed count | \(K=\sum_{t=1}^{n}I_t\) | Number of realized losses above their thresholds |
| Observed frequency | \(K/n\) | Share of tested periods with exceptions |
| Difference in counts | \(K-E[K]\) | Descriptive gap; not by itself a significance test |
Exception counts also discard timing information. Two models can have the same number of exceptions while one has them spread across the testing period and the other has them close together. The count addresses overall frequency; it does not establish that exceptions are independent or reveal whether they cluster. Those are distinct questions.
Common Mistakes
- Using the confidence level as the exception probability. At 99% confidence, the continuous-model exception probability is \(1-0.99=0.01\), not 0.99.
- Comparing counts without the period length. Always state \(n\), the expected count, and the observed frequency; a raw count alone is incomplete.
- Applying one tail probability to varying confidence levels. Calculate the appropriate \(p_t\) for each group or period and sum them.
- Ignoring ties for discrete losses. With a strict exceedance rule, a loss equal to VaR is not an exception. Calculate \(P(X>v_\alpha)\) directly when the model is discrete.
- Treating a difference from expectation as a formal verdict. The expected count is not a cutoff that observed counts must equal. A statistical assessment needs an appropriate test and its assumptions.
- Mixing horizons. Pair each realized loss with a VaR forecast for the same horizon and risk measure.
Check Your Understanding
For each problem, use the stated strict exceedance convention and distinguish expected counts from observed counts.
- A hypothetical model issues 97.5% VaR for 80 independent testing periods with continuous losses. Three losses exceed their thresholds. Find the expected exception count and observed frequency. (Answer: Expected count \(=80(0.025)=2\); observed frequency \(=3/80=3.75\%\).)
- A continuous-loss model is tested over 120 days using 99% VaR. One realized loss exceeds its threshold. Calculate the expected count and observed frequency, and state the difference in counts. (Answer: Expected count \(=120(0.01)=1.2\); observed frequency \(=1/120\approx0.8333\%\); count difference \(=1-1.2=-0.2\).)
- A hypothetical portfolio has loss probabilities \(P(X=0)=0.90\), \(P(X=5{,}000)=0.07\), and \(P(X=30{,}000)=0.03\). At 95% confidence, find VaR and the strict exceedance probability. Over 100 periods, find the expected exception count. (Answer: VaR \(=\$5{,}000\); strict exceedance probability \(=0.03\); expected count \(=3\).)
- Over 100 periods, a continuous-loss model uses 99% VaR on 30 days and 95% VaR on 70 days. Five exceptions are observed. Find the expected count and observed frequency. (Answer: Expected count \(=30(0.01)+70(0.05)=0.3+3.5=3.8\); observed frequency \(=5/100=5\%\).)
- A model has a continuous loss distribution with 98% VaR for 50 periods. No exceptions occur. Calculate the expected count and observed frequency. Explain what the comparison alone establishes. (Answer: Expected count \(=50(0.02)=1\); observed frequency \(=0\%\); the count is one below expectation, but this descriptive difference alone does not establish whether the model is statistically inadequate.)