Why Test VaR Exception Counts?
A VaR backtest compares forecasts with realized losses. For a confidence level of \(q\), a correctly calibrated one-period VaR forecast should be exceeded with probability \(1-q\), under the model’s stated assumptions. The count of exceedances over a fixed evaluation period therefore has a reference distribution. A binomial test uses that distribution to assess whether the observed count is unusually high or low.
This test answers a deliberately narrow question: is the total number of exceptions consistent with the assumed exception probability and independence? It does not establish that a VaR model is correct in every respect. In particular, the count alone does not describe the order or timing of exceptions.
The continuity condition makes the exceedance probability exactly \(1-q\) when VaR is the \(q\)-quantile. For a discrete loss distribution, probability mass at the quantile can make the strict-exceedance probability differ from \(1-q\). In that setting, the appropriate exception probability must be specified from the model and the chosen quantile convention; it should not be assumed to equal \(1-q\) automatically.
Deriving the Binomial Reference Distribution
For the standard exact test, assume that each forecast has the same exception probability \(p_0\) and that the exception indicators are independent across evaluation periods. Under those assumptions, each \(I_t\) is Bernoulli with probability \(p_0\) of being 1. The total count has a binomial distribution:
To see why, fix a particular sequence of \(n\) indicators containing exactly \(k\) exceptions. Independence gives that sequence probability \(p_0^k(1-p_0)^{n-k}\). There are \(\binom{n}{k}\) ways to choose which \(k\) periods have exceptions. These disjoint sequences have the same probability, so adding their probabilities gives the binomial formula.
The binomial model is not justified just because the forecasts have the same nominal confidence level. It also requires independent exception indicators. If exception probabilities vary by period, or if indicators are dependent, the binomial distribution does not generally apply. The test described here is for the stated independent, equally calibrated setting.
Upper-Tail Tests and Critical Regions
A common backtesting concern is too many exceptions, which corresponds to an upper-tail test. The null hypothesis is that the forecasts are correctly calibrated with exception probability \(p_0\). The alternative is that the exception probability is greater than \(p_0\). For a chosen significance level \(\alpha\), an exact upper-tail critical count is
The test rejects the null hypothesis when \(K\ge c\). Because \(K\) can take only integer values, it may not be possible to make the probability of rejection exactly \(\alpha\). The actual size of the test is \(P(K\ge c\mid p_0)\), which is at most \(\alpha\). The next lower cutoff must fail the size requirement: \(P(K\ge c-1\mid p_0)>\alpha\).
Worked Example: Finding a Critical Count for 250 Forecasts
A hypothetical risk team evaluates 250 daily forecasts at a 99% VaR confidence level. Assume the forecasts are correctly calibrated and their exception indicators are independent. At significance level \(\alpha=0.05\), find the upper-tail critical count and the actual size of the test.
Method and assumptions. The exception probability under the null is \(p_0=1-0.99=0.01\), so \(K\sim\operatorname{Binomial}(250,0.01)\). The binomial probability formula gives, for example, \(P(K=5)=\binom{250}{5}(0.01)^5(0.99)^{245}\approx0.0666\). The relevant exact upper-tail probabilities, rounded to four decimal places, are
| Count or tail | Probability |
|---|---|
| \(P(K=5)\) | 0.0666 |
| \(P(K=6)\) | 0.0275 |
| \(P(K=7)\) | 0.0097 |
| \(P(K=8)\) | 0.0030 |
| \(P(K\ge6)\) | 0.0412 |
| \(P(K\ge5)\) | 0.1078 |
Since \(P(K\ge6)=0.0412\le0.05\), a cutoff of 6 satisfies the size requirement. A cutoff of 5 does not, because \(P(K\ge5)=0.1078>0.05\). Therefore the critical region is \(K\ge6\), and the actual size is approximately 0.0412. If the observed count were 6, its upper-tail p-value would be approximately 0.0412, so the null would be rejected at the 5% level.
As an additional tail check, the exact binomial calculation gives \(P(K\ge10)\approx0.0002502\). The tail probability is small because it includes only counts of 10 or more, not counts of 9 or more.
Calculating and Interpreting a P-Value
For an upper-tail test, the p-value is the probability, assuming the null model, of observing the actual count or a still larger count:
A p-value is not the probability that the null hypothesis is true. It is a tail probability calculated under the null model. If it is no greater than the prespecified \(\alpha\), the result is statistically significant for that test. Otherwise, the test does not reject the null; that is not proof that the forecasts are correctly calibrated.
Worked Example: Evaluating Three Exceptions in a Short Test
For a hypothetical set of 20 independent forecasts at 95% confidence, suppose 3 exceptions are observed. Test for too many exceptions at \(\alpha=0.05\), and calculate the upper-tail p-value.
Method and assumptions. Under the null, \(p_0=1-0.95=0.05\) and \(K\sim\operatorname{Binomial}(20,0.05)\). Since the alternative is too many exceptions, use the upper tail. The probabilities for counts 0, 1, and 2 are
Their sum is \(0.3584859+0.3773536+0.1886768=0.9245163\). Therefore \(P(K\ge3)=1-0.9245163=0.0754837\), or 0.0755 to four decimal places. Because \(0.0755>0.05\), the observed count does not reject the null at the 5% level.
The discrete critical region makes the distinction especially clear. The probability of exactly 3 exceptions is \(\binom{20}{3}(0.05)^3(0.95)^{17}\approx0.05958\), so \(P(K\ge4)\approx0.0159\), while \(P(K\ge3)\approx0.0755\). Thus the 5% upper-tail critical region is \(K\ge4\), not \(K\ge3\). The method avoids treating the observed count as significant merely because it is above its expected value \(np_0=1\).
Lower-Tail and Two-Sided Questions
A low exception count can also be examined. For a prespecified alternative of too few exceptions, the lower-tail p-value is \(P(K\le k_{\mathrm{obs}}\mid p_0)\). A two-sided test instead considers departures in either direction. There are multiple conventions for defining an exact two-sided p-value in a discrete distribution, so the convention should be stated. One common equal-tail convention is
This convention doubles the smaller inclusive tail probability and caps the result at 1. It is not the only possible exact two-sided convention. A test should not switch between one- and two-sided interpretations after seeing which tail contains the observation.
Worked Example: Separating a Lower-Tail Result From a Two-Sided Result
A hypothetical evaluation has 30 independent forecasts at 90% confidence and observes no exceptions. Calculate the lower-tail p-value and the equal-tail two-sided p-value. Use \(\alpha=0.05\).
Method and assumptions. The null exception probability is \(p_0=1-0.90=0.10\), so \(K\sim\operatorname{Binomial}(30,0.10)\). With \(k_{\mathrm{obs}}=0\), the lower-tail probability is just \(P(K\le0)=P(K=0)=(0.90)^{30}=0.0423912\).
For a prespecified lower-tail test, \(0.0424<0.05\), so the null is rejected in favor of too few exceptions. For the stated equal-tail two-sided convention, \(P(K\ge0)=1\), and thus \(\min(1,2\min(0.0423912,1))=0.0847824\), or 0.0848 to four decimal places. The two-sided test does not reject at 5%. The different decisions result from different alternatives, not from a change in the observed data.
Common Mistakes and Practical Interpretation
- Using the wrong tail. A test for too many exceptions uses \(P(K\ge k_{\mathrm{obs}})\), not \(P(K\le k_{\mathrm{obs}})\). Define the alternative before calculating the p-value.
- Comparing the count only with its mean. The expected count \(np_0\) is not a critical value. In the 250-forecast example it is 2.5, but whether a count is significant depends on the binomial tail probabilities and the chosen \(\alpha\).
- Ignoring discreteness. An exact critical region can have actual size below the nominal significance level. Report the cutoff and its actual tail probability.
- Assuming calibration guarantees independence. Correct marginal exception probabilities do not by themselves imply independent indicators. The binomial test requires both the stated probability and independence.
- Interpreting non-rejection as validation. A non-significant result means the count is not sufficiently unusual under this test; it does not prove that the model is correct.
The binomial exception-count test is a transparent way to assess aggregate frequency against a specified reference probability. Its conclusion is conditional on the model assumptions and the chosen tail. It is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Check Your Understanding
For each problem, use the exact binomial model under the stated assumptions. Round reported probabilities to four decimal places.
- A hypothetical model generates 10 independent forecasts at 90% confidence and has 3 exceptions. Find the upper-tail p-value and decide whether to reject at \(\alpha=0.05\). (Answer: \(P(K\ge3)=1-\sum_{j=0}^{2}\binom{10}{j}(0.10)^j(0.90)^{10-j}\approx0.0702\); do not reject.)
- For 20 independent forecasts at 95% confidence, find the smallest upper-tail critical count at \(\alpha=0.05\). (Answer: \(K\ge4\), with actual size \(P(K\ge4)\approx0.0159\); \(P(K\ge3)\approx0.0755\).)
- A hypothetical test uses 100 independent forecasts at 99% confidence and observes 4 exceptions. Calculate the upper-tail p-value and test at 5%. (Answer: \(1-\sum_{j=0}^{3}\binom{100}{j}(0.01)^j(0.99)^{100-j}\approx0.0184\); reject.)
- For 30 independent forecasts at 90% confidence, 0 exceptions are observed. Give the lower-tail p-value and the equal-tail two-sided p-value defined in this tutorial. (Answer: lower-tail \(0.0424\); equal-tail two-sided \(0.0848\).)
- For 250 independent forecasts at 99% confidence, identify the exact 5% upper-tail critical region using the binomial probabilities. (Answer: reject for \(K\ge6\); actual size \(P(K\ge6)\approx0.0412\), while \(P(K\ge5)\approx0.1078\).)