Why the Order of Exceptions Matters
A VaR backtest does more than count how often losses exceed a forecast. The order of those exceptions matters too. A model might have a count near its expected level but produce several exceptions in a short run. That pattern can indicate that losses are behaving differently from the model’s assumptions, even when the overall frequency alone does not look unusual.
Let \(L_t\) be the observed loss for period \(t\), and let \(V_t\) be the VaR forecast made for that period at confidence level \(c\). Define an exception indicator \(I_t\) to be 1 when \(L_t>V_t\), and 0 otherwise. The ordered observations \(I_1,\ldots,I_n\) are the exception sequence. The convention for ties should be specified and used consistently; for a continuous, correctly calibrated loss distribution, the probability of a tie at VaR is zero.
If the VaR forecast is correctly calibrated at confidence level \(c\), the exception probability is \(q=1-c\). Under independent, identically distributed exceptions, the total number \(K=\sum_{t=1}^n I_t\) has a binomial distribution with parameters \(n\) and \(q\). A binomial frequency test assesses whether the count is unusually large or small under those assumptions. It does not, by itself, assess whether the exceptions are independent over time.
Frequency and Clustering Are Different Questions
Frequency concerns the marginal probability of an exception: how often does \(I_t=1\)? Clustering concerns serial dependence: does observing an exception change the chance of an exception in the next period? These questions are related but not interchangeable. For example, a sequence with three exceptions can have the same count whether the exceptions are consecutive or widely separated.
A simple way to examine adjacent periods is to count transitions from each indicator value to the next. Let \(n_{ij}\) be the number of consecutive pairs \((I_t,I_{t+1})\) for which \(I_t=i\) and \(I_{t+1}=j\), where \(i,j\in\{0,1\}\). The four transition counts are \(n_{00}, n_{01}, n_{10}, n_{11}\), and their sum is \(n-1\), not \(n\).
| Transition | Interpretation |
|---|---|
| \(0\to0\) | A nonexception is followed by a nonexception |
| \(0\to1\) | A nonexception is followed by an exception |
| \(1\to0\) | An exception is followed by a nonexception |
| \(1\to1\) | An exception is followed by an exception |
The count \(n_{11}\) is direct evidence of adjacent exceptions, but it should be interpreted relative to the total transitions and the overall exception rate. A large count alone is not a complete test: if exceptions are common, some \(1\to1\) transitions are expected even without dependence.
A Transition-Based Independence Test
A standard diagnostic compares an independence model with a first-order Markov alternative. Under the independence model, the probability of an exception in the next period does not depend on the current indicator. Under the alternative, the probability of a next-period exception may differ according to whether the current period was an exception.
The transition likelihood under the Markov alternative is
The independence likelihood is
The likelihood-ratio statistic for independence is \(LR_{\mathrm{ind}}=2(\log L_{\mathrm{M}}-\log L_{\mathrm{I}})\). With sufficiently many transitions and under the null hypothesis of independence, it is commonly compared with a chi-square distribution with one degree of freedom. At a 5% significance level, the approximate critical value is 3.841.
The transition likelihood uses \(n-1\) pairs, so it conditions on the first indicator and does not use it as a transition destination. This is why the estimated common transition probability is based on \(n_{01}+n_{11}\), the number of exceptions among the \(n-1\) destination observations, rather than automatically using the exception count among all \(n\) observations.
Worked Examples
Worked Example: A Modest Count but a Run of Exceptions
A hypothetical risk team tests a 95% one-day VaR forecast over 20 days. The sequence contains 17 nonexceptions followed by 3 exceptions. Assess the count under an independent frequency model with \(q=0.05\), then calculate the transition-based independence statistic.
Method. First evaluate the upper-tail binomial probability for the count, then form transition counts and compare the two maximized transition likelihoods. This keeps the frequency question separate from the clustering question.
For the count test, \(K=3\) and \(K\sim\operatorname{Binomial}(20,0.05)\) under independent, correctly calibrated exceptions. The upper-tail probability is
At the 5% level, this count alone does not reject the independent frequency model. For the 19 consecutive transitions, the counts are \(n_{00}=16\), \(n_{01}=1\), \(n_{10}=0\), and \(n_{11}=2\). Thus \(\hat{\pi}=3/19\), \(\hat{\pi}_{01}=1/17\), and \(\hat{\pi}_{11}=1\).
The independence log-likelihood is
Under the Markov alternative, all transitions from an exception lead to another exception, so that row contributes \(2\log(1)=0\). The other row gives
Therefore, using the unrounded log-likelihoods,
This exceeds 3.841, suggesting exception dependence under the large-sample reference test. The contrast is the main point: the count test does not reject, while the sequence has strong clustering evidence. The sequence is short and some transition counts are small, so the chi-square approximation warrants caution; a formal application with sparse transitions may require a more suitable calibration. Common mistake avoided: treating a nonrejected count test as proof that exceptions are independent.
Worked Example: The Same Count, Spread Across the Sequence
A different hypothetical 12-day backtest has the sequence \(0,1,0,0,0,1,0,0,0,1,0,0\). There are three exceptions, each separated from the next. Calculate the transition counts and independence statistic.
Method. Count adjacent pairs directly from left to right. The transitions are \(01,10,00,00,01,10,00,00,01,10,00\), giving \(n_{00}=5\), \(n_{01}=3\), \(n_{10}=3\), and \(n_{11}=0\). There are 11 transitions, of which 3 have an exception as their destination, so \(\hat{\pi}=3/11\).
After a nonexception, the estimated probability of an exception is \(3/8\). After an exception, it is 0. Thus
The independence log-likelihood is
It follows that
This is below 3.841, so the approximate test does not reject independence at the 5% level. The exceptions are not adjacent in this sequence, and the statistic does not provide strong evidence of first-order clustering. This conclusion is not proof of independence, especially with only 12 observations. Common mistake avoided: concluding that a particular exception count must imply clustering without inspecting the order of the observations.
Worked Example: Why Dependence Changes Count Uncertainty
Suppose two consecutive hypothetical VaR forecasts each have exception probability \(q=0.05\), but persistent conditions make the probability of exceptions on both days \(P(I_1=1,I_2=1)=0.02\). Determine whether the two-day exception count has the binomial variance.
Method. Use the marginal probabilities and the specified joint probability to compute the covariance, then apply the variance formula for a sum.
Each indicator has variance \(q(1-q)=0.05(0.95)=0.0475\). Their covariance is
For \(K=I_1+I_2\), the mean remains \(E[K]=0.05+0.05=0.10\), but
If the indicators were independent, the binomial variance would instead be \(2(0.05)(0.95)=0.0950\). Also, the probability of two exceptions is 0.02 under the stated dependence, compared with \(0.05^2=0.0025\) under independence. The count still has the same expected value, but its distribution is not the binomial distribution. Common mistake avoided: assuming that correct marginal exception probabilities alone are enough to justify a binomial test.
Interpreting the Diagnostic Carefully
The transition test is a focused diagnostic, not a complete model assessment. A significant result indicates evidence against the tested independence restriction; it does not identify the economic cause of dependence or prove that a particular model feature is responsible. Conversely, a nonsignificant result can reflect limited data or low power rather than an absence of dependence.
The test’s chi-square reference is an approximation. It is most credible when the sample contains enough transitions of each type to estimate the transition probabilities reliably. If a row has no observations, or if counts are very small, estimates can lie at a boundary and the approximation can be poor. Report the transition table and the sample size alongside the statistic, and interpret sparse-sample results cautiously.
- Check frequency separately. A count test concerns the number of exceptions under its stated assumptions; it does not inspect their ordering.
- Check the sequence. Transition counts make the time-order information explicit and help reveal repeated exceptions.
- Check assumptions. A binomial count test requires independent Bernoulli indicators with a common exception probability, not merely a correctly specified marginal rate.
- Check the approximation. A small number of transitions can make a chi-square reference unreliable.
A common exam trap is to use the observed exception fraction in a binomial test and then interpret the result as a test of clustering. That calculation answers only a frequency question. Another is to count all \(n\) indicators when constructing transition probabilities: there are only \(n-1\) consecutive pairs, and their destination indicators exclude the initial observation.
Check Your Understanding
For each problem, distinguish what the exception count says from what the ordering or dependence assumptions say.
- A hypothetical 8-day sequence is \(0,0,1,0,0,1,0,0\). How many exceptions are there, and how many \(1\to1\) transitions occur? (Answer: 2 exceptions; 0 \(1\to1\) transitions.)
- A 50-day backtest has a 2% target exception probability. State the expected exception count under the independent model and one additional assumption needed to use a binomial count test. (Answer: Expected count \(=50(0.02)=1\); the indicators must be independent Bernoulli trials with common exception probability 0.02.)
- For the sequence \(0,0,0,1,1,0,0,0\), calculate \(n_{00},n_{01},n_{10},n_{11}\) and the approximate independence statistic. (Answer: \(n_{00}=4,n_{01}=1,n_{10}=1,n_{11}=1\); \(LR_{\mathrm{ind}}\approx0.5992\).)
- With independent exceptions and \(q=0.10\) over 10 periods, calculate \(P(K\geq3)\). (Answer: \(1-[0.9^{10}+10(0.1)(0.9^9)+45(0.1)^2(0.9^8)]=0.0702\).)
- A risk analyst reports a nonsignificant binomial count test and concludes that exceptions are not clustered. Identify the logical error and the relevant sequence diagnostic. (Answer: A count test assesses frequency, not ordering; inspect the transition counts and, when assumptions and sample size permit, use a transition-based independence test.)