Why Tail Fit Needs Its Own Diagnostics
A model can describe the center of a loss distribution reasonably well and still misrepresent the relatively few large losses that determine a high-level risk estimate. For example, a fitted model may reproduce the overall average and most of the observed losses, yet predict too few losses above its 99th-percentile VaR or understate how large those exceedances tend to be. Tail-fit diagnostics therefore focus on losses near and beyond the target risk threshold, not only on overall fit.
Two distinct questions are useful. First, does the fitted model assign a plausible probability to losses exceeding its target VaR? Second, when losses do exceed a relevant threshold, are their sizes consistent with the model's predicted tail severity? A count comparison addresses the first question; an average-excess or tail-average comparison addresses the second. Neither comparison alone establishes that a model is correct.
Exceedance Counts and Tail Severity
Suppose a model specifies a threshold \(q\), and \(X_1,\ldots,X_n\) are independent observations from the model's loss distribution. If the model's probability of exceeding \(q\) is \(r=P(X>q)\), then the number \(K\) of sample losses exceeding \(q\) has a binomial distribution with parameters \(n\) and \(r\). Thus its expected value is \(nr\), and its standard deviation is \(\sqrt{nr(1-r)}\). For a fitted \(p\)-level VaR in a continuous model, the model predicts \(r=1-p\), so the expected number of exceedances is \(n(1-p)\).
That expected count is not a quota that a sample must match exactly. Random samples vary. A useful diagnostic compares the observed count with the expected count and its binomial variability, while also considering whether the model parameters were estimated from the same observations. In that latter case, the simple binomial calculation is a reference point, not a complete account of uncertainty.
Counts do not describe the sizes of exceedances. If \(q\) is a threshold, the excess is \(X-q\) conditional on \(X>q\). A model may predict a reasonable number of exceedances but still underestimate their average excess. Conversely, a large observed count does not by itself establish that the observed exceedances are unusually severe.
For a continuous model with \(q=q_p\), the conditional mean in this identity is \(\operatorname{TVaR}_p\). In a finite sample, the average of all observations above a fixed fitted threshold estimates a conditional mean at that threshold; it is not automatically the empirical TVaR at the sample's own quantile. That distinction matters when the observed exceedance count differs from \(n(1-p)\).
Why These Comparisons Work
The count result follows by defining an indicator \(I_j\) that equals 1 when \(X_j>q\) and 0 otherwise. Under independent, identically distributed observations, each indicator is Bernoulli with probability \(r\). Since \(K=\sum_{j=1}^{n} I_j\), the sum has a binomial distribution, with mean \(nr\) and variance \(nr(1-r)\). This justifies comparing an observed count with that distribution's center and spread, rather than demanding an exact match.
The tail-average identity follows by writing \(X=q+(X-q)\) on the event \(X>q\), then taking the conditional expectation of both sides. For a continuous distribution, the upper \(1-p\) fraction of probability lies above \(q_p\), so that conditional mean is the TVaR definition. With an atom at the quantile, conditioning only on \(X>q_p\) can omit some of the probability needed to form the upper \(1-p\) tail; the positive-part formula in the definition handles that case.
State the tail probability \(1-p\), and specify whether the threshold is a fitted model VaR or an empirical quantile.
Count losses strictly above the stated threshold. Compare that count with \(nP(X>q)\), allowing for sampling variation.
Use the actual losses above the same threshold to calculate their mean excess or conditional average. Do not substitute a top-tail average calculated at a different threshold.
Report frequency and severity findings separately, and qualify conclusions if only partial order statistics or a small number of tail observations are available.
Worked Diagnostics
Worked Example: What Partial Upper-Tail Data Can Establish
A hypothetical sample contains 100 losses. Its 95th ordered loss is $500, and its five largest losses are $500, $600, $700, $900, and $2,500. A fitted model reports a 95th-percentile VaR of $447.21 and a 95th-percentile TVaR of $894.43. Determine what the supplied information establishes about exceedance frequency and severity.
Method. Compare the reported model threshold with the ordered losses, then calculate only those empirical summaries supported by the stated data. The key trap is treating the five largest observations as all the losses above the fitted threshold.
The fitted threshold is $447.21. The 95th ordered loss is $500, and $500>$447.21. Therefore, observations at ranks 95 through 100 are all above the fitted threshold: there are at least six exceedances. The exact count is not determined because the values at lower ranks are not supplied; some or all may also exceed $447.21.
The five largest losses sum to \(500+600+700+900+2{,}500=5{,}200\) dollars. Their average is \(5{,}200/5=\$1{,}040\). This is the empirical average of the five largest losses, or the upper 5% average under the stated ordering. It is not the average of all losses exceeding $447.21, because at least six losses exceed that threshold and the additional exceedances are unknown.
The model's 95% VaR implies a 5% exceedance probability, or five expected exceedances in 100 observations if the model were correct and the threshold fixed. The sample has at least six exceedances; the exact count cannot be compared with five from the supplied information. Similarly, the upper-5% empirical average of $1,040 is not a direct comparison with the fitted conditional mean above $447.21. To make that comparison, all observations above $447.21 and their total would be needed. This is the supported conclusion; stronger claims about count agreement or conditional severity would overstate the data.
Worked Example: Checking Frequency and Severity Under an Exponential Fit
A hypothetical fitted model assumes \(X\) is exponential with mean $100, so its survival function is \(P(X>x)=e^{-x/100}\) for \(x\ge0\). In a sample of 1,000 independent losses, the model's 99th-percentile VaR threshold has 16 observed exceedances, whose total excess above that threshold is $2,200. Calculate the fitted threshold and TVaR, then compare the observed count and severity with the fit.
Model and method. The exponential model has rate \(1/100\) per dollar. Solve its survival probability for the VaR, use the memoryless property for the conditional tail mean, and assess count variation with a binomial reference. A common mistake is to compare the observed excess total directly with TVaR rather than first finding the average excess.
At the 99th percentile, the survival probability is \(1-p=0.01\). Hence \(e^{-q_{0.99}/100}=0.01\), so \(q_{0.99}=-100\ln(0.01)=100\ln(100)=\$460.52\), rounded to cents. For an exponential loss, conditional on exceeding \(q\), the mean excess is $100. Therefore, \(\operatorname{TVaR}_{0.99}=460.517\ldots+100=\$560.52\), rounded to cents.
The model expects \(1{,}000(0.01)=10\) exceedances. The binomial standard deviation is \(\sqrt{1{,}000(0.01)(0.99)}=\sqrt{9.9}=3.146\). The observed count of 16 is 6 above expectation, or \(6/3.146=1.91\) standard deviations above it. This is a diagnostic signal, not by itself a definitive rejection of the model.
The observed mean excess is \(2{,}200/16=\$137.50\). Thus the observed average loss among these exceedances is \(\$460.52+\$137.50=\$598.02\), rounded to cents. This exceeds the fitted TVaR of $560.52 by $37.50. The frequency and severity comparisons both point above the model's predictions, but they answer different questions and should be reported separately.
Worked Example: Overall Fit Can Conceal a Weak 99% Tail
A hypothetical sample of 1,000 losses is fitted with a normal model \(X\sim N(100,20^2)\), where the mean and standard deviation are in thousands of dollars. Suppose central summaries and a central fit plot appear acceptable. At the fitted model's 99th-percentile threshold, 22 observations exceed the threshold, and their average loss is 180 thousand dollars. Assess the tail evidence. Use \(z_{0.99}=2.3263\) and \(\phi(2.3263)=0.02665\), where \(\phi\) is the standard normal density.
Method. Calculate the fitted threshold and conditional tail mean, then compare the observed exceedance count and observed mean at that same fitted threshold. The trap is to infer tail reliability from central fit or to label the observed conditional mean as empirical 99% TVaR without checking its quantile convention.
The fitted threshold is \(q_{0.99}=100+20(2.3263)=146.526\) thousand dollars, or $146,526 when expressed in dollars. The model predicts a 1% exceedance probability, so the expected number in 1,000 observations is 10. Its binomial standard deviation is \(\sqrt{1{,}000(0.01)(0.99)}=\sqrt{9.9}=3.146\). The observed count is 12 above expectation, a standardized difference of \(12/3.146=3.81\).
For a normal model, the conditional mean above the threshold is \(\mu+\sigma\phi(z)/(1-p)\). Here this gives \(100+20(0.02665/0.01)=100+53.30=153.30\) thousand dollars, or $153,300. The observed average of $180 thousand among losses above the fitted threshold is $26.70 thousand higher. This is a conditional average at the fitted threshold; because the observed exceedance count is 22 rather than 10, it is not automatically the empirical 99% TVaR.
The count and conditional average both indicate that the fitted tail may understate risk, even though central fit looks acceptable. The evidence does not prove which alternative model is appropriate, and the normal assumption may not adequately represent a heavy upper tail. A prudent conclusion is that the fit needs specific scrutiny near the 99% threshold before relying on its VaR or TVaR.
Interpreting Tail-Fit Evidence
A tail diagnostic is most informative when the model threshold, empirical threshold, and averaging rule are clearly distinguished. For example, counting observations above the fitted \(q_p\) evaluates calibration at that model threshold. Averaging the largest fraction \(1-p\) of the sample instead describes an empirical upper-tail average. When the observed number above \(q_p\) differs from \(n(1-p)\), those quantities generally use different observations and need not agree.
Tail counts also become noisy as the target probability grows. At a 99.5% VaR level, a sample of 200 observations has only one expected exceedance under a continuous correctly specified model. The absence of an exceedance in that sample would not establish that the model is safe; nor would one exceedance alone determine the shape of the far tail. Sparse data limit what any empirical diagnostic can resolve.
Useful practice is to inspect more than one tail level, record both counts and exceedance sizes, and compare empirical and fitted survival probabilities near the intended risk threshold. A pattern of repeated excess counts or unusually large average excesses deserves attention even when the overall fit appears adequate. These are diagnostic warnings, not automatic proofs of model failure: sampling variation, parameter estimation, data quality, and the intended use of the estimate all matter.
Check Your Understanding
For each problem, identify the relevant tail calculation and keep model-threshold comparisons distinct from empirical quantile summaries.
- A Pareto Type I loss has survival function \(P(X>x)=(50/x)^3\) for \(x\ge50\). Find its 99th-percentile VaR and TVaR. (Answer: VaR \(=\$232.08\); TVaR \(=\$348.12\).)
- A fitted exponential loss model has mean $80. Find its 95th-percentile VaR and TVaR. (Answer: VaR \(=\$239.66\); TVaR \(=\$319.66\).)
- A model's target threshold has exceedance probability 0.025. For 400 independent observations, find the expected exceedance count and binomial standard deviation. (Answer: Expected count \(=10\); standard deviation \(=\sqrt{9.75}\approx3.122\).)
- A fitted model gives a 90th-percentile threshold of $120. In a sample of 200, 12 losses are strictly above $120. State the model-expected count and observed exceedance fraction. (Answer: Expected count \(=20\); observed fraction \(=12/200=0.0600\), compared with the model probability \(0.1000\).)
- A sample of 20 losses is ordered, with its 18th, 19th, and 20th values equal to $40, $60, and $100. Using the empirical quantile \(x_{(\lceil np\rceil)}\), find the 90th-percentile VaR and the average of the largest 10% of observations. (Answer: Empirical VaR \(=\$40\); upper-10% average \(=(60+100)/2=\$80\).)