Why Compare Quantiles?
A severity model is often used to describe not just the center of a loss distribution but also loss levels associated with particular probabilities. For example, a model's 90th percentile is the loss level that the model assigns to the largest 10% of losses. Comparing such model quantiles with quantiles calculated directly from observed losses gives an interpretable check on how well the model represents different parts of the distribution.
A difference is not automatically evidence that the model is wrong. A sample quantile varies from sample to sample, even when observations truly come from the fitted distribution. Conversely, differences that are consistently small at some probability levels but large and directional at others can point to a model that describes one part of the loss distribution better than another. The comparison is a diagnostic, not by itself a formal test.
This inverse-empirical-CDF convention selects an observed loss rather than interpolating between observations. Some software uses an interpolation convention instead, so a comparison should state how empirical quantiles were calculated. Throughout this tutorial, the inverse-empirical-CDF convention is used. Ties are allowed; they can make the empirical distribution jump by more than \(1/n\).
Model Quantiles and Quantile Differences
Let \(F_\theta\) be a parametric severity distribution with parameter or parameter vector \(\theta\). Its \(p\)-quantile is \(Q_\theta(p)=\inf\{x:F_\theta(x)\ge p\}\). When a model has been fitted, write \(\widehat\theta\) for the estimated parameter and compare \(\widehat Q_n(p)\) with \(Q_{\widehat\theta}(p)\) at the same probability level.
A positive \(D(p)\) means the empirical quantile is above the fitted-model quantile; a negative value means it is below. It is useful to keep the sign, rather than comparing only absolute differences, because a sequence of differences that tends to become positive at higher probability levels may be more informative than a single large difference. Differences can also be expressed relative to the model quantile as \(D(p)/Q_{\widehat\theta}(p)\), provided that the model quantile is positive.
Compare matched probability levels, such as 25%, 50%, 75%, and 90%. Comparing a sample median with a model's 90th percentile does not diagnose agreement at either probability level. In a table, report the probability level, the empirical quantile, the model quantile, and their signed difference. State units and rounding consistently.
Why Empirical Quantiles Vary
Suppose, temporarily, that observations \(X_1,\ldots,X_n\) are independent and identically distributed from a continuous distribution \(F\), and let \(q_p\) be its true \(p\)-quantile. The number of sample observations at or below \(q_p\) is a sum of \(n\) independent indicator variables. Each indicator equals 1 with probability \(p\), so the count has a binomial distribution with parameters \(n\) and \(p\).
If \(k=\lceil np\rceil\), the empirical \(p\)-quantile is at or below \(q_p\) exactly when at least \(k\) observations are at or below \(q_p\). Thus:
This gives a direct justification for why the empirical quantile need not equal the true quantile in a finite sample: the number of observations falling on either side of \(q_p\) is random. The formula assumes independent observations from the same continuous distribution and refers to a fixed true quantile. It does not, on its own, account for estimating model parameters from the same data or provide a formal comparison test.
Worked Comparisons
Worked Example: A Lognormal Model at the 90th Percentile
A hypothetical sample of 10 losses, in dollars, has ordered values \(18,24,31,38,44,51,63,72,90,140\). A fitted lognormal model is specified by \(\ln X\sim N(\ln 50,0.4^2)\). Compare the empirical and model 90th percentiles and interpret the signed difference. Use \(z_{0.90}=1.28155\), where \(z_p\) is the \(p\)-quantile of the standard normal distribution.
Method. Use the stated empirical-quantile convention to select the sample order statistic, then transform the standard-normal quantile back to the loss scale. This avoids confusing the model median with its 90th percentile.
Empirical quantile. Here \(n=10\) and \(p=0.90\), so the selected rank is \(\lceil np\rceil=\lceil9\rceil=9\). The ninth ordered loss is \(\widehat Q_{10}(0.90)=90\) dollars.
Model quantile. Since \(\ln X\) is normal with mean \(\ln 50\) and standard deviation \(0.4\), the model quantile is obtained by exponentiating the corresponding normal quantile:
Difference and interpretation. The signed difference is \(90.00-83.48=6.52\) dollars. Using the unrounded model value, the difference is approximately \(6.517\) dollars; both amounts are shown to the nearest cent as \(6.52\) dollars. The sample's 90th percentile is above the fitted model's 90th percentile, but one sample quantile does not establish systematic model mismatch.
Check. The model median is \(50\) dollars, because the median of \(\ln X\) is \(\ln 50\). A 90th percentile above \(50\) is consistent with the positive normal quantile used here.
Worked Example: Comparing Several Quantiles Under an Exponential Model
A hypothetical sample of eight losses, in thousands of dollars, is \(10,12,14,18,22,30,45,80\), in increasing order. Compare its empirical quantiles at \(p=0.25,0.50,0.75,\) and \(0.875\) with those from a fitted exponential model with mean \(25\) thousand dollars.
Method. Select each empirical order statistic using \(\lceil np\rceil\). For an exponential distribution with mean \(25\), the rate is \(\lambda=1/25=0.04\) per thousand dollars and the distribution function is \(F(x)=1-e^{-0.04x}\) for \(x\ge0\). Solving \(p=1-e^{-0.04x}\) gives \(Q(p)=-\ln(1-p)/0.04\).
At \(p=0.25\), the empirical rank is \(\lceil8(0.25)\rceil=2\), so the empirical quantile is \(12\). The model quantile is \(-\ln(0.75)/0.04=7.192\), or \(7.19\) thousand dollars. The difference is \(12-7.192=4.808\), or \(4.81\).
At \(p=0.50\), the rank is \(\lceil4\rceil=4\), giving empirical quantile \(18\). The model quantile is \(-\ln(0.50)/0.04=17.329\), or \(17.33\). The difference is \(18-17.329=0.671\), or \(0.67\).
At \(p=0.75\), the rank is \(\lceil6\rceil=6\), giving empirical quantile \(30\). The model quantile is \(-\ln(0.25)/0.04=34.657\), or \(34.66\). The difference is \(30-34.657=-4.657\), or \(-4.66\).
At \(p=0.875\), the rank is \(\lceil7\rceil=7\), giving empirical quantile \(45\). The model quantile is \(-\ln(0.125)/0.04=51.986\), or \(51.99\). The difference is \(45-51.986=-6.986\), or \(-6.99\).
The comparison is summarized below. Quantiles and differences are rounded to two decimal places in thousands of dollars.
| Probability \(p\) | Empirical quantile | Model quantile | Empirical minus model |
|---|---|---|---|
| 0.25 | 12.00 | 7.19 | 4.81 |
| 0.50 | 18.00 | 17.33 | 0.67 |
| 0.75 | 30.00 | 34.66 | -4.66 |
| 0.875 | 45.00 | 51.99 | -6.99 |
Interpretation and check. The empirical quantile is above the model at the lower two levels and below it at the upper two. The sign changes, so it would be inaccurate to say the model is uniformly too high or too low. The two upper empirical quantiles are below the corresponding model quantiles, but the sample is small and the empirical quantiles are variable. The values are in thousands, and each difference equals the empirical entry minus the model entry.
Worked Example: Reading a Pattern Across a Lognormal Fit
A hypothetical set of eight ordered losses, in dollars, is \(12,18,25,32,45,70,110,180\). Compare its empirical quantiles at \(p=0.25,0.50,0.75,\) and \(0.875\) with a fitted model satisfying \(\ln X\sim N(\ln 40,0.5^2)\). State what the direction of the differences suggests, and what it cannot establish.
Method. As in the preceding example, the empirical rank is \(\lceil8p\rceil\). For the model, use \(Q(p)=\exp(\ln40+0.5z_p)\). The standard-normal quantiles used are \(z_{0.25}=-0.67449\), \(z_{0.50}=0\), \(z_{0.75}=0.67449\), and \(z_{0.875}=1.15035\).
At \(p=0.25\), the rank is 2, so the empirical quantile is \(18\). The model quantile is \(\exp(\ln40+0.5(-0.67449))=40\exp(-0.337245)\approx28.55\). The difference is \(18-28.55=-10.55\) dollars.
At \(p=0.50\), the rank is 4, so the empirical quantile is \(32\). The model quantile is \(\exp(\ln40)=40.00\). The difference is \(32-40=-8.00\) dollars.
At \(p=0.75\), the rank is 6, so the empirical quantile is \(70\). The model quantile is \(40\exp(0.337245)\approx56.04\). The difference is \(70-56.04=13.96\) dollars.
At \(p=0.875\), the rank is 7, so the empirical quantile is \(110\). The model quantile is \(40\exp(0.5(1.15035))=40\exp(0.575175)\approx71.10\). The difference is \(110-71.10=38.90\) dollars.
Interpretation and check. The empirical quantiles are below the model at the lower two levels and above it at the upper two, with a larger positive difference at the highest level. This pattern is consistent with the sample having a more pronounced upper tail than the fitted model represents, but it is not proof of that conclusion: there are only eight observations, and the largest empirical quantiles are particularly sensitive to which losses happened to be sampled. The model median is \(40\), matching its log-scale location parameter, which checks the calculation at \(p=0.50\).
Interpreting Comparisons Carefully
A comparison is more informative when it uses several probability levels and retains both the sign and scale of each difference. Differences that are small near the center but increase in one direction toward higher quantiles may flag a potential mismatch in the upper portion of the distribution. Differences that switch signs suggest that the model may be positioned differently from the sample across the distribution, rather than simply shifted up or down. These descriptions are clues for further assessment, not automatic conclusions.
The binomial argument above explains why the empirical quantile fluctuates, but it assumes the true distribution is fixed and known. In practice, a fitted model usually has parameters estimated from data. If the same observations are used both to fit the model and to calculate empirical quantiles, those two sets of quantiles are not independent. A close in-sample match is not independent validation, and a visible difference is not by itself a calibrated significance result. A separate validation sample can provide a more independent comparison, although its empirical quantiles also vary.
Quantile comparisons complement other model assessments. They communicate differences at specific probability levels in the original loss units, while a distribution-level goodness-of-fit assessment addresses a broader question. No single probability level captures the entire distribution. In particular, results at a few levels should not be extrapolated beyond the observed sample or treated as a precise characterization of extremely rare losses.
Common Mistakes
- Using the wrong empirical rank. Under the convention used here, the rank is \(\lceil np\rceil\), not automatically the nearest integer to \(np\), and not an interpolated value. State the convention when comparing results.
- Comparing different probability levels. Compare the empirical and model quantiles at the same \(p\); otherwise the difference does not have the intended interpretation.
- Forgetting the distribution's scale. The exponential quantile uses the rate \(\lambda\), while a lognormal quantile must be exponentiated after calculating the normal-scale quantile.
- Calling one difference proof of model failure. A single empirical quantile can depart from its model counterpart because of sampling variation. Look at several levels and consider sample size and fitting context.
- Ignoring units or signs. Record whether losses are in dollars or thousands of dollars, and define differences consistently as empirical minus model (or state another convention explicitly).
Check Your Understanding
For each question, use the inverse-empirical-CDF convention where a sample is given. Round dollar amounts to the nearest cent when appropriate.
- A hypothetical ordered sample of six losses is \(8,11,15,19,27,44\) dollars. Find the empirical median. (Answer: \(15\) dollars.)
- A sample of five ordered losses is \(6,10,14,24,35\) thousand dollars. Find the empirical 80th percentile. Compare it with the 80th percentile of an exponential model with mean \(20\) thousand dollars, and report empirical minus model. (Answer: empirical \(24.00\) thousand dollars; model \(32.19\) thousand dollars; difference \(-8.19\) thousand dollars.)
- A fitted lognormal model has \(\ln X\sim N(\ln 30,0.3^2)\). Using \(z_{0.90}=1.28155\), calculate its 90th percentile. (Answer: approximately \(44.06\) dollars.)
- A hypothetical ordered sample of eight losses, in thousands of dollars, is \(5,9,12,16,21,29,40,65\). Compare its empirical 62.5th percentile with the 62.5th percentile of an exponential model with mean \(24\) thousand dollars. Give the rank, both quantiles, and empirical minus model, to two decimals. (Answer: rank \(5\); empirical \(21.00\) thousand dollars; model \(23.54\) thousand dollars; difference \(-2.54\) thousand dollars.)
- Across several matched probability levels, a sample's empirical quantiles are below fitted-model quantiles at 25% and 50%, but above them at 90%. What can this pattern suggest, and why is it not conclusive by itself? (Answer: It may suggest that the sample and fitted distribution differ in their shape or relative spread, especially toward the upper part of the distribution; sampling variation, particularly in upper empirical quantiles, means the pattern alone does not establish systematic model mismatch.)