Graphical Diagnostics For Candidate Severity Models

Why Graphical Diagnostics Matter

A single summary of model fit can conceal where a candidate severity model succeeds and where it fails. Two models may have similar information-criterion values while one tracks the center of the observed losses well and misses the largest losses. Graphical diagnostics make the location and direction of discrepancies visible. They complement, rather than replace, numerical model-selection criteria.

This tutorial develops three related tools for positive loss data: an empirical-versus-fitted distribution plot, a quantile plot, and a tail plot. The plots are diagnostic summaries, not formal hypothesis tests. Their interpretation depends on using the same observations and the same fitted model throughout, and on recognizing that the empirical tail is especially variable when only a few large losses are observed.

Definition: For observed losses $x_1,\ldots,x_n$, the empirical distribution function is $F_n(x)=n^{-1}\sum_{j=1}^n I(x_j\le x)$, where $I(\cdot)$ is 1 when its condition is true and 0 otherwise. A fitted severity model with parameter estimate $\hat\theta$ has fitted distribution function $F_{\hat\theta}(x)$ and fitted survival function $S_{\hat\theta}(x)=1-F_{\hat\theta}(x)$.

Empirical-Versus-Fitted Distribution Plots

An empirical-versus-fitted distribution plot places the fitted cumulative probability on one axis and the empirical cumulative probability on the other. If the fitted model describes the data well, the plotted points should generally be near the identity line, where the two probabilities are equal. A systematic pattern of departures can reveal a mismatch even when the overall fit looks acceptable.

When all observed values are distinct and ordered as $x_{(1)}<\cdots<x_{(n)}$, the empirical distribution function at $x_{(i)}$ is $i/n$. This right-continuous definition counts the observation at the point itself. Tied observations produce jumps larger than $1/n$; the empirical function still counts all observations less than or equal to the plotted value.

Worked Example: Comparing an Empirical Distribution With an Exponential Fit

A hypothetical portfolio has four complete, positive loss observations of \$30, \$60, \$90, and \$120. For this diagnostic, fit an exponential severity model by maximum likelihood. For an exponential model with mean parameter $\theta$, $F_\theta(x)=1-\exp(-x/\theta)$ for $x\ge0$. The complete-data maximum-likelihood estimate is the sample mean. Construct the empirical-versus-fitted comparison at each observed loss.

Step 1: Fit the model. The sample total is $30+60+90+120=300$, so the fitted mean is $\hat\theta=300/4=75$ dollars. The fitted distribution is $F_{\hat\theta}(x)=1-\exp(-x/75)$.

Step 2: Find empirical probabilities. The observations are already ordered and distinct, so their empirical probabilities are $1/4=0.25$, $2/4=0.50$, $3/4=0.75$, and $4/4=1.00$.

Step 3: Calculate fitted probabilities and signed differences. Use the difference $F_{\hat\theta}(x_{(i)})-F_n(x_{(i)})$, calculating with unrounded values before rounding the results.

\begin{array}{c|c|c|c} x_{(i)} & F_n(x_{(i)}) & 1-\exp(-x_{(i)}/75) & F_{\hat\theta}(x_{(i)})-F_n(x_{(i)})\\ 30 & 0.2500 & 0.3297 & 0.0797\\ 60 & 0.5000 & 0.5507 & 0.0507\\ 90 & 0.7500 & 0.6988 & -0.0512\\ 120 & 1.0000 & 0.7981 & -0.2019 \end{array}

Interpretation. The fitted cumulative probability is above the empirical probability at the first two observations and below it at the last two. The largest observation has empirical probability 1 but fitted probability only about 0.7981. This indicates a visible discrepancy at the upper endpoint of this small sample; it does not by itself establish that the model is unsuitable. The calculation avoids a common plotting-position error: the empirical distribution function at the sample maximum is 1, not $n/(n+1)$.

Quantile Plots and Their Interpretation

A quantile plot compares observed ordered losses with quantiles from the fitted model. For a continuous fitted distribution, its quantile function $F_{\hat\theta}^{-1}(p)$ is the loss value at cumulative probability $p$. Plot the fitted quantile on the horizontal axis and the corresponding observed sample quantile on the vertical axis. If the model fits well, points should lie roughly along the identity line $y=x$.

A plotting position assigns a probability to each ordered observation. One common choice for a quantile plot is $p_i=(i-0.5)/n$. Other plotting-position conventions are also used. The important distinction is that these are plotting probabilities for a quantile comparison; they do not change the definition of the empirical distribution function $F_n(x)$.

Derivation: If a continuous model $F$ is correct, the probability integral transform gives $F(X)\sim\operatorname{Uniform}(0,1)$. Inverting this relation gives $X=F^{-1}(U)$, where $U$ is uniform. Thus, an observation whose plotting probability is $p_i$ is compared with the model quantile $F_{\hat\theta}^{-1}(p_i)$. Under a well-fitting model, the observed and fitted quantiles should be similar, which explains why the points should be near $y=x$. This is an approximate graphical comparison: finite samples fluctuate, and the parameters have been estimated rather than known.

The direction of a departure matters. With observed quantiles on the vertical axis, points above the identity line mean that the observed losses are larger than the fitted quantiles at those plotting probabilities. A sustained upward bend at high quantiles suggests that the fitted model gives too little weight to large losses. Points below the line indicate observed quantiles smaller than the corresponding fitted quantiles. A pattern across several adjacent points is generally more informative than one isolated extreme observation.

Worked Example: Reading an Exponential Quantile Plot

A hypothetical equipment-loss sample, in dollars, is $35,70,95,150,400$. Fit an exponential model to these complete observations, then calculate the fitted quantiles and deviations needed for a quantile plot. Use $p_i=(i-0.5)/n$, with the observed loss on the vertical axis.

Step 1: Estimate the model. The losses sum to $35+70+95+150+400=750$. The maximum-likelihood estimate of the exponential mean is $\hat\theta=750/5=150$ dollars.

Step 2: Assign plotting probabilities. For $n=5$, the plotting probabilities are $0.1,0.3,0.5,0.7,0.9$.

Step 3: Calculate fitted quantiles. For an exponential model with mean $\hat\theta$, $F_{\hat\theta}^{-1}(p)=-\hat\theta\ln(1-p)$. Calculate each fitted value and subtract it from the observed loss.

\begin{array}{c|c|c|c} p_i & x_{(i)} & -150\ln(1-p_i) & x_{(i)}-F_{\hat\theta}^{-1}(p_i)\\ 0.1 & 35 & 15.8041 & 19.1959\\ 0.3 & 70 & 53.5012 & 16.4988\\ 0.5 & 95 & 103.9721 & -8.9721\\ 0.7 & 150 & 180.5959 & -30.5959\\ 0.9 & 400 & 345.3878 & 54.6122 \end{array}

Interpretation. The first two observed quantiles exceed their fitted counterparts, the next two fall below, and the largest observed loss again exceeds its fitted quantile. On the plot, the last point is at approximately $(345.39,400)$, well above the identity line. This suggests that the fitted exponential distribution may understate the largest loss in this sample, but one upper observation is not enough to establish a stable tail pattern.

Check. The fitted mean is exactly the sample mean, \$150, as required by the complete-data exponential maximum-likelihood estimate. The quantile calculations use the specified plotting probabilities, not the empirical-CDF values $i/n$.

Tail Plots: Inspecting Survival Probabilities

For severity work, the survival function $S_X(t)=P(X>t)$ describes the probability that a loss exceeds a threshold $t$. A tail plot compares empirical exceedance proportions with fitted survival probabilities across selected thresholds. Using the strict exceedance definition, the empirical survival at threshold $t$ is $n^{-1}\sum_{j=1}^n I(x_j>t)$. State the threshold convention clearly: counting losses greater than $t$ is not the same as counting losses greater than or equal to $t$ when observations equal the threshold.

A plot of empirical and fitted survival probabilities can use either survival probabilities directly or their logarithms, provided the chosen scale is clear. On a logarithmic scale, a fitted exponential survival function is linear in the threshold because $\log S_{\hat\theta}(t)=-t/\hat\theta$. This is a visual property of the exponential model, not a general rule that every severity model should appear linear on that scale. If an empirical survival probability is zero, its logarithm is undefined, so it cannot be plotted as a finite point on a log-survival graph.

Worked Example: Comparing Exceedance Rates With an Exponential Tail

A hypothetical set of eight complete losses, in dollars, is $25,50,75,100,150,250,500,850$. Fit an exponential severity model using the complete-data maximum-likelihood estimate. Compare empirical and fitted survival probabilities at thresholds \$300, \$600, and \$800, counting losses strictly greater than each threshold.

Step 1: Fit the model. The total is $25+50+75+100+150+250+500+850=2{,}000$. Thus, $\hat\theta=2{,}000/8=250$ dollars, and the fitted survival function is $S_{\hat\theta}(t)=\exp(-t/250)$.

Step 2: Count exceedances and calculate fitted probabilities. At \$300, two losses exceed the threshold; at \$600 and \$800, only the \$850 loss exceeds the threshold. Calculate the empirical-to-fitted ratio using the unrounded fitted survival probability.

\begin{array}{c|c|c|c|c} t & \#\{x_j>t\} & \widehat S_n(t) & \exp(-t/250) & \widehat S_n(t)/S_{\hat\theta}(t)\\ 300 & 2 & 0.2500 & 0.3012 & 0.8300\\ 600 & 1 & 0.1250 & 0.0907 & 1.3779\\ 800 & 1 & 0.1250 & 0.0408 & 3.0666 \end{array}

For example, at \$600 the ratio is $0.125/\exp(-2.4)$, which is approximately 1.3779. At \$800 it is $0.125/\exp(-3.2)$, approximately 3.0666; these ratios are calculated from the unrounded exponential values, not from the displayed survival probabilities.

Interpretation and check. At \$300 the empirical exceedance proportion is below the fitted survival probability. At the higher thresholds it is above the fitted value, so the largest observed loss appears relatively more prominent than the exponential model predicts. The empirical values at \$600 and \$800 are both based on the same single loss. The apparent excess is therefore highly sensitive to that observation and should not be treated as precise evidence about the population tail. The counts, empirical proportions, and ratios are consistent with the stated strict-exceedance rule.

How to Use the Plots Together

Each plot emphasizes a different view of the same fitted severity model. The empirical-versus-fitted distribution plot summarizes cumulative probabilities over the observed range. The quantile plot shows how fitted and observed loss magnitudes compare at selected probability levels. The tail plot focuses directly on the probability of exceeding selected loss thresholds. Reading the plots together helps distinguish a broad distributional mismatch from one concentrated near the largest observations.

Confirm the data basis.
Use the same loss observations, units, and treatment of incomplete observations that were used to fit the candidate model. If the fitting procedure accounts for censoring or truncation, do not compare its fitted distribution with an empirical summary that silently treats those observations as complete and uncensored.

Choose and label the comparison.
For a distribution plot, identify whether the axes are empirical and fitted cumulative probabilities. For a quantile plot, state which axis contains observed losses and which contains fitted quantiles. For a tail plot, specify the threshold convention and whether the vertical scale is ordinary or logarithmic.

Look for a pattern, not just a point.
Assess whether departures from the reference line or equality of the two probabilities are systematic, where they occur, and whether neighboring points support the same interpretation.

Qualify the conclusion.
Note when a tail feature depends on very few observations. Use the plots to describe the location and direction of a mismatch, not to claim that a model has been formally accepted or rejected.

Common Mistakes and Exam Traps

Mixing empirical-CDF values with quantile plotting positions. At the sample maximum, the right-continuous empirical CDF is 1. A quantile plot may instead assign that observation a plotting position below 1. These are different constructions, so substituting one for the other can make an endpoint look artificially well fitted.
Rounding before subtracting or dividing. Calculate differences and ratios from unrounded probabilities, then round the result. Subtracting or dividing displayed rounded values can change the final reported digit.
Reversing the axes or the interpretation. In this tutorial's quantile plots, observed losses are vertical and fitted quantiles are horizontal. A point above the identity line means the observed loss is larger than the fitted quantile, not smaller.
Overinterpreting one extreme loss. Empirical tail estimates change in increments of $1/n$. With eight observations, one exceedance corresponds to $0.1250$; it is a coarse and uncertain estimate, not a stable description of all large-loss probabilities.
Treating a plot as a formal test. A visible departure is useful diagnostic evidence, but a graph alone does not provide a formal significance level or a model-selection rule.

Key takeaway: Graphical diagnostics make model-fit discrepancies interpretable by showing where they occur and in which direction. Use consistent definitions and plotting conventions, examine patterns across the distribution, and treat tail conclusions based on sparse data with particular caution.

Check Your Understanding

For each problem, calculate or interpret the requested diagnostic using the stated conventions.

A sample of four losses is $20,40,60,80$. A fitted exponential model has mean \$50. At $x=\$40$, calculate the empirical CDF, fitted CDF, and fitted-minus-empirical difference. (Answer: $F_n(40)=0.5000$, $F_{\hat\theta}(40)=0.5507$, difference $=0.0507$.)
A fitted exponential severity model has mean \$80. Find its fitted quantile at plotting probability $p=0.75$. (Answer: $-80\ln(0.25)=\$110.90$, to the nearest cent.)
Five losses are $10,30,60,90,210$, and the fitted exponential mean is \$80. At threshold \$100, calculate the strict empirical exceedance proportion, fitted survival probability, and empirical-to-fitted ratio. (Answer: $0.2000$, $\exp(-1.25)=0.2865$, and approximately $0.6981$.)
On a quantile plot with observed losses on the vertical axis, several consecutive high-quantile points lie above the identity line. What direction of discrepancy does this suggest? (Answer: The observed high losses exceed the fitted quantiles, suggesting the model may understate upper quantiles.)
A sample has no observations greater than its maximum. What happens to the empirical survival probability above that maximum, and can its logarithm be plotted as a finite value? (Answer: The empirical survival is $0$; its logarithm is undefined, so it cannot be plotted as a finite log-survival value.)

What You'll Learn