Goodness-of-Fit Tests for Continuous Severity Models

Why Use Goodness-of-Fit Statistics?

A fitted severity distribution is a useful summary only if it is compatible with the observed losses. A goodness-of-fit statistic measures one aspect of the discrepancy between the sample and a proposed continuous distribution. The Kolmogorov–Smirnov (K–S) statistic measures the largest vertical difference between the empirical distribution function and the model CDF. The Anderson–Darling (A–D) statistic also measures discrepancies across the distribution, but gives relatively greater weight to the tails.

These statistics help quantify discrepancies; neither one proves that a model is correct. Their interpretation depends on how the model was specified, how its parameters were obtained, and whether the data satisfy the assumptions behind the calculation. The methods here are aligned with the SOA Exam FAM (Fundamentals of Actuarial Mathematics) syllabus.

Setup: Let \(X_1,\ldots,X_n\) be independent observations from a continuous severity distribution with CDF \(F\). Write the ordered observations as \(X_{(1)}\le\cdots\le X_{(n)}\). For a goodness-of-fit calculation, evaluate the proposed model CDF at each ordered observation. The ordinary formulas below presume a complete, uncensored, untruncated sample from a continuous model. Ties from rounding, censoring at policy limits, or truncation above a deductible require care; applying the ordinary formulas as though the observations were a complete continuous sample may not give a valid test.

The Kolmogorov–Smirnov Statistic

The empirical distribution function (ECDF) at a value \(x\) is the proportion of sample observations less than or equal to \(x\). At the \(i\)th ordered observation, the ECDF jumps from \((i-1)/n\) to \(i/n\). Compare both sides of that jump with the proposed model CDF. If the model CDF is \(F\), define

D^+ = \max_{1\le i\le n}\left(\frac{i}{n}-F(X_{(i)})\right),\qquad D^- = \max_{1\le i\le n}\left(F(X_{(i)})-\frac{i-1}{n}\right), \qquad D=\max(D^+,D^-).

The maximum accounts for the whole ECDF, including the interval before its first jump and the interval after its last jump. A large \(D\) means that the largest observed vertical separation is substantial. The statistic is not a direct measure of the average discrepancy: one unusually large gap can determine its value.

Why these two one-sided differences? At \(X_{(i)}\), the ECDF immediately after its jump is \(i/n\), while immediately before it is \((i-1)/n\). The largest positive gap between the ECDF and the model is therefore found by comparing \(i/n\) with \(F(X_{(i)})\); the largest gap in the opposite direction compares \(F(X_{(i)})\) with \((i-1)/n\). Taking the larger maximum gives the supremum of the absolute ECDF–model difference.

Worked Example: Finding the K–S Statistic

A hypothetical severity sample has five ordered losses. Under a fully specified continuous candidate model, its CDF values at those losses are \(0.08,\ 0.31,\ 0.54,\ 0.83,\ 0.96\). Calculate \(D^+\), \(D^-\), and \(D\). No parameter is estimated from these five observations.

Method. Compare each model CDF value with both ECDF levels at the corresponding jump. Since \(n=5\), the after-jump levels are \(0.20,\ 0.40,\ 0.60,\ 0.80,\ 1.00\), and the before-jump levels are \(0,\ 0.20,\ 0.40,\ 0.60,\ 0.80\).

The after-jump differences are \(0.20-0.08=0.12\), \(0.40-0.31=0.09\), \(0.60-0.54=0.06\), \(0.80-0.83=-0.03\), and \(1.00-0.96=0.04\). Their maximum is \(D^+=0.12\).

The before-jump differences are \(0.08-0=0.08\), \(0.31-0.20=0.11\), \(0.54-0.40=0.14\), \(0.83-0.60=0.23\), and \(0.96-0.80=0.16\). Their maximum is \(D^-=0.23\).

Thus \(D=\max(0.12,0.23)=0.23\). The largest discrepancy occurs just before the fourth ECDF jump. Check: \(D\) equals the larger of the two separately calculated maxima.

Common mistake avoided: using only the after-jump differences misses the larger discrepancy of \(0.23\). Both sides of every ECDF jump must be considered.

The Anderson–Darling Statistic

The A–D statistic is another measure of disagreement between an empirical distribution and a continuous model. For ordered observations and model CDF \(F\), its finite-sample formula is

A^2=-n-\frac{1}{n}\sum_{i=1}^{n}(2i-1) \left[\ln F(X_{(i)})+\ln\left(1-F(X_{(n+1-i)})\right)\right].

Every term uses a model probability between zero and one. The logarithms require \(0<F(X_{(i)})<1\); a zero or one value makes the formula undefined. For a continuous model, finite observations strictly inside the model’s support ordinarily have probabilities in that range. Check the support and the computed probabilities before applying the formula.

The statistic has a tail-weighting interpretation. If \(U=F(X)\), then under the proposed continuous model \(U\) is uniform on \([0,1]\). A population version of the A–D discrepancy integrates the squared difference between the empirical and uniform CDFs with weight \(1/[u(1-u)]\). That weight increases near zero and one. The finite-sample expression above is the corresponding ordered-data calculation. Thus, discrepancies near either tail can matter more to A–D than discrepancies of the same size near the center.

Worked Example: Calculating A–D and K–S from Model Probabilities

A hypothetical continuous severity model is fully specified. At three ordered sample losses, its CDF values are \(0.05,\ 0.50,\ 0.95\). Calculate \(A^2\) and \(D\), rounding each final statistic to four decimal places.

Method. For K–S, compare the three probabilities with both ECDF levels. For A–D, substitute the ordered probabilities in the formula, using weights \(2i-1=1,3,5\).

For K–S, the after-jump differences are \(1/3-0.05=0.2833\), \(2/3-0.50=0.1667\), and \(1-0.95=0.0500\). The before-jump differences are \(0.05-0=0.0500\), \(0.50-1/3=0.1667\), and \(0.95-2/3=0.2833\). Therefore \(D^+=0.2833\), \(D^-=0.2833\), and \(D=0.2833\).

For A–D, the first weighted contribution is \(1[\ln(0.05)+\ln(1-0.95)]=2\ln(0.05)\approx-5.991465\). The second is \(3[\ln(0.50)+\ln(1-0.50)]=6\ln(0.50)\approx-4.158883\). The third is \(5[\ln(0.95)+\ln(1-0.05)]=10\ln(0.95)\approx-0.512933\). The sum is approximately \(-10.663281\), so \(A^2=-3-(-10.663281/3)\approx0.554427\).

Rounded to four decimal places, \(D=0.2833\) and \(A^2=0.5544\). Check: each term inside a logarithm is between zero and one, so the weighted logarithmic sum is negative; the resulting \(A^2\) is positive.

Common mistake avoided: the A–D formula pairs the lower-tail probability at \(X_{(i)}\) with the upper-tail probability at \(X_{(n+1-i)}\). Pairing both terms at \(X_{(i)}\) for every \(i\) gives the wrong weights and generally the wrong statistic.

Estimating Parameters Changes the Test

The statistic can be calculated whether a model is fully specified in advance or has parameters estimated from the sample. The distinction matters for inference. With a fully specified continuous null distribution, the probability-integral transform gives \(F(X_j)\sim\text{Uniform}(0,1)\) under the null. This supports standard null distributions for the goodness-of-fit statistics, subject to the test’s assumptions.

When parameters are estimated from the same observations, the fitted CDF is not fixed independently of the sample. The observations helped choose the CDF against which they are being compared, which affects the distribution of the resulting statistic. Consequently, a critical value or p-value for a fully specified null cannot automatically be used after fitting parameters. The appropriate calibration depends on the model and estimation procedure. The statistic itself remains a useful descriptive measure, but a formal test needs calibration that reflects parameter estimation.

Worked Example: K–S with an Estimated Exponential Mean

Four hypothetical complete severity observations, measured in thousands of dollars, are \(1,\ 2,\ 3,\ 4\). Assume a continuous exponential model with mean \(\theta\), so \(F(x)=1-e^{-x/\theta}\) for \(x\ge0\). Estimate \(\theta\) by maximum likelihood from these observations, then calculate the K–S statistic using the fitted CDF. Round the final statistic to four decimals.

Estimate the parameter. For a complete exponential sample, the maximum-likelihood estimate of the mean is the sample mean: \(\hat{\theta}=(1+2+3+4)/4=10/4=2.5\) thousand dollars. The fitted rate is \(1/\hat{\theta}=0.4\) per thousand dollars.

Evaluate the fitted CDF. At \(x=1\), \(F_{\hat{\theta}}(1)=1-e^{-0.4}\approx0.329680\). At \(x=2\), \(F_{\hat{\theta}}(2)=1-e^{-0.8}\approx0.550671\). At \(x=3\), \(F_{\hat{\theta}}(3)=1-e^{-1.2}\approx0.698806\). At \(x=4\), \(F_{\hat{\theta}}(4)=1-e^{-1.6}\approx0.798103\).

Calculate both maxima. The after-jump differences are \(0.25-0.329680=-0.079680\), \(0.50-0.550671=-0.050671\), \(0.75-0.698806=0.051194\), and \(1-0.798103=0.201897\). Thus \(D^+=0.201897\). The before-jump differences are \(0.329680-0=0.329680\), \(0.550671-0.25=0.300671\), \(0.698806-0.50=0.198806\), and \(0.798103-0.75=0.048103\). Thus \(D^-=0.329680\), and \(D=0.3297\) to four decimals.

Check and interpret. The overall maximum is the larger of \(0.201897\) and \(0.329680\), namely \(0.329680\). This is a statistic calculated against a mean estimated from the same four observations. It is not appropriate to assess its significance using the calibration for a fully specified exponential distribution without accounting for that estimation.

Common mistake avoided: the model mean is \(2.5\) thousand dollars, not \(2.5\) dollars, and the fitted rate is its reciprocal. Keeping the units consistent is essential when evaluating the exponential CDF.

Choosing and Interpreting the Statistics

Feature	Kolmogorov–Smirnov	Anderson–Darling
What it summarizes	Largest absolute vertical ECDF–CDF gap	Weighted discrepancy across ordered model probabilities
Relative tail emphasis	Does not apply a special tail weight	Greater weight near probabilities zero and one
Useful reading	Identifies the worst single gap	Responds more strongly to tail departures
Parameter estimation	Changes the null calibration	Changes the null calibration

A–D tail sensitivity is useful for severity work because the tails can have material consequences for aggregate losses and high quantiles. It does not mean A–D is always the better choice: the statistic answers a different question, and its increased tail emphasis can make it respond strongly to a small number of extreme observations. K–S provides a simple maximum-gap summary, while A–D emphasizes a weighted overall discrepancy. Reading both can reveal different features of the same sample.

Neither statistic identifies why a model disagrees with the data. A large discrepancy may reflect a poor distributional family, an unsuitable parameter estimate, an unusual observation, or data features such as rounding. Probability plots provide a visual diagnostic alongside these numerical summaries. For censored, truncated, or deductible-modified observations, use a method whose likelihood and test statistic reflect how the observations were collected rather than treating them as ordinary complete losses.

Key takeaway: Calculate K–S by checking both sides of every ECDF jump; calculate A–D using the ordered-data formula and valid interior CDF probabilities. Interpret A–D as more tail-sensitive, and do not use fully specified-model test calibration automatically when parameters were estimated from the same sample.

Check Your Understanding

For each problem, calculate the requested statistic or identify the correct testing implication.

A fully specified continuous model gives CDF values \(0.20,\ 0.60,\ 0.80\) at three ordered observations. Calculate \(D^+\), \(D^-\), and \(D\), rounding to four decimals. (Answer: \(D^+=0.2000,\ D^-=0.2667,\ D=0.2667\))
A fully specified continuous model gives ordered CDF values \(0.25,\ 0.75\) for a sample of size two. Calculate \(A^2\), rounding to four decimals. (Answer: \(A^2=0.2493\))
For \(n=4\), a fully specified model has ordered CDF values \(0.12,\ 0.38,\ 0.61,\ 0.88\). Calculate \(D\), rounding to four decimals. (Answer: \(D=0.1400\))
For \(n=3\), a fully specified continuous model has ordered CDF values \(0.20,\ 0.50,\ 0.80\). Calculate \(A^2\), rounding to four decimals. (Answer: \(A^2=0.2031\))
A severity model’s parameters were estimated from the same complete sample used to calculate its K–S statistic. Is the fully specified-model K–S calibration automatically valid? (Answer: No; the null calibration must account for parameter estimation.)

What You'll Learn