Selecting A Goodness-of-Fit Test For Loss Data

Introduction

A goodness-of-fit test is useful only if its assumptions match the information actually recorded. Exact continuous losses, rounded payments, counts in categories, and grouped observations do not provide the same evidence about a proposed model. Even when two tests can be calculated, they may be sensitive to different departures from the model.

This tutorial gives a practical way to choose among the Kolmogorov-Smirnov (KS), Anderson-Darling (AD), and chi-square goodness-of-fit procedures. The central questions are: What form are the data in? What kind of model departure matters? Can the test be calibrated under the way the model was fitted?

Definition: A goodness-of-fit test compares observed loss data with a specified distribution under a null hypothesis. The KS and AD procedures are principally used for exact observations from a continuous distribution. The chi-square procedure compares observed counts with expected counts in categories and is especially useful for discrete or grouped data. A test statistic measures a discrepancy; its reference distribution or a valid calibration method is needed to obtain a p-value or critical-value decision.

What the Three Tests Measure

Let \(X_1,\ldots,X_n\) be independent observations from a continuous null distribution with cumulative distribution function \(F\). Write the ordered observations as \(X_{(1)}\le\cdots\le X_{(n)}\), and let \(F_n(x)\) be the empirical distribution function: the proportion of observations no greater than \(x\).

The KS statistic measures the largest vertical distance between the empirical and proposed cumulative distribution functions. It is useful when the data are exact continuous observations and a broad, unlocalized discrepancy is of interest. A maximum discrepancy in the middle of the distribution can matter as much to KS as one near a tail.

D_n=\sup_x |F_n(x)-F(x)|

The AD statistic also compares the empirical distribution with the proposed distribution, but gives more influence to differences in the tails. For a continuous null distribution with fully specified \(F\), one common form is:

A^2=-n-\frac{1}{n}\sum_{i=1}^{n}(2i-1) \left[\ln F(X_{(i)})+\ln\left(1-F(X_{(n+1-i)})\right)\right].

The chi-square goodness-of-fit statistic compares observed category counts \(O_j\) with expected counts \(E_j=np_j\), where \(p_j\) is the null probability of category \(j\). It is suited to discrete outcomes or data available only in groups. It does not use the exact position of an observation within its category.

\chi^2=\sum_{j=1}^{k}\frac{(O_j-E_j)^2}{E_j}.

For a fully specified null model and \(k\) categories, the usual large-sample reference distribution has \(k-1\) degrees of freedom, provided the categories are exhaustive, mutually exclusive, and the expected counts are adequate. If parameters are estimated from the same data, the reference degrees of freedom and calibration must account for that estimation; a simple adjustment is not always valid for every fitting procedure.

Why AD Gives More Weight to Tails

The distinction between KS and AD can be understood by transforming each loss with the null distribution: \(U_i=F(X_i)\). If the continuous null model is correct and fully specified, then \(U_i\) has a uniform distribution on \([0,1]\). A discrepancy near 0 or 1 corresponds to a tail discrepancy in the original loss scale.

The AD statistic is based on a weighted squared difference between the empirical distribution of these transformed values and the uniform distribution. Its weighting factor is \(1/[u(1-u)]\). This factor is larger near either endpoint than near the center, so a given-sized discrepancy in a tail receives more weight.

A^2 = n\int_0^1 \frac{\left(F_n^U(u)-u\right)^2}{u(1-u)}\,du,

where \(F_n^U\) is the empirical distribution function of the transformed observations. The denominator explains the tail emphasis: as \(u\) approaches 0 or 1, \(u(1-u)\) becomes small. KS instead takes the maximum unweighted absolute discrepancy, so it emphasizes the single largest gap rather than systematically upweighting tail gaps.

A Practical Selection Framework

Identify the observed data form.
Exact continuous losses permit continuous-data procedures. Discrete outcomes or counts in pre-defined groups naturally lead to chi-square. Rounding, ties, truncation, or censoring require additional care because they change what the observations reveal.

Identify the departure of interest.
Use KS when the largest overall CDF discrepancy is the target. Prefer AD when discrepancies in either tail deserve extra emphasis. Use chi-square when the available information is category counts.

Check whether the null distribution was fitted.
Critical values depend on how the null model was specified. If parameters were estimated from these observations, use a calibration appropriate to that estimation rather than automatically applying the fully specified-model reference values.

Check data limitations and expected counts.
For chi-square, combine categories only in a defensible way if expected counts are too small. For continuous tests, do not treat tied or censored values as if they were exact continuous observations without an appropriate adjustment.

Data and objective	Natural starting choice	Main caution
Exact continuous losses; largest CDF gap is the concern	KS	Standard critical values require the appropriate null calibration; fitted parameters change it.
Exact continuous losses; tail departures are especially important	Anderson-Darling	Use the calibration for the specified AD statistic and fitting procedure.
Discrete losses or observations reported only as category counts	Chi-square	Expected counts and degrees of freedom must be handled correctly.
Rounded, tied, or censored observations	Adapted or calibrated procedure	Ordinary continuous-test assumptions may not hold.

Worked Examples

Worked Example: Exact Continuous Losses with Tail Concern

A hypothetical insurer considers an exponential loss model with scale parameter 100 dollars, so \(F(x)=1-\exp(-x/100)\) for \(x\ge 0\). Five exact losses are constructed as \(X_i=-100\ln(1-u_i)\) for \(u_i=0.10,0.20,0.40,0.70,0.90\). The parameters are treated as fully specified, and the analyst is particularly concerned about tail fit. Select a suitable test and calculate its statistic. Also calculate KS to illustrate the difference in emphasis.

Method. The losses are exact and continuous, so both KS and AD are candidates. Tail emphasis makes AD the more directly aligned choice. Because \(F(X_i)=u_i\), the ordered fitted probabilities are \(0.10,0.20,0.40,0.70,0.90\).

AD calculation. The five weighted paired terms inside the sum are:

\begin{aligned} 1[\ln(0.10)+\ln(0.10)]&=-4.605170,\\ 3[\ln(0.20)+\ln(0.30)]&=-8.440232,\\ 5[\ln(0.40)+\ln(0.60)]&=-7.135582,\\ 7[\ln(0.70)+\ln(0.80)]&=-4.058729,\\ 9[\ln(0.90)+\ln(0.90)]&=-1.896489. \end{aligned}

The sum is \(-26.136202\). Therefore,

A^2=-5-\frac{-26.136202}{5}=0.227240.

KS calculation. At the ordered observations, \(i/n\) is \(0.20,0.40,0.60,0.80,1.00\). The absolute differences from the fitted probabilities are \(0.10,0.20,0.20,0.10,0.10\), so the right-side maximum is \(0.20\). The left-limit differences \(u_i-(i-1)/n\) are \(0.10,0.00,0.00,0.10,0.10\), so \(D_n=0.20\). The KS statistic is a maximum-gap summary; it does not itself give the tail differences extra weight.

The appropriate choice for the stated objective is AD. Neither statistic alone gives a decision: a p-value or critical-value comparison requires the correct null calibration. The common mistake is to choose KS merely because the data are continuous; the correction is to consider the intended sensitivity as well as the data form.

Worked Example: Discrete Loss Categories

A hypothetical warranty plan records each covered repair cost in one of four discrete categories. Under a specified model, the category probabilities are \(0.50,0.25,0.15,0.10\). Among 120 repairs, the observed counts are \(55,34,19,12\). Choose a test and calculate its statistic.

Method. The observations are discrete categories, so use chi-square rather than an ordinary continuous KS or AD procedure. The expected counts are \(E_j=np_j\):

(120)(0.50)=60,\quad (120)(0.25)=30,\quad (120)(0.15)=18,\quad (120)(0.10)=12.

All expected counts are at least 12. The four contributions are:

\begin{aligned} \frac{(55-60)^2}{60}&=\frac{25}{60}=0.4167,\\ \frac{(34-30)^2}{30}&=\frac{16}{30}=0.5333,\\ \frac{(19-18)^2}{18}&=\frac{1}{18}=0.0556,\\ \frac{(12-12)^2}{12}&=0. \end{aligned}

Thus \(\chi^2=0.4167+0.5333+0.0556+0=1.0056\). The categories are exhaustive and the probabilities are specified, so the conventional degrees of freedom are \(4-1=3\). A final test decision still requires comparison with the appropriate reference distribution or calculation of a p-value.

A common error is to use each observed count as its own expected count, which would make every contribution zero. Expected counts come from the null probabilities, not from the observed data; here the first expected count is 60, not 55.

Worked Example: Grouped Continuous Loss Data

A hypothetical analyst has only counts in four pre-established loss intervals; the individual loss amounts are unavailable. Under a fully specified continuous model, the interval probabilities are \(0.20,0.30,0.30,0.20\). For 100 claims the observed counts are \(16,28,34,22\). Select a test and calculate the statistic.

Method. Because only grouped counts are available, the chi-square procedure uses the information retained in the data. Expected counts are \(100(0.20)=20\), \(100(0.30)=30\), \(100(0.30)=30\), and \(100(0.20)=20\). Each is adequate for the usual approximation.

The contributions are:

\begin{aligned} \frac{(16-20)^2}{20}&=\frac{16}{20}=0.8000,\\ \frac{(28-30)^2}{30}&=\frac{4}{30}=0.1333,\\ \frac{(34-30)^2}{30}&=\frac{16}{30}=0.5333,\\ \frac{(22-20)^2}{20}&=\frac{4}{20}=0.2000. \end{aligned}

The statistic is \(0.8000+0.1333+0.5333+0.2000=1.6666\), or \(1.6667\) using unrounded contributions. With four categories and a fully specified model, the usual degrees of freedom are 3.

The common mistake is to apply KS to the four category counts as though they were the original exact losses. The correction is to use chi-square when only grouped counts are retained; a continuous-data statistic would require the individual observations or a specifically justified grouped-data adaptation.

Calibration and Data Limitations

Choosing the statistic is only part of choosing the test. Its reference distribution must match the null hypothesis and the data collection process. In particular, standard KS critical values for a fully specified continuous distribution do not automatically apply when parameters are estimated from the same observations. The same principle applies to AD and chi-square: parameter fitting can change the null distribution of the statistic. Use an appropriate calibration method, such as a suitable simulation procedure, when required by the fitting setup.

The ordinary continuous KS and AD formulas presume observations from a continuous model. Rounded losses can create ties even when an underlying loss is continuous. If recorded amounts are rounded to the nearest dollar, the observed data are not exact draws from a continuous distribution at the recorded precision. A test should account for the rounding mechanism, or a defensible grouped procedure should be used. Likewise, a right-censored loss is not an exact observed loss equal to its censoring threshold; ordinary formulas applied as if it were exact can distort the comparison.

For chi-square, categories must be defined so that every observation belongs to exactly one category, and the null model must supply the probability of each category. Small expected counts can make the usual chi-square approximation poor. Combining categories may help, but the grouping should preserve meaningful distinctions and should not be altered after inspecting the observed discrepancies merely to obtain a preferred result.

Common Selection Errors

Choosing only by familiarity. KS is not automatically the right test for every continuous loss sample. Decide whether maximum overall discrepancy or tail sensitivity better matches the question.
Using continuous-test critical values for discrete data. Discrete distributions have jumps, so the usual continuous-null calibration does not apply automatically.
Discarding information unnecessarily. Converting exact continuous losses into broad bins allows chi-square testing but loses within-bin information. Use grouped testing when the data are genuinely grouped or when grouping is otherwise justified.
Ignoring how parameters were estimated. A fully specified-model calibration may be invalid after fitting parameters to the same observations.
Treating censored or rounded values as exact. Use a procedure or calibration that reflects how each value was observed.

Key takeaway: Start with the data form: exact continuous observations suggest KS or AD, while discrete or grouped counts suggest chi-square. Then choose based on the target departure—largest overall CDF gap, tail discrepancy, or category-frequency mismatch—and verify that the null calibration reflects estimation and observation limitations.

Check Your Understanding

For each situation, choose the most appropriate starting procedure or explain why an ordinary version is unsuitable.

A fully specified continuous severity model is assessed using exact, unrounded losses. The analyst is concerned mainly with the largest overall difference between empirical and model CDFs. (Answer: KS.)
A continuous severity model is assessed using exact observations, and the pricing team places particular importance on detecting an incorrect upper tail. (Answer: Anderson-Darling.)
A discrete benefit has three outcomes with probabilities \(0.60,0.30,0.10\). In 200 observations the counts are \(118,62,20\). Calculate the chi-square statistic. (Answer: Expected counts \(120,60,20\); \(\chi^2=0.0333+0.0667+0=0.1000\).)
A continuous model is assessed from 80 observations recorded only as counts in four intervals. The model probabilities are \(0.25,0.25,0.30,0.20\), and observed counts are \(18,22,26,14\). Calculate the statistic and choose the procedure. (Answer: Chi-square; expected counts \(20,20,24,16\), giving \(0.2000+0.2000+0.1667+0.2500=0.8167\).)
A fitted continuous loss model is assessed using payments rounded to the nearest ten dollars, with many ties. Is an ordinary continuous KS test with fully specified-model critical values automatically valid? (Answer: No; rounding creates ties and parameters were fitted, so the procedure needs calibration or adaptation for both features.)
Every observation in a portfolio is right-censored at its policy limit, and only the limit is recorded for those claims. Should the analyst enter each limit as an exact loss into the ordinary AD formula? (Answer: No; a censored threshold is not an exact loss, so an ordinary uncensored AD calculation is unsuitable without a censoring-aware method.)

What You'll Learn