From a fitted severity model to a defensible assessment
A goodness-of-fit assessment is a sequence of linked decisions, not just a test statistic. The analyst specifies a severity model, estimates its parameters, compares observed losses with the fitted distribution, and calibrates each diagnostic for how the parameters were estimated. An apparently small statistic can be misleading if its reference distribution assumes known parameters while the parameters were fitted to the same data.
This tutorial integrates two common diagnostics for continuous loss data: the Kolmogorov–Smirnov (KS) test, which measures the largest discrepancy between cumulative distributions, and Pearson’s chi-square test, which compares observed and expected counts in classes. The worked calculations use exponential models. The same workflow applies more broadly, but test calibration and class construction must match the model and data.
What must be calibrated
If a continuous null distribution is completely specified independently of the test sample, standard one-sample KS critical values apply. If the same observations are used to estimate parameters and calculate \(D_n\), the usual fully specified-distribution critical values generally do not apply: fitting tends to make the empirical distribution look closer to the fitted distribution. A parametric bootstrap can account for this effect by simulating from the fitted model and refitting the parameters in every simulated sample.
For Pearson’s statistic, a common large-sample reference distribution is chi-square with \(k-1-p\) degrees of freedom, where \(p\) parameters have been estimated from the same data. This approximation requires suitable expected counts and regularity conditions. It is not a universal correction: if the parameters were estimated on an independent fitting sample, and the test sample is independent, the fitted model is fixed conditional on that fitting sample, so the usual degrees of freedom are \(k-1\). Sparse expected counts can make the chi-square approximation unreliable even when the degrees of freedom are chosen correctly.
A practical alternative for both diagnostics when parameters are estimated from the test data is a parametric bootstrap. Simulate samples of size \(n\) from the fitted model, refit the model to each simulated sample using the original estimation procedure, and recalculate the statistic. If \(B\) simulations are made and \(b\) simulated statistics are at least as large as the observed statistic, the commonly used corrected estimate of the upper-tail p-value is \((b+1)/(B+1)\). Re-estimation is essential: simulating from the fitted model but not refitting fails to reproduce the parameter-fitting effect.
Why the KS statistic compares both sides of each jump
At \(x_{(i)}\), the empirical cumulative distribution function immediately after the observation is \(i/n\); immediately before it is \((i-1)/n\). The fitted continuous distribution has value \(F_{\hat\theta}(x_{(i)})\) at that point. The greatest discrepancy can occur on either side of the jump, which gives \(D_n^+\) and \(D_n^-\) in the definition. Taking only \(i/n-F_{\hat\theta}(x_{(i)})\), for instance, would overlook a large discrepancy in the opposite direction.
Worked Example: Fit on one sample, test on another
Worked Example: Fit on one sample, test on another
A hypothetical analyst fits an exponential severity model using an independent training sample of 10 losses whose total is $10. The analyst then tests a separate validation sample of 20 losses: \(0.10,0.20,0.25,0.35,0.45,0.55,0.65,0.68,0.69,0.75,0.90,1.10,1.20,1.30,1.40,1.45,1.50,1.60,1.80,3.08\). Calculate the fitted parameter, a KS statistic, and a chi-square statistic using four equal-probability classes. Assess both tests at the 5% level.
Method. Write the exponential model in scale form, so \(F_\theta(x)=1-e^{-x/\theta}\) for \(x\ge0\), where \(\theta=E[X]\). The maximum-likelihood estimate is the training-sample mean. Thus \(\hat\theta=10/10=1\). Since the validation sample is independent of the data used to estimate \(\hat\theta\), treat the fitted distribution as fixed for these validation tests.
KS calculation. The largest negative-side discrepancy is at \(x_{(7)}=0.65\): \(F_{\hat\theta}(0.65)=1-e^{-0.65}\approx0.4780\), so \(F_{\hat\theta}(x_{(7)})-6/20\approx0.4780-0.3000=0.1780\). Checking the ordered observations gives \(D_n^+\approx0.1153\), with its maximum at \(x_{(19)}=1.80\), and \(D_n^-\approx0.1780\). Therefore \(D_n\approx0.1780\). The approximate 5% critical value for a fully specified continuous model at \(n=20\) is 0.294. Since \(0.1780<0.294\), the KS test does not reject.
Chi-square calculation. Four equal-probability exponential classes have boundaries at the 25th, 50th, and 75th percentiles: \(-\ln(0.75)\approx0.2877\), \(-\ln(0.50)\approx0.6931\), and \(-\ln(0.25)\approx1.3863\). The classes are \([0,0.2877]\), \((0.2877,0.6931]\), \((0.6931,1.3863]\), and \((1.3863,\infty)\). Each has probability 0.25, so each expected count is \(20(0.25)=5\). The observed counts are \(3,6,5,6\).
Because the model was estimated on an independent sample, use \(k-1=3\) degrees of freedom, not a parameter-estimation subtraction. The chi-square upper-tail p-value is approximately 0.753. All expected counts are 5, meeting the common minimum guideline for this approximation. Neither test rejects at 5%. This is evidence that the validation data are not inconsistent with the fitted exponential model on these diagnostics; it does not prove that the model is correct.
Common mistake avoided: Subtracting one degree of freedom here because one parameter was estimated would confuse independent fitting with fitting on the test sample. The validation observations played no part in estimating \(\hat\theta\).
When the same observations estimate and test the model
In many analyses, one sample serves both purposes. The model parameter, expected class counts, and test statistics are then linked through the data. Standard KS critical values for a fully specified distribution are not valid in general. For Pearson’s statistic, the familiar \(k-1-p\) degrees-of-freedom rule is an approximation, not a substitute for checking expected counts and the estimation procedure. A bootstrap provides a direct way to reflect the entire procedure: each simulated sample must be fitted anew before its test statistic is calculated.
Worked Example: Calibrate tests after fitting to the test sample
Use the same 20 losses listed in the previous example, but now suppose they are the only data available. Fit an exponential model to these losses, then assess it with the KS and four-class Pearson statistics. For the KS bootstrap, take as given that 231 of 999 simulated-and-refitted statistics were at least as large as the observed statistic.
Method and fitted parameter. The sum of the listed losses is 20, so the sample mean is \(20/20=1\) and the exponential maximum-likelihood estimate is \(\hat\theta=1\). The ordered values and fitted CDF values are unchanged from the preceding calculations. Hence \(D_n^+\approx0.1153\), \(D_n^-\approx0.1780\), and \(D_n\approx0.1780\).
KS calibration. The stated bootstrap count includes the required refitting in every replicate. With \(b=231\) and \(B=999\),
At the 5% level this bootstrap result does not reject the exponential model. The standard fully specified-distribution KS critical value is not used: the data being tested also determined the fitted scale.
Pearson calibration. The observed counts remain \(3,6,5,6\), with expected counts 5 each and \(X^2=1.2\). Here four classes are used and one parameter is estimated from the same sample, so the conventional approximate degrees of freedom are \(4-1-1=2\). The chi-square upper-tail probability at 1.2 with 2 degrees of freedom is \(e^{-1.2/2}=e^{-0.6}\approx0.549\). This test also does not reject at 5%.
Conclusion. Both diagnostics are consistent with retaining the exponential model as a candidate. The KS p-value is bootstrap-calibrated; the chi-square p-value uses the usual degrees-of-freedom approximation. Their numerical p-values need not match because the statistics measure different discrepancies and their calibrations differ.
Common mistake avoided: Using the ordinary KS critical value because the fitted distribution is “exponential” ignores that its scale was estimated from these same 20 losses.
Interpreting agreement and disagreement
The KS statistic responds to the largest cumulative discrepancy at any point. Pearson’s statistic aggregates discrepancies across the chosen classes, with each squared difference scaled by its expected count. As a result, a localized cumulative departure can be prominent for KS without producing a large grouped statistic. Conversely, Pearson’s result can depend noticeably on class boundaries, while KS does not require binning. Neither test is universally more informative, and a non-rejection is not proof of fit.
A defensible assessment records the fitted model and parameter estimate, the data used for fitting, the statistic, the calibration method, and the conclusion at the stated significance level. If test results conflict, report the conflict and investigate which part of the distribution drives it. Do not select whichever p-value supports a preferred model. Graphical diagnostics and the practical implications of tail fit can add context, but they do not change the calibration of a formal test.
Worked Example: A KS warning despite a modest chi-square result
A hypothetical analyst fits a one-parameter exponential severity model to 40 losses and uses four equal-probability classes. The observed class counts are \(4,10,10,16\), with expected count 10 in each class. The ordered losses give \(D_n=0.23\). A parametric bootstrap with 999 simulated-and-refitted samples gives 29 statistics at least as large as the observed KS statistic. Interpret the two diagnostics at the 5% level.
Method and Pearson calculation. One parameter was estimated from the same sample, so the conventional Pearson reference degrees of freedom are \(4-1-1=2\). The statistic is
The upper-tail probability for a chi-square variable with 2 degrees of freedom at 7.2 is \(e^{-7.2/2}=e^{-3.6}\approx0.0273\). Thus the Pearson test rejects at 5%.
KS calculation and calibration. The problem supplies \(D_n=0.23\) from the ordered losses; it is not possible to reconstruct that statistic from class counts alone. The bootstrap p-value is \((29+1)/(999+1)=0.030\), so the KS test also rejects at 5%. The two tests agree on this assessment, but they reach it through different summaries: cumulative discrepancy for KS and class-count discrepancy for Pearson.
Conclusion and trap. Both diagnostics flag a lack of fit under their stated calibrations. A common error would be to use the fully specified-model KS critical value despite estimating the exponential parameter from these 40 losses. The bootstrap result avoids that error. These results warrant further investigation of the model and the losses; they do not identify a replacement model by themselves.
Integrated workflow and exam traps
State the distribution, parameters, and whether the test observations also estimate those parameters.
For KS, check both sides of each empirical jump. For Pearson, define exhaustive classes and calculate their probabilities under the fitted model.
Use standard fully specified-model KS calibration only when parameters are fixed independently. Otherwise use a suitable adjusted calibration, such as a parametric bootstrap. For Pearson, state and justify the degrees of freedom and check expected counts.
Compare p-values or statistics with their appropriate reference distributions, report rejection or non-rejection, and explain what the diagnostics do and do not establish.
- Wrong: “One parameter was estimated, so subtract one degree of freedom” in a validation sample independent of the fitting sample. Correction: Conditional on the independent fit, use \(k-1\) degrees of freedom for the Pearson test.
- Wrong: Use standard KS critical values after fitting the model to the same observations. Correction: Use a calibration that accounts for estimation, such as a bootstrap that refits every replicate.
- Wrong: Treat non-rejection as proof that the severity model is true. Correction: Say only that the test does not provide sufficient evidence against the model at the selected significance level.
Check Your Understanding
For each problem, identify the appropriate calculation or calibration before interpreting the result.
- An exponential model has fitted scale \(\hat\theta=2.5\). For \(n=100\), find the expected counts in three classes with cumulative probabilities 0.3935 and 0.6321 at their boundaries. (Answer: 39.35, 23.86, and 36.79, approximately.)
- A Pearson test uses 5 classes and estimates 2 parameters from the same sample. What is the conventional approximate degrees of freedom? (Answer: 2.)
- A bootstrap uses \(B=999\) refitted samples, of which \(b=47\) statistics equal or exceed the observed statistic. Find the corrected p-value. (Answer: 0.0480.)
- A validation sample has \(n=25\), and its model parameters were fitted on an independent sample. What is the approximate 5% fully specified-model KS critical value using \(1.36/\sqrt{n}\)? (Answer: 0.272.)
- A chi-square test does not reject, but a properly calibrated KS bootstrap test does reject. What is the most defensible immediate conclusion? (Answer: The diagnostics conflict; report the discrepancy and investigate the pattern of losses rather than declaring the model adequate.)