Why Goodness-of-Fit Tests Can Disagree
A severity model can look plausible under one goodness-of-fit test and questionable under another. This is not necessarily a calculation error. The Kolmogorov-Smirnov (KS), Anderson-Darling (AD) and chi-square tests summarize discrepancies differently: they emphasize different parts of the distribution, and the chi-square test also depends on how observations are grouped. Sample size and parameter estimation further affect the evidence.
The central task is to interpret the disagreement, not to select whichever test produces the preferred conclusion. First check whether the tests address the same null model and whether their p-values are calibrated for the way the model was fitted. Then ask where the data depart from the model and whether that part of the distribution matters for the intended actuarial use.
When several tests are applied to the same fitted severity model, their results are not independent votes. Each test looks at a different summary of the same observations. A small p-value from one test can identify a type of discrepancy to investigate, while a larger p-value from another may simply reflect that the second test is less sensitive to that discrepancy.
What Each Test Emphasizes
For a continuous model with cumulative distribution function \(F\), the KS statistic is the largest absolute gap between the empirical distribution function \(F_n\) and \(F\):
The KS statistic captures the largest vertical discrepancy, wherever it occurs. It does not deliberately give extra weight to the tails. The AD statistic instead gives relatively greater weight to discrepancies near the ends of the distribution. This can make it especially informative when the lower or upper tail is important. The Pearson chi-square statistic compares observed counts \(O_j\) with expected counts \(E_j\) in selected categories:
Because the chi-square test works with category counts, it does not use the exact position of every observation within a category. Its result can change when category boundaries change. The KS and AD tests also require appropriate calibration when parameters have been estimated from the same data; using a null distribution that assumes known parameters can give misleading p-values.
A Short Justification for Different Sensitivities
The formulas themselves explain why the tests can disagree. KS reduces all discrepancies to their largest absolute value. If the largest gap is modest, KS may not reject even when there are several smaller discrepancies. AD accumulates discrepancies across the distribution and weights the tails more heavily. A tail departure that is not the single largest vertical gap can therefore matter more to AD than to KS.
For chi-square, grouping changes the information being compared. Suppose several original categories are combined. The combined observed and expected counts are sums of their component counts. Opposing deviations can cancel: one category may have more observations than expected while its neighbor has fewer. The grouped comparison then sees a smaller discrepancy than the ungrouped comparison. This follows directly from comparing a sum of deviations with the separate squared deviations in \(X^2\).
Thus, a disagreement may point to a meaningful feature of the data, such as a localized departure or tail mismatch, rather than a contradiction among the tests. The grouping and fitting choices must still be defensible; one should not keep changing them until a desired test result appears.
Worked Example: A Tail Concern Missed by a Broad Check
A hypothetical insurer fits a two-parameter lognormal severity model to 250 independent paid losses. The estimation procedure is repeated in the calibration used for each test. At \(\alpha=0.05\), the reported p-values are KS \(0.120\), AD \(0.018\), and chi-square \(0.090\). Interpret the results.
Step 1: Check the setup. The problem states that all tests assess the same fitted model and that the calibration accounts for parameter estimation. Thus, the p-values can be compared as evidence against the same null model. The significance level is \(0.05\).
Step 2: Apply the decision rule. For KS, \(0.120>0.05\), so do not reject. For AD, \(0.018<0.05\), so reject. For chi-square, \(0.090>0.05\), so do not reject.
Step 3: Interpret the pattern. The AD result is evidence that the fitted model misses some aspect of the distribution, with particular reason to inspect the tails. The KS result says that its largest empirical-versus-model gap is not statistically significant at this level. The chi-square result says that the chosen grouped counts did not provide sufficient evidence to reject. Neither non-rejection proves the lognormal model is correct.
Method and mistake avoided. Interpret the tests by sensitivity and inspect tail diagnostics before drawing a conclusion. A common mistake is to claim “two tests accept the model, so the model fits.” Tests do not accept a model, and a majority vote ignores the AD evidence.
Category Choices and Parameter Fitting
A chi-square result is conditional on its categories. Boundaries should represent the question being studied, and expected counts must be adequate for the chosen approximation. Combining categories can be necessary when expected counts are sparse, but it can also hide a discrepancy if the combined categories have deviations in opposite directions. Any comparison across alternative groupings should explain why each grouping is relevant, rather than treating the most favorable result as decisive.
Parameter fitting requires equal care. If a distribution's parameters are estimated from the observations, the fitted model has already adapted to those data. The reference distribution for a test statistic may therefore differ from the one used when parameters are known in advance. For a fair interpretation, calibration should reproduce the fitting process (for example, by refitting in each simulated sample when simulation is used). A p-value from an inappropriate calibration is not repaired by comparing it with other p-values.
Worked Example: A Grouping Hides Opposing Deviations
A fully specified hypothetical severity model assigns probability \(0.25\) to each of four ordered loss bands. A sample of 400 losses has observed counts \(120, 80, 100, 100\). Compare the four-band chi-square test with a two-band test that combines bands 1 and 2 and combines bands 3 and 4. Assume observations are independent and the expected counts are adequate.
Step 1: Four-band calculation. Each expected count is \(400(0.25)=100\). The contributions are \( (120-100)^2/100=4\), \( (80-100)^2/100=4\), and \(0\) for each of the last two bands. Hence \(X^2=4+4+0+0=8\). There are \(4-1=3\) degrees of freedom because the four probabilities are specified and sum to one. The upper-tail p-value is approximately \(0.0460\), so this test rejects at \(0.05\).
Step 2: Two-band calculation. The combined observed counts are \(120+80=200\) and \(100+100=200\). Each combined expected count is \(100+100=200\). Thus both contributions are zero, and \(X^2=0\). There are \(2-1=1\) degree of freedom, and the p-value is \(1.0000\); this test does not reject.
Step 3: Explain the disagreement. The first two bands have deviations \(+20\) and \(-20\). Combining them cancels those deviations, so the two-band test cannot detect the difference between those bands. The tests used different groupings, not conflicting calculations.
Method and mistake avoided. Compare the observed and expected counts at each stated grouping before interpreting the statistic. The trap is to combine categories and then assume the resulting non-rejection means there was no discrepancy in the original data.
Sample Size and Actuarial Purpose
A test's p-value reflects both the pattern of departure and the amount of information in the sample. With a large sample, even a small departure from the model can be statistically detectable. With a small sample, a material tail mismatch may go undetected because the test has limited power. Consequently, neither “significant” nor “not significant” by itself measures the practical importance of a model error.
The relevant part of the distribution also depends on the task. A model used to estimate an ordinary expected claim payment may be judged differently from one used to price excess-of-loss protection or quantify a high loss percentile. Tail discrepancies can have a substantial effect on a high layer even when they affect few observations. Conversely, a statistically detectable central discrepancy may have little effect on a particular decision. The analyst should connect diagnostics to the model's intended use and quantify the consequence where possible.
Worked Example: Statistical Evidence and Tail Impact
A hypothetical portfolio has 10,000 independent claims. A fitted severity model is calibrated with parameter refitting in each test simulation. At the 5% level, the reported p-values are KS \(0.032\), AD \(0.004\), and chi-square \(0.110\). In the upper-tail region, the model assigns probability \(0.010\), while the empirical proportion is \(0.012\). Interpret the evidence for a decision involving high losses.
Step 1: Test decisions. KS rejects because \(0.032<0.05\). AD rejects because \(0.004<0.05\). Chi-square does not reject because \(0.110>0.05\). The results indicate evidence against the fitted model, especially under the tail-sensitive AD test; the chi-square non-rejection does not erase that evidence.
Step 2: Quantify the stated tail difference. The model's expected number of observations in that region is \(10{,}000(0.010)=100\). The empirical proportion corresponds to \(10{,}000(0.012)=120\) observations. The difference is \(120-100=20\) observations, or \(0.002\) of the full sample: \(0.012-0.010=0.002\).
Step 3: Relate it to the purpose. If the model supports a decision sensitive to high losses, the excess of 20 empirical tail observations warrants investigation and may be material. The information given does not establish its dollar impact, so that impact must be assessed using the relevant payment or layer structure rather than inferred from the p-values alone.
Method and mistake avoided. Separate statistical significance from practical consequence, and examine the tail when the decision concerns high losses. The trap is to dismiss the discrepancy because \(0.002\) appears numerically small, or to claim a particular financial impact without the needed loss amounts.
A Practical Interpretation Sequence
A disciplined interpretation can be organized as follows. It does not replace the calculations; it prevents a test result from being given more meaning than it supports.
Check that the tests use the same observations, model and significance level, and that calibration reflects whether parameters were estimated.
Relate KS to the largest gap, AD to tail-sensitive discrepancies, and chi-square to the selected category counts.
Review category boundaries, expected counts and parameter fitting. Explain choices on statistical or actuarial grounds, not by which p-value they produce.
Consider sample size and the model's intended use. Quantify how the observed discrepancy affects the relevant expected payment, layer or risk measure when the required information is available.
A good conclusion is specific: state which tests reject at the chosen level, identify the aspect of fit that may be responsible, and say what further diagnostic or consequence analysis is needed. This approach is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus because it interprets test results in the context of their assumptions and actuarial use.
Check Your Understanding
For each problem, state what the test results do and do not justify.
- At \(\alpha=0.05\), a fitted model has calibrated KS and AD p-values \(0.070\) and \(0.020\). Which test rejects, and what part of the distribution merits focused inspection? (Answer: AD rejects; inspect the tails in particular. KS does not reject, and its result does not prove the model is correct.)
- A fully specified model has four equally likely categories and \(n=200\). Observed counts are \(60,40,50,50\). Calculate the four-category Pearson statistic and identify whether combining the first two categories would preserve their discrepancy. (Answer: Expected counts are 50 each; \(X^2=2+2+0+0=4\). Combining the first two gives observed 100 and expected 100, so their opposing deviations cancel.)
- A chi-square test uses \(k=6\) categories with two estimated model parameters. Under the usual regularity and adequate-count conditions, what degrees of freedom are used, and why should the p-value be treated cautiously if the parameters were not accounted for in calibration? (Answer: \(6-1-2=3\) degrees of freedom; calibration that ignores fitting may not give the appropriate null distribution.)
- In a sample of 5,000 claims, a model predicts a tail probability of \(0.006\), while the empirical proportion is \(0.008\). How many observations correspond to each proportion, and what conclusion about dollar impact is justified? (Answer: 30 predicted and 40 empirical, a difference of 10 observations; dollar impact cannot be determined without the loss amounts and relevant payment structure.)
- KS, AD and chi-square p-values are \(0.14\), \(0.01\) and \(0.20\), respectively. The chi-square categories combine adjacent bands, and the severity parameters were fitted from the data. Name two checks required before concluding that the AD result is compelling evidence of model failure. (Answer: Verify that AD calibration accounts for parameter fitting, and inspect tail diagnostics and the chi-square category grouping; the results should be interpreted as evidence, not as proof of failure.)