Goodness-Of-Fit Tests Versus Information Criteria

Two Different Questions About Model Fit

A goodness-of-fit test and an information criterion can be applied to the same candidate severity models, but they answer different questions. A goodness-of-fit test asks whether the observed data provide sufficient evidence against a specified null model at a chosen significance level. AIC and BIC compare candidate models by balancing fit against model complexity; they identify a relative ranking, not a null-hypothesis conclusion.

This distinction matters in actuarial work. A model may not be rejected by a test but still rank behind another model under AIC or BIC. Conversely, a criterion may rank one model first even though a goodness-of-fit test indicates that its fit is inadequate for the intended use. The methods are complementary, not interchangeable.

Definition: A goodness-of-fit test specifies a null hypothesis about the data-generating distribution and uses a statistic whose null distribution is known or calibrated. Its conclusion is to reject or fail to reject that null at a stated significance level. An information criterion assigns a score to each fitted candidate model on comparable data; lower scores indicate a preferred relative trade-off between fit and complexity. Neither conclusion, by itself, proves that a model is true or suitable for every actuarial purpose.

The Three Goodness-of-Fit Tests

Let $X_1,\ldots,X_n$ be independent observations from a continuous null distribution with cumulative distribution function $F_0$. If the null distribution is fully specified, the probability integral transform gives $U_j=F_0(X_j)$, which is uniform on $(0,1)$ under the null. The tests below measure different features of departure from that null.

Kolmogorov–Smirnov test: Let $X_{(1)}\le\cdots\le X_{(n)}$ be the ordered observations, and set $U_{(i)}=F_0(X_{(i)})$. Define $$ D^+=\max_{1\le i\le n}\left(\frac{i}{n}-U_{(i)}\right),\qquad D^-=\max_{1\le i\le n}\left(U_{(i)}-\frac{i-1}{n}\right),\qquad D=\max(D^+,D^-). $$ The statistic is the largest vertical distance between the empirical distribution function and the specified null distribution. Critical values must match the test setup.

Anderson–Darling test: For the same fully specified continuous null, with $U_{(1)}\le\cdots\le U_{(n)}$, define $$ A^2=-n-\frac{1}{n}\sum_{i=1}^{n}(2i-1)\left[\ln U_{(i)}+\ln\left(1-U_{(n+1-i)}\right)\right]. $$ A larger value indicates greater evidence against the null. Its weights give relatively more influence to departures in the tails than the Kolmogorov–Smirnov statistic does. The critical values must be calibrated for the null and for whether parameters were estimated.

Chi-square goodness-of-fit test: If observations are assigned to $k$ mutually exclusive categories with observed counts $O_j$ and null expected counts $E_j$, then $$ X^2=\sum_{j=1}^{k}\frac{(O_j-E_j)^2}{E_j}. $$ For a multinomial model with fixed category probabilities, the usual degrees of freedom are $k-1$. If $r$ parameters are estimated from the grouped data, the usual adjustment is $k-1-r$, subject to the model and estimation method supporting that calibration. Expected counts that are too small can make the usual chi-square approximation unreliable; categories may need to be combined or another calibration used.

The Kolmogorov–Smirnov statistic measures the single largest discrepancy, whereas Anderson–Darling accumulates discrepancies with extra weight in the tails. The chi-square test instead compares counts in chosen categories. Consequently, bin choice affects the chi-square statistic, while the KS and AD statistics use the ordered observations. In all three cases, the statistic alone is not a complete test conclusion: a critical value or p-value from the correct null distribution is needed.

Why the Statistics Measure Departures

For the KS statistic, the empirical distribution function just before $X_{(i)}$ has value $(i-1)/n$, and just after it has value $i/n$. At the observation itself, the null distribution function is $F_0(X_{(i)})=U_{(i)}$. The discrepancies on the two sides of that jump are therefore $U_{(i)}-(i-1)/n$ and $i/n-U_{(i)}$. Taking the largest of both kinds of discrepancies over all observations gives exactly $D$.

Anderson–Darling's tail emphasis can also be understood from the variance of the empirical distribution function. At a point where the null probability is $t$, its approximate variance is $t(1-t)/n$. This variance is small near 0 and 1. Weighting squared discrepancies by a quantity proportional to $1/[t(1-t)]$ therefore gives tail departures more influence than an unweighted comparison would. The formula's logarithmic terms produce this emphasis while yielding a statistic that can be calibrated under the null.

Worked Example: Applying the Kolmogorov–Smirnov Test

A hypothetical insurer examines five independent claim amounts under a fully specified continuous candidate severity model. The ordered probability integral transforms are $0.10, 0.28, 0.51, 0.72, 0.90$. For this fully specified null, use the supplied 5% critical value $0.565$. Calculate the statistic and make the test decision.

Method. Apply the KS statistic to the ordered transforms. Because the null distribution is fully specified and continuous, the transforms are compared with the uniform distribution on $(0,1)$.

Calculate $D^+$. The values $i/5-U_{(i)}$, for $i=1,\ldots,5$, are:

0.20-0.10=0.10,\quad 0.40-0.28=0.12,\quad 0.60-0.51=0.09,\quad 0.80-0.72=0.08,\quad 1.00-0.90=0.10.

Thus, $D^+=0.12$.

Calculate $D^-$. The values $U_{(i)}-(i-1)/5$ are:

0.10-0.00=0.10,\quad 0.28-0.20=0.08,\quad 0.51-0.40=0.11,\quad 0.72-0.60=0.12,\quad 0.90-0.80=0.10.

Thus, $D^-=0.12$, so $D=\max(0.12,0.12)=0.12$. Since $0.12<0.565$, fail to reject the null hypothesis at the 5% level. This is not proof that the candidate model is correct; with only five observations, the test may have limited ability to detect departures.

Common mistake avoided: Using only $D^+$ could miss the largest discrepancy on the other side of the empirical distribution. Both $D^+$ and $D^-$ must be calculated before taking their maximum.

Worked Example: Calculating an Anderson–Darling Statistic

For a hypothetical sample of four independent losses, the ordered probability integral transforms under a fully specified continuous model are $0.20,0.40,0.60,0.80$. Calculate $A^2$, rounding the final result to four decimal places.

Method. Substitute the ordered transforms into the Anderson–Darling formula. The values are symmetric: $1-U_{(5-i)}=U_{(i)}$ for each $i$. Thus, each bracketed pair of logarithms is twice the logarithm of the corresponding lower-half value.

Calculate the weighted log sum. The sum in the formula is:

\begin{aligned} \sum_{i=1}^{4}(2i-1)\left[\ln U_{(i)}+\ln(1-U_{(5-i)})\right] &=2\left[\ln(0.20)+3\ln(0.40)+5\ln(0.60)+7\ln(0.80)\right]\\ &=2\left[-1.609437912-2.748872196-2.554128119-1.562005238\right]\\ &\approx -16.948886930. \end{aligned}

Now $n=4$, so:

A^2=-4-\frac{-16.948886930}{4} =-4+4.237221733 \approx 0.2372.

The statistic is nonnegative, as expected. A formal reject-or-fail-to-reject conclusion still requires a critical value or p-value calibrated to this null and sample size. The calculation alone does not supply one; using a critical value for a different test setup would not justify a decision.

Common mistake avoided: The raw weighted log sum is negative, but the AD statistic is not that sum. The formula includes both the leading $-n$ and division of the sum by $n$; omitting either can produce an impossible negative statistic.

Worked Example: A Chi-Square Decision for Grouped Losses

A hypothetical model assigns probabilities $1/3$ to each of three claim-size categories. In a sample of 30 independent claims, the observed counts are $6,12,12$. The category probabilities are fixed in advance, not estimated from these data. At the 5% level, use the chi-square critical value $5.991$ with 2 degrees of freedom.

Expected counts. Each category has expected count $30(1/3)=10$. All expected counts are at least 5, so the usual approximation is reasonable for this example.

Calculate the statistic.

X^2=\frac{(6-10)^2}{10}+\frac{(12-10)^2}{10}+\frac{(12-10)^2}{10} =\frac{16}{10}+\frac{4}{10}+\frac{4}{10} =2.4.

There are $k=3$ categories and no estimated parameters, so the degrees of freedom are $k-1=2$. Because $2.4<5.991$, fail to reject the null at the 5% level. The data do not show sufficient evidence against the specified category probabilities at this significance level; they do not establish that those probabilities are true.

Common mistake avoided: Subtracting an additional degree of freedom for a fitted parameter would be incorrect here: no parameter was estimated from the grouped observations.

Information Criteria Rank Rather Than Test

For fitted models based on the same data and likelihood convention, let $\ell$ denote the maximized log-likelihood, $k$ the number of free estimated parameters, and $n$ the sample size. The standard criteria are:

Information criteria: AIC is $-2\ell+2k$, and BIC is $-2\ell+k\ln n$. A lower value ranks better under that criterion within the candidate set. The penalties differ: AIC adds $2k$, while BIC adds $k\ln n$. These scores are not p-values, and neither has a critical value that divides models into “rejected” and “not rejected.”

These comparisons require consistent likelihood calculations and comparable data, including the same observations and compatible treatment of censoring or truncation. AIC and BIC values calculated on different data sets or under inconsistent likelihood conventions do not provide a fair ranking. Earlier material on information criteria establishes the formulas and their penalties; the key point here is how their conclusions differ from those of goodness-of-fit tests.

Worked Example: AIC and BIC Can Rank Models Differently

Two hypothetical severity models are fitted by maximum likelihood to the same $n=80$ complete losses. Model $M_1$ has maximized log-likelihood $-100$ and $k_1=1$ free parameter. Model $M_2$ has maximized log-likelihood $-97.5$ and $k_2=3$. Calculate both criteria, to three decimal places, and state the relative rankings.

AIC calculations. For $M_1$, $-2\ell_1=200$, so $\operatorname{AIC}_1=200+2(1)=202$. For $M_2$, $-2\ell_2=195$, so $\operatorname{AIC}_2=195+2(3)=201$. AIC ranks $M_2$ first because $201<202$.

BIC calculations. Since $\ln(80)\approx4.382026635$,

\operatorname{BIC}_1=200+1(4.382026635)\approx204.382, \qquad \operatorname{BIC}_2=195+3(4.382026635)\approx208.146.

BIC ranks $M_1$ first because $204.382<208.146$. Thus, AIC and BIC disagree on the relative preference in this comparison. Neither result is a hypothesis-test decision, and neither establishes that its preferred model is adequate.

Common mistake avoided: It would be incorrect to say that AIC rejects $M_1$ or that BIC accepts $M_1$. The criteria rank models; they do not return reject-or-fail-to-reject conclusions.

Interpreting the Evidence Correctly

A non-rejection from a KS, AD, or chi-square test means only that the test did not find sufficient evidence against its null at the chosen significance level. It does not show that the null is true. The outcome depends on sample size, the type of departure, and the test's calibration. For example, the KS statistic is driven by the largest absolute discrepancy, while AD puts relatively more emphasis on the tails; a test can therefore be more sensitive to some departures than to others.

AIC and BIC answer a different, comparative question. The lowest value is preferred among the models evaluated under consistent conditions, but the magnitude of the score alone does not establish absolute fit. A model can rank first in a candidate set even if every candidate is a poor description of the data. This is why an actuarial assessment should distinguish absolute evidence against a specified model from relative preference among candidates.

The calibration details matter. The standard KS null distribution assumes a fully specified continuous distribution; estimating parameters from the same data changes the reference distribution. The same caution applies to AD. For chi-square tests, estimating parameters affects degrees of freedom, and small expected counts can invalidate the usual approximation. Always match the reference distribution to the test actually performed.

State the question.
Decide whether the goal is to test a specified null model or rank candidate models relative to one another.

Check the setup.
For a goodness-of-fit test, identify the null distribution, whether parameters were estimated, and whether the test's assumptions and calibration apply.

Calculate and interpret.
Compare a test statistic with its appropriate reference value, or compare AIC/BIC scores computed on consistent data. Do not turn a relative ranking into a null-hypothesis conclusion.

Key takeaway: KS, Anderson–Darling, and chi-square tests assess evidence against a specified null distribution using calibrated test statistics. AIC and BIC rank candidate models by fit-complexity trade-offs. A model's non-rejection is not proof of truth, and a model's lowest information criterion is not proof of adequate fit.

Check Your Understanding

For each problem, distinguish the calculation from the conclusion the method supports.

Four ordered probability integral transforms under a fully specified continuous null are $0.15,0.30,0.55,0.80$. Calculate $D^+$, $D^-$, and $D$. (Answer: $D^+=0.20,\ D^-=0.15,\ D=0.20$.)
Three ordered transforms for a fully specified continuous null are $0.25,0.50,0.75$. Calculate the AD statistic to four decimal places. (Answer: $A^2\approx0.2694$; a test decision additionally requires an appropriate calibrated critical value or p-value.)
For three fixed categories, observed counts are $8,14,8$, and the null probabilities are $1/3$ each. At the 5% level, use the critical value $5.991$ with 2 degrees of freedom. Calculate the statistic and conclusion. (Answer: $X^2=2.4$; fail to reject the null.)
Two models are fitted to the same 50 observations. Model A has $\ell=-60,k=1$; Model B has $\ell=-57,k=3$. Calculate AIC and BIC to three decimals, using $\ln(50)\approx3.912023$, and identify each criterion's preferred model. (Answer: AIC$_A=122$, AIC$_B=120$, so AIC prefers B; BIC$_A\approx123.912$, BIC$_B\approx125.736$, so BIC prefers A.)
A candidate model is not rejected by a goodness-of-fit test, but another model has the lower AIC on the same data. Does either result prove which model is true? (Answer: No. Non-rejection is not proof of the null, and lower AIC is only a relative ranking within the candidate set.)

What You'll Learn