Why Goodness-of-Fit Tests Need a Framework
An actuarial model is useful only if it describes the relevant data well enough for the task. A severity model, for example, may be used to estimate expected payments or assess tail risk. A goodness-of-fit test provides a structured way to ask whether observed data are inconsistent with a proposed distribution. It does not, by itself, establish that a model is true or decide whether the model is useful for every purpose.
The essential pieces are distinct: hypotheses describe the claim being tested; a test statistic summarizes the observed discrepancy; a reference distribution describes how that statistic behaves under the null hypothesis; and a decision rule compares the evidence with a chosen significance level. Confusing these pieces can lead to a correct calculation but an incorrect conclusion.
Hypotheses: Specified Models and Fitted Families
Suppose \(X_1,\ldots,X_n\) are observed losses. A fully specified null model might say that the losses are independent and identically distributed according to a particular distribution \(F_0\), with all its parameters fixed. In that case, the null hypothesis is \(H_0: F_X=F_0\). The alternative is \(H_1: F_X\ne F_0\), meaning that the data-generating distribution is not the proposed distribution.
A different question arises when a model family is proposed but its parameters are unknown. For example, a lognormal family can be written as \(F_{\mu,\sigma}\), where \(\ln X\sim N(\mu,\sigma^2)\) and \(\sigma>0\). The composite null is that \(F_X=F_{\mu,\sigma}\) for some allowable \(\mu\) and \(\sigma\). Estimates such as \(\hat\mu\) and \(\hat\sigma\) may be calculated from the same observations, but their use does not turn the hypothesis into the claim that the parameters equal those estimates with certainty.
The calibration of a test must match the null hypothesis and the procedure used to produce the statistic. In particular, the reference distribution for a statistic calculated after estimating parameters from the tested data need not be the same as its reference distribution when all parameters are fixed in advance. A supplied or derived calibration must account for that distinction. A formula for a test statistic alone is not enough to justify a rejection decision.
Significance Levels, p-Values, and Decisions
For many goodness-of-fit procedures, larger values of the statistic indicate greater discrepancy from the null. If the reference cumulative distribution function is \(G_0\), an upper-tail p-value for observed statistic \(t_{\mathrm{obs}}\) is
when the reference distribution is continuous. For a discrete statistic, the probability is calculated with the stated tail convention, commonly \(P_{H_0}(T\ge t_{\mathrm{obs}})\). The p-value is a probability under the null model: it measures how often a statistic at least as extreme as the observed one would arise if the null model and test calibration were correct.
Before examining the result, the analyst chooses a significance level \(\alpha\), such as 0.05. The level controls the probability of rejecting a true null hypothesis, subject to the test's calibration. The usual p-value rule is to reject \(H_0\) when \(p\le\alpha\); otherwise, do not reject \(H_0\). Equivalently, a critical-value rule rejects when the statistic falls in the designated rejection region. A decision at equality follows the stated rule, including its use of \(\le\).
Write the null distribution or family, identify which parameters are fixed or estimated, and state the alternative.
Determine what the statistic measures and use its reference distribution under the stated null and estimation procedure.
Calculate the p-value or compare the observed statistic with the correct critical value.
State whether the null is rejected at the chosen level and explain what that does, and does not, say about the model.
Why the p-Value Rule Controls the Error Rate
A significance level is not merely a customary cutoff. It has a probability interpretation when the test is calibrated. Under a continuous null reference distribution, the p-value is the upper-tail probability associated with the observed statistic. If \(T\) has continuous null CDF \(G_0\), then \(G_0(T)\) is uniformly distributed on \([0,1]\) under the null. Consequently, \(1-G_0(T)\), the upper-tail p-value, is also uniform under \(H_0\).
It follows that, when \(H_0\) is true,
Thus the rule “reject when \(p\le\alpha\)” rejects a true null with probability \(\alpha\) for this continuous, correctly calibrated case. For discrete tests, the attainable p-values may not fill the interval continuously; a valid rule generally has \(P_{H_0}(p\le\alpha)\le\alpha\). This justification depends on correct calibration. It does not rescue a test whose reference distribution ignores parameter fitting or other features of the sampling procedure.
Worked Example: Testing a Fully Specified Severity Model
A hypothetical portfolio has \(n=60\) independent, identically distributed claim severities. The proposed model is exponential with mean \(\theta=400\) dollars, fixed in advance. A suitably calibrated upper-tail discrepancy statistic is \(T=5.00\); under this null model its reference distribution is chi-square with 4 degrees of freedom. Test at \(\alpha=0.05\). For this reference distribution, use \(P(T\ge t)=e^{-t/2}(1+t/2)\).
Method and hypotheses. This is a goodness-of-fit test of a fully specified distribution. The hypotheses are \(H_0: X_i\) are iid exponential with mean \(\theta=400\), against \(H_1\): the severities do not follow that specified distribution. The statistic and its null calibration are supplied; no separate statistic construction is required.
Calculate the p-value. Substitute \(t_{\mathrm{obs}}=5.00\) into the supplied upper-tail formula:
Decision and interpretation. Since \(0.2873>0.05\), do not reject \(H_0\) at the 5% level. This sample does not provide sufficient evidence, by this test, to reject the specified exponential model. It does not prove that the model is correct.
Check. The statistic is not unusually large under its stated reference distribution: its upper-tail probability is greater than 0.05. The conclusion agrees with the p-value rule.
Common mistake avoided. Treating \(T=5.00\) as a probability or comparing it directly with 0.05 is incorrect. The statistic must first be assessed using its reference distribution; the resulting p-value is the quantity compared with \(\alpha\).
Fitting Parameters Changes What Must Be Calibrated
When parameters are estimated from the same observations used in a goodness-of-fit test, the test statistic may be affected by that estimation. For example, fitting a location or scale parameter can make the fitted distribution track some features of the observed data. The null behavior of a discrepancy statistic can therefore differ from the behavior it would have if those parameters had been fixed before observing the sample.
This does not mean that fitted models cannot be tested. It means the statistic's reference distribution must correspond to the actual procedure: the model family, the estimation method, the sample design, and the statistic. If a problem states that the reference distribution has already been calibrated for parameter fitting, use it as stated. If it supplies only a known-parameter calibration, do not silently assume that calibration applies after estimation.
Worked Example: Decision for a Fitted Distribution Family
A hypothetical insurer models positive repair costs using the lognormal family, parameterized by \(\ln X\sim N(\mu,\sigma^2)\), with \(\mu\) and \(\sigma\) estimated from the same \(n=90\) observed costs. For the specified estimation procedure, the problem supplies a calibrated upper-tail reference distribution: chi-square with 4 degrees of freedom. The observed statistic is \(T=8.00\). Decide at both the 5% and 10% significance levels. Use \(P(T\ge t)=e^{-t/2}(1+t/2)\).
Method and hypotheses. The null is composite: \(H_0\) says the observations come from some lognormal distribution in the stated family, with parameters estimated by the stated procedure. The alternative is that they do not come from any distribution in that family. The provided calibration expressly applies after fitting; it is not being assumed from a fixed-parameter test.
Calculate the p-value.
Make both decisions. At \(\alpha=0.05\), \(0.0916>0.05\), so do not reject the lognormal-family null. At \(\alpha=0.10\), \(0.0916\le0.10\), so reject it. These decisions are not contradictory: the 10% rule rejects for a broader range of evidence against the null than the 5% rule.
Check. The p-value lies between the two levels: \(0.05<0.0916\le0.10\). Therefore, non-rejection at 5% and rejection at 10% follow directly from the decision rule.
Common mistake avoided. It would be unjustified to use a known-parameter reference distribution merely because the fitted family is lognormal. The calibration supplied here is explicitly for the stated fitting procedure.
What a Rejection or Non-Rejection Means
A rejection at level \(\alpha\) is evidence against the null model under the specified test and calibration. It does not identify the exact feature responsible for the discrepancy, nor does it automatically show that every alternative model is better. A non-rejection means only that the test did not find sufficiently strong evidence against the null at that level. It is not the probability that \(H_0\) is true, and it is not a declaration that the model has been proved adequate.
The p-value also is not \(P(H_0\mid\text{data})\). It is calculated under the assumption that \(H_0\) holds, and asks how often the statistic would be at least as extreme as observed. A small p-value can result from a substantial mismatch, a large sample detecting a modest mismatch, or both. Model adequacy therefore requires judgment about the intended actuarial use as well as a formal test result.
Worked Example: Interpreting a Borderline Result
A hypothetical administrator assesses whether a proposed distribution describes claim counts per policy-year. The null specifies a Poisson distribution with mean \(\lambda=1.20\), fixed from an external assumption before the sample is examined. An upper-tail statistic has a supplied null reference distribution and yields \(p=0.032\). State the decision at \(\alpha=0.01\) and \(\alpha=0.05\), and interpret the result without overstating it.
Method and hypotheses. The null is that the observed policy-year counts are generated by the specified Poisson distribution with mean 1.20; the alternative is that they are not. The p-value is supplied, so compare it with each significance level.
At 1%. Since \(0.032>0.01\), do not reject \(H_0\) at the 1% level.
At 5%. Since \(0.032\le0.05\), reject \(H_0\) at the 5% level.
Interpretation and check. The result is evidence against the specified Poisson model at 5%, but not at 1%. Because \(0.01<0.032\le0.05\), both decisions follow the p-value rule. The p-value does not mean there is a 3.2% probability that the Poisson model is true.
Common mistake avoided. Reporting “there is a 3.2% chance the null is correct” reverses the conditioning. The 3.2% is a probability of a statistic at least this extreme, calculated assuming the null model.
Common Exam Traps
- Using the wrong hypothesis language. The alternative for a general goodness-of-fit test is that the distribution does not match the proposed model or family; it is not automatically a specific named alternative.
- Mixing up the statistic and its reference distribution. A statistic summarizes the observed sample. Its reference distribution is the null behavior used to assess how extreme that summary is.
- Ignoring estimation. A null calibration for fixed parameters cannot automatically be transferred to a test where parameters were estimated from the same data.
- Calling non-rejection acceptance. “Do not reject” is a statement about evidence at a chosen level, not proof of the model.
- Changing the significance level after seeing the result. The chosen \(\alpha\) determines the decision threshold and should be set before interpreting the test result.
Check Your Understanding
For each problem, use the stated calibration and report the requested decision or interpretation.
- A specified severity model has a calibrated upper-tail statistic with chi-square reference distribution on 4 degrees of freedom. The observed statistic is \(T=4.00\). Using \(P(T\ge t)=e^{-t/2}(1+t/2)\), calculate the p-value to four decimals and decide at \(\alpha=0.05\). (Answer: \(p=0.4060\); do not reject.)
- A fitted distribution family has a reference calibration that accounts for parameter estimation. Its observed statistic has p-value 0.018. State the decision at \(\alpha=0.01\) and \(\alpha=0.05\). (Answer: Do not reject at 1%; reject at 5%.)
- A test rejects a null model at the 5% significance level. Does this establish that the null model is false, or that the alternative model is correct? Give the strongest justified conclusion. (Answer: It is evidence against the null at the 5% level; it proves neither that the null is false nor that a particular alternative is correct.)
- A claim severity model has a statistic whose p-value is 0.074. An analyst says, “There is a 7.4% chance that the model is correct.” Explain the error and state the decision at \(\alpha=0.10\). (Answer: The p-value is a null-conditional probability of a statistic at least as extreme as observed, not the probability the model is correct; reject at 10%.)
- Parameters of a proposed distribution were estimated from the tested sample. The only supplied reference distribution is stated to apply when parameters are fixed in advance. Can it be used without qualification to make the fitted-model decision? (Answer: No; the calibration must account for the parameter-estimation procedure, or its validity for the fitted case must otherwise be established.)