Why Non-Nested Models Need Different Comparisons

Why the Relationship Between Models Matters

A larger maximized likelihood can indicate which of two models assigns greater likelihood to the observed data. It does not, by itself, say whether that difference is statistically significant. For a likelihood-ratio test, the relationship between the models matters: the usual chi-square result is derived for a particular structure in which one model is a restriction of the other.

This distinction is important in actuarial work. A claim-count analyst might consider Poisson and geometric models, while a severity analyst might consider exponential and lognormal models. Such candidate pairs can describe different distributional shapes without either model being a special case of the other. Comparing their fitted likelihoods may provide a descriptive ranking, but treating twice their difference as an ordinary likelihood-ratio statistic with degrees of freedom taken from parameter counts is not justified.

Definition: Two models are nested when every distribution allowed by one model is also allowed by the other. The smaller model is then a restriction of the larger model. Two models are non-nested when neither model is a restriction of the other. Non-nested models may still overlap at particular distributions, but neither contains the full set of distributions allowed by the other.

For a nested comparison, write the restricted model as \(H_0\) and the larger model as \(H_1\). If their maximized log likelihoods are \(\ell_0\) and \(\ell_1\), respectively, the likelihood-ratio statistic is

\Lambda=2(\ell_1-\ell_0).

Under the null hypothesis and suitable regularity conditions, \(\Lambda\) has an approximate chi-square distribution with degrees of freedom equal to the number of independent restrictions. That calibration is not a general rule for any two fitted models. In particular, subtracting their parameter counts does not create valid degrees of freedom when the models are non-nested.

Why the Usual Chi-Square Result Requires Nesting

In a regular nested comparison, the null hypothesis places the true parameter on a restricted part of the larger model's parameter space. The larger model can reproduce every distribution in the null model, so fitting it cannot produce a lower maximized likelihood. Near the null value, the log likelihood can be approximated by a quadratic function of the parameter displacement. The likelihood-ratio statistic is then approximately the squared length of the score component in the directions released by the restrictions.

Specify the null restriction.
Let the larger model have \(p\) free parameters and suppose the null imposes \(q\) independent restrictions. Under regularity conditions, the null parameter lies inside the larger model's parameter space rather than on a problematic boundary.

Compare the two fits.
The restricted fit is one available fit within the larger model, so \(\ell_1\geq \ell_0\). The statistic \(2(\ell_1-\ell_0)\) measures the improvement from releasing the \(q\) restrictions.

Use the null distribution.
The local quadratic approximation makes the statistic approximately a sum of squares of \(q\) independent standard normal quantities. That sum has a chi-square distribution with \(q\) degrees of freedom.

The key step is not merely that both models have parameters. It is that the null model is represented within the alternative model and that the improvement can be attributed to a specified set of restrictions. For two non-nested models, there is no corresponding set of restrictions to release. The difference in their parameter counts does not identify a number of score directions, and there is no universal chi-square calibration for their log-likelihood difference.

Takeaway: A likelihood difference between non-nested models can describe relative fit on the data at hand. It is not automatically a likelihood-ratio test statistic with a chi-square reference distribution. A calibrated inferential conclusion requires a method justified for the particular non-nested comparison and its assumptions.

Worked Examples

Worked Example: Poisson Versus Geometric Claim Counts

A hypothetical insurer observes claim counts \(0,1,1,2\) for four equal-exposure policies. Compare a Poisson model with a geometric model on \(0,1,2,\ldots\), using each model's maximum-likelihood estimate. Calculate twice the difference in maximized log likelihoods, and decide whether an ordinary chi-square likelihood-ratio test is justified.

Method. Fit each model to the same four observed counts, calculate its maximized log likelihood, and then assess whether the models are nested before considering a test calibration.

Poisson fit. For a Poisson model with mean \(\lambda\), the maximum-likelihood estimate is the sample mean. The total count is \(0+1+1+2=4\), so \(\hat{\lambda}=4/4=1\). The log likelihood is

\ell_{\mathrm{Pois}} =\sum_{j=1}^{4}\left(x_j\log\hat{\lambda}-\hat{\lambda}-\log(x_j!)\right) =0+(-1)+(-1)+(-1-\log 2) =-4-\log 2 \approx -4.693147.

Here the zero-count term is \(-1\), each one-count term is \(-1\), and the two-count term is \(-1-\log 2\). The displayed sum is therefore \(-1-1-1-1-\log 2=-4-\log 2\).

Geometric fit. Use the parameterization \(P(X=k)=p(1-p)^k\), for \(k=0,1,\ldots\). Its mean is \((1-p)/p\), so the maximum-likelihood estimate is \(\hat p=n/(n+\sum x_j)=4/(4+4)=0.5\). The total of the four observed counts is 4, giving

\ell_{\mathrm{Geom}} =4\log(0.5)+4\log(0.5) =8\log(0.5) \approx -5.545177.

The first four log terms come from the four factors \(p\); the other four come from the total count multiplying \(\log(1-p)\). Thus \(\ell_{\mathrm{Pois}}-\ell_{\mathrm{Geom}}\approx -4.693147-(-5.545177)=0.852030\), and twice this difference is \(2(0.852030)=1.704060\). The Poisson model has the larger maximized log likelihood for this sample.

Inference check. The Poisson family is not a restriction of this geometric family, nor is the geometric family a restriction of the Poisson family. Both have one fitted parameter, but that does not provide a chi-square test with zero degrees of freedom—or a test with one degree of freedom. The value \(1.704060\) is a descriptive likelihood difference here; the ordinary nested-model chi-square calibration does not follow. Common mistake avoided: assigning degrees of freedom by subtracting parameter counts even though neither model is nested in the other.

Worked Example: Exponential Versus Fixed-Shape Gamma Severity

A hypothetical portfolio has three uncensored positive claim severities, measured in thousands of dollars: \(1,2,3\). Compare an exponential severity model with an exponential rate parameter \(\lambda\) against a gamma model with fixed shape 2 and rate parameter \(\beta\). Use the maximum-likelihood estimates, calculate twice the difference in maximized log likelihoods, and state what the comparison supports.

Exponential fit. The total severity is \(1+2+3=6\), so the exponential maximum-likelihood estimate is \(\hat\lambda=3/6=0.5\) per thousand dollars. The exponential density is \(f(x)=\lambda e^{-\lambda x}\) for \(x>0\). Therefore

\ell_{\mathrm{Exp}} =3\log(0.5)-0.5(1+2+3) =-2.079442-3 \approx -5.079442.

Gamma fit. With shape fixed at 2, the gamma density is \(f(x)=\beta^2 x e^{-\beta x}\) for \(x>0\), since \(\Gamma(2)=1\). The rate estimate is \(\hat\beta=2n/\sum x_j=6/6=1\). The fitted log likelihood is

\ell_{\mathrm{Gamma}(2)} =6\log(1)+\log(1)+\log(2)+\log(3)-1(6) =\log 6-6 \approx -4.208241.

The gamma model's maximized log likelihood is larger by \(-4.208241-(-5.079442)=0.871201\). Twice the difference is \(2(0.871201)=1.742402\). On these data, the fixed-shape gamma model has the larger maximized log likelihood.

Neither of these two fixed-shape families contains the other: changing the rate does not turn a shape-2 gamma distribution into an exponential distribution or vice versa. Accordingly, \(1.742402\) is not automatically a statistic with a chi-square distribution. The calculation supports a fit ranking for this dataset, not a calibrated test of whether one model is significantly better. Common mistake avoided: treating a difference of maximized log likelihoods as inferential evidence without first checking the model relationship.

Worked Example: Reading a Non-Nested Fit Summary

For a hypothetical set of 30 uncensored losses, a fitting procedure reports maximized log likelihoods of \(-82.75\) for a lognormal severity model and \(-83.40\) for a Pareto severity model. Assume both summaries use the same observations and the same likelihood convention. Interpret the difference without claiming a calibrated test.

Step 1: Calculate the observed log-likelihood difference. The lognormal value exceeds the Pareto value by \(-82.75-(-83.40)=0.65\).

Step 2: Calculate the doubled difference. The numerical quantity often used in a nested comparison is \(2(0.65)=1.30\).

Step 3: Check the model relationship and conclusion. A lognormal distribution is not a restriction of the Pareto family, and a Pareto distribution is not a restriction of the lognormal family. The value \(1.30\) therefore has no ordinary nested-model chi-square calibration just because it can be calculated. The defensible conclusion is that the lognormal model has the larger maximized log likelihood in this stated comparison. This does not establish a statistically significant difference, prove either model correct, or show which model will predict future losses better. Common mistake avoided: converting a numerical fit difference directly into a p-value without a justified reference distribution.

Fit Ranking Is Not Calibrated Inference

A fit ranking answers a limited descriptive question: among the candidate fits being compared, which one achieves the larger maximized likelihood for the observations and likelihood definition used? A test asks a different question: under a specified null hypothesis, how unusual is a statistic at least this extreme? Answering the second question requires a justified null distribution, not just a computed number.

Before interpreting a non-nested comparison, check the following:

Are the observations and likelihood definitions comparable? A likelihood based on one set of losses should not be ranked directly against one based on a different set or conditioning scheme. A likelihood for exact observations is not interchangeable with one that accounts for a different observation process.
Are the models truly non-nested? Inspect their distributions and parameter spaces. The fact that model names differ does not prove non-nesting; likewise, counting parameters does not prove that one model contains the other.
What conclusion is supported? A larger maximized likelihood is a fit ranking under the stated comparison. It is not, by itself, a p-value, evidence of statistical significance, or a guarantee of better predictions outside the observed data.
What calibration is appropriate? If inferential comparison is required, use a procedure whose assumptions and reference distribution have been established for the particular models and data structure. Do not borrow the nested chi-square result merely because the same algebraic difference appears.

A common exam trap is to see two likelihoods, form \(2(\ell_A-\ell_B)\), and select degrees of freedom from the difference in parameter counts. The arithmetic may be correct while the inference is not. The first question is structural: can one candidate be obtained by imposing restrictions on the other? If not, the standard nested likelihood-ratio chi-square conclusion is unavailable.

Key takeaway: For non-nested models, keep the numerical fit comparison separate from the inferential question. Report which fitted model has the larger likelihood when that ranking is relevant, but do not attach an ordinary likelihood-ratio chi-square p-value unless the required nested structure and regularity conditions justify it.

Check Your Understanding

For each problem, determine what the likelihood comparison does and does not establish.

A Poisson model and a geometric model are fitted to the same claim-count observations. Their maximized log likelihoods are \(-24.6\) and \(-25.3\), respectively. Calculate twice the difference in favor of the Poisson fit and state whether the ordinary nested chi-square calibration follows. (Answer: \(2[{-24.6}-(-25.3)]=1.4\); the Poisson fit ranks higher, but the ordinary nested-model chi-square calibration does not follow.)
Two non-nested severity models have maximized log likelihoods \(-40.2\) and \(-41.1\), based on the same data and likelihood convention. What is the log-likelihood difference in favor of the first model, and what is the doubled difference? (Answer: \(0.9\) and \(1.8\); this is a descriptive ranking, not automatically a calibrated test.)
Two non-nested models have two and three free parameters, respectively. An analyst proposes a chi-square reference distribution with one degree of freedom because the parameter counts differ by one. Is that justification sufficient? (Answer: No. For non-nested models, the parameter-count difference does not supply valid likelihood-ratio degrees of freedom.)
A model \(H_0\) is obtained from \(H_1\) by setting two independent parameters equal to specified constants. Assume the null value is interior and the usual regularity conditions hold. What is the usual asymptotic reference distribution for twice the log-likelihood improvement under \(H_0\)? (Answer: A chi-square distribution with 2 degrees of freedom, because \(H_0\) imposes two independent restrictions on the nested model.)
Two candidate models have been fitted to different policy subsets, and one has the larger maximized log likelihood. Can that ranking be interpreted as evidence that it fits better on a common basis? (Answer: Not from that information alone; the comparison must use a suitably comparable dataset and likelihood definition before interpreting the ranking.)

What You'll Learn