Why Deviance Differences Do Not Test Nonnested Models

Introduction

A smaller deviance means a model fits better according to its maximized likelihood, provided the models are fitted to the same response data under the same likelihood. But a difference in deviances is not automatically a test statistic. The familiar chi-square reference distribution for a likelihood-ratio statistic depends on a crucial feature: the models must be nested, with one representing a restricted version of the other.

For nonnested models, the difference can have either sign, and there is generally no degrees-of-freedom rule that turns it into a chi-square statistic. The comparison can still describe which model attained the larger likelihood. What it does not provide, by itself, is a calibrated test of one model against the other.

Deviance Differences and the Likelihood Ratio

Let \(M\) be a model fitted to the observed response data, with maximized log-likelihood \(\hat{\ell}_M\). Suppose the same likelihood admits a saturated model with log-likelihood \(\ell_{\mathrm{sat}}\), the largest log-likelihood available when the fitted response means are unrestricted. The deviance of \(M\) is

The saturated model is a common reference: it is based on the same observations and likelihood, not a separate fitted candidate. For two models \(M_A\) and \(M_B\) with this same reference, subtraction gives

Thus the difference is twice a log-likelihood difference. Its sign says which candidate has the higher maximized likelihood: a positive value favors \(M_B\), while a negative value favors \(M_A\). This algebra does not establish a sampling distribution or a test.

If \(M_0\) is nested in \(M_1\), then \(M_1\) can fit at least as well: \(\hat{\ell}_{M_1}\geq\hat{\ell}_{M_0}\). Consequently,

Under the regularity conditions for Wilks' theorem, if \(M_0\) is true and the models have parameter dimensions \(d_0\) and \(d_1\), this likelihood-ratio statistic converges in distribution to \(\chi^2_{d_1-d_0}\). The usual conditions include a fixed dimension difference, an identifiable smooth model, an interior true parameter, and nonsingular Fisher information. Boundary or other nonregular cases require different calibration.

For nonnested models, there is no containment argument guaranteeing a nonnegative difference, and the difference in parameter dimensions is not the Wilks degrees of freedom. Wilks' theorem does not give a chi-square reference distribution for this comparison.

Worked Comparisons

Why the Nonnested Difference Has No General Chi-Square Law

The failure is not merely that a particular choice of degrees of freedom is difficult to identify. Even in a very simple regular Gaussian example, the deviance difference has a signed distribution that cannot be chi-square.

Proof. Under the true mean \((0,0)\), \(Y_1\) and \(Y_2\) are independent \(N(0,1)\) variables. In model \(M_A\), maximizing over \(\beta\) fits \(Y_1\), leaving residual sum of squares \(Y_2^2\). In model \(M_B\), maximizing over \(\gamma\) fits \(Y_2\), leaving residual sum of squares \(Y_1^2\). The common Gaussian likelihood therefore gives

Swapping \(Y_1\) and \(Y_2\) leaves their joint distribution unchanged and changes the sign of the difference. Also, \(P(Y_2^2=Y_1^2)=0\), since the pair has a continuous joint density and the equality describes a set of area zero. It follows that the difference is negative with probability \(1/2\) and positive with probability \(1/2\). A chi-square variable is nonnegative with probability one, so this difference cannot have a chi-square distribution. \(\square\)

This exact counterexample shows why a nested-model argument cannot simply be reused. In a nested test, the larger model includes the restricted one, so the maximized likelihood cannot decrease, and the null hypothesis corresponds to a restriction within the larger model. For nonnested models, neither feature is guaranteed. A difference can be negative, and there is no parameter-dimension difference that determines its null distribution.

What the Difference Can and Cannot Say

When both models use a common likelihood and saturated reference, their deviance difference is useful as a fit comparison: it is exactly twice the difference in maximized log-likelihoods. A lower deviance identifies the candidate with the better maximized likelihood for the observed data. This descriptive fact is valuable, but it is not a hypothesis test and does not quantify sampling uncertainty.

The common-likelihood qualification matters. If candidate models use different response likelihoods, different data, or different saturated references, their deviances need not subtract to twice a likelihood-ratio statistic at all. Even with a common likelihood, the identity alone does not provide a chi-square distribution. A test requires a null hypothesis and a justified reference distribution.

A common mistake is to say, “Both models are GLMs, so the difference in deviances is chi-square, with degrees of freedom equal to the difference in parameter counts.” The correction is to check nesting first. For a regular nested comparison, the dimension difference gives the Wilks degrees of freedom asymptotically. For two nonnested models, neither the sign nor the null distribution is supplied by that rule. The fitted likelihoods can be compared numerically, but inference needs a method justified for the specific competing hypotheses.