Valid Comparisons Require a Common Likelihood

Why a Common Likelihood Matters

A likelihood-ratio comparison asks how much better the larger model can fit the observed responses than the smaller model. That question is meaningful only when both fits describe the same observed response data under a compatible likelihood. If one fit uses a different set of observations, or gives the observations different likelihood weights, the two maximized likelihoods need not be values of the same function. Subtracting them does not produce the usual likelihood-ratio statistic.

For nested GLMs, the familiar identity between a difference in residual deviances and a likelihood-ratio statistic relies on this shared basis for comparison. Nesting is necessary, but it is not enough: the response vector, observation-level likelihood contributions, conditioning, and weighting conventions must also agree.

Definition: Two GLM fits have a common likelihood for a comparison when they are likelihoods for the same observed response data, under the same response-distribution and likelihood conventions. In particular, they use the same observations, response values, offsets or exposures, and likelihood weights. The models may have different linear predictors and parameter spaces. Likelihoods written with different but common, parameter-independent factors are also compatible if the same factor applies to both fits.

Write \(\ell_F(\theta_F)\) and \(\ell_R(\theta_R)\) for the log-likelihoods under a full model \(F\) and a reduced model \(R\). When the models have a common likelihood, these are two restrictions of the same log-likelihood function for the observed data. The log-likelihoods may be maximized over different parameter spaces, but the data and the likelihood contributions being evaluated are held fixed.

The Deviance-Difference Identity

Suppose the reduced model is nested in the full model: every distribution permitted by the reduced model is also permitted by the full model. Assume both models use the same likelihood for the same response data, and that their relevant likelihood maxima exist. Let \(\hat\ell_F\) and \(\hat\ell_R\) denote their maximized log-likelihoods. Let \(\ell_{\mathrm{sat}}\) be the maximized log-likelihood in the same saturated model for these data, using the same likelihood convention.

Theorem: With the residual deviances defined by \(D_F=2(\ell_{\mathrm{sat}}-\hat\ell_F)\) and \(D_R=2(\ell_{\mathrm{sat}}-\hat\ell_R)\), the deviance difference is \(D_R-D_F=2(\hat\ell_F-\hat\ell_R)\). It is the likelihood-ratio statistic for the full model against the reduced model.

Proof. By the definitions, both deviances use the same saturated-model log-likelihood:

\begin{aligned} D_R-D_F &=2(\ell_{\mathrm{sat}}-\hat\ell_R) -2(\ell_{\mathrm{sat}}-\hat\ell_F)\\ &=2\hat\ell_F-2\hat\ell_R\\ &=2(\hat\ell_F-\hat\ell_R). \end{aligned}

For a likelihood-ratio test, the ratio of the maximized likelihood under the reduced model to that under the full model is \(\exp(\hat\ell_R-\hat\ell_F)\). Taking minus twice its logarithm gives \(2(\hat\ell_F-\hat\ell_R)\), as stated. \(\square\)

The same-data and nesting conditions also explain why the statistic cannot be negative when both maxima are global. The reduced parameter space is contained in the full parameter space, so maximizing the same function over the larger space cannot produce a smaller maximum.

Result: If \(R\) is nested in \(F\), both models maximize the same log-likelihood over nested parameter spaces, and global maxima exist, then \(\hat\ell_F\geq\hat\ell_R\). Consequently, \(D_R-D_F\geq 0\).

Proof. Let \(\Theta_R\subseteq\Theta_F\) be the two parameter spaces, viewed as restrictions of the same likelihood. For every \(\theta\in\Theta_R\), its likelihood value is available in \(\Theta_F\). Therefore,

\hat\ell_F =\sup_{\theta\in\Theta_F}\ell(\theta) \geq \sup_{\theta\in\Theta_R}\ell(\theta) =\hat\ell_R.

Multiplying the resulting nonnegative difference by two gives the claim. \(\square\)

Worked Comparisons

The examples below separate three issues that are easy to conflate: whether the response observations match, whether the models are nested, and whether the likelihood weights match. A deviance difference has its usual likelihood-ratio interpretation only when all relevant conditions hold.

Worked Example: Nested Bernoulli Models on the Same Six Responses

Suppose six independent Bernoulli responses are observed. Group A has outcomes \(1,1,0\); group B has outcomes \(1,0,0\). Compare a reduced model with one common success probability to a full model with a separate probability for each group. Both models use all six responses and the ordinary Bernoulli likelihood.

The reduced model has three successes and three failures. Its maximum-likelihood estimate is \(\hat p=3/6=1/2\), so its maximized log-likelihood is

\hat\ell_R =3\log(1/2)+3\log(1/2) =6\log(1/2) \approx -4.1589.

In the full model, the estimates are \(\hat p_A=2/3\) and \(\hat p_B=1/3\). Thus

\begin{aligned} \hat\ell_F &=2\log(2/3)+\log(1/3) +\log(1/3)+2\log(2/3)\\ &=4\log(2/3)+2\log(1/3)\\ &\approx -3.8191. \end{aligned}

For the numerical check, \(4\log(2/3)\approx-1.6219\) and \(2\log(1/3)\approx-2.1972\), which sum to \(-3.8191\). The full-model fit improves the maximized log-likelihood, and the likelihood-ratio statistic is

2(\hat\ell_F-\hat\ell_R) =2(-3.8191+4.1589) \approx 0.6796.

This is a valid nested-model comparison: both fits use the same six outcomes and the same Bernoulli likelihood, while the reduced model is obtained by constraining the two group probabilities to be equal.

Worked Example: Dropping an Observation Breaks the Comparison

Keep the data and models from the previous example, but suppose the full model is fitted to all six responses while the reduced model is fitted after dropping the final response in group B. That final response is \(0\). The two fitted models are no longer based on the same response sample.

On the five remaining responses, there are three successes and two failures. The reduced-model estimate is \(\hat p=3/5=0.6\), giving

\hat\ell_{R,5} =3\log(0.6)+2\log(0.4) \approx -3.3651.

The full model, if fitted to those same five responses, would have group A probability \(2/3\) and group B probability \(1/2\). Its log-likelihood would be \(2\log(2/3)+\log(1/3)+2\log(1/2)\approx-3.2958\). These two five-response fits can be compared; their likelihood-ratio statistic is approximately \(2[-3.2958-(-3.3651)]=0.1384\).

But using the full-model log-likelihood from all six observations together with the five-response reduced-model log-likelihood would give

2(\hat\ell_{F,6}-\hat\ell_{R,5}) =2(-3.8191+3.3651) \approx -0.9080.

That negative value is not a paradox and is not a likelihood-ratio statistic: it subtracts log-likelihoods computed for different response samples. The nesting argument does not apply because these are not maximizations of one common function over nested parameter spaces. The correction is to fit both models to the same observations, or to choose a different method designed for the actual missing-data or sample-selection problem.

Worked Example: Changing Poisson Likelihood Weights Changes the Statistic

Consider independent Poisson responses \(Y_1=1\) and \(Y_2=3\). Compare an intercept-only model, which has a common mean, with a model having a separate mean for each response. First use unit likelihood weights, then change the first observation's weight to 2 while leaving the second at 1. A weighted log-likelihood is \(\sum_i w_i\log f_i(y_i)\); changing \(w_i\) changes the likelihood being maximized.

With unit weights, the common-mean estimate is \(\hat\lambda=(1+3)/2=2\). Under the separate-means model, the estimates are 1 and 3. Since the Poisson log-likelihood contribution is \(y\log\lambda-\lambda-\log(y!)\), the likelihood-ratio statistic is

\begin{aligned} 2(\hat\ell_F-\hat\ell_R) &=2\left[ 1\log(1/2)-(1-2) +3\log(3/2)-(3-2) \right]\\ &=2\left[-\log 2+1+3\log(3/2)-1\right]\\ &=2\left[3\log(3/2)-\log 2\right] \approx 1.0465. \end{aligned}

Now assign weights \(w_1=2\) and \(w_2=1\). The common weighted-mean estimate is \((2\cdot1+1\cdot3)/(2+1)=5/3\). Under the separate-means model, the fitted means remain 1 and 3. The weighted log-likelihood difference between the separate-means and common-mean models is

\begin{aligned} \hat\ell_F-\hat\ell_R &=2\left[\log(1/(5/3))-(1-5/3)\right]\\ &\quad+\left[3\log(3/(5/3))-(3-5/3)\right]\\ &=2\left[\log(3/5)+2/3\right] +3\log(9/5)-4/3\\ &\approx 0.7417. \end{aligned}

The weighted likelihood-ratio statistic is therefore \(2(0.7417)\approx1.4834\), not \(1.0465\). Each pair of fits is internally comparable when both models use the same weights, but a deviance difference from one weighting convention cannot be treated as the likelihood-ratio statistic for fits using another. Whether weights represent replication, precision, or another construction must also be interpreted according to the model that defines the likelihood.

What May Differ—and What Must Match

A common likelihood does not mean the two models have identical predictors or parameter values. That would leave nothing to test. The full model may add regression terms, and its parameter space may have higher dimension. The requirements concern the response data and the likelihood specification used to evaluate each candidate model.

A parameter-independent factor can be omitted from a likelihood without changing its maximizer or a likelihood ratio, provided it is the same factor for both models. For example, a combinatorial term in a grouped-binomial likelihood depends on the observed counts but not on the success probability. It may be dropped from both log-likelihoods and still cancels in their difference. It is not safe to omit or alter constants differently across the two fits and then assume they cancel.

In practice, check the following before interpreting a deviance difference:

Response records: Are the same observational units and response values included in both fits?
Response likelihood: Are both fits based on the same family and the same likelihood convention, including any conditioning that defines the response distribution?
Weights and exposure: Are likelihood weights, offsets, and exposures handled consistently? A changed weight changes the contribution of an observation.
Common reference: If using residual deviances, do both use the same saturated-model likelihood for the same data and convention?
Nesting: Is the reduced model genuinely a restriction of the full model, rather than merely a different fitted specification?

These checks matter before any calibration of the statistic. A chi-square approximation, or another reference distribution, cannot repair a subtraction that was not a likelihood-ratio statistic in the first place.

Common Mistake

Wrong reasoning: “The models are nested, so subtracting their reported deviances always gives a likelihood-ratio statistic.” This overlooks that a reported deviance depends on the responses and the likelihood convention used to calculate it.

Correction: First verify that both fits use the same response sample and compatible likelihood contributions, including weights. Then confirm that one model is nested in the other. Only under those conditions does the common saturated-model term cancel and the deviance difference equal \(2(\hat\ell_F-\hat\ell_R)\). A negative difference under these conditions calls for checking the fit, optimization, or stated nesting; a negative number produced by comparing different samples is not evidence against the nesting theorem, because its assumptions were not met.

Key takeaway: A valid nested deviance comparison is a comparison of two maximized fits to the same response data under one common likelihood. Matching sample, response definition, and weights is part of the mathematical basis of the test—not just a bookkeeping preference.

Check Your Understanding

For each question, identify what must be checked before interpreting the comparison.

Two nested binomial GLMs use the same counts but one fit omits a likelihood constant. Under what condition can their likelihood-ratio statistic still be computed from the reported log-likelihoods?
A full model is fitted to 80 patients and a reduced model to 77 after three records with missing covariates are dropped. Why is subtracting their maximized log-likelihoods not the usual nested-model likelihood ratio?
Explain why maximizing a common log-likelihood over a nested larger parameter space cannot produce a smaller maximized value.
In the Poisson example, what changed when the first observation's likelihood weight changed from 1 to 2?
If two residual deviances use different saturated-model conventions, what must be established before their difference can be identified with a likelihood-ratio statistic?