Likelihood-Ratio Screening And Its Limits

What a Likelihood-Ratio Screen Measures

When two generalized linear models (GLMs) use the same response observations and likelihood, a likelihood-ratio comparison asks whether the larger model fits sufficiently better to justify its additional parameters. This is a focused question about a specified pair of models. It is not, by itself, a guarantee that a model chosen after many comparisons will perform well or support ordinary confirmatory inference.

Let $M_R$ be a reduced model and $M_F$ a full model. The models are nested when every distribution allowed by $M_R$ is also allowed by $M_F$. Write their maximized log-likelihoods as $\ell_R$ and $\ell_F$. A likelihood-ratio screening test compares the reduced-model null hypothesis with the full model's additional freedom.

Definition: For nested models fitted to the same data with comparable likelihoods, the likelihood-ratio statistic is $$ G^2 = 2(\ell_F-\ell_R). $$ The null hypothesis states that the reduced model is adequate within the full model; equivalently, the additional parameters satisfy the restrictions defining $M_R$. If those restrictions impose $q$ independent constraints, the nominal degrees of freedom are $q$.

Because the reduced model is contained in the full one, maximizing over the larger parameter space cannot lower the likelihood: $\ell_F \geq \ell_R$, so $G^2 \geq 0$. A larger value indicates greater improvement in fit relative to the number of restrictions being tested. The statistic has a chi-square reference distribution only under suitable conditions; it is not automatically chi-square merely because two models are nested.

Connection to Deviance

For models with a common likelihood and response data, residual deviance is defined relative to the same saturated model. This gives a useful way to calculate a nested comparison.

Result: If $\ell_S$ is the maximized log-likelihood of the same saturated model for both fits, and $D_R=2(\ell_S-\ell_R)$ and $D_F=2(\ell_S-\ell_F)$, then $$ G^2 = D_R-D_F. $$

Derivation. Subtract the two deviances and cancel their shared saturated-model term:

D_R-D_F =2(\ell_S-\ell_R)-2(\ell_S-\ell_F) =2(\ell_F-\ell_R) =G^2.

This identity explains why a decrease in residual deviance is the likelihood-ratio statistic for a nested comparison. It also makes clear why the data, response specification, and likelihood must be held fixed: otherwise the shared reference term, or even the meaning of the likelihood difference, may not be the same.

Worked Example: Adding a Group Effect to a Logistic Model

Suppose two independent groups each contain 20 people. There are 8 events in group A and 14 in group B. Compare a logistic model with one common event probability against a model with a separate probability for each group. For a binomial count $y$ out of $n$, the log-likelihood, apart from the common binomial coefficient, is $y\log(p)+(n-y)\log(1-p)$.

Under the reduced model, the common maximum-likelihood estimate is $\hat p=22/40=0.55$. Its log-likelihood, omitting the common binomial coefficients, is $22\log(0.55)+18\log(0.45)=-27.526$.

Under the full model, the fitted probabilities are $8/20=0.40$ and $14/20=0.70$. Its log-likelihood is $8\log(0.40)+12\log(0.60)+14\log(0.70)+6\log(0.30)=-25.678$. The omitted binomial coefficients are the same in both models and cancel.

Thus $G^2=2[-25.678-(-27.526)]=3.696$. The full model adds one parameter, so the nominal reference is $\chi^2_1$; its upper-tail probability is approximately $0.055$. At a prespecified $0.05$ threshold, this comparison does not reject the common-probability model. The result is close to the threshold, but it does not establish that the group probabilities are equal.

When the Chi-Square Reference Applies

The standard reference result is Wilks' theorem. For a fixed-dimensional, regular parametric model, under the null hypothesis and as the sample size grows, twice the log-likelihood difference between a full model and a correctly specified nested null model converges in distribution to a chi-square variable. In a GLM comparison, the degrees of freedom are the number of independent restrictions, not necessarily the raw number of coefficient labels if some are redundant.

Theorem (Wilks' theorem, nested-model form): Suppose the observations are independent, the true parameter under the null is an interior point of a fixed-dimensional parameter space, the likelihood is sufficiently smooth near that parameter, and the Fisher information is finite and nonsingular. Suppose also that the null restrictions are smooth and impose $q$ independent constraints, and that the maximum-likelihood estimators behave regularly. Then, under the null, $$ G^2 \xrightarrow{d} \chi^2_q. $$

The theorem is an asymptotic statement: its chi-square calibration can be inaccurate in small samples, in sparse data, or when estimates are unstable. It can also fail in nonregular cases, such as a null value on a parameter-space boundary or a non-identifiable parameter. For example, separation can prevent finite maximum-likelihood estimates in logistic regression. The standard likelihood-ratio calibration also requires a genuine likelihood; a quasi-likelihood fit alone does not supply the ordinary likelihood-ratio test.

In practice, check that the models use the same observations, response distribution, and likelihood, that one is genuinely nested in the other, and that the difference in parameter dimension matches the restrictions being tested. These checks do not prove that the asymptotic approximation is excellent, but they identify basic reasons it might be inapplicable.

Worked Example: Comparing Poisson Rate Models

Suppose two groups have independent Poisson counts: group A has 12 events over 6 exposure units, and group B has 30 events over 10 exposure units. Compare a common-rate model with a model having separate rates. The exposure is included as an offset, so the fitted rates are events per exposure unit.

Under the common-rate model, the maximum-likelihood rate is total events divided by total exposure: $\hat\lambda_R=42/16=2.625$. Under the separate-rate model, the fitted rates are $\hat\lambda_A=12/6=2$ and $\hat\lambda_B=30/10=3$.

For a Poisson count $y$ with exposure $e$ and rate $\lambda$, the log-likelihood contribution is $y\log(\lambda)-e\lambda-\log(y!)$. In the difference between the two model log-likelihoods, the factorial terms cancel, and the exposure-rate terms sum to $-42$ in both models. Therefore, $G^2=2[12\log(2)+30\log(3)-42\log(2.625)]$. The bracket is $41.2761-40.5334=0.7427$, giving $G^2=1.485$.

The separate-rate model adds one parameter, so the asymptotic reference is $\chi^2_1$, with upper-tail probability about $0.223$. This comparison does not provide strong evidence against a common rate at conventional thresholds. It does not show that the two rates are exactly equal.

Why Repeated Screening Changes the Interpretation

A single prespecified test at level $\alpha$ has probability $\alpha$ of rejection when its null hypothesis is true, under the test's calibration assumptions. If many candidate terms or comparisons are screened, the chance of at least one false rejection can be much larger. The tests need not be independent for the following upper bound to apply.

Result: For $m$ tests, each with a valid null rejection probability no greater than $\alpha$, the probability of at least one false rejection is at most $m\alpha$. If all $m$ null tests are independent and each has rejection probability exactly $\alpha$, that probability is $1-(1-\alpha)^m$.

Proof. Let $A_j$ be the event that test $j$ falsely rejects its null. The event of at least one false rejection is the union $\bigcup_{j=1}^m A_j$. The union bound gives $P(\bigcup_{j=1}^m A_j)\leq\sum_{j=1}^m P(A_j)\leq m\alpha$. If the events are independent and each has probability $\alpha$, then the probability that none occurs is $P(\bigcap_{j=1}^m A_j^c)=\prod_{j=1}^m(1-\alpha)=(1-\alpha)^m$. Taking the complement gives $1-(1-\alpha)^m$. $\square$

Worked Example: Twenty Candidate Terms

Imagine screening 20 candidate terms, with every null hypothesis true, using a $0.05$ cutoff. If the tests were independent and each correctly calibrated, the probability of at least one false rejection would be $1-(1-0.05)^{20}=1-0.95^{20}\approx 0.642$. The exact independence assumption is often unrealistic for correlated predictors, but the calculation shows why the chance of a false lead can rise quickly.

Without assuming independence, the union bound gives an upper bound of $20(0.05)=1$, which is not informative here. A Bonferroni threshold of $0.05/20=0.0025$ for each test does guarantee that the probability of at least one false rejection is at most $0.05$, provided each individual test is valid. This control comes at a cost: stricter thresholds make it harder to detect real effects.

Searches, Thresholds, and Their Limits

A stepwise search repeatedly uses data to decide which model to test next. Even if each individual likelihood-ratio calculation is correct for its stated pair of models, the final model is not the result of one prespecified test. The sequence of comparisons depends on earlier outcomes, and the selected terms are more likely to have looked unusually favorable in this particular sample. Consequently, a nominal $p$-value from the last comparison should not automatically be read as though that comparison had been specified in advance.

The same issue arises if an analyst tries many subsets, transformations, or stopping rules and reports only the most significant comparison. The effective search includes the choices that were considered and discarded, not just the final pair shown in a table. Dependence among tests affects the size of the inflation, but does not make the general problem disappear.

A threshold such as $0.05$ is a decision convention, not a boundary between a real and a nonexistent effect. A result with $p=0.049$ and one with $p=0.051$ do not represent a meaningful discontinuity in evidence. Choosing a more lenient threshold retains more candidate terms but also permits more false positives; choosing a stricter threshold reduces false positives at the potential cost of missing real effects. The consequences depend on the scientific purpose and the full selection process.

Likelihood-ratio screening is most defensible when the candidate comparison and its threshold are specified before examining the outcome, the nested models are appropriate, and the test's calibration is credible. When many comparisons are scientifically necessary, the multiplicity strategy should be chosen to match the goal, such as controlling the chance of any false rejection or using a prespecified exploratory screen followed by an independent evaluation. A screening result should not be confused with a complete assessment of predictive performance.

Common Mistakes

Comparing unrelated fits. A likelihood-ratio statistic is not justified merely because two models have log-likelihood numbers. They must be nested and based on comparable likelihoods for the same data.
Treating the chi-square approximation as exact. Wilks' theorem is asymptotic and requires regularity. Sparse data, boundary parameters, or unstable estimation can undermine the reference distribution.
Interpreting a nonsignificant result as equality. Failure to reject says the data did not provide sufficient evidence against the reduced model at the chosen threshold; it does not prove the restriction true.
Reading the final stepwise $p$-value as a prespecified test. A search guided by the observed data changes the interpretation of the reported comparison and can exaggerate apparent evidence.
Treating a threshold as a scientific law. The conclusion depends on the chosen cutoff, the number of comparisons, and the consequences of false positives and false negatives.

Key takeaway: A likelihood-ratio statistic quantifies the improvement from a full nested GLM over its reduced counterpart. Its chi-square reference relies on regularity and a specified comparison; repeated, data-guided screening makes the usual single-test threshold an incomplete account of the evidence.

Check Your Understanding

Answer these questions using the definitions and reasoning above.

Two nested models use different sets of observations. Why is their log-likelihood difference not automatically a likelihood-ratio statistic?
A full model adds three independent parameters to a reduced GLM. Under the regularity conditions, what is the asymptotic reference distribution for the likelihood-ratio statistic under the null?
Explain why a $p$-value just above $0.05$ does not establish that the reduced model is true.
For 10 independent tests at level $0.05$, write the probability of at least one false rejection when all null hypotheses are true.
Why can the final likelihood-ratio $p$-value from a data-guided stepwise search be misleading as a single-test measure of evidence?