Likelihood-Ratio Tests For Nested Designs

Why Compare Nested Models?

A count-model analysis often asks whether a set of predictors improves the description of a gene’s expression after accounting for library size and other design variables. A likelihood-ratio test answers this by comparing two models fitted to the same observations: a reduced model that imposes the null restriction and a full model that allows the effect being tested. The central calculation is a difference in maximized log likelihoods, not a comparison of raw counts or fitted means alone.

For example, to test whether expression differs by condition while adjusting for batch, the reduced model includes batch but omits condition; the full model includes both. This is a test of the condition term as a group, and it can extend naturally to multi-level factors or several coefficients tested together.

Negative Binomial Models and the Test Statistic

For one gene, let \(Y_i\) be the observed count in sample \(i\), with mean \(\mu_i\) and dispersion \(\alpha\). The negative binomial model used here has \(\operatorname{Var}(Y_i)=\mu_i+\alpha\mu_i^2\), where \(\alpha\geq 0\). A log-link model with a library-size offset is \(\log(\mu_i)=o_i+x_i^\mathsf{T}\beta\), where \(o_i\) is the known offset and \(x_i\) is the design row. The reduced and full models use the same counts and offsets; their design matrices differ in the predictors under test.

This is the log likelihood for independent negative binomial observations under the stated parameterization. When \(\alpha=0\), the limiting model is Poisson; the negative binomial likelihood expression above is for \(\alpha>0\). Write \(\widehat{\ell}_R\) and \(\widehat{\ell}_F\) for the maximized log likelihoods under the reduced and full models, respectively. If dispersion is estimated rather than fixed, each maximization includes that nuisance parameter according to the analysis procedure; the same dispersion treatment must be used in both fits.

Because the reduced model is contained in the full model, maximizing over the larger parameter space cannot produce a smaller log likelihood: \(\widehat{\ell}_F\geq\widehat{\ell}_R\). Thus \(\Lambda\geq0\). Small numerical negatives can occur from optimization tolerances; a substantial negative value signals a fitting, data, or model-comparison problem.

Degrees of Freedom and the Chi-Square Reference

Let \(r_F\) and \(r_R\) be the ranks of the full and reduced design matrices. If the models differ only in regression terms and share the same nuisance parameters, the number of added independent coefficients is \(q=r_F-r_R\). More generally, \(q\) is the difference in the number of independently estimated parameters in the full and reduced models. Under regularity conditions, the null distribution of \(\Lambda\) is approximately chi-square with \(q\) degrees of freedom.

The degrees of freedom count independent parameter directions, not samples, genes, or the total number of columns as written before checking for redundancy. If a categorical condition has three levels and the model uses two independent contrasts, testing the entire condition effect against a model with no condition term has \(q=2\). Testing one particular contrast instead has \(q=1\).

The approximation relies on a correctly specified likelihood, independent observations conditional on the model, identifiable parameters, and a sufficiently regular setting for large-sample likelihood theory. The null parameter must not lie on a problematic boundary, and the models must be fitted comparably. In RNA-seq workflows, dispersion may be estimated or stabilized using information across genes; when a dispersion estimate is plugged in rather than jointly maximized, the usual chi-square calibration is an approximation under that procedure, not an exact finite-sample guarantee.

Worked Comparisons

Why the Reference Distribution Is Appropriate

The likelihood-ratio statistic measures the loss in fit when the parameter restrictions of the reduced model are imposed. To see why a chi-square reference appears, consider a regular model near the null parameter value. A second-order Taylor expansion of the log likelihood around its maximum makes the local log-likelihood loss approximately a quadratic form in parameter differences. Under the null, the \(q\) directions freed in the full model contribute approximately \(q\) independent squared standard-normal components. Their sum has a chi-square distribution with \(q\) degrees of freedom.

This is an asymptotic justification: the approximation improves as the information for the gene’s parameters grows, but a small number of samples or very sparse counts may make it less reliable. It also explains why counting the extra independent coefficients matters. Counting a redundant design column as an extra parameter would use the wrong reference distribution.

Assumptions, Diagnostics, and Alternatives

Before interpreting an LRT, confirm that the models use the same gene-level observations, offsets, response scale, and dispersion convention. Check that the full model is genuinely nested in the reduced one and that the design matrices are identifiable. A condition perfectly confounded with batch, for example, cannot be separated from batch by adding both terms; the intended coefficient directions are not estimable.

Inspect fit warnings, convergence status, extreme fitted means, and whether the fitted model is compatible with the count structure. A model that fails to converge does not provide trustworthy maximized likelihoods. A large apparent improvement driven by one unusual sample can also be fragile; the LRT itself does not diagnose that observation. Such checks complement the test rather than changing its definition. Very low counts may provide little information, and sparse data can weaken the large-sample chi-square approximation.

A Wald test, covered in the earlier material on Wald tests for count-model coefficients, evaluates an estimated coefficient or contrast relative to its estimated standard error. It is convenient for a single specified contrast. The LRT instead compares fitted likelihoods and is particularly natural for a joint test of several coefficients, such as all contrasts for a multi-level condition. In large regular samples the methods often give similar conclusions, but they need not agree in small or weakly informative samples. Neither method removes the need to specify the scientific hypothesis and design correctly.

Common Pitfalls and Limitations

• Using the wrong degrees of freedom. The difference in parameter counts is the number of independent added directions, not the number of labels or raw design columns. Check design rank.
• Subtracting rounded likelihoods. Compute \(2(\widehat{\ell}_F-\widehat{\ell}_R)\) from full-precision fit values and round only the final statistic. Early rounding can change a reported statistic and p-value.
• Comparing non-nested fits. The standard nested-model chi-square result does not apply just because two models have different predictors. Verify that the reduced model is obtained by restrictions on the full model.
• Treating a non-significant result as proof of no effect. A large p-value can reflect limited information. It is not the probability that the null is true.
• Forgetting the scope of the test. A joint condition test asks whether the added condition terms collectively improve fit. It does not show which contrast is responsible, nor does it establish causation.

The likelihood-ratio test is a per-gene test statistic and p-value. When many genes are tested, the resulting collection of p-values requires appropriate multiplicity handling before making genome-wide claims; the single-gene chi-square reference does not itself control the overall false-positive rate.