Likelihood-Ratio Tests for Nested Genetic Models

Why Compare Nested Genetic Models?

A genetic analysis often asks whether a simpler model is adequate compared with a more flexible one. For example, do three genotype groups need separate mean trait values, or can they share one mean? Does disease risk vary by genotype, or is a common penetrance sufficient? A likelihood-ratio test answers such questions by comparing how well the two models fit the same observed data after each model’s parameters have been estimated.

The test rewards the more flexible model for its improved fit, but also accounts for how many additional parameters that model uses. Its large-sample reference distribution provides a way to judge whether the improvement is more than would typically arise from sampling variation under the simpler model.

Definition: Two models are nested when every distribution permitted by the null model $H_0$ is also permitted by the alternative model $H_a$. The null model is therefore a restricted version of the alternative. Write $\ell_0$ for the maximized log likelihood under $H_0$, and $\ell_1$ for the maximized log likelihood under $H_a$. The likelihood-ratio statistic is $$ T = -2\log\left(\frac{L_0}{L_1}\right) = 2(\ell_1-\ell_0), $$ where $L_0$ and $L_1$ are the corresponding maximized likelihoods. Since the alternative includes the null model, $L_1 \geq L_0$, so $T \geq 0$.

The statistic measures the improvement in fit on a log-likelihood scale. A large value means the unrestricted model fits substantially better relative to the constraints imposed by $H_0$. It does not, by itself, measure the biological importance of that improvement.

The Chi-Squared Reference Distribution

Under regularity conditions, if $H_0$ is true, the null parameter lies away from the boundary of its allowed values, and the models are identifiable and sufficiently smooth, then $T$ has an approximate chi-squared distribution in large samples. Its degrees of freedom, $df$, equal the number of independent parameter restrictions imposed by $H_0$ relative to $H_a$. Equivalently, for ordinary models with identifiable parameters, $df$ is the number of free parameters in the alternative minus the number in the null.

Theorem: For a regular, nested comparison with $q$ independent restrictions, the likelihood-ratio statistic has the asymptotic null distribution $T \sim \chi^2_q$. The p-value is the upper-tail probability $P(\chi^2_q \geq T_{\mathrm{obs}})$, calculated assuming $H_0$ is true.

“Large-sample” matters: the chi-squared distribution is generally an approximation, not an exact finite-sample distribution. The usual result may fail when parameters are on a boundary, a parameter is not identifiable under the null, or the likelihood has other irregular features. In those settings, the reference distribution needs separate justification; the ordinary degrees-of-freedom rule should not be applied automatically.

Calculating and Interpreting the Test

Both maximized likelihoods must describe the same observations under comparable likelihood specifications. In practice, calculations are often easier using maximized log likelihoods, since $T=2(\ell_1-\ell_0)$ avoids multiplying many small likelihood contributions. The following sequence keeps the comparison explicit.

State the models.
Identify the null restrictions and the larger alternative. Confirm that the models are nested and use the same data.

Fit each model.
Find the maximum likelihood estimates under each set of constraints, then record $\ell_0$ and $\ell_1$.

Form the statistic and degrees of freedom.
Calculate $T=2(\ell_1-\ell_0)$, and count the independent restrictions imposed by the null.

Assess evidence.
Use the upper tail of the appropriate $\chi^2_{df}$ distribution, provided the large-sample conditions are reasonable. Interpret the p-value as evidence against $H_0$, not as the probability that $H_0$ is true.

Worked Example: Comparing Genotype-Specific Trait Means

Suppose, hypothetically, that 30 unrelated individuals have a quantitative trait measured in the same units. Genotypes 0, 1, and 2 are all represented. Compare a model with one common mean against a model with a separate mean for each genotype. Assume independent normally distributed trait values with a common variance, and a full-rank mean design. The fitted residual sums of squares are $RSS_0=106.6667$ under the common-mean model and $RSS_1=60.0000$ under the genotype-specific model.

The alternative adds two free mean parameters: the common mean is replaced by three genotype-specific means. Thus the null imposes $df=2$ restrictions. Under the normal model with an unknown common variance, its maximum-likelihood estimate is $RSS/n$. The maximized likelihood for either model is proportional to $(RSS/n)^{-n/2}$, with the remaining factors equal because the sample size is the same. Therefore,

T = n\log\left(\frac{RSS_0}{RSS_1}\right) = 30\log\left(\frac{106.6667}{60.0000}\right) = 17.2609.

Using the large-sample $\chi^2_2$ reference distribution gives $P(\chi^2_2\geq17.2609)\approx0.0001786$. This small p-value is evidence against a common mean in favor of genotype-specific means under the stated model. It is not evidence, by itself, that genotype causes the trait difference; study design and other sources of bias matter for causal claims.

Examples With Genetic Count Likelihoods

The same method applies to likelihoods built from discrete genetic data. For independent Bernoulli observations, a group with $y$ events out of $n$ has likelihood proportional to $p^y(1-p)^{n-y}$. The maximum likelihood estimate is $\hat{p}=y/n$. With several groups, the null may constrain their probabilities to be equal, while the alternative estimates them separately.

Worked Example: Testing a Transmission Probability

Imagine 30 independent, informative transmissions in a hypothetical sample, with an allele of interest transmitted 18 times. Compare $H_0:p=0.5$, the specified Mendelian transmission probability, with $H_a$, in which the transmission probability $p$ is freely estimated between 0 and 1. Assume the transmissions are independent Bernoulli observations and the null value is an interior point of the alternative parameter space.

Under $H_a$, $\hat{p}=18/30=0.6000$. Under $H_0$, $p=0.5000$. The binomial coefficient is the same in both likelihoods and cancels in their ratio, so

T = 2\left[18\log\left(\frac{0.6000}{0.5000}\right) +12\log\left(\frac{0.4000}{0.5000}\right)\right] = 2[18\log(1.2)+12\log(0.8)] = 1.2081.

The alternative has one free parameter and the null has none, so $df=1$. The upper-tail probability is $P(\chi^2_1\geq1.2081)\approx0.2717$. These data do not provide strong evidence against $p=0.5$ at conventional significance levels. This conclusion does not prove that the probability is exactly 0.5; it describes the evidence in this sample under the model and its assumptions.

Worked Example: Comparing Common and Genotype-Specific Penetrance

Consider hypothetical independent binary disease outcomes in three genotype groups, with 20 individuals per group. The observed numbers with disease are 2, 10, and 18, respectively. Compare a common penetrance under $H_0$ with three genotype-specific penetrances under $H_a$. Assume each outcome is Bernoulli, the individuals are independent, and each group has a probability strictly between 0 and 1.

Under the alternative, the maximum likelihood penetrances are $0.1000$, $0.5000$, and $0.9000$. Under the null, the common estimate is the pooled proportion, $(2+10+18)/60=0.5000$. The null imposes two independent equality restrictions, so the asymptotic reference distribution has $df=2$. The log-likelihood contributions can be compared directly to give

\begin{aligned} T=2\{& 2\log(0.1/0.5)+18\log(0.9/0.5)\\ &+10\log(0.5/0.5)+10\log(0.5/0.5)\\ &+18\log(0.9/0.5)+2\log(0.1/0.5)\}\\ =2\{&2[2\log(0.2)+18\log(1.8)]\} =29.4451. \end{aligned}

The middle group contributes zero because its fitted penetrance is the same under both models. For $df=2$, the upper-tail probability is $P(\chi^2_2\geq29.4451)\approx4.03\times10^{-7}$. Under the assumptions stated, the data provide strong evidence against a common penetrance. The result supports genotype-associated differences in disease probability in this sample; it does not establish a causal effect.

Why the Chi-Squared Approximation Is Reasonable

The asymptotic result can be understood through the local shape of the log likelihood near its maximum. Under regularity conditions, small changes in an estimated parameter produce an approximately quadratic change in log likelihood. Under the null, the $q$ parameter directions that the null restricts behave, after appropriate scaling, like $q$ independent standard normal quantities. The likelihood-ratio statistic is approximately the sum of their squares, which has a $\chi^2_q$ distribution.

This explains both the use of the chi-squared reference and the degrees of freedom: each independent restriction contributes one squared standardized component. If the restrictions are redundant, the nominal number of written constraints overcounts the effective degrees of freedom. If regularity fails, the quadratic approximation or the standard normal limit for the restricted directions may not hold, so the usual chi-squared calibration can be wrong.

Common Mistakes and Practical Checks

Reversing the likelihood ratio. Since the alternative contains the null, its maximized likelihood cannot be smaller. Use $T=2(\ell_1-\ell_0)$, which is nonnegative; reversing the subtraction can produce a negative statistic.
Counting all alternative parameters as degrees of freedom. Count only the independent restrictions separating the null from the alternative. A null with one shared penetrance versus three group-specific penetrances imposes two restrictions, not three.
Treating the chi-squared result as exact for any sample. The standard result is asymptotic and relies on regularity. Sparse genetic categories, boundary estimates, or nonidentifiable parameters can make the approximation unreliable.
Calling a p-value the chance that the null is true. A p-value is a probability of a statistic at least as extreme as the observed one, calculated under the null model. It is not $P(H_0\mid\text{data})$.
Confusing statistical evidence with biological importance. A small p-value does not quantify the size or clinical relevance of a genotype effect. Consider estimated parameters and their units alongside the test.

Key takeaway: For a regular nested comparison, maximize the likelihood under both models, calculate $T=2(\ell_1-\ell_0)$, and use a chi-squared reference distribution with degrees of freedom equal to the number of independent null restrictions. Check nesting, model assumptions, and large-sample conditions before interpreting the p-value.

Check Your Understanding

Try each question before reviewing the worked examples.

In a nested comparison, why must the maximized likelihood under the alternative be at least as large as that under the null?
A null model constrains four genotype-specific means to be equal, while the alternative estimates all four means. How many independent restrictions does the null impose?
What does a likelihood-ratio p-value of 0.03 mean when calculated using a $\chi^2_2$ reference distribution?
Name two situations in which the standard chi-squared approximation may not be reliable.
Why should the likelihoods in a likelihood-ratio test be calculated from the same observations?