Log-Likelihood Deviance And Evidence

Why Express a Likelihood Comparison as Deviance?

A likelihood ratio compares how well two statistical models can explain the same observed data. It is often easier to interpret the comparison after taking logarithms: the resulting deviance is nonnegative for properly nested models with finite, positive maximized likelihoods, and larger values indicate a greater loss of fit when the null-model restriction is imposed. This is a relative measure of fit, not a probability by itself.

The transformation is useful because log-likelihoods turn products into sums, and because under suitable regularity conditions a deviance has an approximate chi-square distribution when the null hypothesis is true. That approximation can support a p-value calculation, but it requires assumptions beyond merely computing the deviance.

Definition: Let $L(\theta)$ be the likelihood for the observed data under a statistical model. For a null model nested within a larger model, let $L_0$ be the maximum likelihood under the null restriction and $L_1$ the unrestricted maximum likelihood. If both maxima are finite and positive, the likelihood ratio and deviance are $$ \Lambda=\frac{L_0}{L_1},\qquad D=-2\ln(\Lambda)=2\{\ell_1-\ell_0\}, $$ where $\ell_0=\ln L_0$ and $\ell_1=\ln L_1$. The unrestricted model contains every parameter choice allowed by the null model, so $L_1\geq L_0>0$, $0<\Lambda\leq1$, and $D\geq0$.

The likelihoods in the ratio must be evaluated for the same observed data under the same sampling framework. Terms that do not depend on the model parameter often cancel in the ratio, but only when they are common to both likelihoods. The definition also requires care with supports: if a maximum likelihood is zero, or if a finite maximum does not exist, the ratio or deviance may be undefined under this definition. Nesting alone does not guarantee positive finite maxima.

How the Transformation Works

Taking logarithms gives a direct connection between relative likelihood and deviance. If the null model achieves a maximum likelihood that is a fraction $\Lambda$ of the unrestricted maximum, then

\ell_0-\ell_1=\ln\left(\frac{L_0}{L_1}\right)=\ln(\Lambda), \qquad D=2(\ell_1-\ell_0)=-2\ln(\Lambda).

Since $\Lambda$ is at most one, its logarithm is at most zero; multiplying by $-2$ produces a nonnegative deviance. If $\Lambda=1$, the restriction does not reduce the maximum likelihood and $D=0$. As $\Lambda$ decreases toward zero, the deviance increases, expressing a larger relative loss in maximized likelihood under the null restriction.

The factor of two is part of the conventional likelihood-ratio test statistic. It also gives the standard asymptotic chi-square result under regularity conditions. It is not a rescaling that turns the ratio into a probability.

Find the two maxima.
Maximize the likelihood subject to the null restriction to obtain $L_0$, and maximize over the larger model to obtain $L_1$.

Form the ratio or log-likelihood difference.
Use $\Lambda=L_0/L_1$, or subtract the maximized log-likelihoods. Keep the order clear: the null value is in the numerator.

Compute deviance.
Use $D=-2\ln(\Lambda)=2(\ell_1-\ell_0)$. The two calculations should agree apart from rounding.

Interpret and calibrate separately.
Describe the relative fit from $\Lambda$ or $D$. Obtain a p-value only after determining the appropriate null distribution of the test statistic.

Worked Examples

Worked Example: A Binomial Null Restriction

A hypothetical quality audit examines 12 independently selected devices, each classified as either passing or failing. Assume each device has the same pass probability $p$, with independent outcomes. Eight devices pass. Test the fit of the null model $H_0:p=0.5$ against the unrestricted binomial model. Calculate the likelihood ratio and deviance.

Method. Compare the null likelihood with the likelihood maximized over $0\leq p\leq1$. The unrestricted maximum occurs at $\hat p=8/12=2/3$. The binomial coefficient is the same in both likelihoods and cancels from their ratio.

Under the null, the likelihood is $\binom{12}{8}(0.5)^{12}$. Under the unrestricted model it is $\binom{12}{8}(2/3)^8(1/3)^4$. Thus

\Lambda =\frac{(0.5)^{12}}{(2/3)^8(1/3)^4} \approx 0.50682.

Equivalently, calculate the log ratio:

\ln(\Lambda) =12\ln(0.5)-8\ln(2/3)-4\ln(1/3) =-8.317766-(-7.638169) =-0.679597.

Therefore,

D=-2(-0.679597)=1.359194\approx1.3592.

The null-model maximum is about $50.68\%$ of the unrestricted maximum, so the null restriction reduces the maximum likelihood by about $49.32\%$. As a check, using the rounded ratio $0.5068$ gives $-2\ln(0.5068)\approx1.3593$; the small difference from $1.3592$ is due to rounding the ratio before taking its logarithm. The unrounded calculations agree.

Common mistake avoided: The ratio is $L_0/L_1$, not $L_1/L_0$. Reversing it would produce a value greater than one and a negative value after multiplying its logarithm by $-2$, inconsistent with the nested-model comparison.

Worked Example: A Poisson Rate Comparison

In a hypothetical monitoring study, 18 events occur during 10 units of exposure. Assume the event count follows a Poisson model with constant rate $\lambda$ per exposure unit. Compare $H_0:\lambda=1.5$ with the unrestricted model, whose maximum-likelihood estimate is $18/10=1.8$. Find the deviance and likelihood ratio.

For total count $y=18$ over exposure $t=10$, the log-likelihood, apart from no omitted parameter-dependent terms, is

\ell(\lambda)=18\ln(\lambda)-10\lambda-\ln(18!).

The factorial term is common to both models and cancels. The log-likelihood difference is

\begin{aligned} \ell_1-\ell_0 &=18\ln(1.8/1.5)-10(1.8-1.5)\\ &=18\ln(1.2)-3\\ &=18(0.1823216)-3\\ &=3.281789-3\\ &=0.281789. \end{aligned}

Thus the deviance and likelihood ratio are

D=2(0.281789)=0.563578\approx0.5636, \qquad \Lambda=\exp(-0.281789)\approx0.7544.

The null maximum is approximately $75.44\%$ of the unrestricted maximum. The check $-2\ln(0.7544)\approx0.5637$ differs in the last displayed digit because the ratio has been rounded; the unrounded values give $D\approx0.5636$.

Common mistake avoided: The observed count is 18 over 10 exposure units, so the unrestricted rate is $1.8$, not 18. Using the count as the rate would ignore the exposure and compare the wrong models.

Worked Example: A Normal Mean Restriction

A hypothetical instrument records five independent readings $1,2,3,4,5$. Assume the readings have a normal distribution with unknown mean $\mu$ and known variance $\sigma^2=4$. Compare $H_0:\mu=2$ with the unrestricted model. Compute the deviance and interpret the likelihood ratio.

The unrestricted maximum-likelihood estimate is the sample mean, $\bar x=(1+2+3+4+5)/5=3$. For known variance, the log-likelihood difference can be found from the two sums of squared deviations. Under the null mean 2, that sum is $1+0+1+4+9=15$. At the unrestricted estimate 3, it is $4+1+0+1+4=10$.

The normal log-likelihood has a squared-deviation term divided by $2\sigma^2$. The terms involving the variance and normalizing constant cancel, giving

\ell_1-\ell_0 =\frac{15-10}{2(4)} =\frac{5}{8} =0.625.

Therefore,

D=2(0.625)=1.25, \qquad \Lambda=\exp(-0.625)\approx0.5353.

The null model achieves about $53.53\%$ of the unrestricted maximum likelihood. As a check, $-2\ln(0.5353)\approx1.250$ to the shown precision.

Common mistake avoided: The log-likelihood contains the squared-deviation difference divided by $2\sigma^2$, while the deviance is twice the log-likelihood difference. Omitting either the known variance or the factor of two gives the wrong statistic.

Deviance Is Not a Probability or a P-Value

The likelihood ratio is a ratio of two maximized likelihood values for the observed data. Although it lies between zero and one under the stated conditions, it is not the probability that the null hypothesis is true, nor the probability of observing the data under the null. A likelihood evaluated at observed data is not generally a probability assigned to that exact data value, especially for continuous observations.

The deviance is a transformation of that ratio. It is a nonnegative statistic measuring how much better the unrestricted model fits in relative maximum-likelihood terms. A p-value answers a different question: under the null model, how probable is a test statistic at least as extreme as the observed statistic, according to the specified test procedure?

Theorem: For a regular nested likelihood-ratio test with a null parameter in the interior of the parameter space, identifiable parameters, suitable smoothness, and the usual large-sample conditions, the null distribution of $D$ is approximately chi-square with degrees of freedom equal to the difference in the number of free parameters between the unrestricted and null models. This is an asymptotic approximation, not a consequence of the definition of deviance alone. Boundary restrictions, nonidentifiability, small samples, and other failures of regularity can invalidate the usual approximation.

When the approximation is justified, the p-value is the upper-tail probability of the corresponding chi-square distribution at the observed deviance. For example, a calculation of $D$ alone does not determine a p-value: one must also identify the degrees of freedom and justify the reference distribution. In some models, an exact null distribution or another calibration method is needed instead.

Also distinguish the likelihood ratio from a Bayesian Bayes factor. Both can compare models, but a likelihood ratio uses maximized likelihoods; a Bayes factor compares likelihoods integrated over parameter distributions. They are not interchangeable.

Common Exam Traps

Reversing the ratio. For the convention used here, $\Lambda=L_0/L_1$. A ratio above one signals that the numerator, model definitions, or maximization has been mishandled when the models are nested and the maxima are finite and positive.
Calling deviance a p-value. Deviance is the observed test statistic. A p-value requires a null distribution and a tail calculation.
Assuming nonnegative deviance without checking the setup. The result $D\geq0$ follows when the unrestricted model contains the null model and both maximized likelihoods are finite and positive. If a maximum is zero or does not exist, the ratio may not be defined as stated.
Using parameter count without checking regularity. A difference in free-parameter counts gives the usual chi-square degrees of freedom only under the regularity conditions for the asymptotic result.
Rounding too early. Keep log-likelihood differences or the ratio at adequate precision until the final result. A rounded likelihood ratio can make a recomputed deviance differ in its last reported digit.

Key takeaway: For finite, positive maximized likelihoods in nested models, $\Lambda=L_0/L_1$ measures the null maximum relative to the unrestricted maximum, while $D=-2\ln(\Lambda)$ expresses the same comparison on a deviance scale. Neither is itself a probability or p-value; calibration is a separate step.

Check Your Understanding

For each problem, identify the model comparison, calculate the requested likelihood ratio or deviance, and keep the distinction between fit comparison and statistical calibration in view.

A hypothetical batch has 6 independently tested components, of which 4 pass. Under a binomial model, compare $H_0:p=0.5$ with unrestricted $p$. Find $D$ and $\Lambda$. (Answer: $D\approx0.6796,\ \Lambda\approx0.7119$)
A hypothetical Poisson process records 12 events over 4 exposure units. Compare $H_0:\lambda=2$ with unrestricted $\hat\lambda=3$. Find $D$ and $\Lambda$. (Answer: $D\approx1.7312,\ \Lambda\approx0.4206$)
Four independent normal measurements have known variance $\sigma^2=9$ and sample mean $\bar x=5$. Compare $H_0:\mu=3$ with unrestricted $\mu$. Find the deviance and likelihood ratio. (Answer: $D=16/9\approx1.7778,\ \Lambda=\exp(-8/9)\approx0.4111$)
A hypothetical multinomial sample has 20 observations in two categories, with counts 12 and 8. Compare the null probabilities $(0.5,0.5)$ with unrestricted category probabilities. Find the deviance. (Answer: $D=2\{12\ln(1.2)+8\ln(0.8)\}\approx0.8054$)
A regular nested model comparison has a calculated deviance of 4.2 and two additional free parameters in the unrestricted model. Using the asymptotic chi-square reference with 2 degrees of freedom, find the approximate upper-tail p-value. (Answer: $P(\chi^2_2\geq4.2)=\exp(-2.1)\approx0.1225$)

What You'll Learn