Exact Likelihood Ratio Tests For Bernoulli Data

Why Use a Likelihood-Ratio Test for Bernoulli Data?

Suppose a process produces a sequence of binary outcomes, such as whether a hypothetical device passes inspection or whether a hypothetical policyholder files a claim during a specified period. If each outcome is modeled as a Bernoulli trial, the success count summarizes the data relevant to the success probability. A likelihood-ratio test asks whether the observed count is substantially less plausible under a specified probability than under the best-fitting probability.

For a finite sample, the test statistic can be calculated directly from the count. Its exact null distribution can also be found from the binomial probabilities, without relying on a large-sample approximation. This is especially useful when a sample is small or the null probability is near zero or one.

Model and hypotheses: Let $X_1,\ldots,X_n$ be independent Bernoulli random variables with the same success probability $p$, where $0<p<1$. The null hypothesis specifies $p=p_0$, with $0<p_0<1$; the unrestricted alternative allows any $p\in[0,1]$. Write $K=\sum_{i=1}^n X_i$ for the number of successes. Then $K\sim\operatorname{Binomial}(n,p)$, and the likelihood based on the observations is $L(p)=p^K(1-p)^{n-K}$.

The Bernoulli Likelihood-Ratio Statistic

The likelihood-ratio statistic compares the likelihood at the null value with the largest likelihood over all allowable values of $p$. We use the convention that the likelihood ratio has the null likelihood in its numerator; the statistic $T$ is minus twice its logarithm. A small likelihood ratio, or equivalently a large $T$, indicates that the null value fits poorly compared with the unrestricted best fit.

Definition: For observed count $K=k$, let $\hat p=k/n$ be the unrestricted maximum likelihood estimate. The likelihood ratio and its deviance form are $$ \Lambda=\frac{L(p_0)}{L(\hat p)} =\frac{p_0^k(1-p_0)^{n-k}}{\hat p^{\,k}(1-\hat p)^{n-k}}, \qquad T=-2\log\Lambda =2\left[k\log\left(\frac{\hat p}{p_0}\right) +(n-k)\log\left(\frac{1-\hat p}{1-p_0}\right)\right]. $$ Terms of the form $0\log 0$ in the expanded expression are interpreted by continuity as zero.

The binomial coefficient $\binom{n}{k}$ is absent because the likelihood here is for the observed sequence of Bernoulli outcomes. If instead the likelihood is written for the count $K$, that coefficient appears in both numerator and denominator and cancels. Either way, the likelihood ratio is the same.

Why the Maximum Likelihood Estimate Is the Sample Proportion

For $0<k<n$, take the logarithm of the likelihood: $\ell(p)=k\log p+(n-k)\log(1-p)$. Differentiating gives $\ell'(p)=k/p-(n-k)/(1-p)$. Setting this equal to zero yields $k(1-p)=(n-k)p$, hence $p=k/n$. Also, $\ell''(p)=-k/p^2-(n-k)/(1-p)^2<0$, so this stationary point is a maximum. If $k=0$, the likelihood $(1-p)^n$ is maximized at $p=0$; if $k=n$, the likelihood $p^n$ is maximized at $p=1$. Thus $\hat p=k/n$ holds for all counts, including the endpoints.

Substituting $\hat p$ into the likelihood comparison gives the stated formula for $T$. The statistic is nonnegative: the unrestricted model includes the null value, so $L(\hat p)\geq L(p_0)$, which implies $0<\Lambda\leq1$ and $T\geq0$. When the count equals its null expectation $np_0$, the estimate equals $p_0$ and $T=0$.

Worked Example: Nine Trials and a Specified Success Probability

A hypothetical quality check records $n=9$ independent pass/fail outcomes, with $k=6$ passes. Test the specified value $H_0:p=1/3$ against an unrestricted success probability by calculating the MLE, likelihood ratio, and statistic. This example uses the likelihood-ratio calculation and avoids the common error of dropping a success or failure contribution.

Step 1: Find the unrestricted estimate. The observed pass proportion is $\hat p=k/n=6/9=2/3$. The failure proportion is $1-\hat p=1/3$.

Step 2: Form the likelihood ratio. Under the null, the likelihood is $(1/3)^6(2/3)^3$. At the unrestricted estimate, it is $(2/3)^6(1/3)^3$. Therefore,

\Lambda =\frac{(1/3)^6(2/3)^3}{(2/3)^6(1/3)^3} =\frac{1}{8}.

Step 3: Calculate the statistic. Using the count formula,

\begin{aligned} T &=2\left[6\log\left(\frac{2/3}{1/3}\right) +3\log\left(\frac{1/3}{2/3}\right)\right]\\ &=2[6\log 2+3\log(1/2)]\\ &=2[3\log 2] =6\log 2 \approx 4.1589. \end{aligned}

The result checks against the likelihood ratio: $-2\log(1/8)=6\log 2\approx4.1589$. In particular, the three observed failures contribute to the statistic; omitting them would give the wrong result.

Exact Calibration Under the Null

Under $H_0:p=p_0$, the count has the known finite-sample distribution $K\sim\operatorname{Binomial}(n,p_0)$. For an observed count $k_{\mathrm{obs}}$, calculate $T(k)$ for each possible count $k=0,1,\ldots,n$. The exact likelihood-ratio p-value is the null probability of all counts whose statistic is at least as large as the observed statistic. Including equality means that all outcomes tied with the observed statistic are counted.

Exact likelihood-ratio p-value: For observed statistic $t_{\mathrm{obs}}=T(k_{\mathrm{obs}})$, define $$ p_{\mathrm{exact}} =P_{p_0}\{T(K)\geq t_{\mathrm{obs}}\} =\sum_{\{k:T(k)\geq t_{\mathrm{obs}}\}} \binom{n}{k}p_0^k(1-p_0)^{n-k}. $$ A nonrandomized test at level $\alpha$ rejects when $p_{\mathrm{exact}}\leq\alpha$. Its null rejection probability is at most $\alpha$; because $K$ is discrete, it need not equal $\alpha$.

For a specified interior probability $p_0$, $T(k)$ decreases toward its minimum near $np_0$ and increases as the count moves farther away. But the exact set of counts in the tail is determined by comparing the actual statistic values, not by assuming equal distances from $np_0$. When $p_0$ is not one half, counts on opposite sides can have different statistics even if they are equally far from the null expectation.

Worked Example: A Symmetric Exact Tail for Eight Trials

A hypothetical monitor records $n=8$ independent binary outcomes and observes $k=6$ successes. Under $H_0:p=1/2$, calculate the likelihood-ratio statistic and its exact p-value. The method is to compare statistics for all possible counts, then sum their binomial probabilities; this avoids treating a continuous approximation as an exact calculation.

The unrestricted estimate is $\hat p=6/8=3/4$. Thus

\begin{aligned} T(6) &=2\left[6\log\left(\frac{3/4}{1/2}\right) +2\log\left(\frac{1/4}{1/2}\right)\right]\\ &=2[6\log(3/2)+2\log(1/2)]\\ &\approx 2[2.432790-1.386294] \approx 2.0930. \end{aligned}

Because $p_0=1/2$, the statistic is symmetric: $T(k)=T(8-k)$, and it increases as the count moves away from 4. Consequently $T(k)\geq T(6)$ exactly for $k\leq2$ or $k\geq6$. Under the null, each count has probability $\binom{8}{k}/2^8$. The combined probability of these tails is

p_{\mathrm{exact}} =\frac{\binom80+\binom81+\binom82+\binom86+\binom87+\binom88}{256} =\frac{1+8+28+28+8+1}{256} =\frac{74}{256} \approx 0.2891.

The exact tail includes both observed and tied counts. The calculation checks by symmetry: the lower tail has 37 equally likely outcomes and the upper tail has 37, for $74$ of the $256$ possible sequences.

Worked Example: Unequal Tails When the Null Probability Is One Quarter

A hypothetical sensor is tested in $n=8$ independent trials. It records $k=5$ successes, and the specified null probability is $p_0=1/4$. Calculate $T$, identify every count in the exact likelihood-ratio tail, and decide whether to reject at level $0.05$. This example uses direct count comparisons and avoids assuming that equal distances from $np_0$ produce equal tail statistics.

The unrestricted estimate is $\hat p=5/8$. The statistic is

\begin{aligned} T(5) &=2\left[5\log\left(\frac{5/8}{1/4}\right) +3\log\left(\frac{3/8}{3/4}\right)\right]\\ &=2[5\log(5/2)+3\log(1/2)]\\ &\approx 2[4.581454-2.079442] \approx 5.0040. \end{aligned}

Checking the possible counts on the low side, the endpoint $k=0$ gives $T(0)=2[8\log(1/(3/4))]=16\log(4/3)\approx4.6029$, which is less than the observed statistic. Counts $k=1,2,3$ are closer to $np_0=2$ and have smaller statistics. On the high side, the statistic increases beyond $k=5$. Thus the exact tail consists of $k=5,6,7,8$ only.

Under the null, $K\sim\operatorname{Binomial}(8,1/4)$, so

\begin{aligned} p_{\mathrm{exact}} &=P_{1/4}(K\geq5)\\ &=\frac{\binom85 3^3+\binom86 3^2+\binom87 3+\binom88}{4^8}\\ &=\frac{56(27)+28(9)+8(3)+1}{65536}\\ &=\frac{1789}{65536} \approx 0.0273. \end{aligned}

Since $0.0273<0.05$, reject $H_0$ at the stated level. The lower endpoint $k=0$ is not included, even though it is on the opposite side of the null expectation: its statistic is below the observed threshold.

Common Mistakes and Exam Traps

Keep the likelihood ratio, its logarithm, and the deviance statistic distinct. Here $\Lambda=L(p_0)/L(\hat p)$ is at most one, $\log\Lambda$ is at most zero, and $T=-2\log\Lambda$ is nonnegative. Reversing the likelihood ratio without changing the sign produces a different convention and can lead to an incorrect test statistic.

Leaving out failures. The likelihood includes both $k$ successes and $n-k$ failures. In the first worked example, the failure term $3\log(1/2)$ changes $12\log 2$ to $6\log 2$.
Confusing $T$ with $\Lambda$. The likelihood ratio is not the statistic $T$; the relationship is $T=-2\log\Lambda$.
Using the wrong tail rule. The exact p-value sums null probabilities for which $T(k)\geq T(k_{\mathrm{obs}})$, not just probabilities for counts greater than the observed count.
Assuming exact size equals the nominal level. A nonrandomized test has only finitely many possible count outcomes, so its rejection probability may be less than the chosen level.

The exact procedure uses the binomial distribution implied by the model. A large-sample chi-square approximation may be useful in other settings, but it is not the calculation made by an exact test. The key practical steps are to calculate $T$ from the observed count, compare $T$ across possible counts, and sum the corresponding binomial probabilities.

Check Your Understanding

For each problem, use the Bernoulli likelihood-ratio statistic and, where requested, its exact binomial tail. Probabilities are rounded to four decimals.

A hypothetical inspection process has $n=6$, $k=4$, and $p_0=1/2$. Find $\hat p$ and $T$. (Answer: $\hat p=2/3$, $T=2[4\log(4/3)+2\log(2/3)]\approx0.6796$.)
A hypothetical binary outcome is observed zero times in $n=5$ independent trials. For $p_0=0.2$, calculate $T$. (Answer: $T=2[5\log(1/0.8)]=10\log(1.25)\approx2.2314$.)
For $n=4$, $p_0=1/2$, and observed $k=0$, determine the exact likelihood-ratio p-value. (Answer: $P(K=0\text{ or }4)=2/16=0.1250$.)
For $n=6$, $p_0=1/4$, and observed $k=4$, identify the exact likelihood-ratio tail and calculate its probability. (Answer: the tail is $K\geq4$, with probability $[15(9)+6(3)+1]/4^6=154/4096\approx0.0376$.)
A hypothetical trial has $n=5$, $k=3$, and $p_0=1/2$. Find $\hat p$, calculate $T$, and state the exact p-value. (Answer: $\hat p=3/5$, $T=2[3\log(6/5)+2\log(4/5)]\approx0.2014$, and $p_{\mathrm{exact}}=1.0000$, since every possible count has statistic at least this small observed value.)

What You'll Learn