Comparing the Three Tests for a Bernoulli Success Probability

One Sample, Three Tests

For a single Bernoulli success probability, the Wald, score, and likelihood-ratio tests all compare the same hypotheses, $H_0:p=p_0$ against $H_1:p\ne p_0$. Their large-sample null calibration is the same, but the calculations use different features of the likelihood: the Wald test uses the unrestricted estimate and its estimated variance, the score test evaluates the slope of the log-likelihood under the null, and the likelihood-ratio test compares the maximized likelihoods.

This simple model makes the distinctions visible. It also provides a useful check on calculations from restricted and unrestricted fits: the three statistics should generally be close in a sufficiently informative sample, but they need not be identical.

Model and notation: Let $X_1,\ldots,X_n$ be independent and identically distributed Bernoulli random variables with $P(X_i=1)=p$, where the parameter space is $p\in[0,1]$. Let $K=\sum_{i=1}^n X_i$ be the number of successes and $\hat p=K/n$ the unrestricted maximum-likelihood estimate. For the tests below, the specified null value satisfies $p_0\in(0,1)$.

The Likelihood and the Three Statistics

The likelihood, apart from factors that do not depend on $p$, is $L(p)=p^K(1-p)^{n-K}$. When $0<K<n$, differentiating its log gives $K/p-(n-K)/(1-p)=0$, whose solution is $K/n$. This is the maximum because the second derivative is $-K/p^2-(n-K)/(1-p)^2<0$. If $K=0$, the likelihood is $(1-p)^n$, maximized at $p=0$; if $K=n$, it is $p^n$, maximized at $p=1$. Thus the closed parameter space includes the endpoint maximum-likelihood estimates.

Under $H_0$, the variance of $\hat p$ is $p_0(1-p_0)/n$. Under the unrestricted model, its plug-in estimate is $\hat p(1-\hat p)/n$. These variances, the null score, and the likelihood ratio yield the following statistics.

Test statistics: For $0<K<n$, $$ T_W=\frac{(\hat p-p_0)^2}{\hat p(1-\hat p)/n} =\frac{n(\hat p-p_0)^2}{\hat p(1-\hat p)}, \qquad T_S=\frac{n(\hat p-p_0)^2}{p_0(1-p_0)}, $$ $$ T_{LR}=2\left\{K\log\left(\frac{\hat p}{p_0}\right) +(n-K)\log\left(\frac{1-\hat p}{1-p_0}\right)\right\}. $$ The Wald statistic is undefined when $K=0$ or $K=n$, because its estimated variance is zero. The likelihood-ratio formula at those endpoints is interpreted by continuity, with a zero count contributing $0\log 0=0$. Under the usual large-sample calibration, each statistic is compared with a $\chi^2_1$ distribution.

The score expression follows directly from the log-likelihood $\ell(p)=K\log p+(n-K)\log(1-p)$. Its derivative at the null is $U(p_0)=K/p_0-(n-K)/(1-p_0)=(K-np_0)/[p_0(1-p_0)]$. The Fisher information is $I(p_0)=n/[p_0(1-p_0)]$, so $U(p_0)^2/I(p_0)=T_S$. The likelihood-ratio statistic is $2\{\ell(\hat p)-\ell(p_0)\}$, which gives the displayed expression after substituting $K=n\hat p$.

Worked Example: Three Tests for 14 Successes in 50 Trials

Suppose $n=50$, $K=14$, and the null hypothesis is $p_0=0.20$. Then $\hat p=14/50=0.28$, and $1-\hat p=0.72$.

Wald statistic. The estimated variance of $\hat p$ is $0.28(0.72)/50=0.2016/50=0.004032$. Therefore, $$ T_W=\frac{(0.28-0.20)^2}{0.004032} =\frac{0.0064}{0.004032} \approx 1.5873. $$

Score statistic. Under the null, the variance of $\hat p$ is $0.20(0.80)/50=0.16/50=0.0032$. Hence, $$ T_S=\frac{(0.28-0.20)^2}{0.0032} =\frac{0.0064}{0.0032} =2. $$

Likelihood-ratio statistic. The log-likelihood difference is $$ 14\log(0.28/0.20)+36\log(0.72/0.80) =14\log(1.4)+36\log(0.9). $$ Numerically, $\log(1.4)\approx0.3364722$ and $\log(0.9)\approx-0.1053605$, so this difference is approximately $4.7106113-3.7929786=0.9176327$. Thus $T_{LR}=2(0.9176327)\approx1.8353$.

Large-sample p-values. Using the upper tail of $\chi^2_1$, the p-values are approximately $0.208$ for Wald, $0.1573$ for score, and $0.1755$ for likelihood ratio. For example, the likelihood-ratio p-value is $P(\chi^2_1\ge 1.8353)\approx0.1755$. All three are above $0.05$, so none rejects at the five-percent level.

Why Their Large-Sample Results Agree

The statistics have different finite-sample formulas, but under an interior null they share a common leading term. The argument uses the central limit theorem and a local expansion of the log-likelihood. These conditions matter: $p_0$ must be strictly between zero and one, and the observations must follow the stated independent, identically distributed Bernoulli model.

Theorem: If $X_1,\ldots,X_n$ are independent and identically distributed Bernoulli$(p_0)$ variables with fixed $p_0\in(0,1)$, then each of the Wald, score, and likelihood-ratio statistics converges in distribution to $\chi^2_1$ under $H_0$. For the Wald statistic, define any fixed finite value on $K=0$ and $K=n$ to obtain a statistic on the full sample space. This extension does not affect the limit.

Standardize the sample proportion.
The central limit theorem gives $Z_n=\sqrt{n}(\hat p-p_0)/\sqrt{p_0(1-p_0)}\xrightarrow{d}N(0,1)$. Consequently, $T_S=Z_n^2\xrightarrow{d}\chi^2_1$.

Compare the Wald statistic.
The law of large numbers gives $\hat p\xrightarrow{p}p_0$. Because $p_0(1-p_0)>0$, the ratio $p_0(1-p_0)/[\hat p(1-\hat p)]\xrightarrow{p}1$. Thus, whenever the Wald formula is defined, $T_W=T_S\,p_0(1-p_0)/[\hat p(1-\hat p)]$, so Slutsky’s theorem gives the same $\chi^2_1$ limit.

Expand the likelihood ratio.
For $q\in(0,1)$, define $g(q)=q\log(q/p_0)+(1-q)\log((1-q)/(1-p_0))$. Then $T_{LR}=2ng(\hat p)$, $g(p_0)=g'(p_0)=0$, and $g''(p_0)=1/[p_0(1-p_0)]$.

To complete the likelihood-ratio step, choose a closed interval around $p_0$ lying strictly inside $(0,1)$. On this interval $g'''$ is bounded. Taylor’s theorem therefore gives $g(\hat p)=(\hat p-p_0)^2/[2p_0(1-p_0)]+R_n$, where $|R_n|\le C|\hat p-p_0|^3$ for some finite constant $C$, whenever $\hat p$ lies in that interval. The probability that it lies there tends to one, and the central limit theorem implies $\hat p-p_0=O_p(n^{-1/2})$. Hence $nR_n=O_p(n^{-1/2})=o_p(1)$, and $$ T_{LR} =\frac{n(\hat p-p_0)^2}{p_0(1-p_0)}+o_p(1) =T_S+o_p(1). $$ It follows that $T_{LR}\xrightarrow{d}\chi^2_1$. Finally, under $H_0$, $P(K=0)+P(K=n)=(1-p_0)^n+p_0^n\to0$. Assigning any fixed value to the Wald statistic on these endpoint events therefore cannot change its limiting distribution. This proves the theorem. $\square$

What the Finite-Sample Comparison Shows

The first example illustrates a general feature of the tests: Wald uses the estimated variance $\hat p(1-\hat p)/n$, while score uses the null variance $p_0(1-p_0)/n$. The likelihood-ratio test instead measures the log-likelihood loss from imposing the null. Their statistics are not identical, even though their first-order large-sample behavior agrees. In the example, the statistic ordering is $T_W<T_{LR}<T_S$, and the corresponding p-values are close but distinct.

Worked Example: A Second Interior Sample

Let $n=20$, $K=7$, and $p_0=0.25$. Then $\hat p=7/20=0.35$. The Wald statistic is $$ T_W=\frac{20(0.35-0.25)^2}{0.35(0.65)} =\frac{0.2}{0.2275} \approx0.8791. $$ The score statistic is $$ T_S=\frac{20(0.35-0.25)^2}{0.25(0.75)} =\frac{0.2}{0.1875} \approx1.0667. $$ For the likelihood ratio, $$ T_{LR}=2\{7\log(0.35/0.25)+13\log(0.65/0.75)\} =2\{7\log(1.4)+13\log(0.866\overline{6})\} \approx0.990. $$ The values differ, with the score statistic largest in this sample. Each would be compared with $\chi^2_1$ for a large-sample p-value; these modest values do not provide strong evidence against the null.

Worked Example: An Endpoint Sample

Suppose $n=10$, $K=0$, and $p_0=0.5$. The unrestricted maximum-likelihood estimate is $\hat p=0$ because the parameter space includes zero. The Wald statistic is undefined: its estimated variance is $0(1-0)/10=0$, while $\hat p-p_0=-0.5$.

The score statistic remains defined: $$ T_S=\frac{10(0-0.5)^2}{0.5(0.5)} =\frac{2.5}{0.25}=10. $$ For the likelihood ratio, the zero-success term contributes zero by continuity, and $$ T_{LR}=2\left\{0+10\log\left(\frac{1}{1-0.5}\right)\right\} =20\log 2 \approx13.8629. $$ These calculations define the score and likelihood-ratio statistics for the observed endpoint sample, but the usual large-sample $\chi^2_1$ calibration is not a guarantee of accurate finite-sample inference. In particular, the Wald statistic cannot be computed from its plug-in variance here.

Common Pitfall: Treating the Three Statistics as Interchangeable

A tempting but incorrect argument is: “The tests have the same asymptotic distribution, so their statistics and p-values must agree for this sample.” The theorem establishes a shared limiting distribution as $n$ grows under the null; it does not make the finite-sample formulas equal. The example with 14 successes gives three different statistics and three different p-values. Calculate each statistic from its own definition, then apply the same $\chi^2_1$ reference distribution when the large-sample approximation is appropriate.

A second mistake is to calculate a Wald statistic at an endpoint by substituting a zero estimated variance and then report an ordinary finite value. Division by zero does not yield the Wald formula. In a limiting-distribution proof, one may define an arbitrary finite extension on the endpoint events because their probability tends to zero under an interior null. For an actual endpoint dataset, however, report that the plug-in Wald statistic is undefined rather than presenting the extension as a data-based calculation.

Key takeaway: For a Bernoulli probability, Wald uses the unrestricted estimated variance, score uses the null variance, and the likelihood-ratio test uses the log-likelihood difference. Under an interior null, all three have a $\chi^2_1$ limit, but their finite-sample values—and their endpoint behavior—can differ.

Check Your Understanding

Use the definitions and reasoning above to answer these questions.

For a Bernoulli sample with $n=40$, $K=12$, and $p_0=0.25$, calculate $\hat p$ and write the Wald and score statistics before evaluating them.
Why does the score statistic use $p_0(1-p_0)$ where the Wald statistic uses $\hat p(1-\hat p)$?
State the conditions under which the three statistics converge to $\chi^2_1$.
For $K=0$, explain why the unrestricted maximum-likelihood estimate is zero in the closed parameter space and why the plug-in Wald statistic is undefined.
Why does first-order asymptotic equivalence not imply identical finite-sample p-values?