Exact And Asymptotic Calibration In Binomial Models

Why Calibration Is Discrete in a Binomial Model

A likelihood-ratio statistic measures how much better the data fit an unrestricted model than a specified null model. In a binomial model, the statistic can take only finitely many values. Consequently, a nonrandomized test generally cannot attain every desired significance level exactly. Its achieved size is the actual probability of rejection under the null, and may be below the nominal level.

This tutorial considers independent Bernoulli observations with a common success probability. The null specifies that probability, while the alternative allows it to vary freely. We use the discrete null distribution to select likelihood-ratio critical regions, and compare them with the large-sample chi-square calibration.

Definition: Let $X_1,\ldots,X_n$ be independent Bernoulli random variables with common success probability $p$, and let $K=\sum_{i=1}^n X_i$. Then $K\sim\operatorname{Bin}(n,p)$. For testing $H_0:p=p_0$ against the two-sided alternative $H_1:p\ne p_0$, the likelihood-ratio statistic is $$ D(K)=-2\log\lambda(K) =2\left[K\log\left(\frac{K}{np_0}\right)+(n-K)\log\left(\frac{n-K}{n(1-p_0)}\right)\right]. $$ At $K=0$ or $K=n$, a term of the form $0\log 0$ is defined to be zero. Under $H_0$, the exact size of a rejection region $R$ is $P_{p_0}(K\in R)$, calculated using $K\sim\operatorname{Bin}(n,p_0)$.

Where the Binomial Deviance Comes From

For an observed count $K=k$, the binomial likelihood is proportional to $p^k(1-p)^{n-k}$; the binomial coefficient is the same for every value of $p$ and cancels from the likelihood ratio. The unrestricted maximum-likelihood estimate is $\hat p=k/n$. When $0<k<n$, the likelihood ratio comparing the null with the unrestricted estimate is

\lambda(k)=\frac{p_0^k(1-p_0)^{n-k}}{(k/n)^k(1-k/n)^{n-k}}.

Taking $-2$ times the logarithm gives the deviance in the definition. At the endpoints, the same expression follows by using the endpoint maximum-likelihood estimates $\hat p=0$ or $\hat p=1$, with the zero-term convention. The statistic is nonnegative: the unrestricted model includes the null model, so its maximized likelihood cannot be smaller.

The test rejects for large values of $D(K)$. Under standard large-sample conditions, including a null probability strictly between zero and one, Wilks' theorem gives an approximate chi-square distribution with one degree of freedom. This approximation does not change the fact that the exact null distribution is discrete.

Choosing a Nonrandomized Critical Region

To calibrate exactly at a target level $\alpha$, first calculate $D(k)$ for each possible count $k=0,\ldots,n$, then rank the counts by their statistic values. A likelihood-ratio critical region has the form $R=\{k:D(k)\ge c\}$. Select the cutoff so that $P_{p_0}(D(K)\ge c)\le\alpha$, taking the largest such region under the likelihood-ratio ordering. If including the next statistic value would push the null probability above $\alpha$, stop there. All counts tied at the cutoff are included together.

Compute the statistic values.
Evaluate $D(k)$ for every count in the binomial sample space. Sort distinct values from largest to smallest.

Accumulate null probabilities.
For each statistic value, add the probabilities of all counts at that value, using $P_{p_0}(K=k)=\binom{n}{k}p_0^k(1-p_0)^{n-k}$.

Stop before exceeding the target.
Include a whole tied group if its addition keeps the cumulative probability at or below $\alpha$. The resulting probability is the achieved exact size.

A chi-square calibration instead uses the approximate cutoff $\chi^2_{1,1-\alpha}$, the $1-\alpha$ quantile of a chi-square distribution with one degree of freedom, and rejects when $D(K)$ exceeds that cutoff. Its achieved size in a finite binomial model is still found by summing the exact binomial probabilities of the counts that pass the cutoff; it need not equal $\alpha$.

Worked Example: Exact and Chi-Square Calibration at a Five-Percent Level

A hypothetical quality audit records whether each of $8$ independently selected components passes a stress check. Under the null, the pass probability is $p_0=0.5$. Test $H_0:p=0.5$ against $p\ne0.5$ at nominal level $\alpha=0.05$. Find the nonrandomized exact likelihood-ratio region and the achieved size under the chi-square cutoff.

Method. Compute the statistic for each possible pass count and use the exact $\operatorname{Bin}(8,0.5)$ probabilities to assemble the critical region. Here $np_0=4$, so $$ D(k)=2\left[k\log\left(\frac{k}{4}\right)+(8-k)\log\left(\frac{8-k}{4}\right)\right]. $$ The statistic is symmetric around $k=4$. The values at $k=0,1,2,3,4$, rounded to four decimal places, are $11.0904,\ 5.0620,\ 2.0930,\ 0.5053,\ 0.0000$; the values at $k=5,6,7,8$ repeat those at $k=3,2,1,0$.

Under the null, the two endpoint counts have total probability $$ P(K=0\text{ or }8)=\frac{1+1}{2^8}=\frac{2}{256}=0.0078125. $$ Including the next tied group, $k=1,7$, adds $2\binom{8}{1}/256=16/256=0.0625$, making the cumulative size $18/256=0.0703125$, which exceeds $0.05$. Thus the exact nonrandomized region is $R=\{0,8\}$, with achieved size $0.0078$ to four decimals.

For the chi-square calibration, the five-percent cutoff is approximately $3.8415$. The statistics exceeding this cutoff occur at $k=0,1,7,8$. Their exact null probability is $18/256=0.0703125$, so the chi-square-calibrated test has achieved size $0.0703$, rather than $0.05$.

Check. The exact region’s probability is below the nominal level, and adding the next entire tied group would exceed it. The chi-square region includes precisely that next group. Common mistake avoided: reporting the chi-square cutoff as if it guaranteed a five-percent finite-sample size.

An Asymmetric Null Probability

When $p_0\ne0.5$, the statistic values and their null probabilities need not be symmetric. Also, the rejection region is not necessarily a pair of equally sized tails in the count $K$. It is the likelihood-ratio statistic—not distance from the mean alone—that determines which outcomes enter a likelihood-ratio region.

Worked Example: Ranking Counts Under an Asymmetric Binomial Null

A hypothetical monitoring system flags an event independently on each of $12$ operating days. The null event probability is $p_0=0.25$. Construct a nonrandomized likelihood-ratio test at target level $\alpha=0.10$, and compare its exact size with the chi-square calibration.

Method. Under $H_0$, $K\sim\operatorname{Bin}(12,0.25)$, with mean $np_0=3$. For example, $$ D(6)=2\left[6\log(6/3)+6\log(6/9)\right] =12\log(4/3)=3.4522. $$ Other relevant values, rounded to four decimals, are $D(0)=6.9044$, $D(1)=2.2175$, $D(5)=1.5899$, $D(7)=5.9843$, and $D(8)=9.2058$. The remaining high-count values $D(9),\ldots,D(12)$ are also greater than $D(6)$. No count from $1$ through $5$ has a statistic as large as $D(6)$.

The candidate region $D(K)\ge D(6)$ is therefore $K=0$ or $K\ge6$. Its exact probability is $$ P(K=0)+P(K\ge6)=0.75^{12}+P(K\ge6) =0.0316764+0.0544022=0.0860786. $$ The achieved exact size is $0.0861$ to four decimals. The next statistic group to add is $k=1$, whose probability is $$ P(K=1)=\binom{12}{1}(0.25)(0.75)^{11}=0.1267054. $$ Adding it would raise the size above $0.10$, so the selected region is the largest likelihood-ratio region that does not exceed the target.

The chi-square cutoff at level $0.10$ is approximately $2.7055$. Since $D(1)=2.2175<2.7055$ while $D(6)=3.4522>2.7055$, the chi-square rule selects the same counts, $K=0$ or $K\ge6$. Its finite-sample achieved size is therefore also $0.0861$, not exactly $0.10$.

Check. The result includes the low-count outcome $K=0$ as well as sufficiently high counts. Common mistake avoided: assuming that a two-sided likelihood-ratio region must use matching upper and lower count cutoffs around $np_0$.

Exact Likelihood-Ratio P-Values

The same discrete distribution gives an exact p-value for an observed count $k_{\mathrm{obs}}$. It is the null probability of obtaining a likelihood-ratio statistic at least as large as the observed one: $$ p_{\mathrm{exact}}=P_{p_0}\left(D(K)\ge D(k_{\mathrm{obs}})\right). $$ The inequality includes ties. This is a tail probability in the statistic, not necessarily the probability of counts beyond the observed count in one direction.

Worked Example: An Exact P-Value for an Endpoint Observation

A hypothetical screening program tests $5$ independent samples. Under the null, each sample has probability $p_0=0.4$ of showing a marker. Suppose no samples show the marker, so $k_{\mathrm{obs}}=0$. Calculate the exact likelihood-ratio p-value and compare it with the one-degree-of-freedom chi-square approximation.

Method. Here $np_0=2$. At the observed count, $$ D(0)=2\left[0+5\log\left(\frac{5}{3}\right)\right] =10\log(5/3)=5.1083. $$ For the other endpoint, $D(5)=10\log(1/0.4)=10\log(2.5)=9.1629$, which is also at least as large as the observed statistic. The remaining counts have smaller statistic values: $D(1)=0.9152$, $D(2)=0$, $D(3)=0.8109$, and $D(4)=3.3480$. Therefore only $K=0$ and $K=5$ enter the exact p-value.

Under the null, $K\sim\operatorname{Bin}(5,0.4)$, so $$ p_{\mathrm{exact}}=P(K=0)+P(K=5) =(0.6)^5+(0.4)^5 =0.07776+0.01024=0.08800. $$ The exact p-value is $0.0880$ to four decimals. The chi-square approximation is $P(\chi^2_1\ge5.1083)$, approximately $0.0238$.

Check. The exact tail includes the opposite endpoint because its deviance is larger than the observed deviance. Common mistake avoided: treating the exact two-sided p-value as just $P(K\le0)$, or treating the chi-square tail probability as the exact answer.

Interpretation and Common Traps

Exact calibration is especially useful when the sample size is modest or the null probability is near an endpoint, because the actual statistic distribution may be far from its continuous chi-square approximation. Even when the chi-square rule happens to select the same counts as an exact rule, the achieved size must still be checked from the binomial probabilities. A nominal level describes the target used to set a rule; it does not, by itself, state the rule’s finite-sample rejection probability.

Confusing nominal and achieved size. A chi-square cutoff labeled “five percent” need not give five-percent rejection probability in a discrete sample.
Splitting a tie without permission. A nonrandomized statistic-threshold rule includes all counts with the same statistic value. Selecting only part of a tied group would require a different rule, such as randomization.
Using a count-tail probability instead of a statistic-tail probability. For a two-sided likelihood-ratio test, sum over every count satisfying $D(k)\ge D(k_{\mathrm{obs}})$, even if those counts lie on both sides of the observed count.
Forgetting the model. These formulas assume independent Bernoulli trials with one common success probability and compare a fixed null value with the unrestricted binomial model. A one-sided alternative or a different sampling model requires its own likelihood-ratio statistic and calibration.

Key takeaway: In a binomial likelihood-ratio test, calculate the deviance for every possible count and use the exact null probabilities to determine the achieved size or p-value. A chi-square cutoff is an approximation; its finite-sample size is whatever the discrete binomial calculation gives.

Check Your Understanding

For each problem, use the specified binomial likelihood-ratio ordering and report probabilities to four decimals.

For $K\sim\operatorname{Bin}(4,0.5)$, test $p=0.5$ against a two-sided alternative with target level $0.10$. What is the nonrandomized likelihood-ratio critical region with size at most $0.10$? (Answer: reject for no outcomes; achieved size $0.0000$, since the endpoint group alone has probability $2/16=0.1250$.)
For $K\sim\operatorname{Bin}(6,0.5)$ and target level $0.10$, compare the exact nonrandomized region with the region using the $\chi^2_1$ cutoff $2.7055$. (Answer: exact region $K=0,6$, size $2/64=0.0313$; chi-square region $K=0,1,5,6$, size $14/64=0.2188$.)
For $K\sim\operatorname{Bin}(5,0.4)$, an observed count is $K=4$. Find the exact likelihood-ratio p-value. (Answer: $P(K=0)+P(K=4)+P(K=5)=0.07776+0.07680+0.01024=0.1648$.)
For $K\sim\operatorname{Bin}(10,0.2)$, an observed count is $K=0$. Find the exact likelihood-ratio p-value, accounting for ties. (Answer: $P(K=0)+P(K\ge5)=0.1073742+0.0327935=0.1402$; $D(5)=D(0)$, so $K=5$ is included.)

What You'll Learn