Comparing Exact and Z-Test Results

Why Compare the Two Methods?

A one-proportion test can be carried out with an exact binomial distribution or with a normal approximation that produces a z statistic. Both methods address the same null hypothesis, but they do not assign exactly the same probabilities to the observed data. When the p-value is far from the significance level, this difference may not change the conclusion. Near the threshold, it can.

The comparison is useful because it shows what the z procedure approximates and what “exact” means here. The binomial model assigns probabilities to whole-number counts, so its p-values reflect the discreteness of the data. The z procedure uses a continuous normal curve to approximate those probabilities. Neither method makes a p-value the probability that \(H_0\) is true.

The Two Methods for the Same Hypothesis

Model and assumptions: Let \(X\) be the number of successes in \(n\) independent Bernoulli trials, each with the same success probability \(p\). Under the null hypothesis \(H_0:p=p_0\), \(X\sim\operatorname{Bin}(n,p_0)\). The exact test uses this binomial model. The one-proportion z test additionally uses a normal approximation to the null distribution; a common check is that both \(np_0\) and \(n(1-p_0)\) are at least 10. The trials must be independent and the success probability must be constant for either binomial procedure to apply.

Write \(\hat{p}=x/n\) for the observed sample proportion. In a one-proportion z test, the standard error is calculated under the null, not by substituting \(\hat{p}\):

SE_0=\sqrt{\frac{p_0(1-p_0)}{n}}, \qquad z=\frac{\hat{p}-p_0}{SE_0}.

For a two-sided alternative \(H_a:p\ne p_0\), the uncorrected z-test p-value is \(2P(Z\ge |z|)\), where \(Z\) has a standard normal distribution. It measures the probability, under the normal approximation and \(H_0\), of a standardized result at least as far from zero as the observed statistic.

For the exact two-sided test in this tutorial, use the probability-ordering definition: add the null probabilities of all possible counts whose probability under \(H_0\) is no greater than the probability of the observed count. In symbols, if \(X\sim\operatorname{Bin}(n,p_0)\) and the observed count is \(x\), the exact p-value is \(P\{P(X=k)\le P(X=x)\}\), summing over possible \(k\). This definition is important because two-sided exact p-values can be defined in more than one way. When \(p_0=0.5\), the binomial distribution is symmetric, so in the examples below this probability-ordering p-value equals the sum of the two inclusive tail probabilities.

Why the Results Can Differ

The exact test works with the actual probabilities of the integer counts \(0,1,\ldots,n\). A count cannot fall between two integers, and the probability in each tail changes in steps as the observed count changes. The z test instead maps the count to a point on a continuous normal curve. That curve can approximate the binomial distribution when the null expected counts are sufficiently large, but the approximation need not match the probability of the discrete tail exactly.

The z statistic follows directly from standardizing the count under \(H_0\). Under the binomial model, the null mean of \(X\) is \(np_0\) and its null variance is \(np_0(1-p_0)\). Since \(\hat{p}=X/n\), dividing the centered sample proportion by its null standard error is equivalent to standardizing the count:

\frac{\hat{p}-p_0}{\sqrt{p_0(1-p_0)/n}} = \frac{X/n-p_0}{\sqrt{p_0(1-p_0)/n}} = \frac{X-np_0}{\sqrt{np_0(1-p_0)}}.

The final expression is the binomial count centered at its null mean and divided by its null standard deviation. Treating that standardized count as approximately standard normal gives the z-test p-value. The equality explains the connection between the methods; the approximation enters when the standardized binomial distribution is treated as continuous and normal.

Worked Example: Different Decisions at a Borderline Threshold

Suppose a hypothetical quality check records 15 successes in 20 independent trials. Test \(H_0:p=0.5\) against \(H_a:p\ne0.5\), using \(\alpha=0.03\). The outcomes are binary, the number of trials is fixed, and each trial is assumed independent with the same success probability. Under the null, the expected counts are \(20(0.5)=10\) successes and \(20(0.5)=10\) failures, so the usual z-test expected-count check is satisfied.

Z test. The observed proportion is \(\hat{p}=15/20=0.75\). The null standard error and statistic are

SE_0=\sqrt{\frac{0.5(0.5)}{20}}=0.1118, \qquad z=\frac{0.75-0.5}{0.1118}=2.236.

Thus the two-sided z-test p-value is \(2P(Z\ge2.236)\approx0.02535\). Since \(0.02535<0.03\), the z test rejects \(H_0\) at the 3% significance level.

Exact test. Under \(H_0\), \(X\sim\operatorname{Bin}(20,0.5)\). By symmetry, counts at least as extreme in probability as 15 are \(X\ge15\) and \(X\le5\). The upper-tail probability is

P(X\ge15) = \frac{\binom{20}{15}+\binom{20}{16}+\binom{20}{17}+\binom{20}{18}+\binom{20}{19}+\binom{20}{20}}{2^{20}} = \frac{21{,}700}{1{,}048{,}576} \approx0.02069.

The two-sided exact p-value is \(2(0.02069)\approx0.04139\). Since \(0.04139>0.03\), the exact test does not reject \(H_0\) at the 3% significance level. The two methods give different decisions at this threshold. Their p-values are both calculated under the null, but the exact calculation sums discrete binomial probabilities while the z calculation uses a continuous approximation.

Worked Example: A Second Borderline Comparison

In a separate hypothetical set of 12 independent trials, suppose 10 are successes. Test \(H_0:p=0.5\) against \(H_a:p\ne0.5\) at \(\alpha=0.03\). Under the null, the expected numbers of successes and failures are both \(12(0.5)=6\). These expected counts are below the common threshold of 10, so the z approximation check is not met. The exact binomial test remains available under the stated independent, constant-probability model.

For the z calculation, \(\hat{p}=10/12=0.8333\), and

SE_0=\sqrt{\frac{0.5(0.5)}{12}}=0.1443, \qquad z=\frac{0.8333-0.5}{0.1443}=2.309.

The two-sided z-test p-value is \(2P(Z\ge2.309)\approx0.02092\). If used, this approximation would lead to rejection at \(\alpha=0.03\), since \(0.02092<0.03\), but its expected-count condition is not satisfied.

Under \(H_0\), \(X\sim\operatorname{Bin}(12,0.5)\). The counts whose null probability is no greater than that of the observed count 10 are \(0,1,2,10,11,12\). The exact p-value is therefore

P(X\in\{0,1,2,10,11,12\}) = \frac{2(1+12+66)}{2^{12}} = \frac{158}{4096} \approx0.03857.

Because \(0.03857>0.03\), the exact test does not reject at the 3% level. Here the count data are especially discrete, and the normal approximation condition is not met; the z p-value consequently gives a different borderline conclusion. The exact result should be used when the binomial model holds and the z approximation check fails.

Worked Example: Different P-Values, Same Decision

Suppose a hypothetical set of 20 independent trials has 14 successes. Test \(H_0:p=0.5\) against \(H_a:p\ne0.5\) at \(\alpha=0.05\). The null expected counts are 10 successes and 10 failures, so the usual expected-count check is met.

The observed proportion is \(14/20=0.70\). The z calculation is

SE_0=\sqrt{\frac{0.5(0.5)}{20}}=0.1118, \qquad z=\frac{0.70-0.50}{0.1118}=1.789.

The two-sided z-test p-value is approximately \(0.0737\), which is greater than \(0.05\). For the exact test, symmetry gives tails at \(X\ge14\) and \(X\le6\). Since \(\sum_{k=14}^{20}\binom{20}{k}=60{,}460\), the exact p-value is

2P(X\ge14) = \frac{2(60{,}460)}{2^{20}} = \frac{120{,}920}{1{,}048{,}576} \approx0.1153.

This p-value is also greater than \(0.05\), so both tests fail to reject \(H_0\) at the 5% level. The numerical p-values differ, but here that difference does not change the decision.

How to Compare Results Responsibly

A p-value is compared with the chosen \(\alpha\) to make a test decision: reject \(H_0\) when the p-value is at most \(\alpha\); otherwise, do not reject \(H_0\). It is not correct to say that a non-rejection proves \(H_0\), or that a p-value is the probability that \(H_0\) is true. Report the method, the p-value, the decision, and the conclusion in the context of the alternative hypothesis.

In practice, compare methods in this order:

Specify the model and hypotheses.
Confirm that the observations can be treated as independent Bernoulli trials with one common success probability, and identify \(p_0\) and the direction of \(H_a\).

Check the z approximation.
For a one-proportion z test, use the null expected counts \(np_0\) and \(n(1-p_0)\). If the usual check is not met, do not rely on the z approximation merely because software reports a z value.

Use a consistent exact definition.
For a two-sided exact result, state which definition is being used. Here it is probability ordering; another definition can produce a different two-sided p-value.

Compare each p-value with the same \(\alpha\).
Explain whether each method rejects or does not reject, and note if their conclusions differ. Do not choose a method only because its p-value supports a preferred conclusion.

The exact test is exact in the sense that it uses the binomial probabilities under its model, not in the sense that its assumptions are automatically true or that its result is always “better” for every purpose. The z test can be a useful approximation when its conditions are met. When a decision is close to \(\alpha\), the difference between a continuous approximation and discrete tail probabilities deserves explicit attention.

Common Mistake

Common mistake: “The z p-value is smaller, so it is the correct result and the exact test must be too conservative.” This does not follow. The tests calculate tail probabilities differently, and a smaller p-value is not evidence that its method is appropriate. Check the model and the z expected counts, specify the exact two-sided definition, and interpret each p-value under its own procedure. A borderline decision should be reported transparently rather than resolved by selecting whichever answer is preferred.

Check Your Understanding

Use the definitions and comparison steps above to answer these questions.

In a one-proportion z test, why is \(p_0\), rather than \(\hat{p}\), used in the standard error?
For \(n=20\) and \(p_0=0.5\), what are the null expected numbers of successes and failures? Does the common expected-count check hold?
Why can an exact binomial p-value differ from an uncorrected z-test p-value even when both tests use the same data and hypotheses?
If an exact p-value is \(0.041\) and a z-test p-value is \(0.026\), what can be concluded at \(\alpha=0.05\), and what should not be concluded about the truth of \(H_0\)?
Why should a two-sided exact p-value definition be stated when comparing results across methods or software?