Diagnosing the Normal Approximation Near Its Limits

Why Check the Approximation?

A one-proportion z test uses a normal distribution to approximate the behavior of a sample proportion under a null hypothesis. That approximation can work well, but it is not automatic: a large sample may still produce very few expected successes if the null proportion is small, and a small sample may have reasonable expected counts when the null proportion is near one half. Before interpreting a z statistic or its p-value, assess whether the null distribution is plausibly close enough to normal for that calculation to be trustworthy.

The key diagnostic is the number of successes and failures expected if the null hypothesis is true. Sample size matters through those expected counts, while a proportion near zero or one can make the distribution strongly unbalanced. These checks are guides, not guarantees: passing them does not prove that every tail probability is accurate.

The Model Behind the One-Proportion z Test

Suppose each of \(n\) observational units has a binary outcome, classified as a success or failure. Let \(p\) be the population success proportion, and let \(X\) count the successes in the sample. Under the null hypothesis \(H_0:p=p_0\), if the outcomes are independent and each has the same success probability \(p_0\), then \(X\) has a binomial distribution. The sample proportion is \(\hat{p}=X/n\).

Model and assumptions: For a one-proportion z test, the outcome is binary, the sample size \(n\) is fixed, and observations are independent with a common success probability. Under \(H_0:p=p_0\), \(X\sim\operatorname{Binomial}(n,p_0)\). A random sample or suitable random assignment supports the intended inference; for sampling without replacement from a finite population, independence is an approximation that is generally reasonable when the sampling fraction is small.

Under this null model, the expected number of successes is \(np_0\), and the expected number of failures is \(n(1-p_0)\). The null standard error of the sample proportion is \(\sqrt{p_0(1-p_0)/n}\). The one-proportion z statistic compares the observed \(\hat{p}\) with \(p_0\), in units of that standard error:

z=\frac{\hat{p}-p_0}{\sqrt{p_0(1-p_0)/n}}

This standardization is useful when the null distribution of \(\hat{p}\) is adequately approximated by a normal distribution. A common conservative screening guideline is that both null expected counts, \(np_0\) and \(n(1-p_0)\), should be at least 10. Some settings use a different cutoff, so follow the convention specified for the analysis. In any case, a cutoff is a practical warning rule rather than a mathematical boundary between “valid” and “invalid.”

Diagnostic: Calculate \(np_0\) and \(n(1-p_0)\) using the null proportion, not \(\hat{p}\). If either expected count is small, the binomial distribution may be too discrete or skewed for a normal approximation to give dependable tail probabilities. An exact binomial test is an alternative to consider.

Why Expected Counts Matter

The binomial count \(X\) is a sum of \(n\) independent zero-or-one outcomes. Under \(H_0\), its mean is \(np_0\) and its variance is \(np_0(1-p_0)\). Since \(\hat{p}=X/n\), dividing the mean and variance of \(X\) by \(n\) and \(n^2\), respectively, gives

E(\hat{p})=p_0,\qquad \operatorname{Var}(\hat{p})=\frac{p_0(1-p_0)}{n}

The z statistic centers \(\hat{p}\) at its null mean and divides by its null standard deviation. The normal approximation is motivated by the central limit theorem: sums of many independent contributions often have an approximately normal distribution. But when \(p_0\) is near zero, most contributions are failures and successes are rare; when \(p_0\) is near one, the reverse is true. The count distribution can then be asymmetric, especially if the expected count in the rarer category is small.

The expected-count check reflects both sample size and balance. Increasing \(n\) raises both expected counts, but if \(p_0\) is extremely small, \(n\) may need to be quite large before \(np_0\) is no longer small. Conversely, \(p_0=0.50\) gives equal expected counts, each \(n/2\). Thus no universal sample-size cutoff can replace checking the null expected counts.

Worked Example: A Balanced Null Model

In a hypothetical quality check, a company tests whether the success rate for a process differs from \(p_0=0.50\). A random sample of \(n=200\) items includes \(x=112\) successes. Assume the sampled items are independent and the outcome for each item is binary. Assess the normal approximation and calculate the z statistic.

Under the null, the expected success and failure counts are

np_0=200(0.50)=100,\qquad n(1-p_0)=200(0.50)=100

Both counts exceed 10, so the expected-count guideline supports using the normal approximation. The observed sample proportion is \(\hat{p}=112/200=0.560\). The null standard error and z statistic are

SE=\sqrt{\frac{0.50(0.50)}{200}} =\sqrt{0.00125} \approx 0.03536

z=\frac{0.560-0.500}{0.03536} \approx 1.697

The conditions support using a one-proportion z test for this hypothetical sample, subject to the stated sampling assumptions. This diagnostic does not itself determine a conclusion about the null hypothesis; the test’s alternative and p-value would also matter.

Worked Example: A Small Sample With a Rare Null Outcome

Suppose a hypothetical process has a claimed success proportion of \(p_0=0.10\). In a random sample of \(n=25\) independent trials, \(x=5\) are successes. Find the null expected counts and the z statistic, then assess whether the normal approximation is a good choice.

The null expected counts are

np_0=25(0.10)=2.5,\qquad n(1-p_0)=25(0.90)=22.5

The expected number of successes is well below 10, so the guideline is not met. The sample proportion and null standard error are

\hat{p}=\frac{5}{25}=0.20,\qquad SE=\sqrt{\frac{0.10(0.90)}{25}} =\sqrt{0.0036}=0.060

z=\frac{0.20-0.10}{0.060} =\frac{0.10}{0.060} \approx 1.667

The arithmetic produces a z statistic, but that alone does not make its normal reference distribution reliable. Under the null, only 2.5 successes are expected, so the binomial count is relatively discrete and the normal approximation may give a poor tail probability. An exact binomial test is a sensible alternative to consider; its probability calculation is a separate method.

Large Samples and Unbalanced Null Proportions

A large \(n\) is not enough if the null proportion is very close to zero or one. For example, with \(n=1{,}000\) and \(p_0=0.005\), the expected number of successes is only 5. The failures are plentiful, but the sparse success count can still make a normal approximation questionable. The next example shows that the z statistic can be calculated even when the diagnostic advises caution.

Worked Example: Many Observations but Few Expected Successes

In a hypothetical inspection of \(n=1{,}000\) independent items, suppose the null success proportion is \(p_0=0.005\), and \(x=12\) successes are observed. Check the null expected counts and calculate the z statistic.

The null expected counts are

np_0=1{,}000(0.005)=5,\qquad n(1-p_0)=1{,}000(0.995)=995

Despite the large sample size, the expected success count is below 10. The observed proportion is \(\hat{p}=12/1{,}000=0.012\). The null standard error is

SE=\sqrt{\frac{0.005(0.995)}{1{,}000}} =\sqrt{0.000004975} \approx 0.002230

Therefore,

z=\frac{0.012-0.005}{0.002230} =\frac{0.007}{0.002230} \approx 3.139

The calculation is arithmetically defined, but the expected-count screen flags the normal approximation because successes are sparse under \(H_0\). The value \(x=12\) is the observed count; it does not replace the null expected count of 5 in the diagnostic. An exact binomial test can be used to avoid relying on the normal approximation.

Null Expectations Versus Observed Outcomes

The expected-count check is based on \(p_0\) because a hypothesis test evaluates how unusual the data would be under the null model. Substituting the observed \(\hat{p}\) into the expected-count check answers a different question and can be misleading. For instance, a sample may happen to contain few successes even when the null model predicts many; that unusual observation is evidence to evaluate, not a reason to redefine the null distribution after seeing the data.

Likewise, a highly unbalanced observed sample does not by itself show that the normal approximation fails. The null expected counts can be adequate even when the observed count is far from its null expectation. However, an observation deep in a tail is a setting where small differences between discrete binomial probabilities and a continuous normal approximation may matter. Passing the expected-count guideline is support for the approximation, not a guarantee that every extreme-tail p-value will be accurate.

Check the data and design.
Confirm that the outcome is binary and that the sampling or assignment design supports independent observations with a common success probability.

Use the null value.
Calculate \(np_0\) and \(n(1-p_0)\), where \(p_0\) is specified by \(H_0\).

Assess the approximation.
If both counts meet the applicable guideline, the normal approximation is more defensible. If a count is small, treat the z-test tail probability with caution and consider exact testing.

Keep the conclusion conditional on the method.
Describe whether the z-test approximation is suitable before interpreting its p-value; do not confuse a calculated z statistic with proof that its reference distribution is appropriate.

Why the Diagnostic Matters

A p-value is calculated from a reference distribution. If the reference distribution poorly represents the null behavior of the statistic, the reported p-value may not accurately measure how unusual the data are under \(H_0\). Checking expected counts is therefore part of assessing the method, not an optional arithmetic detail. When counts are sparse or the null proportion is extreme, exact binomial testing provides an alternative based on the binomial model rather than its normal approximation.

The diagnostic does not decide whether \(H_0\) is true, and it does not replace checking the study design. It answers a narrower question: whether using a normal approximation for the one-proportion test is reasonably supported by the null expected counts. A result near a practical cutoff deserves judgment rather than a mechanical label, and an analysis should report the method actually used.

Common Mistake

Wrong reasoning: “The sample has 1,000 observations, so a z test must be reliable.” Sample size by itself does not establish adequate approximation. With \(p_0=0.005\), that sample expects just 5 successes, as the worked example showed.

Correction: Calculate both \(np_0\) and \(n(1-p_0)\) under the null. A large sample can still have sparse expected outcomes when \(p_0\) is highly unbalanced. Conversely, do not automatically reject the approximation merely because the observed sample is unbalanced: use the null expected counts for the standard screen, and take particular care when interpreting extreme-tail results.

Key takeaway: The one-proportion z test relies on a normal approximation to a binomial null distribution. Check the null expected successes and failures, not just the total sample size or the observed counts. When either expected count is small, an exact binomial test is an alternative.

Check Your Understanding

Answer each question using the null model and the approximation diagnostic.

For \(n=80\) and \(p_0=0.50\), what are the null expected success and failure counts? Does each meet the guideline of 10?
For \(n=400\) and \(p_0=0.01\), calculate both null expected counts. Explain why a sample size of 400 does not settle the approximation question.
Why should the expected-count check use \(p_0\), rather than the observed \(\hat{p}\)?
If both null expected counts are at least 10, does that guarantee that every extreme-tail z-test p-value is accurate? Explain.
Name an alternative to consider when a null expected count is small.