Planning Sample Size for a Proportion Test

Why Plan Sample Size for a Proportion Test?

A study measuring a proportion might ask whether a new reminder changes the fraction of patients who attend appointments, or whether a product defect rate has fallen below a specified level. Before collecting data, researchers need to decide how many independent observations to gather. Too small a sample may often miss a practically important difference; a planned sample size connects the study’s resources to its intended ability to detect that difference.

For this calculation, specify the proportion under the null hypothesis, a particular alternative proportion worth detecting, the significance level \(\alpha\), and the desired power \(1-\beta\). The resulting sample size is a planning approximation, not a promise that a future study will achieve exactly its target power.

The One-Proportion Z-Test Model

Suppose \(n\) independent observations each have a binary outcome, such as “attended” or “did not attend.” Let \(X\) be the number with the outcome of interest, and let \(\hat{p}=X/n\) be the observed sample proportion. Under a population proportion \(p\), \(X\) has a binomial distribution with parameters \(n\) and \(p\). When the normal approximation is reasonable, \(\hat{p}\) is approximately normal with mean \(p\) and standard deviation \(\sqrt{p(1-p)/n}\).

For a test of \(H_0:p=p_0\), the one-proportion z statistic uses the null standard error. Under the normal approximation, it is compared with standard normal critical values. For a one-sided test, the critical value is \(z_{1-\alpha}\); for a two-sided test, the critical values are \(-z_{1-\alpha/2}\) and \(z_{1-\alpha/2}\). Here \(z_q\) is the \(q\) quantile of the standard normal distribution.

The test statistic’s standard error under \(H_0\) is \(\sqrt{p_0(1-p_0)/n}\). But when calculating power at \(p_1\), the sampling distribution of \(\hat{p}\) has standard deviation \(\sqrt{p_1(1-p_1)/n}\). The two quantities serve different purposes: the first sets the test’s rejection boundary, and the second describes how sample proportions vary at the alternative.

Deriving a One-Sided Sample-Size Formula

First consider an upper-tailed test, \(H_0:p=p_0\) against \(H_a:p>p_0\), with a planned alternative \(p_1>p_0\). The normal-approximation rejection boundary for \(\hat{p}\) is \(p_0+z_{1-\alpha}\sqrt{p_0(1-p_0)/n}\). To attain power \(1-\beta\), require this boundary to lie approximately \(z_{1-\beta}\) alternative standard deviations below the mean \(p_1\). Thus:

Move \(p_0\) to the right and collect the terms containing \(1/\sqrt{n}\). Squaring both sides then gives the planning formula. For a lower-tailed test, the same expression applies using the absolute difference between the planned proportions and the corresponding one-sided critical quantile.

This formula treats the target power as the probability of crossing the rejection boundary in the planned direction. For a one-sided test, it follows directly from the normal approximation. The calculated value is generally not an integer, so round it up to ensure the planned sample is at least as large as the calculation.

Planning a Two-Sided Test

If departures in either direction matter, use \(H_a:p\ne p_0\). The test allocates \(\alpha/2\) to each tail, so its critical values are \(\pm z_{1-\alpha/2}\). A convenient planning formula replaces \(z_{1-\alpha}\) in the one-sided formula with \(z_{1-\alpha/2}\). For a specified alternative \(p_1\), it is a common approximation that treats power in the direction of \(p_1-p_0\) as the relevant tail and neglects the very small probability of rejection in the opposite tail.

The two-sided normal approximation also makes it possible to check the power more directly. Its rejection region for \(\hat{p}\) is below or above \(p_0\) by \(z_{1-\alpha/2}\sqrt{p_0(1-p_0)/n}\). Under the planned alternative, approximate power is the sum of the probabilities that a normal variable with mean \(p_1\) and standard deviation \(\sqrt{p_1(1-p_1)/n}\) falls below the lower boundary or above the upper boundary. This check includes both tails and can confirm whether the simplified planning formula is adequate.

Checking the Approximation and Choosing Inputs

A sample-size formula is only as useful as its inputs and assumptions. The null proportion \(p_0\) comes from the hypothesis being tested. The alternative proportion \(p_1\) should represent a specific difference that would matter for the study, not an effect selected merely because it gives a convenient sample size. The direction of the test must also match the research question and be chosen before observing the data.

After calculating \(n\), check expected counts using both planning proportions: evaluate \(np_0\), \(n(1-p_0)\), \(np_1\), and \(n(1-p_1)\). Meeting the common threshold of 10 supports the normal approximation but does not make the binomial distribution exactly normal. If the proportions are very close to 0 or 1, or the expected counts are small, the approximation may not be dependable even when the arithmetic in the formula is correct. A normal-based sample size should then be treated cautiously.

The following sequence keeps the planning calculation tied to the intended test:

Common Mistake and Key Takeaway

A frequent error is to use the null standard deviation for both parts of the calculation. For example, substituting \(\sqrt{p_0(1-p_0)}\) in place of \(\sqrt{p_1(1-p_1)}\) in the power term assumes that sample proportions vary under the alternative as they do under the null. They generally do not: under \(p_1\), the variance of \(\hat{p}\) is \(p_1(1-p_1)/n\). Use the null proportion to locate the test’s rejection boundary and the alternative proportion to describe the distribution when calculating power.

Sample-size planning for a proportion test therefore depends on a complete set of choices: the null value, a meaningful alternative, the test direction, \(\alpha\), and target power. The result is an approximate design calculation whose conditions and expected counts must be checked, not a sample size that guarantees a particular outcome.