Why Proportion-Test Power Needs Two Standard Errors
A proportion is an average of Bernoulli observations: each observation records whether an event occurred. Examples include whether a hypothetical policyholder files a claim during a specified period or whether a patient responds to a treatment. When planning a test, the analyst must specify both a null value and a plausible alternative value. Those values generally give different variances—and therefore different standard errors.
The distinction is essential. The null standard error determines the test's rejection boundary at the chosen significance level. The alternative standard error describes how the statistic varies when the alternative is true, and is used to calculate power. Using the null standard error for both steps can produce a noticeably inaccurate result.
The calculations below use normal approximations. They assume independent Bernoulli observations within each sample, a fixed sample size, and a specified alternative proportion (or pair of proportions). The approximation is more credible when the expected numbers of successes and failures are not small under both the null and the planned alternative. If these conditions are poor, a normal approximation may be unreliable.
One-Sample Proportion Test
Let \(X_1,\ldots,X_n\) be independent Bernoulli observations with common success probability \(p\), and let \(\widehat p=X/n\), where \(X=\sum_{j=1}^n X_j\). Under a null value \(p_0\), the approximate standard error of \(\widehat p\) is \(\sqrt{p_0(1-p_0)/n}\). For an upper-tailed test of \(H_0:p=p_0\) against \(H_1:p>p_0\), a normal-approximation rejection rule is
where \(z_{1-\alpha}\) is the standard normal quantile with cumulative probability \(1-\alpha\). For power at a planned alternative \(p_1>p_0\), use the alternative standard error \(\sqrt{p_1(1-p_1)/n}\). Thus the approximate power is
where \(\Phi\) is the standard normal cumulative distribution function. This calculation treats \(\widehat p\) as approximately normal under the alternative. A lower-tailed test reverses the rejection direction. For a two-sided test, both tails outside the null-based boundaries contribute to power.
Worked Example: Power for an Upper-Tailed One-Sample Test
A hypothetical program is evaluated by recording whether each of 100 independent participants completes a training module. Test \(H_0:p=0.20\) against \(H_1:p>0.20\) at \(\alpha=0.05\). For planning, suppose the true completion proportion is \(p_1=0.32\). Approximate the power using \(z_{0.95}=1.6449\), with no continuity correction.
Method. First find the rejection boundary under the null, then calculate the probability of exceeding that boundary under the alternative.
Under the null, \(n=100\), \(p_0=0.20\), and \(1-p_0=0.80\). The null standard error is
The critical value on the \(\widehat p\) scale is
Under the alternative, \(p_1=0.32\) and \(1-p_1=0.68\), so
Therefore, the approximate power is
The approximate power is 0.8773. As a check on the normal approximation, the expected success and failure counts are \(20\) and \(80\) under the null, and \(32\) and \(68\) under the alternative. The common mistake this method avoids is using \(SE_0=0.0400\) again in the final probability; power is evaluated under \(p_1\), so the final standard error is \(SE_1\).
Two-Sample Tests of Proportions
Suppose two independent groups have sample sizes \(n_1,n_2\) and success probabilities \(p_1,p_2\). Write \(\widehat p_1,\widehat p_2\) for their sample proportions and \(D=\widehat p_1-\widehat p_2\). Under the null hypothesis \(p_1=p_2\), the variance of \(D\) is estimated using a pooled proportion in the usual equality test. For planning under a specified alternative, the variance is instead
The two terms add because the groups are independent. For a planned alternative, this gives the alternative standard error. For an equal-allocation design, \(n_1=n_2=n\), and a two-sided equality test at level \(\alpha\), a common planning approximation uses \(\bar p=(p_1+p_2)/2\) in the null standard error:
This use of \(\bar p\) is a planning approximation to the pooled null proportion. In an actual test, the pooled estimate is calculated from the observed samples. If the null rejection boundaries for \(D\) are \(-c\) and \(c\), and the planned alternative has mean difference \(\Delta=p_1-p_2\), the approximate two-sided power is
The first term is the probability of falling below the lower boundary under the alternative; the second is the probability of exceeding the upper boundary. Keeping those two regions separate helps prevent counting the same tail twice.
Worked Example: Two-Sided Power With Two Independent Proportions
Two independent groups of 80 participants each receive different versions of a hypothetical reminder message. The outcome is whether a participant completes a follow-up survey. Test equality of the two proportions at \(\alpha=0.05\). For planning, take \(p_1=0.40\), \(p_2=0.60\), and \(z_{0.975}=1.96\). Approximate the power without a continuity correction.
Method. Use the pooled planned proportion to obtain the null rejection boundary, then use the two planned proportions to obtain the alternative standard error. Here \(\Delta=p_1-p_2=-0.20\).
The planned pooled proportion is \(\bar p=(0.40+0.60)/2=0.50\). Thus the null standard error and two-sided critical boundary are
Under the alternative, the two group variances give
The lower-tail probability is
The upper-tail probability is
Adding the disjoint rejection-tail probabilities gives approximate power \(0.7196\). The result is plausible: the alternative difference is in the lower tail, so most power comes from that tail. Expected successes and failures in the groups are \(32,48\) and \(48,32\), respectively, making the normal approximation reasonable for this planning calculation.
Common mistake and correction. Writing the upper-tail term as \(1-\Phi((-0.20+0.15495)/0.07746)\) gives \(1-\Phi(-0.5816)\approx0.7196\). Adding that to the lower-tail probability produces about \(1.4392\), which cannot be a probability. The upper-tail boundary is \(D>c\), so its standardized value is \((c-\Delta)/SE_1=(0.15495+0.20)/0.07746\), not \((-0.20+c)/SE_1\). Using the correct upper tail gives approximately \(0.0000\), not \(0.7196\).
Sample-Size Planning
For a one-sided one-sample test, a useful approximate planning equation makes the two sources of variation explicit. Let \(p_0\) be the null proportion, \(p_1\) the planned alternative, and \(1-\beta\) the target power. For an upper-tailed test, the rejection boundary is approximately \(p_0+z_{1-\alpha}\sqrt{p_0(1-p_0)/n}\). Requiring that boundary to be \(z_{1-\beta}\) alternative standard errors below \(p_1\) gives
Multiplying by \(\sqrt n\) and squaring yields
This is an approximation, not an exact integer-sample guarantee. Round the result up to a whole observation and, when possible, check the achieved approximate power at that integer. The formula for two-sided tests replaces \(z_{1-\alpha}\) with \(z_{1-\alpha/2}\). For equal-sized independent groups, a corresponding approximation is
where \(n\) is the sample size in each group. Unequal allocation changes the two variance contributions and requires using the specified \(n_1,n_2\) relationship rather than treating the total sample size as if the groups were equal.
Worked Example: One-Sample Size for a Target Power
A hypothetical insurer is considering a redesigned notice and plans to measure whether a recipient pays an invoice by its due date. Under the null, the on-time payment proportion is \(p_0=0.25\); the design should detect \(p_1=0.35\). Find an approximate sample size for an upper-tailed test with \(\alpha=0.05\) and target power \(0.80\). Use \(z_{0.95}=1.6449\) and \(z_{0.80}=0.8416\).
Method. Apply the one-sample planning formula, using the null variance in the significance term and the alternative variance in the power term.
The variance factors are
The two weighted terms are \(1.6449(0.4330)\approx0.7123\) and \(0.8416(0.4770)\approx0.4014\). Therefore,
Rounding up gives a planned sample size of 125. As a check, at \(n=125\), \(SE_0=\sqrt{0.1875/125}\approx0.03873\), so the rejection boundary is \(0.25+1.6449(0.03873)\approx0.3137\). Under \(p_1=0.35\), \(SE_1=\sqrt{0.2275/125}\approx0.04266\); hence power is approximately \(\Phi((0.35-0.3137)/0.04266)=\Phi(0.851)\approx0.8025\), just above the target. The common mistake avoided is substituting the alternative variance into the null cutoff or the null variance into the power calculation.
Worked Example: Equal-Group Size for a Two-Sided Test
A hypothetical study compares two independent reminder formats. The planned response proportions are \(p_1=0.30\) and \(p_2=0.45\). Find an approximate equal sample size per group for a two-sided test at \(\alpha=0.05\) with target power \(0.80\). Use \(z_{0.975}=1.96\) and \(z_{0.80}=0.8416\).
Method. Use the equal-allocation two-sample formula. The average planned proportion is \(\bar p=(0.30+0.45)/2=0.375\), and the absolute difference is \(0.15\).
The null-side factor is
The alternative-side factor is
The weighted terms are \(1.96(0.6847)\approx1.3419\) and \(0.8416(0.6764)\approx0.5693\). Thus
Round up to 163 participants per group, or 326 in total. The denominator uses the squared difference between proportions, while the numerator includes both the null and alternative variability terms. A frequent trap is to report 163 as the total sample size; in this formula it is the size of each group.
Interpretation and Practical Checks
Power depends on the chosen significance level, sample size, and planned alternative. A larger sample typically narrows both standard errors; a larger separation between the null and alternative generally makes rejection more likely. These statements assume the same testing setup and valid model assumptions. Power is not an intrinsic property of a test independent of an alternative: a power statement must identify the alternative proportion or proportions at which it is calculated.
Before accepting a result, check the following:
- Identify whether the test is upper-tailed, lower-tailed, or two-sided, and write its rejection region before calculating power.
- Use the null model to determine the critical boundary at the stated significance level.
- Use the specified alternative proportion or proportions to calculate the standard error for power.
- For a two-sided test, add probabilities for the two disjoint rejection tails, each standardized relative to the alternative mean.
- Check that a reported power lies between 0 and 1 and that expected successes and failures support the normal approximation.
- For sample-size planning, round up and verify the resulting approximate power when the rounded value is important.
Check Your Understanding
For each problem, use a normal approximation without a continuity correction unless otherwise stated.
- A one-sample upper-tailed test uses \(H_0:p=0.40\), \(n=100\), and \(\alpha=0.05\). At the planned alternative \(p=0.52\), approximate the power using \(z_{0.95}=1.6449\). (Answer: approximately 0.7850)
- Two independent groups each have \(n=100\). For a one-sided test of \(H_0:p_1=p_2\), use \(\alpha=0.05\), planned proportions \(p_1=0.55\), \(p_2=0.40\), and \(z_{0.95}=1.6449\). Approximate the power. (Answer: approximately 0.686)
- For a one-sample upper-tailed test, plan to detect \(p_1=0.20\) against \(p_0=0.10\), with \(\alpha=0.05\) and power \(0.90\). Using \(z_{0.95}=1.6449\) and \(z_{0.90}=1.2816\), find the approximate sample size. (Answer: 102)
- Two independent groups will have equal sample sizes. For a two-sided test with \(p_1=0.25\), \(p_2=0.40\), \(\alpha=0.05\), and target power \(0.80\), use \(z_{0.975}=1.96\) and \(z_{0.80}=0.8416\). Find the approximate required sample size per group. (Answer: 152 per group)