Sample Size Planning For A Difference In Proportions

Why Plan Both Group Sizes?

When a study compares two proportions, planning only the total number of observations is not enough. The precision of the estimated difference depends on how many observations fall in each independent group and on the proportions expected in those groups. A design with a large total sample can still be imprecise if most observations are assigned to the group with the smaller sample size.

This tutorial develops sample-size planning for a confidence interval for the difference between two independent population proportions. It is aligned with the SOA VEE Mathematical Statistics syllabus. The calculations use the normal-approximation, or Wald, margin of error. The resulting sample sizes are planning approximations, so the conditions for that approximation and the choice of anticipated proportions matter.

Definition: Let two independent groups have population proportions $p_1$ and $p_2$, sample sizes $n_1$ and $n_2$, and sample proportions $\hat p_1$ and $\hat p_2$. For a two-sided confidence level $1-\alpha$, the approximate margin of error for estimating $p_1-p_2$ is $$ E=z_{1-\alpha/2}\sqrt{\frac{p_1(1-p_1)}{n_1}+\frac{p_2(1-p_2)}{n_2}}, $$ where $z_{1-\alpha/2}$ is the corresponding standard normal quantile. In planning, anticipated values of $p_1$ and $p_2$ are substituted for the unknown proportions.

For example, a 95% confidence level uses $z_{0.975}=1.96$, while a 90% confidence level uses $z_{0.95}=1.645$. A smaller target margin of error requires larger samples: the sample sizes grow in proportion to $1/E^2$. This is why halving the desired margin of error requires about four times as many observations, all else equal.

The model assumes independent observations within and between the groups, a fixed sample size in each group, and a sample proportion that is adequately approximated by a normal distribution. Planning values should make the expected numbers of successes and failures in both groups sufficiently large. If those conditions are doubtful, this Wald-based calculation may not give reliable interval coverage; it should not be treated as a guarantee of exact precision.

Equal Allocation

If both groups are to have the same sample size $n$, substitute $n_1=n_2=n$ into the margin-of-error formula and solve for $n$. The proportions need not be equal; only the sample sizes are equal.

E=z_{1-\alpha/2}\sqrt{\frac{p_1(1-p_1)+p_2(1-p_2)}{n}} \qquad\Longrightarrow\qquad n=\frac{z_{1-\alpha/2}^{\,2}\left[p_1(1-p_1)+p_2(1-p_2)\right]}{E^2}.

The formula first produces a continuous planning value. Operational sample sizes must be whole numbers. If the goal is to ensure that the planned margin of error is no greater than the target, round each required group size up to the next integer and check the margin of error using those integer sizes. Keep the continuous result visible when reporting the calculation; it is not itself an executable sample size.

Worked Example: Equal Groups With Different Anticipated Proportions

A hypothetical evaluation compares the proportion of two independent groups that complete a training module. A pilot study suggests proportions of $p_1=0.30$ and $p_2=0.45$. Find the equal continuous planning size per group for a 95% confidence level and target margin of error $E=0.06$, then determine an integer size that meets the target.

Method. Use the equal-allocation formula with $z_{0.975}=1.96$. The anticipated variance terms are $$ p_1(1-p_1)=0.30(0.70)=0.21,\qquad p_2(1-p_2)=0.45(0.55)=0.2475. $$ Their sum is $0.21+0.2475=0.4575$. Also, $1.96^2=3.8416$ and $0.06^2=0.0036$. Therefore, $$ n=\frac{3.8416(0.4575)}{0.0036} =\frac{1.757532}{0.0036} =488.2033\ldots. $$ The continuous planning result is $488.2$ observations per group to one decimal place. Since a study requires whole observations and must meet the target, plan $489$ observations in each group.

Check. With $n_1=n_2=489$, the approximate margin is $$ 1.96\sqrt{\frac{0.4575}{489}} =1.96\sqrt{0.00093558\ldots} \approx 1.96(0.030587) \approx 0.05995. $$ This is below $0.06$, as required. At the continuous value $488.2033\ldots$, the formula gives the target margin $0.06$. The expected success and failure counts are comfortably above the small-count range in both groups. The common mistake to avoid is using only one group’s variance term; both proportions contribute to the variance of the difference.

Unequal Allocation

Sometimes the study design specifies that one group will be larger. Let $n_2=kn_1$, where $k>0$ is a fixed allocation ratio. Substitution into the margin-of-error formula gives a direct solution for the smaller group:

E^2=z_{1-\alpha/2}^{\,2} \left(\frac{p_1(1-p_1)}{n_1}+ \frac{p_2(1-p_2)}{kn_1}\right), \qquad n_1= \frac{z_{1-\alpha/2}^{\,2} \left[p_1(1-p_1)+p_2(1-p_2)/k\right]}{E^2}, \qquad n_2=kn_1.

If the ratio is a design requirement, round the smaller group up and then choose a whole-number size for the larger group that preserves or exceeds the intended ratio. Recalculate the margin of error from the resulting integer sizes. Rounding both continuous answers independently can slightly alter the ratio, so the final check should use the actual planned values of $n_1$ and $n_2$.

Worked Example: A Two-to-One Allocation

A hypothetical study compares two independent outreach methods. Anticipated completion proportions are $p_1=0.20$ and $p_2=0.35$. The design calls for twice as many participants in group 2 as in group 1. Find the 90% confidence planning sizes for a target margin of error $0.05$, then check a whole-number allocation.

Method. Here $k=2$, $z_{0.95}=1.645$, and the variance terms are $$ p_1(1-p_1)=0.20(0.80)=0.16,\qquad p_2(1-p_2)=0.35(0.65)=0.2275. $$ Thus the bracketed coefficient for $n_1$ is $$ 0.16+\frac{0.2275}{2}=0.16+0.11375=0.27375. $$ Using $1.645^2=2.706025$ and $0.05^2=0.0025$, $$ n_1=\frac{2.706025(0.27375)}{0.0025} =\frac{0.7407743\ldots}{0.0025} =296.3097\ldots. $$ The continuous planning sizes are $296.3$ for group 1 and $2(296.3097\ldots)=592.6$ for group 2. To use whole numbers while preserving the ratio, take $n_1=297$ and $n_2=594$.

Check. The variance contribution is $$ \frac{0.16}{297}+\frac{0.2275}{594} =0.00053872\ldots+0.000382997\ldots =0.00092172\ldots. $$ The resulting margin is $$ 1.645\sqrt{0.00092172\ldots} \approx 1.645(0.03036) \approx 0.04994, $$ which is below $0.05$. The anticipated expected counts are adequate for the normal approximation. The trap is to divide the second variance term by $n_1$, rather than by $n_2=2n_1$; doing so would calculate the precision for equal sizes, not this allocation.

Choosing an Allocation When the Ratio Is Flexible

If the total sample size is fixed or costly, it can be useful to allocate more observations to the group whose sample proportion has the larger anticipated variance. For a fixed total $N=n_1+n_2$, the variance of the estimated difference is minimized when the group sizes are in the ratio $$ \frac{n_1}{n_2}= \frac{\sqrt{p_1(1-p_1)}}{\sqrt{p_2(1-p_2)}}. $$ This is the variance-efficient allocation when observations have equal cost in both groups. It is not generally equal allocation unless the anticipated variance terms are equal.

Justification: Write $a=p_1(1-p_1)$ and $b=p_2(1-p_2)$. For a fixed total $N$, the variance is $a/n_1+b/n_2$, with $n_2=N-n_1$. By the Cauchy–Schwarz inequality, $$ \left(\frac{a}{n_1}+\frac{b}{n_2}\right)(n_1+n_2) \geq (\sqrt a+\sqrt b)^2. $$ Since $n_1+n_2=N$, the variance is at least $(\sqrt a+\sqrt b)^2/N$. Equality holds when $n_1/n_2=\sqrt{a/b}$. Therefore, the allocation ratio above minimizes the approximate variance for a fixed total sample size. This result assumes equal observation costs and the stated anticipated proportions.

Substituting the minimum variance into the margin-of-error formula also gives the continuous minimum total size: $$ N=\frac{z_{1-\alpha/2}^{\,2} \left[\sqrt{p_1(1-p_1)}+\sqrt{p_2(1-p_2)}\right]^2}{E^2}. $$ In practice, round the group sizes to whole numbers and verify the margin directly. If different groups have different recruitment costs or feasibility constraints, this equal-cost allocation is not necessarily the best design.

Worked Example: Variance-Efficient Allocation

A hypothetical comparison of two independent service options anticipates success proportions $p_1=0.10$ and $p_2=0.40$. Determine a variance-efficient allocation and the approximate total sample size for a 95% confidence level and margin of error $0.04$. Assume equal cost per observation.

Method. The variance terms are $$ a=0.10(0.90)=0.09,\qquad b=0.40(0.60)=0.24. $$ Their square roots are $\sqrt{a}=0.3000$ and $\sqrt{b}\approx0.489898$, so the efficient allocation ratio is $$ \frac{n_1}{n_2}=\frac{0.3000}{0.489898}\approx0.61237. $$ The minimum total size is $$ N=\frac{3.8416(0.3000+0.489898)^2}{0.04^2} =\frac{3.8416(0.6239387\ldots)}{0.0016} =\frac{2.396923\ldots}{0.0016} =1498.0768\ldots. $$ The corresponding continuous group sizes are $$ n_1=N\frac{0.3000}{0.789898}\approx568.9,\qquad n_2=N\frac{0.489898}{0.789898}\approx929.2. $$ Rounding up to $n_1=569$ and $n_2=930$ gives a practical allocation close to the efficient ratio.

Check. The planned margin is $$ 1.96\sqrt{\frac{0.09}{569}+\frac{0.24}{930}} =1.96\sqrt{0.00015817\ldots+0.00025806\ldots} \approx1.96(0.020402) \approx0.03999. $$ It is below $0.04$. The allocation puts more observations in group 2 because its anticipated variance $0.24$ is greater than group 1’s $0.09$. Equal allocation would not use the available sample as efficiently under these assumptions.

When the Proportions Are Uncertain

Planning requires values for $p_1$ and $p_2$, but those values may not be known precisely. A relevant pilot study or prior evidence can provide reasonable anticipated values. A difference between the anticipated proportions does not by itself determine the variance: the variance contribution for each group is $p(1-p)$, which is largest at $p=0.5$ and decreases toward zero as $p$ approaches either boundary.

If there is no defensible information about either proportion, using $0.5$ for both is a conservative choice for this Wald variance calculation. It gives the largest possible value, $0.25$, for each group’s variance term. Under equal allocation, the resulting planning formula is $$ n=\frac{z_{1-\alpha/2}^{\,2}(0.25+0.25)}{E^2} =\frac{z_{1-\alpha/2}^{\,2}(0.5)}{E^2}. $$ This is conservative for the estimated variance term, not a guarantee that every feature of the confidence interval will have exact coverage. In particular, the normal-approximation conditions still need attention.

Common Planning Traps

Leaving out one group’s variance. The independent-group difference has variance contributions from both groups. Include both terms, with each divided by its own sample size.
Using the wrong quantile. For a two-sided confidence level $1-\alpha$, use $z_{1-\alpha/2}$, not $z_{1-\alpha}$.
Confusing continuous and integer results. The algebra may produce a noninteger planning value. Report it as the continuous result, then round up for an actual design and check the achieved margin.
Assuming the groups must be equal. Equal allocation is a design choice, not a requirement. If a ratio is specified, include it in the variance formula.
Treating anticipated proportions as observed results. The values used in planning are assumptions for the design calculation; they are not estimates from the final study.

Key takeaway: Choose a confidence level, target margin, anticipated proportions, and allocation rule before solving for sample sizes. Apply the formula to both independent groups, round only when converting to a feasible design, and check the final margin of error using the planned integer sizes.

Check Your Understanding

For each problem, identify the allocation and confidence quantile before calculating. Continuous sizes are shown to one decimal place where relevant; integer designs should be rounded up and checked against the target.

A 90% interval is planned for a difference in proportions with equal group sizes. Anticipated proportions are $0.25$ and $0.40$, and the target margin is $0.08$. Find the continuous size per group and the integer size per group. (Answer: 180.8 continuously; 181 per group.)
No reliable anticipated proportions are available. For a 95% interval with equal allocation and target margin $0.04$, use the conservative choice $p_1=p_2=0.5$. Find the continuous size per group and the integer size. (Answer: 1200.5 continuously; 1201 per group.)
A 95% interval is planned with $n_2=1.5n_1$, anticipated proportions $p_1=0.30$ and $p_2=0.60$, and target margin $0.05$. Find the continuous sizes, then give whole-number sizes that preserve at least the intended allocation. (Answer: $n_1=568.6$, $n_2=852.8$ continuously; $n_1=569$, $n_2=854$ as whole numbers.)
For equal-cost observations, anticipated proportions are $0.20$ and $0.50$. Find the variance-efficient allocation ratio $n_1/n_2$, then the continuous total size for a 95% interval with margin $0.05$. (Answer: $n_1/n_2=0.8$; continuous total $N=1244.7$.)
A design uses anticipated proportions $0.10$ and $0.40$, with a 95% confidence level and equal group sizes. Is a continuous size of 1000 per group enough for a target margin of $0.04$? Calculate the margin and compare it with the target. (Answer: Yes; $1.96\sqrt{(0.09+0.24)/1000}\approx0.0356$, which is below $0.04$.)

What You'll Learn