Planning Sample Sizes for Comparing Two Means

Why Plan Sample Size Before Collecting Data?

A comparison of two means is more useful when its precision is planned rather than left to chance. A study with too few observations may produce a wide confidence interval or have little chance of detecting a meaningful difference. Collecting far more observations than necessary can waste time, money, or participant effort. Sample-size planning connects the question a study is meant to answer with the number of observations needed in each group.

There are two common planning goals. For estimation, choose sample sizes so a confidence interval for the difference in population means is expected to have a specified margin of error. For testing, choose sample sizes so a test has a specified probability of detecting a particular difference, if that difference is truly present. Both goals require an anticipated amount of variability, usually based on relevant earlier data or a pilot study. The resulting numbers are planning estimates, not guarantees.

Planning for a Target Margin of Error

Let \(\mu_1-\mu_2\) be the difference between two population means, and let the independent groups have sample sizes \(n_1\) and \(n_2\). Suppose their population standard deviations are anticipated to be \(\sigma_1\) and \(\sigma_2\). For planning, the standard error of the difference in sample means is estimated by

The standard deviations in this expression are planning values; they are not known population quantities in most applications. For a two-sided confidence interval with confidence level \(1-\alpha\), a large-sample approximation to its margin of error is the standard-normal critical value \(z_{1-\alpha/2}\) multiplied by this standard error. If the groups are planned to have the same size, \(n_1=n_2=n\), setting that margin equal to a desired value \(M\) gives

Because a planned group cannot have a fractional participant, round the calculated value up to the next whole number. This calculation targets an approximate margin of error; it does not promise that the realized confidence interval will have exactly that width. The observed standard deviations and the appropriate finite-sample critical value can differ from the planning assumptions.

Planning for Power to Detect a Difference

For a test, planning requires a specified difference that would be important to detect. Write \(\Delta\) for the magnitude of that difference in the population means, and write \(1-\beta\) for the desired power. Power is the probability that the test rejects \(H_0\) when the stated alternative is true. The significance level \(\alpha\) sets the test’s Type I error rate under \(H_0\); it is not the probability that \(H_0\) is true.

For a two-sided test of \(H_0:\mu_1-\mu_2=0\) against \(H_a:\mu_1-\mu_2\ne0\), a normal approximation uses critical value \(z_{1-\alpha/2}\). If the true difference has magnitude \(\Delta\), the standardized distance of the alternative from zero is approximately \(\lambda=\Delta/SE\). Under that alternative, the test statistic is approximated by a normal random variable with standard deviation 1 and mean \(\lambda\), taking the positive direction to match the true difference. Thus its approximate power is

where \(Z\) has a normal distribution with mean \(\lambda\) and standard deviation 1, and \(\Phi\) is the standard normal cumulative distribution function. A widely used sample-size approximation sets \(\lambda\) to \(z_{1-\alpha/2}+z_{1-\beta}\). It gives the balanced-group formula

This formula is an approximation to the power calculation: it targets the desired probability in the tail in the direction of the specified difference and does not separately calculate the opposite tail. It is a standard, convenient planning rule; if a more exact power target is needed, solve the two-tail power equation for \(n\), usually with software. As with the margin-of-error formula, round up. A different scientifically meaningful \(\Delta\), anticipated variability, \(\alpha\), or desired power can lead to a substantially different sample size.

One-Sided Tests and Unequal Group Sizes

A one-sided test is appropriate only when the research question and analysis plan specify a direction in advance. For a test whose alternative is that \(\mu_1-\mu_2\) is positive, the large-sample critical value is \(z_{1-\alpha}\), not \(z_{1-\alpha/2}\). The corresponding balanced-group approximation is

Choosing a one-sided test after looking at the data, or using it simply to reduce the sample size, does not preserve the planned error rate for an unanticipated effect in the opposite direction. The direction and scientific rationale must be settled before examining the results.

Equal group sizes are convenient, but not always feasible. If the planned ratio is \(n_2=kn_1\), where \(k\) is a positive allocation ratio, then the standard error squared is approximately \((\sigma_1^2+\sigma_2^2/k)/n_1\). Substituting that expression gives a direct estimate for the first group size:

In practice, round each group size to a whole number while maintaining at least the planned allocation. The formula does not account for unequal costs or other design constraints; those may affect which allocation is practical.

Why the Approximation Works—and When to Refine It

The planning formulas follow by translating a target into a standard-error requirement. For a confidence interval, the approximate margin is \(z_{1-\alpha/2}SE\). Requiring it to be no more than \(M\) means \(SE\le M/z_{1-\alpha/2}\). With equal group sizes, the variance of the difference in sample means is approximately \((\sigma_1^2+\sigma_2^2)/n\). Squaring both sides and rearranging yields

The power formula uses the same standard error. Under a true difference \(\Delta\), the test statistic is shifted by about \(\lambda=\Delta/SE\). Requiring that shift to be at least \(z_{1-\alpha/2}+z_{1-\beta}\) yields the stated approximate formula after substituting the balanced-group standard error and rearranging.

These arguments rely on independent observations within and between groups, a quantitative outcome, and a sampling distribution of the difference in means that is adequately approximated by a normal distribution. For small samples, strong skewness or influential outliers can undermine that approximation. The anticipated standard deviations should be reasonable for the population and measurement process being studied. If they are underestimated, the actual standard error will tend to be larger than planned, potentially reducing precision and power.

The formulas use normal critical values for convenient planning. Actual two-sample \(t\) procedures use \(t\) critical values and, for Welch inference, degrees of freedom that depend on the sample sizes and estimated variances. In smaller planned studies, a more accurate calculation can iterate: obtain an initial size using the approximation, calculate the relevant \(t\) critical value and power at that size, then adjust until the target is reached. Statistical software can perform this calculation. Allow separately for expected missing outcomes, withdrawals, or unusable measurements; the required number enrolled may need to exceed the number with complete data.

Common Planning Pitfall

A frequent mistake is to calculate a sample size using the observed difference from a small pilot study as though it were a reliable target effect. Pilot estimates can be highly variable, and an unusually large estimated difference can produce an implausibly small planned study. Instead, specify a difference that would matter scientifically or clinically, and use relevant evidence to choose plausible standard deviations. It is also a mistake to report a sample-size estimate without stating its assumptions: the planned size only supports the stated precision or power under those assumptions.