A Complete Sample-Size Planning Workshop

From a Research Question to a Recruitment Target

A useful sample-size plan does more than produce a number. It explains what population quantity the study will estimate, what result would count as meaningful, and how likely the planned test is to detect that result. It also accounts for people who may be recruited but provide no usable outcome data. These decisions should be made before the results are observed.

This workshop follows the planning chain for a one-sample test. The examples use either a population mean or a population proportion. The calculation depends on the test direction, significance level, target power, meaningful effect, and a planning value for variability. The final recruitment target also depends on expected attrition.

Definition: A sample-size plan specifies a required number of analyzable observations for a stated test, significance level \(\alpha\), target power, and alternative effect, under stated assumptions about variability and the sampling process. If some recruited participants are expected not to contribute analyzable data, the recruitment target must be larger than the required analyzable sample.

The Planning Decisions

Start by making the research question statistical. Name the population parameter and state the null and alternative hypotheses. For a mean, the parameter is \(\mu\); for a population proportion, it is \(p\). The null value, such as \(\mu_0\) or \(p_0\), is the benchmark against which the research question is framed.

Next, choose the direction of the test. Use a one-sided alternative only when the research question identifies a particular direction and an effect in the opposite direction would not count as evidence for the research claim. If departures in either direction matter, use a two-sided alternative. The choice should reflect the question, not which direction makes the sample-size calculation smaller.

Alpha, \(\alpha\): the probability of rejecting the null hypothesis when it is true, for the specified test and its assumptions.
Target power: the planned probability of rejecting the null when a specified alternative is true. Equivalently, \(\beta\) is the probability of failing to reject at that alternative, and power is \(1-\beta\).
Meaningful effect: the smallest difference from the null value that the study is designed to detect, expressed in the parameter’s units or scale.
Variability: a planning value for \(\sigma\) when testing a mean, or for the outcome variability implied by \(p_0\) and the planned alternative when testing a proportion.

The meaningful effect is not necessarily the effect the study will find. It is a design choice: an effect smaller than this may be real, but the plan does not promise the target power for detecting it. A planning value for variability may come from relevant earlier data or a pilot; because it is uncertain, it is sensible to examine more than one plausible value.

Why the Mean-Test Formula Has This Form

For a one-sample mean test, suppose observations are independent and the sample mean has an approximately normal sampling distribution with standard error \(\sigma/\sqrt{n}\). This is justified when the population is normal or, with suitable conditions, when the sample is large enough for a normal approximation. For planning, \(\sigma\) is treated as a specified value. If it must instead be estimated from the study sample, the usual \(t\) test uses an estimated standard deviation, and the normal-based formula below is an approximation.

For an upper one-sided test of \(H_0:\mu=\mu_0\) against \(H_a:\mu>\mu_0\), let the meaningful alternative be \(\mu=\mu_0+\delta\), where \(\delta>0\). Under the null, a normal-approximation rejection rule is to reject when the sample mean exceeds \(\mu_0+z_{1-\alpha}\sigma/\sqrt{n}\). Under the alternative, standardizing this cutoff gives the following power:

P\left(\bar{x}>\mu_0+z_{1-\alpha}\frac{\sigma}{\sqrt{n}}\mid\mu=\mu_0+\delta\right) = \Phi\left(\frac{\delta\sqrt{n}}{\sigma}-z_{1-\alpha}\right).

Here \(z_q\) is the \(q\) quantile of the standard normal distribution, and \(\Phi\) is its cumulative distribution function. To make this power at least \(1-\beta\), require the argument of \(\Phi\) to be at least \(z_{1-\beta}\). Rearranging gives:

\frac{\delta\sqrt{n}}{\sigma}-z_{1-\alpha}\ge z_{1-\beta} \quad\Longrightarrow\quad n\ge \left(\frac{(z_{1-\alpha}+z_{1-\beta})\sigma}{\delta}\right)^2.

For a two-sided mean test, the common planning approximation replaces \(z_{1-\alpha}\) with \(z_{1-\alpha/2}\):

n\approx \left(\frac{(z_{1-\alpha/2}+z_{1-\beta})\sigma}{\delta}\right)^2.

This two-sided expression uses the tail in the direction of the planned effect; it is an approximation to the power of the two-sided test. A final calculation can use the exact test and specified analysis model, particularly when the sample is small. In either case, round the required analyzable sample size up to a whole observation.

Worked Example: Planning a Two-Sided Test of a Mean

Suppose a hypothetical study will measure resting pulse, in beats per minute, for a sample from a defined population. The research question asks whether the population mean differs from 72 beats per minute. Investigators consider a difference of 5 beats per minute meaningful. They plan \(\alpha=0.05\), power \(0.80\), and use a planning standard deviation of 12 beats per minute. Estimate the analyzable sample size, then allow for 15% attrition.

Parameter and direction. The parameter is the population mean resting pulse, \(\mu\). The hypotheses are \(H_0:\mu=72\) and \(H_a:\mu\ne72\), so the test is two-sided. The meaningful difference is \(\delta=5\) beats per minute.

Assumptions for the calculation. The sample should represent the target population, and observations should be independent. The planning value \(\sigma=12\) beats per minute should be reasonably plausible. The normal approximation is appropriate if the population is approximately normal or the sample is large enough for the sample mean to be approximately normal. These are planning assumptions, not facts guaranteed by the formula.

For \(\alpha=0.05\), \(z_{1-\alpha/2}=z_{0.975}\approx1.960\). For power \(0.80\), \(z_{1-\beta}=z_{0.80}\approx0.8416\). Substituting into the two-sided approximation:

n\approx \left(\frac{(1.960+0.8416)(12)}{5}\right)^2 = \left(\frac{33.6192}{5}\right)^2 = 6.72384^2 \approx45.21.

Rounding up, the study needs 46 analyzable participants. If 15% of recruited participants are expected not to provide usable measurements, the recruitment target is:

n_{\text{recruit}} = \left\lceil\frac{46}{1-0.15}\right\rceil = \left\lceil54.12\right\rceil = 55.

The plan is therefore to recruit 55 participants to aim for 46 analyzable observations. This does not guarantee exactly 46 usable records; attrition is uncertain. It does mean that the recruitment target allows for the stated expected loss.

Planning a One-Sample Proportion Test

For a one-sample test of a proportion, the parameter is \(p\). In a normal-approximation calculation, the standard error depends on the proportion: under the null it is based on \(p_0(1-p_0)\), and under a specified alternative it is based on \(p_1(1-p_1)\). For an upper one-sided test, a widely used planning formula is:

n\approx \frac{ \left[ z_{1-\alpha}\sqrt{p_0(1-p_0)} + z_{1-\beta}\sqrt{p_1(1-p_1)} \right]^2 }{ (p_1-p_0)^2 }.

This approximation assumes independent observations with a common probability of success and a normal approximation suitable at both the null and planned alternative. The expected success and failure counts should be large enough for that approximation to be credible. A final design may require an exact or software-based calculation if counts are small or the intended test differs from this approximation.

Worked Example: Planning an Upper-Tail Test of a Proportion

In a hypothetical evaluation, a team wants to test whether the proportion of eligible participants who complete a program is greater than the benchmark \(p_0=0.40\). An increase to \(p_1=0.50\) would be meaningful. They choose a one-sided test with \(\alpha=0.05\), target power \(0.90\), and expect 10% attrition.

Direction and assumptions. The parameter is the population completion proportion \(p\), with \(H_0:p=0.40\) and \(H_a:p>0.40\). The one-sided direction is appropriate because the research question is specifically about an increase. Assume each participant contributes one binary outcome, observations are independent, and the sample can be treated as representative of the target population. For the normal approximation, expected counts should be checked under both \(p_0\) and \(p_1\).

Use \(z_{0.95}\approx1.6449\), \(z_{0.90}\approx1.2816\), \(p_0(1-p_0)=0.40(0.60)=0.24\), and \(p_1(1-p_1)=0.50(0.50)=0.25\):

n\approx \frac{ \left[ 1.6449\sqrt{0.24}+1.2816\sqrt{0.25} \right]^2 }{ (0.50-0.40)^2 } = \frac{[1.6449(0.4899)+1.2816(0.5000)]^2}{0.10^2} \approx \frac{(0.8058+0.6408)^2}{0.0100} \approx209.3.

The plan requires 210 analyzable participants. At this sample size, the expected counts are \(210(0.40)=84\) successes and 126 non-successes under the null, and \(210(0.50)=105\) successes and 105 non-successes under the alternative. These counts support the normal approximation for this illustration.

To allow for 10% attrition, recruit:

n_{\text{recruit}} = \left\lceil\frac{210}{1-0.10}\right\rceil = \left\lceil233.33\right\rceil = 234.

The resulting target is 234 recruits, aiming for at least 210 analyzable outcomes. The planned power applies to detecting an increase from 0.40 to 0.50 under the stated assumptions; it is not a claim that the observed proportion will equal 0.50.

When Variability Is Uncertain

A sample-size result can change substantially when the assumed variability changes. For a mean test, the required sample size is proportional to \(\sigma^2\) when the other planning choices are held fixed. Thus, a 20% increase in the assumed standard deviation increases the unrounded required sample size by a factor of \(1.2^2=1.44\). When a pilot estimate is imprecise, report how the recruitment target changes across plausible values rather than presenting one estimate as certain.

Worked Example: Choosing a Target Across Plausible Variability

A hypothetical study plans a two-sided test of a population mean with \(\alpha=0.05\), power \(0.90\), and a meaningful difference of 4 units. Previous information does not pin down the standard deviation; values from 10 to 14 units are considered plausible. The team expects 12% attrition.

Assume independent, representative observations and a normal population or an adequate normal approximation for the sample mean. For \(\alpha=0.05\), use \(z_{0.975}=1.960\); for power \(0.90\), use \(z_{0.90}=1.2816\). The common multiplier is \(1.960+1.2816=3.2416\).

At \(\sigma=10\) units, the analyzable requirement is:

n\approx \left(\frac{3.2416(10)}{4}\right)^2 = \frac{10.50797(100)}{16} \approx65.67,

so 66 analyzable observations are required. With 12% attrition, the recruitment target at this variability is \(\lceil66/0.88\rceil=75\).

At the more conservative value \(\sigma=14\) units:

n\approx \left(\frac{3.2416(14)}{4}\right)^2 = \frac{10.50797(196)}{16} \approx128.72,

so 129 analyzable observations are required. Allowing for 12% attrition gives \(\lceil129/0.88\rceil=147\) recruits. If the team judges 14 units to be a plausible value that must be covered, 147 is the more defensible recruitment target. The choice should be justified by the evidence about variability and the available resources, not by selecting whichever value produces the smallest sample.

Turning the Calculation Into a Plan

A complete plan should make the reasoning auditable. It states the parameter and hypotheses, the test direction, \(\alpha\), target power, meaningful effect, variability assumption, calculation method, analyzable sample size, and attrition adjustment. The expected data should also be compatible with the assumptions of the chosen test. A sample-size formula cannot repair biased recruitment, dependent observations, a poorly measured outcome, or a mismatch between the research question and the alternative hypothesis.

The recruitment target is the rounded-up number to approach the desired analyzable sample after expected losses. If participants are recruited in clusters, or if the analysis uses a more complex design, the simple formulas here may not apply without further adjustment. Likewise, if the final test uses a different model from the one assumed in planning, its power may differ.

Common Mistake: Treating the Output as a Guarantee

A common error is to say, “This sample size guarantees 80% power,” without naming the effect, variability, test, and assumptions. Power is not a fixed property of a sample size by itself. It is calculated for a specified alternative under a specified model. If the true effect is smaller than the meaningful effect used in planning, or variability is larger than assumed, actual power can be lower.

Another error is to subtract the attrition percentage from the required sample. If 46 analyzable observations are needed and 15% are expected to be lost, recruiting 46 minus 15% would make the target smaller, not larger. The appropriate calculation divides the required analyzable number by the expected retention proportion: \(46/0.85\), then rounds up.

Key takeaway: A defensible recruitment target begins with a well-defined one-sample question and a meaningful alternative. State the assumptions, calculate the required analyzable sample, examine uncertainty in variability, and then inflate for expected attrition.

Check Your Understanding

For each question, explain the planning decision rather than giving only a number.

A study asks whether a population mean differs from a benchmark, with meaningful differences possible in either direction. Which test direction matches the question, and which normal quantile is used for the alpha component of the two-sided mean formula?
In a one-sample proportion plan, why do the normal-approximation sample-size terms use \(p_0(1-p_0)\) and \(p_1(1-p_1)\)?
A mean-test calculation requires 80 analyzable observations, and 20% attrition is expected. What recruitment target should be used, and why is subtracting 20% incorrect?
What does a target power of 90% mean for a specified alternative, and what does it not say about the probability that the null hypothesis is true?
If the assumed standard deviation for a mean test increases while all other planning choices remain unchanged, how does the required sample size change? Explain using the formula.