Using Pilot Data to Plan Sample Size

Why Pilot Data Can Help With Planning

When planning a confidence interval for a population proportion, the desired confidence level and margin of error do not by themselves determine a sample size. The standard error also depends on the population proportion \(p\), which is usually unknown before data collection. A pilot sample provides an initial estimate that can be used to plan, rather than pretending the unknown proportion is already known.

This approach can be more efficient than planning for the most demanding possible value of \(p\). It also introduces a risk: a pilot estimate may be far from the population proportion, especially when the pilot is small. If the pilot suggests less variability than the population actually has, the planned sample may be too small to achieve the desired precision. The calculation is therefore a planning aid, not a guarantee.

Definition: A pilot-based sample-size plan uses the observed sample proportion from preliminary data, \(\hat{p}_{\text{pilot}}\), as an estimate of the unknown population proportion \(p\) when calculating the sample size for a later proportion confidence interval. The pilot data are used to plan the main sample; they are not automatically counted as part of that sample.

The Pilot-Based Planning Formula

For a two-sided confidence interval with confidence level \(C\), let \(z^*\) be the corresponding standard normal critical value. Let \(E\) be the desired margin of error, expressed as a proportion rather than a percentage. A pilot-based plan substitutes \(\hat{p}_{\text{pilot}}\) for the unknown \(p\) in the estimated standard error.

Planning formula: For a one-proportion interval based on the normal approximation, the pilot-based target sample size is calculated as

n_{\text{plan}} = \frac{(z^*)^2\hat{p}_{\text{pilot}}(1-\hat{p}_{\text{pilot}})}{E^2}.

Here \(E\) is the target margin of error and \(z^*\) is the critical value for the chosen confidence level. The result is a planning value; a practical sample size must be a whole number and should be rounded up. The details of rounding and checking the achieved precision are separate considerations.

The formula is based on the usual approximate margin of error for a one-proportion interval, \(z^*\sqrt{p(1-p)/n}\). For planning, replace \(p\) with the pilot estimate and set that approximate margin of error equal to the target \(E\). Solving for \(n\) gives the displayed expression.

E \approx z^*\sqrt{\frac{\hat{p}_{\text{pilot}}(1-\hat{p}_{\text{pilot}})}{n}} \quad\Longrightarrow\quad \frac{E}{z^*} \approx \sqrt{\frac{\hat{p}_{\text{pilot}}(1-\hat{p}_{\text{pilot}})}{n}} \quad\Longrightarrow\quad n \approx \frac{(z^*)^2\hat{p}_{\text{pilot}}(1-\hat{p}_{\text{pilot}})}{E^2}.

This derivation is a justification for the planning rule, not a claim that the future interval will have exactly the target margin of error. It uses an estimated standard error, and the actual sample proportion in the main study may differ from the pilot estimate. The formula also assumes the main study will use a random sample with independent observations, and that a normal-approximation proportion interval is appropriate at the planned sample size. The target population and the way the outcome is defined should be comparable between the pilot and the main study.

Worked Example: Planning From a Pilot Estimate of 0.42

Suppose a hypothetical pilot survey finds that 42 of 100 respondents support a proposal. A larger survey will use a 95% confidence level and aims for a margin of error of 0.04, or 4 percentage points. Find the pilot-based sample-size planning value.

The pilot estimate is \(\hat{p}_{\text{pilot}}=42/100=0.4200\). For a 95% confidence level, use \(z^*=1.96\). The target margin is \(E=0.04\), in proportion units. Substitution gives

n_{\text{plan}} = \frac{(1.96)^2(0.4200)(1-0.4200)}{(0.04)^2} = \frac{3.8416(0.4200)(0.5800)}{0.0016} = \frac{3.8416(0.2436)}{0.0016} = \frac{0.93581376}{0.0016} = 584.8836.

The pilot-based calculation gives a planning value of about 584.9 observations. It uses the estimated variability associated with a proportion of 0.4200 at the stated confidence level and target margin. This value is not a promise that a future interval will have a 0.04 margin of error: the actual main-sample proportion may differ from 0.4200.

How the Pilot Estimate Affects the Plan

The quantity \(\hat{p}_{\text{pilot}}(1-\hat{p}_{\text{pilot}})\) determines the estimated variability in the formula. It is largest when the pilot estimate is 0.50, and it becomes smaller as the estimate approaches 0 or 1. As a result, a pilot estimate near 0.50 produces a larger planned sample size than an estimate nearer a boundary, when the confidence level and target margin are held fixed.

A lower calculated sample size does not necessarily mean the pilot estimate is more trustworthy or the planned survey is safer. It means only that the formula, using that particular estimate, predicts less variability. If a small or unrepresentative pilot produces an estimate near 0 or 1 by chance, the resulting plan may understate the sample size needed.

Worked Example: Comparing Pilot Estimates

Two hypothetical planning teams want 95% confidence intervals with a target margin of error of 0.05. One pilot estimates a support proportion of 0.20; the other estimates 0.50. Compare their planning values.

For the pilot estimate of 0.20, the estimated product is \(0.20(0.80)=0.16\). With \(z^*=1.96\),

n_{\text{plan}} = \frac{(1.96)^2(0.20)(0.80)}{(0.05)^2} = \frac{3.8416(0.16)}{0.0025} = \frac{0.614656}{0.0025} = 245.8624.

For the pilot estimate of 0.50, the estimated product is \(0.50(0.50)=0.25\):

n_{\text{plan}} = \frac{(1.96)^2(0.50)(0.50)}{(0.05)^2} = \frac{3.8416(0.25)}{0.0025} = \frac{0.9604}{0.0025} = 384.16.

The pilot estimate of 0.50 gives the larger planning value because \(0.25\) exceeds \(0.16\). The same planning value would result from a pilot estimate of 0.80, since \(0.80(0.20)=0.16\), just as \(0.20(0.80)=0.16\). This symmetry follows directly from the product \(p(1-p)\); it is not evidence that the two pilots describe the same population.

Uncertainty in a Pilot Estimate

A pilot estimate is itself based on a sample and therefore varies from one pilot to another. Small pilots are especially vulnerable to chance variation. An estimate such as 0.20 from a small pilot may be a reasonable starting point, but it should not be treated as a precise measurement of \(p\). A sample-size formula that inserts a single estimate does not account for this uncertainty.

One useful response is to examine how the plan changes across plausible proportions. Another is to use the conservative planning value \(\hat{p}_{\text{pilot}}=0.50\) when there is not enough confidence in a pilot estimate. Because \(p(1-p)\) cannot exceed 0.25, this choice gives the largest sample-size value under the formula for the same \(z^*\) and \(E\). This conservative approach may require a larger sample, but it avoids relying on a possibly low variability estimate.

Worked Example: A Small Pilot May Understate the Required Precision

A hypothetical pilot records 6 successes among 30 participants, giving \(\hat{p}_{\text{pilot}}=6/30=0.20\). A main study aims for 95% confidence and a margin of error of 0.04. Calculate the pilot-based plan, then consider what would happen if the population proportion were closer to 0.50.

Using the pilot estimate gives

n_{\text{plan}} = \frac{(1.96)^2(0.20)(0.80)}{(0.04)^2} = \frac{3.8416(0.16)}{0.0016} = \frac{0.614656}{0.0016} = 384.16.

For comparison, using 0.50 as the planning proportion gives

n_{\text{plan}} = \frac{(1.96)^2(0.50)(0.50)}{(0.04)^2} = \frac{0.9604}{0.0016} = 600.25.

The difference arises because the pilot-based estimate implies a smaller estimated product, \(0.16\) rather than \(0.25\). If the population proportion were 0.50, the standard error at the sample size suggested by the 0.20 pilot would be larger than the pilot-based calculation anticipates. In the approximate margin-of-error formula, the ratio of the two margins at the same sample size would be

\frac{1.96\sqrt{0.25/n}}{1.96\sqrt{0.16/n}} = \sqrt{\frac{0.25}{0.16}} = 1.25.

Thus the margin of error under the 0.50 scenario would be about 25% larger than the margin predicted using 0.20. This is a sensitivity comparison, not a claim that the population proportion is likely to be 0.50. The small pilot alone may not justify treating 0.20 as stable; a cautious plan would consider the consequences of other plausible proportions or use the conservative value.

Assumptions and Practical Limits

A pilot-based plan is most useful when the pilot and main study target the same population, use the same definition of success, and have similar sampling and response processes. If the pilot comes from a different population or uses a different measurement procedure, its observed proportion may not be a useful estimate for planning. A numerically precise calculation cannot correct a mismatch in study design.

The formula also belongs to planning for a normal-approximation interval. At the planned sample size, the expected numbers of successes and failures implied by the planning proportion should be large enough for that approximation to be reasonable. In the completed study, the observed counts and the sampling design still matter; a planned sample size does not make an inappropriate interval appropriate. The usual checks for random sampling, independence, and the success-failure condition remain relevant.

Finally, the nominal margin of error describes sampling variability under the model. It does not include bias from nonresponse, coverage problems, measurement error, or a poorly defined outcome. A large sample can reduce standard error while leaving those sources of error unresolved.

Common Mistake: Treating the Pilot as the True Proportion

A common mistake is to say, “The pilot estimate is 0.20, so the population proportion is 0.20,” and then regard the resulting sample-size calculation as guaranteed to attain the desired margin of error. The pilot supplies an estimate, not the known population value. Because the planning formula depends on \(p(1-p)\), an inaccurate estimate can materially change the planned sample size and the precision ultimately achieved.

The correction is to state explicitly that the calculation is conditional on the pilot estimate. Check whether the pilot is sufficiently informative and comparable to the main study, and consider how the plan changes for other plausible proportions. If the cost of underestimating variability is important, use a conservative planning value rather than relying on an unstable pilot estimate.

Key takeaway: Pilot data can make sample-size planning more tailored, but the planned precision depends on the pilot estimate being relevant and reasonably stable. Treat a pilot-based sample size as an informed plan, and assess its sensitivity to uncertainty in the estimated proportion.

Check Your Understanding

Answer in context, showing the setup of any calculation.

A pilot sample estimates a population proportion as 0.35. For a fixed confidence level and margin of error, would the formula give a larger planning value using 0.35 or 0.50? Explain using \(p(1-p)\).
Why is a pilot-based sample-size calculation not a guarantee that the main study will achieve the target margin of error?
A pilot finds 9 successes among 30 participants. What is \(\hat{p}_{\text{pilot}}\), and what feature of this estimate should be considered before relying on it?
Why do pilot estimates of 0.20 and 0.80 produce the same sample-size planning value when the other inputs are unchanged?
Give two reasons, apart from pilot-sample size, that a pilot estimate might not be suitable for planning the main study.