Sample Size Considerations for Pilot Studies

Introduction

Pilot studies are small studies conducted before a larger definitive investigation. Their purpose is usually not to provide a definitive test of treatment efficacy. Instead, pilot studies help investigators determine whether the proposed study is feasible, operationally workable, and sufficiently well understood to justify a larger trial.

Examples of pilot-study objectives include determining whether patients can be recruited at the anticipated rate, estimating the proportion of participants who complete an intervention, evaluating adherence, assessing the feasibility of a new endpoint, estimating variability, testing data-collection procedures, and identifying practical problems with the study protocol.

Because the objective is different from that of a definitive clinical trial, the sample size calculation should also be different.

What Is a Pilot Study?

The term pilot study is used for several related designs. The exact terminology varies across clinical research settings, but useful distinctions include:

These categories can overlap. A study may simultaneously evaluate recruitment feasibility, intervention acceptability, and preliminary outcome variability.

The important point is that the statistical objective should be explicit. The sample size should follow from that objective rather than from the label "pilot."

Why Pilot Studies Are Usually Small

A pilot study is intentionally limited in size because its primary purpose is to answer questions about whether and how a larger study should be conducted.

If a pilot study is designed to determine whether a definitive trial can recruit participants at an acceptable rate, it may be unnecessary to enroll hundreds or thousands of subjects merely to obtain a precise estimate of treatment efficacy.

Similarly, if the purpose is to determine whether participants can complete a new assessment procedure, the study may need only enough participants to expose the research team to the operational challenges associated with that procedure.

The First Question: What Is the Pilot Trying to Estimate?

Before selecting a sample size, identify the primary pilot parameter. Common examples include:

• Recruitment rate
• Retention rate
• Intervention adherence
• Completion rate
• Proportion of participants with an evaluable endpoint
• Frequency of adverse events
• Standard deviation of a continuous endpoint
• Intraclass correlation coefficient
• Correlation between repeated measurements
• Feasibility of a diagnostic or laboratory procedure

Once the primary parameter is identified, the sample size can be linked directly to the information required about that parameter.

Feasibility vs. Efficacy

One of the most important distinctions in pilot-study planning is the difference between feasibility and efficacy.

Suppose investigators are preparing a randomized trial of a new intervention. They may want the pilot study to determine whether at least 80% of randomized participants complete the treatment period.

The pilot's primary question is therefore:

That is fundamentally different from asking whether the intervention improves the primary clinical endpoint compared with control.

The definitive trial may require hundreds of participants to answer the efficacy question, while a much smaller pilot may be sufficient to evaluate the feasibility question.

Should a Pilot Study Be Powered for Efficacy?

In many situations, no.

A pilot study that is too small to provide reliable evidence about efficacy should not be presented as though it were a definitive hypothesis test. A nonsignificant treatment comparison in a small pilot does not establish that the treatment is ineffective.

Conversely, a statistically significant result from a small pilot can be unstable and may exaggerate the apparent treatment effect.

The treatment-effect estimate and confidence interval can still be useful descriptively, but the primary purpose of the pilot should remain clearly defined.

Sample Size Based on a Feasibility Proportion

Suppose a pilot study is designed to estimate a feasibility proportion \(p\), such as the proportion of participants who successfully complete an intervention. Using a simple normal approximation, the approximate sample size for a desired confidence interval half-width \(d\) is:

This has the same mathematical structure as the sample size formula for any binomial proportion.

For a 95% confidence interval:

The formula is useful for understanding how precision changes with sample size, although exact or score-based binomial methods may be preferable for small pilot studies.

Worked Example: Estimating a Completion Rate

Suppose investigators expect that approximately 85% of enrolled participants will complete a 12-week intervention. They want the pilot study to estimate the completion rate with a confidence interval half-width of approximately 10 percentage points using a 95% confidence level.

The planning assumptions are:

Using the approximation:

Therefore:

Rounding upward gives:

Approximately 49 evaluable participants would therefore provide the desired approximate precision under this simple normal-approximation calculation.

Inflating for Non-Evaluable Participants

Pilot studies often have a substantial proportion of participants who do not provide the final feasibility endpoint. Suppose the investigators anticipate that 15% of participants will be lost to follow-up or otherwise become non-evaluable.

The enrollment target becomes:

where \(r\) is the expected non-evaluable rate.

For the example:

Thus:

The pilot would therefore plan to enroll approximately 58 participants to obtain about 49 evaluable participants.

Recruitment Feasibility as the Primary Endpoint

Recruitment is one of the most common reasons for conducting a pilot study. Suppose a definitive trial requires an average recruitment rate of at least 0.5 participants per site per month.

A pilot may be designed to determine whether the observed recruitment rate is compatible with that target.

For example, investigators might monitor:

• Number of patients screened
• Number eligible
• Number consented
• Number randomized
• Number randomized per site per month
• Screen-failure rate
• Time required to recruit the target sample

In this situation, the pilot sample size should be selected based on how much information is needed to make the recruitment decision.

Worked Example: Recruitment Feasibility

Suppose a definitive multicenter trial requires an average recruitment rate of at least 0.40 participants per site per month. The pilot will evaluate 6 sites for 4 months.

The expected number of participants under the target recruitment rate is:

Thus, approximately 10 randomized participants would be expected if the target recruitment rate were achieved.

The pilot might therefore establish a progression criterion such as:

at the end of the recruitment period, together with secondary criteria concerning site activation, screening efficiency, and retention.

The exact decision rule should be based on the operational requirements of the definitive trial rather than on an arbitrary statistical convention.

Recruitment Rate and Exposure Time

Recruitment is often better represented as a rate than as a simple proportion. For example:

where:

This allows recruitment performance to be compared across pilot designs with different numbers of sites or different recruitment durations.

Retention and Follow-Up

Retention is another important pilot-study objective. Suppose a definitive study requires at least 85% of randomized participants to complete the primary endpoint.

A pilot may estimate the actual retention rate and identify operational reasons for withdrawal.

Important quantities include:

• Withdrawal before treatment
• Withdrawal during treatment
• Loss to follow-up
• Missed visits
• Missing primary endpoint measurements
• Early treatment discontinuation

These may need to be analyzed separately because a participant who discontinues treatment but provides the primary endpoint is different from a participant who provides no endpoint data.

Estimating a Binary Feasibility Rate

Suppose the pilot observes 17 successful completions among 20 participants:

The point estimate is 85%, but the uncertainty around this estimate can be substantial because the sample is small.

This illustrates an important feature of pilot studies: pilot estimates are often imprecise by design. The objective is to obtain enough information to make an informed decision about the definitive study, not necessarily to estimate every parameter with high precision.

Using a Conservative Proportion

If the expected feasibility proportion is uncertain, a conservative value can be used for planning. For the binomial variance:

the maximum occurs at:

Therefore, using \(p=0.50\) produces a conservative sample size under the normal approximation when there is little prior information.

For example, with a 95% confidence level and ±10% precision:

Thus approximately:

would be required under this approximation.

Why the 30-Per-Arm Rule Is Not a Universal Rule

A commonly repeated recommendation is that pilot studies should enroll approximately 30 participants per treatment arm. This can sometimes be a reasonable pragmatic choice, but it is not a universal statistical requirement.

The appropriate sample size depends on the purpose of the pilot.

The number 30 is therefore better viewed as a possible pragmatic convention than as a statistical law.

Estimating a Standard Deviation

Pilot studies are frequently used to estimate the variability of a continuous endpoint before planning a definitive trial.

Suppose the primary endpoint is a continuous biomarker and the pilot provides an estimate:

The estimated standard deviation can then be used as an input to the definitive sample size calculation.

For example, a two-group comparison of means may use:

where:

• \(\sigma\) = assumed standard deviation
• \(\Delta\) = clinically meaningful treatment difference
• \(\alpha\) = type I error rate
• \(\beta\) = type II error rate

A pilot may therefore provide information about \(\sigma\), but the pilot estimate should not automatically be treated as the definitive population standard deviation.

Why Pilot Variance Estimates Can Be Unstable

Variance estimates from small samples can be highly uncertain. If the pilot contains only a small number of participants, the observed standard deviation may differ substantially from the true population standard deviation.

This can have an important effect on the definitive trial sample size because:

A modest increase in the assumed standard deviation can therefore produce a substantial increase in the required sample size.

Pilot data should generally be interpreted as one source of information about variability, alongside previous studies, external literature, historical data, and clinical knowledge.

Using Pilot Data to Estimate the Standard Deviation

Suppose a pilot study of 25 participants produces:

Suppose the definitive study aims to detect a clinically meaningful difference of 10 units with 90% power at a two-sided 5% significance level. Using the normal approximation:

This gives approximately:

or approximately 136 evaluable participants in total before accounting for dropout.

Estimating an Intraclass Correlation Coefficient

In cluster-randomized or repeated-measures studies, a pilot may be used to estimate the intraclass correlation coefficient (ICC).

The ICC describes the degree of similarity among observations within the same cluster. A common design-effect expression is:

where:

• \(m\) = average cluster size
• \(\rho\) = ICC

The definitive sample size can then be inflated approximately by the design effect.

However, ICC estimates from small pilot studies can also be unstable. It is often prudent to evaluate a range of plausible ICC values when planning the definitive trial.

Pilot Studies for Event Rates

Pilot studies may also provide information about the frequency of relatively uncommon events. For example, investigators may want to estimate the proportion of participants who experience a particular adverse event or protocol-defined endpoint.

Suppose 3 of 40 participants experience an event:

The observed event rate is 7.5%. However, with only 40 participants, the confidence interval around this estimate will be wide.

This is especially important when investigators attempt to use a small pilot to make strong conclusions about rare-event rates.

Safety Events and the Rule of Three

A useful approximation for planning around rare events is sometimes called the rule of three. If no events are observed in \(n\) independent participants, the approximate upper 95% confidence bound for the event probability is:

For example, if 30 participants are observed and no events occur:

Thus, observing zero events in 30 participants does not establish that the event probability is zero. The approximate upper 95% bound is still around 10%.

To have an upper bound near 3%, approximately:

participants with zero observed events would be needed.

Progression Criteria

Many pilot and feasibility studies use explicit progression criteria to determine whether the definitive trial should proceed.

These criteria can be defined before the pilot begins.

For example:

These thresholds are only illustrative. The actual thresholds should be based on the operational requirements of the definitive study.

For example, a study may decide:

• Green: proceed without modification
• Amber: modify procedures and reassess feasibility
• Red: do not proceed to the definitive trial

Sample Size for a Go/No-Go Feasibility Decision

Sometimes the pilot is not primarily designed to estimate a proportion precisely. Instead, the objective is to determine whether a feasibility proportion exceeds a minimum acceptable threshold.

For example:

where 70% is the minimum acceptable completion rate.

The study could use an exact binomial design to identify a minimum number of successes required to proceed.

This is different from simply estimating the completion rate with a specified confidence interval width.

Worked Example: A Go/No-Go Rule

Suppose a pilot enrolls 30 participants and defines successful completion as completion of the intervention and the primary feasibility assessment.

The investigators decide that the definitive trial is considered feasible if at least 24 of 30 participants complete the pilot:

The progression rule is therefore:

where \(X\) is the number of successful completions.

The statistical operating characteristics of this rule should be evaluated under plausible completion rates before the study begins.

For example, investigators may calculate:

• Probability of progressing if the true completion rate is 80%
• Probability of progressing if the true completion rate is 70%
• Probability of stopping if the true completion rate is 60%

This provides a more transparent framework for interpreting the pilot than simply saying that 30 participants were chosen because "30 is standard."

One-Arm Pilot Studies

Many feasibility studies use a single-arm design. For example, all participants may receive the investigational intervention and the study may evaluate whether the intervention can be delivered successfully.

A one-arm pilot may be appropriate when the primary objective is:

• Recruitment
• Retention
• Adherence
• Endpoint feasibility
• Procedure feasibility
• Operational workflow

A control group may be unnecessary if no comparative efficacy question is being addressed.

Randomized Pilot Studies

Randomization can nevertheless be useful in a pilot study. A randomized pilot may help determine whether the randomization process itself is feasible and whether the procedures associated with both treatment arms can be implemented successfully.

Potential objectives include:

• Testing allocation procedures
• Evaluating recruitment into both treatment groups
• Assessing treatment adherence
• Testing blinding procedures
• Evaluating endpoint collection
• Estimating variability within each arm

If treatment arms are included, the sample size should account for the allocation ratio and the amount of information required in each group.

Equal Allocation

Suppose a randomized pilot uses a 1:1 allocation ratio. If the desired total sample size is 40 participants:

This may be operationally convenient, but it does not automatically make the pilot adequately powered to detect a treatment effect.

The primary purpose of the randomization should therefore be clearly described.

Unequal Allocation

Some pilot studies may use unequal allocation. For example, a 2:1 design might provide more exposure to the investigational intervention while maintaining a smaller control group.

If the total sample size is 45:

The allocation ratio should be justified by the study objective and practical considerations.

When a Formal Power Calculation Is Appropriate

A conventional power calculation can be appropriate if the pilot has a genuine hypothesis-testing objective. For example, an investigator might want to test whether a treatment produces a large preliminary effect relative to a control.

The standard two-group sample size relationship for a continuous endpoint can be written approximately as:

However, this should be used only when the pilot's scientific objective actually requires that level of hypothesis-testing structure.

Why Pilot Studies Should Not Usually Be Used to "Prove" Efficacy

Small pilot studies have limited ability to distinguish moderate treatment effects from random variation. This creates several problems.

First, the study may have low statistical power. Second, treatment-effect estimates can be unstable. Third, statistically significant results can overestimate the underlying effect because large observed effects are more likely to reach significance in a small study.

Therefore, the pilot should generally be used to inform the design of the definitive study rather than to replace it.

Internal Pilot Designs

An internal pilot is a design in which early participants are included in the final analysis rather than treated as a separate preliminary study.

For example, investigators may use the first portion of the trial to reassess nuisance parameters such as:

• Standard deviation
• Event rate
• Dropout rate
• Intraclass correlation

The sample size may then be adjusted based on prespecified rules.

Internal pilots can be statistically efficient because participants contribute to the final trial rather than being discarded after the feasibility phase.

External Pilot vs. Internal Pilot

Accounting for Multiple Feasibility Objectives

A pilot study often has several objectives. For example:

• Recruitment ≥80%
• Retention ≥85%
• Adherence ≥80%
• Endpoint completion ≥90%

Different objectives may imply different sample sizes. The final sample size should generally be large enough to support the most demanding primary feasibility objective.

However, investigators should distinguish between the primary pilot objective and secondary descriptive objectives. Not every secondary measure needs its own independent sample size calculation.

Sample Size and Screening

The number of participants enrolled in a pilot is not necessarily the same as the number who must be screened.

Suppose investigators require 50 randomized participants and expect:

Then the approximate number of participants who must be screened is:

Thus, the pilot may require approximately 200 screening opportunities to obtain 50 eligible participants.

A Participant Flow Model

A useful planning framework is:

Suppose:

• 25% of screened participants are eligible
• 80% of eligible participants enroll
• 90% of enrolled participants are evaluable

The overall probability that a screened participant becomes evaluable is:

Therefore, approximately 18% of screened participants are expected to contribute an evaluable outcome.

If 50 evaluable participants are required:

Approximately:

screened participants would therefore be required under these assumptions.

Adaptive Feasibility Criteria

Some pilot studies use staged designs. For example, investigators may initially enroll 15 participants and review predefined feasibility criteria before continuing to 30.

This can be useful when the purpose is to identify major operational problems early.

A staged approach might be:

The stopping and continuation rules should be defined prospectively.

Bayesian Approaches to Pilot Sample Size

Bayesian methods can be particularly useful in pilot studies because prior information may be available from previous studies.

Suppose the completion probability is:

and a beta prior is specified:

After observing \(x\) successes among \(n\) participants, the posterior distribution is:

This framework allows previous evidence to be incorporated directly into the pilot analysis.

Bayesian decision criteria can then be expressed as probabilities such as:

meaning that the study would proceed if the posterior probability that the completion rate is at least 80% is at least 90%.

Why Small Pilot Studies Produce Wide Confidence Intervals

The standard error of a proportion is approximately:

Therefore:

Doubling the sample size does not halve the standard error. To reduce the standard error by approximately one-half, the sample size must increase by approximately four-fold.

This explains why a small pilot can provide useful directional information while still producing relatively wide confidence intervals.

Sample Size and Precision

Suppose a feasibility rate is approximately 80%. Under the normal approximation:

These values illustrate why a pilot with 20 or 30 participants should generally not be interpreted as providing a highly precise estimate of a feasibility proportion.

Sample Size for Continuous Feasibility Outcomes

Not all pilot outcomes are binary. Suppose investigators want to estimate the mean time required to complete a study procedure.

For a continuous outcome with standard deviation \(\sigma\), the approximate sample size for a confidence interval half-width \(d\) is:

For example, if the expected standard deviation is 12 minutes and the desired 95% confidence interval half-width is 4 minutes:

Thus:

participants would be required under this approximation.

Pilot Studies and Missing Data

Pilot studies should specifically monitor missing-data mechanisms. For example, investigators may discover that participants are consistently unable to complete an assessment at a particular visit.

This can reveal problems that would not be apparent from the overall retention rate.

Useful pilot metrics include:

• Percentage of scheduled assessments completed
• Percentage of assessments completed within the prespecified window
• Percentage of missing endpoint measurements
• Percentage of measurements requiring repeat collection
• Frequency of protocol deviations

These operational metrics may be more informative for planning the definitive study than a simple overall dropout percentage.

Common Sample Size Mistakes in Pilot Studies

1. Automatically using 30 participants per arm. There is no universal statistical requirement that every pilot contain 30 participants per treatment group.
2. Powering the pilot for the definitive efficacy endpoint without a clear reason. This can produce an unnecessarily large study while still failing to provide adequate power for the definitive trial.
3. Using the pilot to claim treatment efficacy. A small feasibility study is usually not designed to provide confirmatory efficacy evidence.
4. Ignoring the pilot's actual objective. Recruitment, retention, adherence, and variance estimation require different sample size approaches.
5. Ignoring non-evaluable participants. The number enrolled is not necessarily the number contributing to the primary feasibility endpoint.
6. Assuming pilot estimates are precise. Small studies naturally produce substantial uncertainty.
7. Using pilot variance estimates without considering uncertainty. A small pilot can substantially overestimate or underestimate the true standard deviation.
8. Failing to prespecify progression criteria. Post-hoc decisions about whether a study was feasible can introduce subjectivity.
9. Ignoring recruitment duration and site effects. Recruitment feasibility is often a rate and operational problem rather than a simple proportion.
10. Treating zero observed adverse events as evidence of zero risk. Rare-event probabilities can remain substantial even when no events occur in a small pilot.

A Practical Pilot Study Sample Size Workflow

A useful workflow is:

1. Define the purpose of the pilot.
2. Identify the primary feasibility or estimation parameter.
3. Specify the minimum acceptable level of feasibility, if applicable.
4. Determine whether the objective is estimation or a go/no-go decision.
5. Specify the desired confidence level or decision error rates.
6. Choose an appropriate statistical method.
7. Calculate the number of evaluable participants required.
8. Inflate for anticipated missing or non-evaluable observations.
9. Translate the enrollment target into the number of participants who must be screened.
10. Consider recruitment rate and study duration.
11. Prespecify progression or stopping criteria.
12. Document how pilot results will inform the definitive study.

What the Pilot Protocol Should State

The statistical section of a pilot protocol should clearly explain why the chosen sample size is appropriate. At minimum, it should identify:

• Primary pilot objective
• Primary feasibility endpoint
• Expected feasibility rate or variance, if applicable
• Desired precision or decision criterion
• Confidence level or type I error, when applicable
• Required evaluable sample size
• Expected non-evaluable rate
• Total planned enrollment
• Expected screening-to-enrollment ratio
• Recruitment period
• Progression criteria
• How pilot findings will inform the definitive trial

The protocol should also explicitly state whether the pilot is intended to provide formal efficacy inference or whether treatment outcomes will be summarized descriptively.

Sample Size Planning in R

The simple normal-approximation calculation for a feasibility proportion can be implemented directly in R.

alpha <- 0.05
z <- qnorm(1 - alpha / 2)

p <- 0.85
d <- 0.10

n.evaluable <- ceiling(
  z^2 * p * (1 - p) / d^2
)

n.evaluable

The result is approximately:

n.evaluable
# 49

To account for a 15% non-evaluable rate:

non_evaluable <- 0.15

n.enroll <- ceiling(
  n.evaluable / (1 - non_evaluable)
)

n.enroll
# 58

Thus, approximately 58 participants would be enrolled to obtain approximately 49 evaluable participants under these assumptions.

Planning Screening Requirements in R

Suppose 25% of screened participants are expected to be eligible, 80% of eligible participants are expected to enroll, and 90% of enrolled participants are expected to provide an evaluable endpoint.

n.evaluable <- 50

eligibility.rate <- 0.25
enrollment.rate <- 0.80
evaluable.rate <- 0.90

overall.rate <- eligibility.rate *
                enrollment.rate *
                evaluable.rate

n.screen <- ceiling(
  n.evaluable / overall.rate
)

n.screen

The result is:

n.screen
# 278

Therefore, approximately 278 participants would need to be screened to obtain 50 evaluable participants under these assumptions.

Planning a Definitive Trial From Pilot Data

One of the most valuable functions of a pilot study is to improve the assumptions used for the definitive trial.

For example, suppose the original definitive-trial planning assumptions were:

The pilot suggests that the definitive trial may need:

• A larger sample because variability is higher than expected
• Additional enrollment because retention is lower
• Protocol modifications to improve adherence

This is precisely the type of information a pilot is intended to generate.

Sample Size Inflation Should Be Transparent

Suppose the definitive calculation requires 120 evaluable participants and the pilot suggests a 15% non-evaluable rate. The enrollment target is:

Thus:

The protocol should distinguish clearly between:

and:

This makes the assumptions transparent and prevents confusion during trial conduct.

When the Pilot Sample Size Should Be Larger

A larger pilot may be justified when:

• The primary feasibility parameter is rare
• Several feasibility questions must be evaluated simultaneously
• Recruitment must be evaluated across many sites
• Site-to-site variability is important
• A reliable variance estimate is needed
• A clinically important safety event is being monitored
• The pilot will be used to estimate a parameter for the definitive trial
• A formal progression decision requires substantial statistical assurance

In these situations, the study may appropriately exceed the size of a very small traditional pilot.

When the Pilot Sample Size Should Be Smaller

A smaller pilot may be appropriate when:

• The objective is primarily operational
• The procedure can be tested in a small number of participants
• The endpoint is straightforward to collect
• The pilot is designed to identify major implementation problems
• Resources are limited and the decision is qualitative
• Early stopping for obvious feasibility problems is possible

The goal is not to maximize the sample size. It is to maximize the information relevant to the pilot's decision.

Statistical Interpretation of Pilot Results

Pilot results should generally be interpreted with appropriate uncertainty. For example, observing a completion rate of 90% in 20 participants does not mean that the true completion rate is exactly 90%.

Similarly, observing a treatment difference of 12 units in a small randomized pilot does not establish that the true treatment effect is 12 units.

The pilot should therefore report:

• Point estimates
• Confidence intervals where appropriate
• Participant flow
• Missing-data patterns
• Protocol deviations
• Recruitment metrics
• Adherence and retention
• Operational problems

Why Confidence Intervals Are Often More Useful Than P-Values

For many pilot objectives, the important question is not whether a null hypothesis can be rejected. Instead, investigators want to know how large or uncertain the observed feasibility parameter is.

For example:

This provides much more information for planning than simply reporting:

A confidence interval communicates the uncertainty surrounding the estimate and can be directly compared with operational thresholds.

Powering a Pilot for the Definitive Trial

A common misconception is that a pilot should be powered at 80% or 90% to detect the expected treatment effect of the definitive trial. This is often inefficient.

If the pilot has the same sample size as the definitive trial required for efficacy, then it may no longer function primarily as a pilot.

The appropriate approach is usually to define:

1. The feasibility question
2. The minimum information required to answer that question
3. The decision that will follow from the pilot

The sample size should then be designed around those requirements.

Summary Table of Sample Size Approaches

Common Practical Questions

Does every pilot need a formal sample size calculation?

Not necessarily. A purely operational feasibility exercise may use a pragmatic sample size if the rationale is transparent. However, the sample size should still be justified in relation to the pilot's objectives.

Is 30 participants enough?

Sometimes, but not universally. Thirty participants may be reasonable for certain operational pilots, but it does not guarantee adequate precision for a feasibility proportion or variance estimate.

Should pilot participants be included in the definitive trial?

That depends on the design. External pilots are usually separate from the definitive trial, whereas internal pilot participants can often contribute to the final analysis.

Can pilot data be used for the definitive sample size?

Yes. Pilot data can inform estimates of variability, event rates, retention, adherence, and other nuisance parameters. However, small pilot estimates can be unstable and should generally be considered alongside external evidence.

Should a pilot have a control group?

Not necessarily. If the primary objective is feasibility, a control group may not be required. Randomization and a control group become more useful when the pilot is also testing trial procedures or collecting preliminary comparative outcome data.

Final Practical Workflow

A practical framework for pilot-study sample size planning is:

1. Define the pilot question.
2. Identify the primary feasibility parameter.
3. Decide whether the objective is estimation or a decision.
4. Specify the desired precision or progression threshold.
5. Choose an appropriate statistical method.
6. Calculate the required number of evaluable participants.
7. Inflate for non-evaluable participants.
8. Account for screening and eligibility rates.
9. Account for recruitment rate and study duration.
10. Prespecify progression criteria.
11. Plan how pilot estimates will inform the definitive study.
12. Interpret pilot estimates with appropriate uncertainty.

Summary

Sample size planning for pilot studies is fundamentally different from sample size planning for definitive clinical trials. The pilot is usually designed to answer questions about feasibility, implementation, recruitment, retention, adherence, endpoint collection, safety, or nuisance parameters.

The correct sample size therefore depends on the pilot's primary objective. A pilot designed to estimate a feasibility proportion can use a precision-based binomial calculation:

A pilot designed around a go/no-go decision may instead use an exact binomial decision rule. A pilot intended to estimate a continuous mean can use:

Pilot studies can also provide important information about standard deviations, event rates, ICCs, recruitment rates, adherence, and retention. These estimates can subsequently improve the assumptions used for the definitive trial.

However, pilot estimates are often imprecise because pilot studies are intentionally small. They should therefore not be treated as though they were definitive estimates of treatment efficacy or population parameters.

The most useful pilot studies are those that are explicitly linked to a decision: whether to proceed, modify the protocol, redesign the intervention, change the endpoint, alter recruitment procedures, or abandon the definitive study.

References

Arain, M., Campbell, M.J., Cooper, C.L. & Lancaster, G.A. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology, 10, 67.
Billingham, S.A.M., Whitehead, A.L. & Julious, S.A. (2013). An audit of sample sizes for pilot and feasibility trials being undertaken in the United Kingdom registered in the United Kingdom Clinical Research Network database. BMC Medical Research Methodology, 13, 104.
Eldridge, S.M., Lancaster, G.A., Campbell, M.J., Thabane, L., Hopewell, S., Coleman, C.L. & Bond, C.M. (2016). Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS ONE, 11(3), e0150205.
Hertzog, M.A. (2008). Considerations in determining sample size for pilot studies. Research in Nursing & Health, 31(2), 180–191.
Julious, S.A. (2005). Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics, 4(4), 287–291.
Lancaster, G.A., Dodd, S. & Williamson, P.R. (2004). Design and analysis of pilot studies: recommendations for good practice. Journal of Evaluation in Clinical Practice, 10(2), 307–312.
Leon, A.C., Davis, L.L. & Kraemer, H.C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629.
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L.P., Robson, R., Thabane, M., Giangregorio, L. & Goldsmith, C.H. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10, 1.
Whitehead, A.L., Julious, S.A., Cooper, C.L. & Campbell, M.J. (2016). Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Statistical Methods in Medical Research, 25(3), 1057–1073.