External Pilot Data Versus Internal Pilot Data

Why the Source of Pilot Data Matters

Sample-size planning often requires nuisance parameters, such as the outcome variance, that are not known precisely before a trial begins. A pilot can provide information about these quantities, but the source of the data affects how well the estimate applies to the definitive trial and whether the pilot participants can be included in its final analysis.

An external pilot is conducted separately from the definitive trial. Its participants are not ordinarily members of that trial’s analysis population. An internal pilot consists of the initial participants enrolled under the definitive trial protocol. If the trial proceeds and the data meet prespecified requirements, these participants can remain in the final analysis. The key distinction is not whether the pilot was conducted at the same hospital or by the same research team; it is whether the pilot participants were enrolled as part of the definitive trial.

Definition: Transportability is the extent to which an estimate from pilot data is relevant to the definitive trial’s target population, outcome, measurement procedures, and conduct. Similar-looking studies do not necessarily provide transportable estimates if these features differ.

For illustration, suppose a two-arm trial compares mean outcomes, allocates participants equally, and targets a difference of \(\Delta\) outcome units. Under a normal approximation, assuming independent observations and a common within-arm variance \(\sigma^2\), a usual planning formula for the required number per group is

n \approx \frac{2\sigma^2(z_{1-\alpha/2}+z_{1-\beta})^2}{\Delta^2}.

Here \(\alpha\) is the two-sided significance level, \(1-\beta\) is the target power, and \(z_q\) is the \(q\)-quantile of the standard normal distribution. In practice, a pilot-based estimate such as \(s^2\) may be substituted for \(\sigma^2\); the calculated \(n\) is rounded up. This approximation presumes that the chosen target difference is clinically meaningful and specified independently of an unusually large observed pilot effect. The formula is a planning tool, not a guarantee of power if its assumptions or inputs are poor.

Using an External Pilot Estimate

An external pilot can inform planning before the definitive trial starts. Since its data are separate, the trial team can review them without creating an interim analysis of the definitive trial’s accumulating outcomes. That separation can simplify operational and statistical planning. Its weakness is that the pilot estimate may not reflect the definitive trial’s population, measurement process, or conduct.

Worked Example: Planning From an External Pilot

Suppose a hypothetical external pilot measures a continuous outcome in two groups, with 16 participants per group. The sample standard deviations are \(s_1=9.2\) units and \(s_2=10.7\) units. The planned definitive trial will use equal allocation, a two-sided \(\alpha=0.05\), 80% power, and a target difference of \(\Delta=5\) units. Assume, for this calculation, that the pilot and definitive trial use compatible outcome definitions and that a common variance is reasonable.

The pooled pilot variance is

s_p^2=\frac{(16-1)s_1^2+(16-1)s_2^2}{16+16-2} =\frac{15(9.2^2)+15(10.7^2)}{30} =\frac{1269.6+1717.35}{30} =99.565\ \text{units}^2.

Using \(z_{0.975}=1.960\) and \(z_{0.80}=0.8416\), the estimated group size is

n\approx \frac{2(99.565)(1.960+0.8416)^2}{5^2} =\frac{2(99.565)(7.849)}{25} =62.52.

Rounding up gives a target of 63 participants per group, or 126 in total, before any separate allowance for loss of outcome data. This is a conditional planning result: it relies on the common-variance and transportability assumptions, as well as the specified target difference and power.

A pilot does not establish that its variance estimate equals the variance in the definitive trial. Before using it, assess whether the pilot and trial align on the eligible population, outcome and assessment time, measurement instrument, treatment context, and relevant procedures. Differences in these features can alter the outcome distribution even when the endpoint has the same name. An external pilot is especially useful when that alignment is credible and when the estimate can be obtained early enough to inform trial planning.

Internal Pilots and Reuse of Participants

An internal pilot uses early participants enrolled under the definitive trial’s protocol. If they satisfy the trial’s eligibility, measurement, and data-quality rules, their observed outcomes can contribute both to a prespecified nuisance-parameter estimate and to the final analysis. Reuse avoids treating the pilot as a separate group that must be recruited in addition to the definitive sample.

That benefit depends on protocol consistency. The endpoint definition, assessment schedule, eligibility criteria, treatment procedures, and data-handling rules should remain compatible between the internal-pilot stage and the remainder of the trial. If a substantial change is made, the early participants may no longer represent the same target population or measurement process. Whether their data can still be included then requires a prespecified statistical and clinical justification; it is not settled merely by calling the first stage an internal pilot.

Worked Example: Updating the Target With an Internal Pilot

Consider a hypothetical trial with the same planning assumptions as above: two-sided \(\alpha=0.05\), 80% power, and a target difference of 5 units. After 30 participants per group have completed the outcome assessment, a prespecified, appropriate blinded procedure for estimating the common within-group variance gives \(s^2=144\ \text{units}^2\). Assume the protocol remains unchanged, all 60 participants have usable outcomes, and the design permits the revised target.

The updated group size is

n\approx \frac{2(144)(1.960+0.8416)^2}{5^2} =\frac{2(144)(7.849)}{25} =90.42.

The revised target is therefore 91 per group. Since 30 per group have already been enrolled and are retained in the analysis, the trial needs \(91-30=61\) additional participants per group: 122 more participants, not 182 more. The final target is 182 participants in total. This arithmetic assumes no sample-size cap, no additional loss-to-follow-up allowance, and no reason to exclude the internal-pilot outcomes.

The estimate in this example is explicitly from a procedure intended to estimate the within-group variance. Simply pooling all blinded outcomes and calculating their variance is not automatically equivalent. With equal allocation and treatment means \(\mu-\Delta/2\) and \(\mu+\Delta/2\), the variance of the combined outcome distribution is

\operatorname{Var}(Y) =E\{\operatorname{Var}(Y\mid G)\}+\operatorname{Var}\{E(Y\mid G)\} =\sigma^2+\frac{\Delta^2}{4},

where \(G\) denotes treatment group. The second term arises from the difference in group means. A blinded variance procedure must therefore be chosen and justified for the design; a raw combined variance can include treatment-effect variation as well as within-group variation.

How the Two Sources Compare

Consideration	External pilot	Internal pilot
Participants	Separate from the definitive trial	Enrolled as the initial participants in the definitive trial
Relevance to trial conduct	Depends on similarity of population, protocol, and measurement	Often closely aligned if the protocol is maintained
Use in final analysis	Not ordinarily included	Can be included if prespecified conditions are met
Timing and operations	Can inform planning before the definitive trial starts	Requires a planned review point and operational procedures during the trial
Key concern	Limited transportability	Valid estimation, data handling, and preservation of the planned trial’s inferential properties

The choice is not simply “external is safer” or “internal is more accurate.” An external pilot avoids using outcomes from the definitive trial for its sample-size decision, but it may be a poor match to the trial. An internal pilot can provide more directly relevant data and reuse participants, but its adaptation must be designed in advance. A blinded sample-size decision, an operational firewall, and a method for maintaining the final test address related concerns; none makes an unsuitable variance estimator or incompatible protocol automatically acceptable. The final testing procedure must be appropriate for the particular adaptation, as emphasized in the earlier material on Type I error with nuisance-parameter re-estimation.

Derivation, Diagnostics, and Failure Modes

The planning formula also shows why variance transportability matters. Holding the target difference, significance level, and power fixed, the required group size is proportional to the variance:

\frac{n_{\text{new}}}{n_{\text{old}}} \approx \frac{\sigma^2_{\text{new}}}{\sigma^2_{\text{old}}}.

This follows directly because the factor \(2(z_{1-\alpha/2}+z_{1-\beta})^2/\Delta^2\) is unchanged. Thus, a variance that is 25% larger implies a sample-size target approximately 25% larger under this calculation. A pilot estimate that is too small can therefore produce a trial with less power than intended.

Before adopting either estimate, check the pilot’s design and data rather than treating the reported standard deviation as self-validating. Useful checks include whether the endpoint was measured in the intended units and at the intended time, whether the pilot includes the target range of participants, whether there were major protocol deviations, and whether a small number of extreme values dominate the variance. For an internal pilot, also verify that the estimator and review time were prespecified, the required outcomes are sufficiently complete, and the protocol has not changed in a way that undermines comparability.

Common warning signs include a pilot conducted in a narrowly selected population, a different assay or follow-up schedule, substantial changes to eligibility criteria, or an unplanned decision based on unblinded treatment differences. A sample-size calculation can be arithmetically correct and still be poorly justified if its inputs do not represent the definitive trial. A pilot is also usually too small to guarantee a stable variance estimate. The uncertainty of interim nuisance-parameter estimates requires separate consideration; using an internal pilot does not make that uncertainty disappear.

Common mistake: “The external pilot’s standard deviation is 8, so the definitive trial’s standard deviation is 8 and the resulting sample size is assured.” The correction is that the estimate is evidence for planning only to the extent that the pilot’s population, outcome, measurement, and conduct are transportable. State that assumption, examine plausible differences, and use a prespecified adaptation rule when updating the target from internal-pilot data.

In summary, an external pilot is attractive when it closely matches the planned trial and early, independent planning is valuable. An internal pilot is attractive when the trial needs information under its own protocol and reuse of early participants is important. In either case, the estimate, its target population, the sample-size rule, and the handling of pilot participants should be defined clearly in advance. Neither approach compensates for an inappropriate endpoint, a poorly chosen clinically relevant difference, or an analysis plan that does not support the adaptation.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

A separate pilot estimates an outcome SD of 8 units, while the definitive trial will use a different assessment instrument. What needs to be considered before using the estimate?
An internal pilot has 25 completed participants per group. A valid prespecified calculation gives a target of 80 per group. How many additional participants are required if pilot participants remain eligible for the final analysis?
With all other planning inputs fixed, the variance estimate doubles. Approximately how does the required group size change?
Why might the variance of all blinded outcomes combined be larger than the within-group variance in a two-arm trial?

Solutions. (1) Assess whether the instruments measure the outcome comparably and whether the resulting variance is transportable; a different instrument may change its scale or variability, so the external estimate should not be adopted without justification. (2) \(80-25=55\) additional participants per group, or 110 in total, assuming no other exclusions or cap. (3) The planning formula makes \(n\) proportional to the variance, so the target approximately doubles. (4) The combined variance includes both within-group variability and variability between the treatment-group means; with equal allocation and a mean difference \(\Delta\), it is \(\sigma^2+\Delta^2/4\), not just \(\sigma^2\).