Internal Pilot Designs With Variance Re-Estimation

Why Re-estimate Variance During the Trial?

For a continuous outcome, the sample size required to detect a specified treatment difference depends on the outcome variance. Before a trial begins, that variance is usually uncertain: it may be drawn from earlier studies, a small external pilot, or clinical judgment. If the actual variability is larger than anticipated, a fixed design may have less power than planned. An internal pilot uses outcomes from an initial group of trial participants to update the variance estimate and, if needed, the planned total sample size.

The defining feature is that the pilot participants remain part of the trial’s final analysis. The adaptation is a prespecified rule based on their accrued outcomes, not a separate study whose participants are discarded. This tutorial considers an unblinded internal pilot for a continuous outcome, with equal allocation and a common within-arm variance. It focuses on estimating variance and updating sample size; the calculation alone does not establish the validity of every possible final test or confidence interval.

Model and Variance Estimate

Suppose the primary outcome is measured once per participant, with larger values interpreted according to the trial’s outcome scale. For the planning calculation, assume independent participant outcomes, normal distributions in the two treatment arms, and a common variance \(\sigma^2\). Let the treatment means be \(\mu_1\) and \(\mu_0\), and let \(\delta>0\) be the treatment difference the design is intended to detect. Equal allocation means the planned total sample is twice the per-arm sample size.

At the internal-pilot review, suppose \(m\) participants have completed the outcome assessment in each arm. Write their sample means as \(\bar{x}_1\) and \(\bar{x}_0\), and their sample variances as \(s_1^2\) and \(s_0^2\). The unblinded pooled within-arm variance estimate is

Each arm’s deviations are calculated around its own sample mean. Consequently, a difference between the treatment means is not counted as within-arm variability. Under independent normal outcomes with the same variance in both arms, this estimator is unbiased for \(\sigma^2\), and its degrees of freedom are \(df=2m-2\). This is an estimate of the variance, in squared outcome units; the estimated standard deviation is \(s_{\mathrm{pilot}}=\sqrt{s_{\mathrm{pilot}}^2}\).

The assumptions matter. If the treatment arms have materially different variances, a pooled estimate can obscure that difference. If outcome variability changes over calendar time, the initial participants may not represent later participants. Strong skewness or influential outliers can also make a small pilot variance unstable. These concerns should be examined using prespecified summaries and clinical knowledge, not resolved by choosing a convenient variance estimate after seeing the results.

Deriving the Sample-Size Update

For a balanced two-arm comparison, if each arm has \(n\) participants and the common variance is \(\sigma^2\), the variance of the difference in sample means is \(2\sigma^2/n\). Under a normal-approximation planning calculation, a two-sided test at significance level \(\alpha\) has a standardized rejection threshold of approximately \(z_{1-\alpha/2}\) under the null. Under a target difference \(\delta\), the standardized mean is \(\delta\sqrt{n/(2\sigma^2)}\). Requiring approximate power \(1-\beta\) gives the familiar planning equation

Rearranging first gives \(\delta\sqrt{n/(2\sigma^2)}\approx z_{1-\alpha/2}+z_{1-\beta}\). Squaring both sides and solving for \(n\) yields the required per-arm sample size. The internal-pilot rule substitutes the pilot variance estimate for the unknown \(\sigma^2\), then rounds up because a fraction of a participant cannot be enrolled.

This is a normal-approximation formula. For small samples or designs whose planned analysis uses a \(t\) statistic, a more accurate planning calculation can solve for sample size using the corresponding noncentral \(t\) distribution. The sample-size rule should match the trial’s specified primary analysis and planning convention; the approximation should not be treated as an exact power guarantee.

The update must also respect operational constraints. If \(m\) participants per arm have already accrued, the revised per-arm total should not be below \(m\). A minimum or maximum total may also be prespecified. Such bounds make the rule feasible, but a maximum below the calculated requirement means the target power is not achieved under the planning assumptions.

Applying Bounds and Interpreting the Result

A sample-size update is a decision rule, not just a formula. Before the review, the protocol should specify what happens if the estimate implies fewer participants than have already enrolled, or a total above the available cap. A simple bounded rule sets the final per-arm target to the calculated requirement subject to a floor and cap. The floor must preserve the accrued sample; the cap may limit feasibility but can reduce power below its target.

Diagnostics, Alternatives, and Limitations

The internal-pilot estimate can be noisy, especially when few participants contribute to it. Under the common-variance normal model, the relative standard deviation of the sample variance is \(\sqrt{2/(2m-2)}\). For \(m=8\) per arm this is \(\sqrt{2/14}=0.378\), or about 37.8%. Thus, a marked change in the estimated variance is not surprising in a small pilot and should not automatically be interpreted as evidence that the initial planning value was wrong.

Useful diagnostics include checking the two arm-specific variances, examining outcome distributions and influential observations, confirming that the same outcome definition and measurement process were used, and reviewing whether accrual timing or participant characteristics could make later variability differ. A substantial difference between arm-specific variances challenges the common-variance assumption; a single extreme value may have a large impact on the estimate. These diagnostics do not justify discretionary changes to the prespecified rule. Any departure should be documented and assessed as a design change.

The main alternatives answer related but distinct planning questions. A fixed design uses a variance value chosen before enrollment and avoids a data-dependent sample-size update, but may be underpowered if that value is too small. An external pilot can inform the variance without using main-trial outcomes, but its participants may differ from the trial population or measurement conditions. An internal pilot uses directly relevant accrued outcomes and retains participants, but makes the final sample size depend on trial data.

That dependence is an important limitation. Substituting an estimated variance into a sample-size formula does not by itself prove that the final test has its nominal type I error rate or that a reported interval has its intended coverage. In particular, the final inferential procedure must be evaluated for the complete adaptive design, including the rule determining sample size. The planning calculation also does not guarantee the target power: it relies on the variance estimate, target difference, model assumptions, and chosen approximation.

An internal pilot is most useful when the outcome variance is genuinely uncertain, the review occurs early enough to change recruitment, and the trial can accommodate the possible range of sample sizes. Its scientific value depends on more than the arithmetic: the review timing, variance estimator, target effect, bounds, and final analysis all need to be specified in a way that remains interpretable after the data-dependent update.