Planning With Uncertain Pilot Estimates

Why Pilot Estimates Need Sensitivity Checks

A pilot study can help choose planning inputs, but its estimates are uncertain. A small pilot may produce a sample standard deviation that differs appreciably from the variability in the future study. An estimated effect can also be unstable: the observed difference may overstate or understate the effect a larger study would encounter. If a sample-size calculation treats either estimate as exact, its result can suggest more certainty than the evidence warrants.

A sensitivity analysis makes this uncertainty visible. Rather than relying on one pilot-based estimate, the planner specifies a plausible range for the effect and for variability, then checks how required sample size or achieved power changes across that range. The result is not a guarantee that the future study will meet its target. It shows what the design would deliver under the stated assumptions.

A sensitivity range is not automatically a confidence interval, and its endpoints do not automatically have assigned probabilities. The range should be justified using the pilot’s uncertainty, relevant earlier evidence, measurement knowledge, or scientific judgment. If the planner has not assigned probabilities to the values in the range, it is not appropriate to describe an average across them as the expected power.

A Mean-Test Planning Model

To make the calculations concrete, consider a two-sided, one-sample \(Z\) test of \(H_0:\mu=\mu_0\) against \(H_a:\mu\ne\mu_0\), with significance level \(\alpha\). Suppose independent observations come from a normal population with standard deviation \(\sigma\), treated as known for this planning model. Under a true mean difference \(\delta=\mu-\mu_0\), the standardized test statistic has a normal distribution with mean \(\delta\sqrt{n}/\sigma\) and variance 1. This calculation assumes the study’s observations and analysis are appropriately represented by this model; it does not account for clustering, missing data, or other design features.

Write \(z_{1-\alpha/2}\) for the standard normal quantile with cumulative probability \(1-\alpha/2\), and let \(\Phi\) denote the standard normal cumulative distribution function. The test rejects when the absolute value of its standardized statistic exceeds \(z_{1-\alpha/2}\). For a positive difference, define the noncentrality value \(\lambda=\delta\sqrt{n}/\sigma\). The exact power under this model is

The first term is the probability of crossing the rejection boundary in the direction of the true positive difference. The second is the probability of crossing the opposite boundary. Although that second probability is often small, it is part of the two-sided test’s power and should not be silently treated as zero when calculating exact power.

A common planning approximation focuses on the first term. To obtain target power \(1-\beta\), set \(\Phi(\lambda-z_{1-\alpha/2})\) equal to \(1-\beta\). Solving gives the familiar approximate sample-size formula:

This is a normal-approximation formula, not an exact solution to the two-sided power equation. Rounding its result up to a whole observation is a conventional conservative choice based on the approximation. It does not, by itself, prove that the preceding integer misses the target under the exact two-sided model; that claim requires calculating exact power at that integer.

How Uncertain Inputs Affect the Design

For a fixed sample size, significance level, and positive effect, power depends on the standardized signal \(\lambda=\delta\sqrt{n}/\sigma\). A smaller effect or greater variability reduces \(\lambda\), tending to reduce power. The approximate sample-size formula makes the corresponding planning consequence clear: required \(n\) increases with \(\sigma^2\) and decreases with \(\delta^2\). Thus, under a rectangular set of plausible values, the combination of the smallest positive effect and largest standard deviation is the conservative corner for planning.

That corner is conservative only if the combination is plausible. If scientific knowledge links effect and variability, independently pairing the smallest effect with the largest standard deviation may describe an impossible scenario. In that case, specify and compare plausible joint scenarios instead of mechanically combining endpoints.

Worked Planning Examples

A Short Justification of the Conservative Corner

For fixed \(n\), let \(c=z_{1-\alpha/2}>0\) and consider the power as a function of a positive \(\lambda\): \(g(\lambda)=\Phi(\lambda-c)+\Phi(-\lambda-c)\). Differentiating, using that the derivative of \(\Phi\) is the standard normal density \(\phi\), gives

For \(\lambda\ge 0\) and \(c>0\), \(|\lambda-c|\le \lambda+c\). The standard normal density is symmetric and decreases as the absolute value of its argument increases, so \(\phi(\lambda-c)\ge\phi(\lambda+c)\). Hence \(g'(\lambda)\ge0\): power does not decrease as the standardized signal increases. Since \(\lambda=\delta\sqrt{n}/\sigma\), lowering a positive \(\delta\) or raising \(\sigma\) cannot improve power. This justifies using the lowest plausible positive effect and highest plausible variability as the conservative corner when those values can plausibly occur together.

Common Mistakes and Practical Use

• Treating a pilot estimate as known. A calculation using one estimated standard deviation or effect is conditional on that input. Compare alternatives when pilot uncertainty is material.
• Calling a sensitivity range a probability statement. A range of powers does not say how likely each power is. That interpretation would require a probability model for the planning inputs.
• Combining incompatible endpoints. The worst numerical combination is useful only if it is scientifically plausible. Use coherent scenarios when effect and variability are related.
• Claiming that rounding up proves the previous integer fails. Rounding the normal-approximation result upward is a conventional conservative planning choice. To claim an integer misses the target under the two-sided \(Z\)-test model, evaluate its exact power formula.
• Interpreting planned power as a guarantee. Power is calculated under specified effect, variability, test, and sampling assumptions. The actual study may differ from them.

A useful report gives the model and target, explains the basis for the plausible ranges, and presents results at central and conservative scenarios. That makes the design’s dependence on uncertain inputs explicit and helps readers distinguish a robust plan from one that succeeds only under optimistic assumptions.