Two-Stage Designs With Pilot Data

Why Separate a Pilot From the Main Stage?

A design that performs well for one parameter value may be inefficient for another. This matters when pharmacometric parameters are uncertain and plausible values favor different sampling plans. A pilot stage can provide data about which parameter region is more credible; a second-stage design can then be selected to suit the updated uncertainty. The potential gain is not free: pilot observations use resources that could otherwise have gone to the main stage, and the pilot may be too noisy to change the choice reliably.

A two-stage design makes this trade-off explicit. Before collecting data, the investigator specifies a pilot design, a set of candidate main-stage designs, and a rule for selecting among those candidates after the pilot. The rule is part of the design, not an informal decision made after seeing results. The focus here is the pilot’s value for choosing a main-stage design, not the later problem of reallocating observations among sampling times.

Specify the Criterion Before Comparing Designs

The pilot can be evaluated using many objectives. To make the trade-off concrete, this tutorial uses an additive information-score criterion. Let \(\theta\) denote uncertain model parameters, let \(\pi(\theta)\) be their uncertainty before the pilot, and let \(I_d(\theta)\) be the information score per main-stage observation for candidate design \(d\) when the parameter is \(\theta\). In a local Fisher-information application, \(I_d(\theta)\) can be a scalar information measure for a target parameter or direction. In a multidimensional application, the investigator must first specify an appropriate scalar objective, such as a chosen function of the information matrix.

For the examples, the parameter uncertainty is represented by two plausible scenarios, \(\theta_L\) and \(\theta_H\). The pilot result \(X\) updates their probabilities by Bayes’ rule. If \(P(X\mid\theta)\) is the pilot-result likelihood, then \(P(\theta_j\mid X)=P(X\mid\theta_j)P(\theta_j)/\sum_k P(X\mid\theta_k)P(\theta_k)\). After observing \(X\), the expected score per main-stage observation under design \(d\) is \(\sum_j P(\theta_j\mid X)I_d(\theta_j)\).

Assume a fixed total budget of \(N\) observations, of which \(n_0\) are assigned to the pilot and \(N-n_0\) to the main stage. For illustration, let \(q_0\) denote the expected information-score contribution per pilot observation. Under the additive-score approximation, the expected total score for a pilot-informed rule is

Here \(\mathcal{D}\) is the prespecified set of main-stage candidates. This is an expected score across pilot outcomes. It is not, in general, an exact expected reduction in posterior variance. The additive information-score approximation is most defensible when the score is meaningful for the scientific objective, information contributions are approximately additive, and the pilot and main-stage observations are appropriately modeled. If those conditions are doubtful, evaluate the complete design with a model-based criterion such as expected posterior loss or simulation of the full estimation procedure.

Why Conditioning on Pilot Data Can Help

The gain from adapting the main-stage choice follows from a simple inequality. Write \(U_d(X)=\mathbb{E}\{I_d(\theta)\mid X\}\), the posterior expected score for design \(d\). If the main-stage design must be fixed before seeing the pilot, its best expected score per observation is \(\max_d\mathbb{E}_X[U_d(X)]\). If it may be chosen after observing the pilot, its expected score per observation is \(\mathbb{E}_X[\max_d U_d(X)]\).

To justify the inequality, choose any fixed candidate \(d^*\). For every possible \(X\), \(\max_d U_d(X)\geq U_{d^*}(X)\). Taking expectations gives \(\mathbb{E}_X[\max_d U_d(X)]\geq\mathbb{E}_X[U_{d^*}(X)]\). This holds for every fixed \(d^*\), so the left side is at least the largest expected score of any fixed candidate. The inequality describes the value of having the option to select after learning; it does not establish that a pilot is worthwhile. Pilot cost and score must still be included.

Worked Examples

Assumptions, Diagnostics, and Alternatives

A credible evaluation depends on more than the arithmetic. The parameter scenarios or uncertainty distribution must cover plausible model values; the pilot likelihood must represent how pilot data would arise under those values; and the information score must match the inferential objective. For additive scores, the assumed contribution of each observation should be appropriate for the design and model. If observations are dependent, if pilot and main-stage measurements differ materially, or if the second-stage choice changes the data-generating process, a simple sum may not be adequate.

Useful diagnostics assess both learning and consequences. Simulate pilot data under each plausible parameter region and check whether posterior probabilities move in the expected direction and by a useful amount. Examine how often the selection rule chooses each candidate, the expected total score under each scenario, and the worst-case score. A rule that almost never changes the design may indicate that the pilot is weak or that all candidates perform similarly. A rule that frequently switches based on tiny changes in pilot data may be unstable; test its behavior under modest changes in the prior, likelihood, and candidate scores. These checks are design evaluations, not reasons to discard inconvenient outcomes after the study begins.

A fixed robust design is a main alternative: it is chosen before data collection to perform acceptably across parameter uncertainty. It avoids spending observations on a pilot and does not depend on a data-driven switch, but it cannot tailor the main stage to the observed pilot result. A two-stage design can improve expected performance when the pilot reliably separates situations that favor different candidates. It can be less attractive when the pilot is expensive, the candidates have similar performance, or model misspecification makes the pilot update unreliable. A full simulation-based comparison is preferable when local information scores do not adequately represent estimation quality or the final scientific objective.