Why the Sampling Distribution Needs a Design Index
In a fixed-design trial, the estimator’s sampling distribution is defined by repeatedly running the same design under the same parameter value. In an adaptive trial, the design itself may vary across repetitions: interim data can determine the later sample size, treatment assignments, or analysis population. Therefore, the estimator’s distribution must account both for the data collected and for the rule that maps interim data to a design.
The key distinction is between the full-trial sampling distribution, which averages over all possible interim results and all designs they select, and a conditional sampling distribution, which describes the estimator after conditioning on an interim history or on a selected design. These distributions answer different questions. Neither should be substituted for the other without stating the conditioning.
The last distinction is important. Conditioning on \(D=d\) does not usually mean that the interim data are fixed at one particular value. It means the history is restricted to the set of values that select \(d\). The conditional distribution given \(D=d\) therefore includes the distribution of \(H\) within that set. This is why conditioning only on the adaptation choice can fail to describe the distribution given the complete interim history.
From Conditional Distributions to the Full-Trial Distribution
A useful way to construct the full-trial distribution is to first describe the estimator conditional on the interim history. Given \(H=h\), the design \(D=g(h)\) is fixed, but future observations remain random. The full distribution then averages those conditional distributions over the possible interim histories. Equivalently, one can first group histories by their selected design and average within each design group.
The law of total expectation and the law of total variance give exact relationships for the mean and variance. They apply whenever the relevant expectations and variances are finite; they do not require the estimator to be normally distributed.
The first variance term is the average remaining variability after the interim history is known. The second is variability in the conditional mean across possible interim histories. Omitting the second term generally understates the full-trial variance. When conditioning on the design \(D\) instead, the same decomposition holds with \(D\) in place of \(H\), but the conditional distribution given \(D\) itself still averages over histories selecting that design.
Write how interim data \(H\) determine \(D\), including the possible sample sizes, assignments, or analysis populations.
State how \(\hat{\theta}\) is calculated under each design and the distribution of the interim and later data.
Find the estimator’s distribution given \(H\), or given \(D\) if that is the target. Then average over the conditioning variable to obtain the full-trial distribution.
Worked Distributions
Worked Example: A Two-Design Mixture
Suppose an adaptive rule selects design A with probability \(0.25\) and design B with probability \(0.75\) at the parameter value under consideration. Assume the estimator has the following conditional distributions: under A its mean is 3 units and its variance is 4 square units; under B its mean is 5 units and its variance is 9 square units. Calculate the full-trial mean and variance.
The full-trial mean is the probability-weighted average of the conditional means:
For the variance, first average the conditional variances: \(0.25(4)+0.75(9)=1+6.75=7.75\) square units. The conditional means also vary across designs. Their variance around the full-trial mean is
Thus the full-trial variance is
The conditional variance within designs is 7.75 on average, but the full-trial variance is 8.5 because design-specific conditional means differ. This calculation specifies the first two moments; it does not imply that the full-trial distribution is normal.
Worked Example: Conditioning on an Interim Mean
Suppose \(n_1=20\) participants contribute an interim sample mean \(X\), with \(X\sim N(\theta,\sigma^2/n_1)\), where the outcome standard deviation is known to be \(\sigma=10\) units. An independent stage-two sample mean \(Y\), conditional on the chosen sample size, has distribution \(N(\theta,\sigma^2/n_2)\). The rule chooses \(n_2=20\) if \(X<1\) unit and \(n_2=60\) if \(X\geq 1\) unit. Use the pooled mean of all participants as the estimator. Find its conditional distribution if the observed interim mean is \(x=2\) units.
Since \(x=2\) is at least 1, the selected design has \(n_2=60\). Conditional on the complete interim data summarized by \(X=x\), the later mean is independent of \(X\) and has distribution \(N(\theta,100/60)\). The pooled estimator is
With \(X=x\) fixed, its conditional mean is \((n_1x+n_2\theta)/(n_1+n_2)\), and its conditional variance comes only from \(Y\):
Therefore, conditional on \(X=2\), the estimator is normal with the displayed mean and variance. At \(\theta=0\), for example, this is \(N(0.5,0.9375)\), where the second argument is the variance. This is not the distribution conditional only on selecting \(n_2=60\): that distribution averages over every interim mean \(X\geq1\), rather than fixing \(X=2\).
Worked Example: The Full Distribution With Adaptive Sample Size
Consider an interim trigger that selects a small stage-two sample with probability \(0.25\) and a large one with probability \(0.75\) when \(\theta=0\). Assume the trigger statistic is independent of the later observations under this parameter value. The estimator in this example is the stage-two sample mean only. The outcome variance is \(\sigma^2=16\) square units; the small design has \(n_2=25\), and the large design has \(n_2=100\). Find the conditional and full-trial distributions at \(\theta=0\).
Under the small design, the stage-two mean is normal with mean 0 and variance \(16/25=0.64\) square units. Under the large design, its variance is \(16/100=0.16\) square units. Thus the conditional distributions are \(N(0,0.64)\) and \(N(0,0.16)\), respectively.
Across complete trials, the estimator follows a mixture: a draw from \(N(0,0.64)\) with probability \(0.25\), or a draw from \(N(0,0.16)\) with probability \(0.75\). Both components have mean zero, so the full-trial mean is zero. The full-trial variance is
There is no additional between-design mean term here because both conditional means are zero. The full distribution is nevertheless a mixture with two different variances, not the single normal distribution \(N(0,0.28)\). Reporting only the full-trial variance would lose this shape information. In this example the independence assumption makes the conditional distribution given the design equal to the stated stage-two normal distribution; if the interim statistic and later observations were dependent, that conclusion would need a different conditional model.
Assumptions, Diagnostics, and Comparisons
The calculations above depend on a specified probability model, not merely on the fact that an adaptation occurred. For the pooled-mean example, the model assumed normal outcomes with known common variance, independent stage-one and stage-two observations, and a stage-two sample size determined by the interim mean. Conditional independence of future data from the observed interim data is essential for the stated stage-two distribution. If there is dependence, delayed outcome information, or a changing analysis population, the conditional law must represent it.
A practical diagnostic is to write the estimator’s conditional distribution separately for each possible interim history or design, then verify that averaging those distributions recovers the claimed full-trial quantities. In simulation, repeat the entire trial—including the adaptation rule—rather than holding the selected design fixed. Compare the simulated distribution’s mean and variance with the conditional calculations and the total-variance decomposition. A discrepancy can indicate that the simulation omitted design variability, used the wrong conditioning event, or failed to reproduce the rule.
| Target distribution | What is held fixed? | What randomness remains? | Typical use |
|---|---|---|---|
| Full-trial | No interim history or selected design | Interim data, selected design, and later data | Describe repeated operation of the adaptive procedure |
| Conditional on \(H=h\) | The complete interim history | Later data under \(g(h)\) | Describe uncertainty after observing that history |
| Conditional on \(D=d\) | The selected design only | Histories leading to \(d\), and later data | Summarize trials that select a particular design |
A fixed-design calculation is the main alternative when a design is set in advance. It can be appropriate conditional on a complete interim history if the remaining-data model and estimator are correctly specified. It is not, by itself, the full-trial distribution when the design varies across repetitions. Likewise, conditioning only on the selected design may be useful, but it does not generally reproduce conditioning on the observed interim data.
These distributions are also distinct from questions about whether an estimator is biased, or how to correct an estimate after adaptation. A sampling distribution supplies the probabilities, moments, and shape needed to address such questions; it does not itself settle them. The central limitation is that the distribution depends on the full adaptation rule and data model. If either is incompletely specified, a precise conditional or full-trial distribution cannot be claimed.
Common Mistakes
- Holding the selected sample size fixed in a full-trial calculation. This describes one design-specific distribution, not repeated trials in which the interim data can select different sample sizes.
- Calling the distribution given \(D=d\) the distribution given \(H=h\). The former includes all histories selecting \(d\); the latter fixes the observed history.
- Averaging conditional variances and stopping there. The full variance also includes the variance of conditional means whenever those means vary with interim data or design.
- Assuming a mixture is normal because each component is normal. A mixture of normal distributions with different means or variances is generally not itself normal.
- Using an independence assumption without checking it. The future-data distribution conditional on interim information must reflect any dependence or information overlap in the trial.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the two-design mixture, what is the full-trial variance if both conditional means are instead 4 units, while the design probabilities and variances remain unchanged?
- Why is conditioning on a selected design not generally the same as conditioning on the complete interim history?
- A trial has conditional estimator variances 2 and 6 square units with equal design probabilities, and conditional means 1 and 3 units. Find the full-trial mean and variance.
- In the adaptive sample-size example, what additional assumption allowed the stage-two mean’s distribution to be stated conditional on the interim mean?
Solutions. (1) Both means are 4, so the between-design variance is zero; the full variance is \(0.25(4)+0.75(9)=7.75\) square units. (2) A design can be selected by many interim histories, so conditioning on that design averages over those histories, whereas conditioning on the complete history fixes its observed value. (3) The mean is \(0.5(1)+0.5(3)=2\) units. The within-design variance is \(0.5(2)+0.5(6)=4\); the between-means variance is \(0.5(1-2)^2+0.5(3-2)^2=1\). Thus the full variance is 5 square units. (4) The stage-two observations were assumed independent of the interim data, conditional on the selected sample size, with the specified normal outcome model.