Randomization Variability and Replicability

Why Repeated Trials Can Look Different

A response-adaptive randomization rule changes allocation probabilities in response to accumulating outcome data. Consequently, two trials using the same protocol, sample size, and underlying treatment effects can finish with different allocation counts. Their estimated treatment effects can differ as well. This variation matters for replicability: repeating a study does not generally reproduce its allocation path or its numerical estimate, even when the design is implemented correctly.

There are two distinct questions. Within one trial, how uncertain is its estimate, given the data collected in that trial? Across repeated trials, how much do the allocation counts and estimates vary when the same design is run again? A standard error addresses the first question; the empirical spread of estimates from independently repeated trials addresses the second. Neither quantity can be substituted for the other.

For a concrete setting, suppose each trial enrolls \(n\) participants to two treatments. Let \(N_{1r}\) be the number assigned to treatment 1 in repeated trial \(r\), and let \(N_{0r}=n-N_{1r}\) be the number assigned to treatment 0. Let \(\hat{\theta}_r\) be the treatment-effect estimate from trial \(r\), for a fixed target \(\theta\), such as the difference in mean outcomes between treatments 1 and 0. The protocol, endpoint, target population, sample size, and adaptive rule are held fixed across repetitions. New assignment and outcome randomness is generated each time.

The repeated-trial distribution depends on the conditions being repeated. To interpret it, specify the true outcome-generating process, the randomization algorithm (including any starting allocation, update timing, and constraints), the sample size, and the estimator. For example, a comparison is not an apples-to-apples assessment of replicability if the endpoint or allocation rule changes between runs. If the treatment effect varies across participants, also specify the population and effect estimand: a difference in average outcomes and an effect for a particular subgroup need not have the same repeated-run behavior.

Summarizing Allocation and Estimate Variation

For \(R\) independently generated trial repetitions, a basic summary of treatment 1 allocation is its average count and sample standard deviation:

The average describes typical allocation under the specified conditions; the standard deviation describes how far the counts tend to vary around that average. A range or quantiles help reveal extreme runs that a standard deviation alone can obscure. It is also useful to report the frequency of counts near any design limit, such as a minimum allocation requirement. Allocation counts describe the realized behavior of the design, not uncertainty in the treatment-effect estimate by themselves.

For the effect estimates, the across-trial sample standard deviation \(s_{\hat{\theta}}\) summarizes their observed spread. By contrast, each trial may report its own estimated standard error, \(SE_r\). These answer different questions: \(s_{\hat{\theta}}\) summarizes the variation among repeated estimates, while \(SE_r\) describes estimated uncertainty for one trial’s estimate under its inferential procedure. Even when the average reported standard error resembles the across-trial spread, that similarity does not establish that every trial’s interval procedure is calibrated.

Within-Trial Uncertainty Is Not the Same as Replicate Spread

A trial-specific standard error is calculated from that trial’s data and its analysis procedure. In a simple comparison of two independent sample means with a fixed allocation, a familiar estimated standard error is \(\sqrt{s_1^2/N_1+s_0^2/N_0}\), where \(s_1\) and \(s_0\) are the sample standard deviations in the two arms. That formula describes a standard fixed-allocation analysis under its assumptions; it should not automatically be treated as an exact conditional variance formula for an adaptive trial. Under response-adaptive allocation, the assignment path depends on earlier outcomes, so inferential methods must be appropriate to the design.

The conceptual distinction does not depend on a particular formula for the standard error. For example, a trial might report a confidence interval based on its chosen adaptive-design analysis. That interval conveys uncertainty under the assumptions and calibration of that analysis. The spread of estimates over repeated trials instead describes how the estimator behaves when the entire data-collection process is rerun. Earlier material on confidence interval coverage under adaptation is relevant to whether the intervals have their intended long-run coverage; a small within-trial standard error alone does not settle that question.

A Variance Decomposition for Repeated Trials

A useful way to understand replicate-to-replicate variability is to condition on an aspect of the realized adaptive history. Let \(A\) denote a specified feature of the allocation history, such as a category of final allocation counts or a prespecified history group. The law of total variance gives an exact decomposition:

The first term measures variation in the conditional mean estimate across history groups. The second is the average variation remaining among trials within those groups. This identity applies to any random variable with finite variance. It does not assume that allocation history is independent of outcomes; in an adaptive design, it generally is not. Nor does the second term automatically equal the square of a reported standard error. It is a property of the repeated-trial distribution conditional on \(A\), while a reported standard error is an estimate produced by a particular analysis.

Here is the justification. Write the centered estimate as the sum of its conditional deviation and conditional mean deviation:

The expected square of the first bracket is \(E[\operatorname{Var}(\hat{\theta}\mid A)]\). The expected square of the second bracket is \(\operatorname{Var}(E[\hat{\theta}\mid A])\). The cross-product has expectation zero because \(E[\hat{\theta}-E[\hat{\theta}\mid A]\mid A]=0\). Thus expanding the square gives the stated decomposition. Choosing a different history summary \(A\) changes the two components’ interpretation, but not their sum.

Diagnostics, Alternatives, and Limitations

A repeated-run assessment is only as informative as the scenarios and summaries it includes. When simulation is used, each run should follow the same prespecified design and outcome model, while using fresh random assignment and outcome draws. Report the number of repetitions and enough detail to interpret the distribution, not just its average. Useful diagnostics include:

• the mean, standard deviation, range, and selected quantiles of final allocation counts and effect estimates;
• the fraction of runs near minimum or maximum allocation limits, when such limits exist;
• histograms or other distribution summaries that show skewness, multiple modes, or extreme runs;
• the relationship between allocation history and the estimate, standard error, or interval width; and
• Monte Carlo precision: whether the number of simulated repetitions is sufficient for the tail probabilities or quantiles being reported.

These checks help reveal instability, but they do not by themselves identify its cause. If estimates vary, the variation may reflect outcome noise, assignment randomness, or their interaction through the adaptive rule. Because assignments depend on observed outcomes, treating the allocation path as if it were generated independently of outcomes can be misleading. The variance decomposition is descriptive of the chosen conditioning scheme, not an automatic causal separation of “randomization variance” from “outcome variance.”

A fixed-allocation design is a useful comparison: its allocation counts are set by the protocol, so they do not vary across repetitions with the same planned sample size, though its outcome-based estimates still vary. Comparing adaptive and fixed designs can therefore clarify whether allocation itself is volatile and how estimates behave under the two procedures. The comparison is meaningful only when the data-generating assumptions, sample size, endpoint, estimand, and analysis are aligned. A single repeated-run standard deviation does not summarize every design objective, and it does not establish that one design is preferable.

A common mistake is to say, “The average standard error across runs is 0.75, so repeated estimates will have a standard deviation of 0.75.” That conclusion is not justified merely by the average reported standard error. The standard error is trial-specific and depends on the inferential method; its calibration under adaptation is a separate question. Estimate spread should be calculated directly across independent repetitions, and standard errors should be evaluated for their intended inferential role. Conversely, a wide spread across runs does not mean any one trial’s confidence interval must be wide: a particular run can have a narrow reported interval while the procedure’s estimates vary substantially across runs.