Diagnosing Allocation Variability

Why Diagnose Allocation Variability?

A response-adaptive rule changes assignment probabilities as information accumulates. Consequently, two trials governed by the same rule can follow different allocation paths: early assignments and observed responses differ, which can lead the rule to assign different probabilities later. A single simulated path shows what can happen once; it does not show how much paths vary under the design.

Repeated simulation makes that variation measurable. For each arm, investigators can examine the distribution across simulated trials of its assignment probability at specified enrollment points and of its final enrollment count. These summaries help reveal whether a design produces consistently similar paths or a wide range of possible allocation patterns. They describe the behavior of the specified design under the specified simulation assumptions; they do not establish that the rule will behave the same way under every plausible clinical scenario.

The probability \(q_{rik}\) is a decision made by the rule for a particular assignment, not the probability that arm \(k\) is effective. It can change over time and across replicates. A probability summary therefore needs a clear time point or time window: for example, the probability at participant 50, or the average assignment probability over participants 41 through 60. If a participant count is not reached in every replicate, report how that is handled rather than silently excluding paths.

What to Summarize Across Paths

Suppose \(R\) independent simulated trials are generated under the same design and clinical scenario. For a prespecified probability summary \(Q_{rk}\) for arm \(k\) in replicate \(r\), calculate its across-replicate mean and sample standard deviation:

The mean describes the average of the selected probability summary across simulated trials; the standard deviation describes its spread across those trials. For final enrollment counts, use the same calculations with \(N_{rk}\) in place of \(Q_{rk}\). Counts are in participants, whereas probabilities are dimensionless. Report quantiles or a range as well when they make the shape or tails of the simulated distribution clearer. A standard deviation alone cannot show whether the distribution is skewed or contains a few unusually extreme paths.

State whether the total sample size is fixed or can vary. With fixed total enrollment \(n\) and no dropped or otherwise unavailable arms, counts satisfy \(\sum_k N_{rk}=n\) in every replicate. The count summaries are then linked: a larger count in one arm must be offset by smaller counts in one or more other arms. When total enrollment varies, the counts also reflect variation in how many participants enter the trial.

These summaries measure variability, not whether allocation is systematically different from a chosen target. A design can have little across-path variation while consistently assigning an arm a low probability; it can also have considerable variation while having an average probability equal to a target. Diagnosing whether allocation is imbalanced relative to a target is a distinct question.

Why Adaptive Counts Need Repeated Simulation

Under a fixed assignment probability \(p\) for every participant, independent random assignments to one arm give \(N\sim\mathrm{Binomial}(n,p)\), with \(\mathrm{Var}(N)=np(1-p)\). That formula is a useful reference for a nonadaptive design. A response-adaptive rule is different: the probability for a later assignment can depend on earlier assignments and outcomes. Counts therefore cannot generally be treated as binomial with one fixed probability.

A short sequential argument clarifies part of the distinction. Let \(\mathcal{F}_{i-1}\) represent all information available before assignment \(i\), and suppose the rule sets \(q_i\) for a particular arm using only that information. The assignment indicator \(A_i\) is Bernoulli with conditional probability \(q_i\). Define the assignment innovation \(D_i=A_i-q_i\). Then \(E(D_i\mid\mathcal{F}_{i-1})=0\) and \(E(D_i^2\mid\mathcal{F}_{i-1})=q_i(1-q_i)\). For \(i<j\), \(D_i\) is known by the time immediately before assignment \(j\), so:

This variance describes the accumulated assignment innovations, not the entire across-trial variance of \(N=\sum_i A_i=\sum_i q_i+\sum_i D_i\). The probabilities \(q_i\) themselves vary with the simulated history, and the two sums need not be uncorrelated. This is one reason to estimate the distribution of the final counts directly from complete simulated paths rather than applying a fixed-probability binomial formula.

Simulation Checks and Interpretation

A defensible variability analysis begins by defining what is held constant across replicates. At a minimum, specify the response-generating assumptions, the adaptive rule, the accrual and follow-up process, the total-enrollment or stopping rule, and the probability and count summaries to be saved. Independent replicates should be generated under the same specifications. If the purpose is to compare clinical scenarios, report results separately by scenario rather than mixing paths from different assumptions into one distribution.

Check the simulated records before interpreting summaries. Verify that each participant receives an assignment consistent with the probabilities recorded immediately before assignment; that arm probabilities are valid and sum to 1 when all active arms are included; and that final counts agree with the assignments. Check that a probability at a checkpoint is not accidentally summarized over different numbers of participants across paths. If an arm can be dropped, distinguish paths where the arm remains active from the full set of paths: reporting only survivors answers a conditional question and can hide variation in whether the arm remains available.

The number of replicates matters because a reported simulation summary is itself estimated with Monte Carlo error. For the mean of a simulated quantity with across-path sample standard deviation \(s\), its estimated Monte Carlo standard error is \(s/\sqrt{R}\), assuming independent replicates. Increasing \(R\) makes the estimated mean more precise, but does not reduce the actual variability of paths generated by the design. Tail estimates generally require more replicates than a mean, so quantiles and low-enrollment probabilities should be checked for stability as \(R\) increases.

Common Pitfalls and Alternatives

A single allocation path is useful for illustrating how a rule operates, but it cannot estimate between-path variability. Conversely, a summary table of means and standard deviations can conceal unusual tails, early arm suspension, or a cluster of paths with very low enrollment. Use plots or quantiles alongside numerical summaries when those features matter, and report the number of replicates and the simulation scenario with the results.

A fixed-probability binomial calculation is a useful benchmark only when its assumptions apply: the assignment probability is constant and assignments are independent with that probability. It does not capture probability changes induced by accumulating responses. Repeated simulation is more flexible, but its conclusions depend on the outcome and accrual assumptions supplied to the simulator. Neither method alone answers whether the rule produces acceptable allocation or sound treatment-effect estimates; those require separate design assessments.

A frequent mistake is to treat a standard deviation across simulated paths as uncertainty about the mean summary, or to interpret a low across-path standard deviation as proof that the rule is well calibrated. The correction is to state which distribution the statistic describes. Across-path standard deviation measures design-generated spread under the simulated scenario. Monte Carlo standard error measures precision of a simulation estimate. Neither, on its own, judges whether the resulting allocation is desirable.