Selection Accuracy and Treatment Allocation

What Selection and Allocation Summaries Measure

An adaptive design may change treatment assignments as information accumulates, then recommend a treatment, dose, or population at the end. Its performance therefore includes more than whether a hypothesis test is significant. Two distinct questions are important: How often does the design select an option that is truly best? and How are simulated participants distributed among the available options?

These questions are related but not interchangeable. A design can allocate many participants to an option and still rarely select it as best. Conversely, it may identify the best option reliably while assigning relatively few participants to it early in the trial. Both summaries depend on the data-generating scenario and the exact adaptive rules being simulated.

The true values \(U_j\) are properties of the scenario, not estimates calculated from the outcomes in a particular replicate. For example, a scenario might specify the true response probabilities for all treatments and define the best treatment as the one with the highest probability, subject to any prespecified safety requirement. If the decision criterion is a benefit-risk utility, the utilities—not efficacy alone—determine which option is best.

For replicate \(r\), write \(D_r\) for the final recommendation. If there is one uniquely best option, define \(I_r=1\) when \(D_r\) equals that option and \(I_r=0\) otherwise. Across \(M\) simulated replicates, the estimated selection accuracy is the proportion of replicates in which the indicator is one.

This average has a direct justification. If replicates are generated under the same scenario and protocol, and each replicate follows the intended simulation procedure, then each \(I_r\) records whether selection was correct. Its expected value is the probability of correct selection, so the average across replicates estimates that probability. The estimate describes the simulated design under the specified scenario; it is not a guarantee about a future trial or a probability that the recommendation is correct after observing trial data.

Defining the Target When Options Tie

A tie in the true criterion requires an explicit definition of “correct.” Let \(B\) be the set of all options that attain the best true value. If the design recommends one option, a common definition counts the recommendation as correct whenever it belongs to \(B\). Thus, if two populations are equally best, selecting either counts as accurate. If a protocol instead requires selection of a particular option among tied options, that stricter target should be stated and justified.

The same principle applies when the design can recommend a set rather than a single option. The simulation plan must say whether the recommendation is correct when the recommended set contains at least one truly best option, contains all truly best options, or meets another explicit rule. These definitions answer different questions and can produce different accuracy estimates.

Summarizing Participant Allocation

Selection is an end-of-trial decision. Allocation describes assignments made during the trial. Let \(N_{rj}\) be the number of simulated participants assigned to option \(j\) in replicate \(r\), and let \(N_r=\sum_{j=1}^{K}N_{rj}\) be the total number assigned in that replicate. A replicate-level allocation proportion for option \(j\) is \(N_{rj}/N_r\), provided \(N_r>0\).

There are two useful ways to summarize these proportions across replicates. The pooled allocation proportion divides the total assignments to an option by the total number of assignments across all replicates. The mean replicate-level proportion first calculates a proportion within each replicate and then averages those proportions. When every replicate has the same total sample size, these two summaries agree. With different trial sizes, they can differ: the pooled summary gives more weight to larger trials, whereas the mean replicate-level summary gives each trial equal weight.

Neither summary is universally preferable. The pooled proportion answers, “What fraction of all simulated assignments went to this option?” The mean replicate-level proportion answers, “What was the average within-trial fraction assigned to this option?” Choose according to the design question and state which quantity is reported.

Allocation Across Stages and Populations

An overall allocation proportion can hide how an adaptive rule behaves over time. A design may begin with balanced randomization and later favor options with more promising accumulating evidence. To describe that pattern, summarize assignments separately by prespecified stage, such as an initial learning period and a later adaptive period. State the stage boundaries and use the number assigned during that stage as the denominator. An option’s fraction of assignments in the later stage is not its fraction of assignments over the entire trial.

If options include treatment-population combinations, define the allocation categories at that level when the question concerns both dimensions. Alternatively, report population totals and treatment totals separately. The latter summaries do not reveal the joint allocation pattern: two designs can have the same total assigned to each treatment and population while assigning different treatments within each population.

Checking the Calculation and Interpreting Results

Before reporting either summary, verify the simulation record and the target it represents. A sound calculation requires that the scenario’s true ranking or best set be known, that the recommendation rule be applied consistently, and that the allocation counts match the protocol’s definition of assignment. For example, the denominator may be participants randomized, participants actually treated, or participants included in a particular stage. Those counts are not automatically interchangeable.

Useful checks include confirming that recommendation counts sum to the number of simulated replicates, that allocation counts sum to the declared total assignments, and that proportions across mutually exclusive options sum to one when every participant is included. If some trials produce no recommendation under the specified rules, the analysis must say how those trials enter the accuracy calculation; silently excluding them changes the target. Similarly, if the algorithm recommends multiple options, a single-option accuracy indicator is not adequate until correctness for sets has been defined.

Selection accuracy is scenario-specific. A design can perform well when one option is clearly superior and less well when true effects are close. Reporting accuracy across several clinically relevant scenarios reveals this sensitivity; one favorable scenario cannot stand in for all plausible conditions. Allocation summaries also need context: assigning more participants to an option may reflect an adaptive rule, but does not establish that the option is effective, safe, or truly best.

Common Mistakes and Alternatives

• Ranking options using observed outcomes in each replicate. That measures whether the recommendation agrees with a noisy replicate-specific ranking, not whether it selects the true best option. Compare recommendations with the scenario’s prespecified truth.
• Ignoring ties. Counting only one of two equally best options as correct can understate accuracy for a tie-aware target. Define the best set and the correctness rule explicitly.
• Calling an allocation proportion selection accuracy. A 60% allocation to an option does not mean it is selected correctly in 60% of trials. Calculate the end-of-trial decision indicator separately.
• Mixing denominators. Pooled allocation proportions and averages of within-trial proportions answer different questions when trial sizes vary. Label the chosen summary.
• Reporting only an overall allocation total. This can conceal when an adaptive rule changed assignments. Add stage-specific summaries when timing matters to the design question.

The main alternative to a single accuracy number is a fuller decision summary: report recommendation frequencies for every option alongside the true best set, and report allocation proportions by option and, where relevant, by stage. This makes it possible to see whether low accuracy reflects frequent selection of one competitor or diffuse recommendations, while keeping the selection and exposure questions separate.