Auditing Adaptive Randomization Decisions

Why an Adaptive Decision Needs an Audit Record

In a fixed-allocation trial, the assignment probabilities are generally specified before enrollment and do not change in response to accumulating outcomes. In an adaptive trial, an update may change those probabilities or the set of available arms. Reconstructing what happened therefore requires more than a plot of allocation over calendar time: the record must connect the information available at that moment to the rule applied and the probabilities actually used.

An audit record serves two related purposes. First, it allows an independent reviewer to reproduce a decision from the data and rule in force at the update. Second, it lets the trial team identify operational discrepancies, such as a data extract taken before the declared cutoff or a probability vector that does not match the approved algorithm. An audit record does not itself establish that the model or adaptation rule is scientifically appropriate. It makes the implemented decision traceable.

The trial protocol and operational plan should specify when updates occur, which observations are eligible for each analysis, how pending outcomes are handled, the rule for calculating probabilities, and any conditions for removing an arm. The audit documents a particular application of those specifications; it should not silently create or alter them. If a planned procedure changes, the change and its authorization need their own traceable record.

What Each Update Should Preserve

A useful record is structured so that another analyst can follow the decision from inputs to implementation. At a minimum, capture the following items for every update:

• Update identity and timing: a unique update number, the analysis timestamp and time zone, the prespecified data cutoff, and the time the allocation probabilities became effective.
• Data state: the data-extract version or identifier; the patients and arms included; endpoint status at cutoff; counts of observed, pending, excluded, and withdrawn patients; and any prespecified eligibility or censoring rules applied.
• Model and posterior state: the model specification and version, prior specification, data supplied to the model, and the posterior summaries required by the rule. State the arm-specific denominators and units as well as estimates, so summaries can be interpreted and checked.
• Rule inputs and decisions: all quantities passed to the allocation rule, any thresholds or constraints invoked, the resulting arm-status decisions, and a clear reason for each decision according to the prespecified rule.
• Allocation output and implementation: the full probability vector, its arm order, any rounding or minimum-probability adjustment, the effective start time, and confirmation of how the randomization system received the vector.
• Reproducibility details: code and configuration versions, software environment, and, where applicable, the random-number-generation details needed to replay the computation. Protect access to any seed or other sensitive information according to the trial’s security procedures.

The cutoff deserves particular care. It is a time boundary, not just a label such as “week 8.” The record should make clear which data were available by that boundary and which were eligible for the update. For time-to-event outcomes or delayed responses, patients may be enrolled but not yet have an outcome available. The audit should preserve those statuses rather than make an incomplete outcome look like a nonresponse or omit the patient without explanation. That distinction can affect both posterior calculations and the interpretation of the next allocation probabilities.

Posterior summaries should be sufficient to check the rule’s inputs, but storing summaries alone is usually not sufficient to reproduce the analysis. Preserve the underlying analysis dataset or a controlled reference to it, plus the model and code versions. Record the posterior quantities actually consumed by the rule, not only a polished summary added later. If a rule uses a posterior mean, a probability, and a cap, each of those values and the resulting transformation should be visible.

From Data Cutoff to Effective Probabilities

Consider a simple illustration with binary response. Suppose each arm \(i\) has response probability \(p_i\), with independent prior distributions \(p_i\sim\operatorname{Beta}(1,1)\). If the cutoff includes \(r_i\) responses among \(n_i\) patients with evaluable outcomes, the posterior is \(p_i\mid\text{data}\sim\operatorname{Beta}(1+r_i,1+n_i-r_i)\). This model and its assumptions are used here only to make an audit calculation explicit; a real trial must record its own prespecified model.

For illustration, suppose the allocation rule first computes posterior means, forms weights proportional to those means, then mixes those weights with equal allocation. With \(K\) active arms, define the raw weight \(v_i=E(p_i\mid\text{data})/\sum_j E(p_j\mid\text{data})\), and the final probability \(q_i=0.20/K+0.80v_i\). The rule is fully specified only when the active-arm set, the constants, the order of operations, and any rounding behavior are also known.

For randomized implementation, the probability vector does not determine the realized assignment sequence on its own. The audit should therefore record the vector sent to the system and confirm its effective time, while the randomization system separately retains the assignment events and the probability applicable to each event. Where replaying a random draw is appropriate and permitted, record the relevant algorithm and protected random-state information. Do not expose confidential randomization details in a way that undermines trial conduct.

Delayed Outcomes and Arm Decisions

A cutoff record must distinguish the enrollment state from the endpoint state. For example, a patient may have entered an arm before the cutoff but still have a pending response assessment. The audit should record whether the rule uses only evaluable responses, a time-to-event model with partial follow-up, or another prespecified treatment of incomplete information. It should also preserve counts of pending outcomes so a reviewer can see why the analysis denominator differs from the enrolled count.

Arm decisions should be recorded with the rule inputs and the decision itself, not just as a changed arm list in the final allocation file. Suppose, for illustration, a protocol specifies that an arm can be removed only after at least 20 evaluable outcomes and when the posterior probability that its response rate is below 0.10 exceeds 0.95. The record should show the evaluable count, the posterior distribution, the calculated probability, the threshold, whether the minimum information condition was satisfied, and the resulting decision.

Independent Checks, Failure Modes, and Alternatives

An audit is strongest when it can be checked independently rather than merely read as a narrative. A second analyst or a controlled software check should verify that the cutoff and data version agree; that the reported counts match the eligible records; that posterior summaries follow from the recorded model and data; that all rule thresholds and constraints were applied in the stated order; and that final probabilities are valid and match the values transmitted to the randomization system. For a probability vector, each component must be between 0 and 1 and the components must sum to 1 within the documented numerical tolerance.

Common failure modes include using a data extract later than the declared cutoff, silently counting pending outcomes as nonresponses, recording rounded posterior summaries that are insufficient to reproduce a threshold decision, changing an arm list without documenting when the change took effect, or storing a probability vector without its arm order. A further warning sign is a mismatch between the approved rule version and the code or configuration used. Each discrepancy should be investigated and resolved through a traceable correction; the original entry should not simply be overwritten.

A brief narrative memo can explain an unusual operational event, but it is not a substitute for structured inputs and outputs. Conversely, a machine-readable log can support exact checking but may not explain a data correction or authorized deviation in understandable terms. A robust approach uses both: a structured, versioned record for replay and a concise explanation for exceptions. Another choice is whether updates occur at scheduled cutoffs or event-driven triggers. Either can be auditable if the trigger, effective time, eligible data state, and rule are recorded consistently; the choice itself belongs in the trial’s prespecified operational plan.

The limits matter. A perfectly reproducible update can still rely on a misspecified model, an unsuitable prior, poor data quality, or a rule with undesirable operating characteristics. The audit establishes what information and procedure produced a decision, not that the decision was optimal or that the treatment caused an outcome. Model assessment and design evaluation are separate tasks. The practical value of a good record is that it makes those tasks possible without guessing what happened at a particular update.