What a Design Diagnosis Must Reconcile
An adaptive platform is not adequately diagnosed by asking whether its allocation looked balanced or whether an arm was retained. The relevant question is whether each decision followed the rule that was specified, using the data that were permitted at that time, and whether the implementation record supports that conclusion. A design can follow its rule exactly and still have concerning allocation paths; a design can also look reasonable overall while containing an unauthorized arm removal or a decision based on the wrong data cutoff.
This workshop treats the protocol, the allocation-path record, the arm-status history, the pending-outcome policy and the implementation log as connected evidence. The diagnostic task is to compare them decision by decision, identify the first point at which they disagree, and recommend a revision that is concrete enough to implement and audit. The earlier material on reading allocation-path plots, allocation variability, operational latency and auditing adaptive randomization decisions provides useful context for this review.
Before examining results, establish the decision state at each interim look: which arms were active, which participants had outcomes available by the specified cutoff, what data summary was supplied to the rule, which rule and software version were in force, and what decision was recorded. Without this state, a path plot cannot show whether a probability was appropriate, and a decision log cannot establish whether the correct data were used.
A Compact Audit Framework
For each decision time, compare four linked records. First, inspect the allocation path: the active arms, their assignment probabilities, and the observed assignments. Second, inspect arm retention: the rule for continuing or removing an arm, the evidence available at the decision, and the recorded status change. Third, inspect the pending-outcome policy: which outcomes count as available, what happens when outcomes are delayed, and whether a decision may proceed before a minimum information requirement is met. Fourth, inspect the implementation record: timestamps, data snapshots, software and rule versions, randomization outputs, overrides, and approvals.
These records should agree on both timing and content. For example, a decision labelled “look 2” is not sufficiently documented if the analysis file contains outcomes recorded after the look’s cutoff, or if the allocation engine used a different active-arm list from the approved decision record. Likewise, an allocation probability shown in a report is not necessarily the probability used for a particular assignment; the audit needs the contemporaneous record.
Record the cutoff time, active arms, available outcomes, pending outcomes, and rule and software versions before interpreting the decision.
Apply the written allocation and retention rules to the frozen state. If the rule does not uniquely determine a decision, document the ambiguity rather than guessing at the intended result.
Check the allocation probabilities, arm statuses, data inputs and timestamps against the actual randomization and decision logs.
Separate a chance pattern under a correctly implemented rule from a rule-design weakness or an implementation deviation; specify the corrective action and its owner.
The comparison has a simple mathematical foundation. If a batch of \(N\) assignments uses fixed probabilities \(p_1,\ldots,p_K\) for \(K\) active arms, then the expected number assigned to arm \(j\) is \(Np_j\). Let \(I_{ij}\) equal 1 when assignment \(i\) goes to arm \(j\), and 0 otherwise. Since \(E(I_{ij})=p_j\), linearity of expectation gives \(E(\sum_{i=1}^N I_{ij})=\sum_{i=1}^N E(I_{ij})=Np_j\). This expectation is a benchmark, not a quota: realized counts can differ from it under valid randomization.
In an adaptive design, probabilities may change with the history. For assignment \(i\), write \(p_{ij}\) for the probability assigned to arm \(j\), conditional on the information available immediately before that assignment. Then \(E(I_{ij}\mid\text{history})=p_{ij}\). Consequently, the overall expected count is \(E(\sum_i I_{ij})=\sum_i E(p_{ij})\), not generally one fixed probability multiplied by the total sample size. A diagnosis should therefore compare each segment of a path with the probabilities in force for that segment.
Worked Example: Checking an Allocation Path Against Its Probabilities
Suppose a hypothetical platform has four active arms: control, A, B and C. For a cohort of 120 participants, the approved allocation probabilities are held fixed at \(0.40, 0.30, 0.20,\) and \(0.10\), respectively. The path record shows 46 assignments to control, 39 to A, 23 to B and 12 to C. Is the path inconsistent with the allocation rule?
First check the probabilities: \(0.40+0.30+0.20+0.10=1.00\), so they define a valid allocation distribution over the four active arms. The expected counts are:
The recorded counts add to \(46+39+23+12=120\). They differ from the expected counts by \(-2,+3,-1,0\), respectively. Those differences alone do not demonstrate an error: expected counts are not required realized counts. The audit should next verify that the probabilities were in fact fixed for this cohort, that every assignment was randomized from that distribution, and that no arm-status or probability update occurred partway through it. If those records agree, this path is compatible with random variation. If the system log shows a different probability vector, the issue is implementation fidelity, even if the observed totals look close to expectation.
Arm Retention: Test the Decision, Not Its Appearance
An arm-retention rule needs a decision time, an evidence definition, a threshold or condition, and an explicit consequence. “Remove an arm if it is doing poorly” is not operationally complete. It leaves open which data count, what “poorly” means, whether one or repeated looks are required, and whether removal can later be reversed. The earlier workshop on specifying a multiarm adaptive rule is relevant here: a diagnostic can only determine compliance with a rule that is sufficiently specified.
For the audit, reconstruct the evidence available at the retention decision and apply the rule as written. Do not infer the rule from the arm’s later outcomes, and do not treat a low allocation probability as proof that the arm was required to stop. Allocation and retention are distinct decisions: an arm can receive a smaller share while remaining active, or be removed under a separate criterion.
Worked Example: Finding an Unauthorized Arm Removal
In a hypothetical platform, the approved retention rule says to remove an arm only if its posterior probability of exceeding the prespecified minimum clinically relevant benefit is below \(0.10\) at two consecutive scheduled looks. The recorded probabilities for arm B at three successive looks are \(0.07\), \(0.12\), and \(0.08\). The implementation record shows that B was removed at the third look.
At the first look, \(0.07<0.10\), so B meets the condition once. At the second look, \(0.12\not<0.10\), so the consecutive-look condition is broken. At the third look, \(0.08<0.10\), which is one qualifying look after the break, not two consecutive qualifying looks. Therefore, under the stated rule, B should not be removed at the third look. The audit has identified a discrepancy between the written criterion and the recorded action.
A useful revision is not simply “be more careful.” The trial team should specify how the system records qualifying looks and resets the consecutive-look count, test that logic on boundary cases such as exactly \(0.10\), and require the decision record to show the two qualifying looks supporting any removal. If the intended policy was a single-look threshold, the protocol and software should instead be amended consistently before that policy is used.
Pending Outcomes and Decision Eligibility
Pending outcomes are a design feature to audit, not a nuisance to resolve informally. The protocol should say when an outcome is considered available, whether the decision uses only observed outcomes, whether a minimum number of mature outcomes is required, and what action is permitted when that minimum is not met. “Use the latest data” is not precise enough if the data extract can include participants with incomplete follow-up.
In particular, an outcome that has not yet been observed is not automatically a failure, a success or a missing-at-random value suitable for an unstated imputation. The analysis or allocation procedure must use the handling rule specified in advance. A design may, for example, defer a look until a maturity requirement is met or continue with a previously approved allocation while waiting; which response is appropriate depends on the prespecified rule.
Worked Example: Applying a Minimum-Maturity Requirement
A hypothetical protocol permits an allocation update only after at least 20 participants in each relevant arm have a mature binary response outcome. At the planned look, arm A has 30 enrolled participants: 18 observed responses, 12 outcomes pending. Control has 28 enrolled participants: 20 observed responses, 8 pending. The protocol also says to retain the previous allocation probabilities if the minimum is not met.
Arm A has \(18\) mature outcomes, which is fewer than the required \(20\). Control has \(20\), which meets the requirement. Because the requirement applies to each relevant arm, the look is not eligible for an update. Under this protocol, the action is to retain the previous allocation probabilities and wait for sufficient follow-up in A.
For illustration, the observed response proportions are \(18/30=0.600\) in A and \(20/28=0.714\) in control if all enrolled participants are incorrectly used as denominators while pending outcomes are treated as nonresponses. Among participants with observed outcomes, the corresponding proportions are \(18/18=1.000\) and \(20/20=1.000\). Neither calculation is a substitute for the protocol’s decision rule: the first silently codes pending outcomes as failures, while the second ignores the stated maturity requirement and has no outcome variation at this look. The correct diagnostic finding is that the update is ineligible, not that one of these improvised comparisons should determine allocation.
Implementation Records and Discrepancy Triage
A reproducible implementation record should make it possible to reconstruct what the decision engine knew and did. At a minimum, preserve the data cutoff and extract identifier; active-arm list; arm-level summaries supplied to the rule; calculated allocation probabilities; rule and software versions; the output for each randomization; decision and approval timestamps; and any manual override with its reason and authorization. Where randomization uses a generated random number, retain the auditable record needed under the trial’s security and governance procedures.
A useful discrepancy review starts with the earliest divergence, not the most dramatic later result. For each finding, ask whether the source is (1) an incomplete or ambiguous rule, (2) a data-timing or outcome-availability error, (3) an incorrect software or configuration state, (4) an authorized deviation that was poorly documented, or (5) ordinary variation under the intended randomization. The remedy differs by cause. A software mismatch requires controlled correction and retesting; a vague rule requires protocol clarification; a chance imbalance alone may require monitoring rather than a change to the rule.
Worked Example: Reconciling an Assignment Log
At a hypothetical look, arm C is recorded as removed at 14:00. The decision report lists active arms as control, A and B. At 14:08, the randomization log records an assignment to C, while the configuration snapshot for that assignment lists probabilities \(0.50\) for control, \(0.30\) for A, \(0.20\) for B and \(0\) for C.
The first check is the probability sum: \(0.50+0.30+0.20+0=1.00\). The reported distribution gives C zero probability, so a valid draw from that distribution cannot assign a participant to C. The assignment is therefore inconsistent with the displayed configuration. The timing is also relevant: the removal was recorded eight minutes before the assignment, so the event cannot be explained merely by a later status change.
The team should preserve the contemporaneous logs, determine whether the assignment was made by a stale randomization service, a manual process, or a transcription error, and check whether any other assignments were affected. The concrete revision is to make arm removal update and verify the randomization service’s active-arm configuration before another assignment can occur, with a logged confirmation and an explicit escalation path if that confirmation fails. The audit should not quietly relabel the recorded assignment as an assignment to another arm.
Diagnostics, Alternatives, and Limits
Several warning signs deserve immediate follow-up: an allocation probability vector that does not sum to one; a positive probability assigned to a removed arm; a removed arm that continues to receive assignments; an interim decision timestamp earlier than its stated data cutoff; inconsistent active-arm lists across the decision report and randomization service; a retention action unsupported by the required looks; or an undocumented override. These are checks for internal consistency, not statistical tests of whether the design is beneficial.
By contrast, a realized allocation count that differs from its expectation is not, by itself, proof of faulty randomization. The distinction between allocation probability and evidence is essential: a probability describes the chance of assignment under the rule, not the strength of evidence for treatment benefit. Likewise, a path that looks unusually concentrated may warrant investigation of the probabilities and implementation, but its appearance alone does not establish that the rule was violated.
A path-by-path audit is complementary to, not a replacement for, simulation of operating characteristics. Simulation can assess the repeated-trial behavior of a fully specified design under stated scenarios; it does not establish what happened in the live implementation. A log audit can establish whether recorded decisions match the configured rule; it does not show how the design performs across hypothetical trials. Independent replay of a decision using the archived data snapshot is another useful check, but it depends on the replay using the same rule version, data definitions and numerical conventions.
The diagnosis also has limits. It cannot repair an outcome definition that was poorly chosen, establish that a treatment caused an observed outcome, or determine the validity of a joint efficacy-safety decision. Nor does a correct implementation guarantee desirable recruitment, precision or clinical value. Those questions require their own prespecified criteria and analyses. The purpose here is narrower: make the adaptive process interpretable, reproducible and faithful to its stated rules.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A fixed-probability cohort of 80 participants uses probabilities \(0.50\) and \(0.50\). The realized counts are 34 and 46. Does this discrepancy alone establish an implementation error?
- A retention rule requires a posterior probability below \(0.05\) at two consecutive looks. The values are \(0.04\), then \(0.06\), then \(0.03\). Is removal justified at the third look?
- A decision requires 15 mature outcomes per arm. One arm has 14 mature outcomes and 6 pending. What should an auditor check before accepting an allocation update?
- A removed arm has zero allocation probability in the configuration, but the assignment log shows a participant allocated to it. Name two records to inspect first and one concrete corrective action.
Solutions. (1) No. The expected counts are \(80(0.50)=40\) per arm, but realized counts need not equal expectations. Verify the probabilities and assignment-level log before concluding there was an error. (2) No. The value \(0.06\) breaks the consecutive-look sequence, so \(0.03\) is only one qualifying look after that break. (3) Check the prespecified pending-outcome policy, the cutoff and maturity status; if the rule requires 15 mature outcomes in every arm, the update is not yet eligible, and the stated fallback action should be followed. (4) Inspect the contemporaneous configuration snapshot and randomization-service log, including timestamps. A concrete action is to block randomization until the active-arm configuration is updated and verified after removal.