Protocol Deviations In Randomization-Based Inference

Why the Executed Assignment Mechanism Matters

In a response-adaptive trial, each assignment may depend on information available at that point, such as earlier responses and the current algorithm state. A randomization test therefore relies not just on the allocation rule written in the protocol, but on the probability law that actually generated the assignments. If a probability was changed without being recorded, a randomization distribution calculated from the protocol may not describe the trial that occurred.

A deviation does not automatically make randomization-based inference impossible. If the executed rule and its relevant history can be reconstructed, the reference distribution can be based on that rule. The difficulty is that an unrecorded or unreconstructable change leaves the assignment probabilities uncertain. In that case, using the planned probabilities as if they were the actual ones does not repair the problem.

For binary assignments, the conditional probability of a complete assignment path \(z=(z_1,\ldots,z_n)\), under the executed mechanism, is

Here each \(\pi_i\) must be evaluated using the history that would arise on the path being considered. In an adaptive design, it is generally not correct to hold later probabilities at their observed values while changing earlier assignments. The algorithm must be replayed on each candidate path, using the stipulated information available at each step.

When a Deviation Changes the Reference Distribution

A randomization-based test specifies a statistic \(T(Z,Y)\), where \(Y\) denotes the outcomes, and compares its observed value with values generated by the assignment mechanism under the null hypothesis. Under a sharp null of no effect for any participant, the potential outcomes are fixed and do not depend on assignment. The reference distribution is then obtained by varying assignments according to the mechanism that actually operated, while retaining the fixed outcomes and replaying any adaptive rule.

This procedure requires that the assignment law used for the replay is the true law conditional on the information being held fixed. In practice, that means knowing the relevant decision history and how each assignment was generated. A probability recorded in the protocol is not a substitute for a probability that changed during operation. Nor does a log of final treatment labels alone usually reveal whether a label came from a random draw, an override, or a skipped step.

A short justification explains why the law matters. Under the sharp null, hold the outcomes fixed and let \(Z\) be generated from the correct executed distribution. For an upper-tail test, define \(p(Z)=P\{T(Z',Y)\geq T(Z,Y)\mid Y\}\), where \(Z'\) is a fresh draw from that same distribution. Ordering possible statistic values from largest to smallest, the probability mass assigned to outcomes with a tail probability at most a given threshold cannot exceed that threshold; ties make the usual nonrandomized p-value conservative. Thus the reference p-value is valid under the null. If the calculation instead draws \(Z'\) from a different, planned distribution, this argument no longer applies.

Adaptive use of recorded prior responses is not, by itself, a deviation that invalidates inference. Those responses belong in the history, and the algorithm’s response to them belongs in the replay. In contrast, an unrecorded decision based on information unavailable to the stated randomization rule may make the actual assignment probabilities unknowable. If assignment depended on a clinician’s unrecorded prognosis, for example, the recorded protocol probabilities need not describe assignment conditional on the information that drove it.

Known Adaptive Updates and Documented Overrides

A deviation can still permit valid randomization inference when its operation is sufficiently recorded. If the assignment probability changes according to a known rule using recorded information, include that information in the assignment history and apply the rule on every simulated path. If an override is deterministic once a recorded pre-assignment condition is met, the reference distribution must respect that condition; it must not create assignments that the executed rule could not produce.

The next example isolates an adaptive update. It is not a claim that every mid-trial change is acceptable: the rule and the information that triggered it must be known well enough to reproduce the assignment law.

Records, Diagnostics, and Limits

A usable assignment record should make it possible to reconstruct both what happened and the information available at each decision. Depending on the design, this includes:

• participant eligibility, assignment order, timestamps, and the final assigned treatment;
• the planned and executed conditional probabilities, or enough information to reproduce them;
• the algorithm version, state, inputs, and randomization output at each step;
• the prior responses and covariates used to update allocation probabilities;
• skipped steps, manual overrides, treatment switches, reasons, and when each decision occurred.

Diagnose problems by comparing the assignment log against the algorithm’s expected transitions. Check that each probability is attached to the correct decision and history, that the recorded assignment is consistent with the draw or documented override, and that candidate-path probabilities sum to 1. Investigate unexplained changes, missing timestamps, duplicated or out-of-order records, and deviations whose triggers were not recorded. A mismatch is a signal to establish the actual mechanism, not a reason to silently substitute planned probabilities.

The inference target also matters. Randomization inference based on \(Z_i\) concerns the randomized assignment. If a participant assigned to treatment instead received control, an analysis of assigned treatment and an analysis of treatment received answer different questions. Replacing \(Z_i\) with treatment received does not preserve the original randomization distribution; inference about receipt requires additional assumptions and a method appropriate to that target.

When a deviation rule is documented, one can often condition on recorded pre-assignment information and replay the resulting mechanism. When an override is based on unrecorded information, or when the assignment law cannot be reconstructed, the intended design alone does not identify the correct randomization reference distribution. Sensitivity analyses under explicitly stated plausible mechanisms may show how conclusions vary, but they do not turn an unknown assignment law into a known one. Randomization-based inference is also limited to the null and statistic being tested; a valid sharp-null test does not automatically supply inference for a different, weaker treatment-effect null.

Common Mistake

Wrong reasoning: “The protocol used probability 0.5, so I can generate the randomization distribution using fair coins, even though some assignments were overridden.” This confuses the intended procedure with the procedure that generated the data. Correction: use the executed conditional probabilities and documented decision rules, replaying adaptive updates on each candidate path. If the executed rule cannot be reconstructed, state that limitation; do not present a protocol-based reference distribution as if it were the trial’s actual one.