Distinguishing Temporal Drift From Random Imbalance

Why a Time Pattern Needs a Reference

An arm-by-time table can show that one treatment received a larger share of assignments early and a smaller share later. That pattern alone does not tell us whether allocation departed from the design. The intended probability of assigning a participant to that arm may have changed over time, for example because the response-adaptive algorithm updated its probabilities. Even if the intended probability stayed fixed, random assignments can produce uneven counts across periods.

The relevant comparison is therefore between observed assignments and the probabilities in force immediately before those assignments. The probabilities provide a reference for ordinary random variation under the implemented randomization mechanism. A persistent discrepancy may warrant investigation, but a diagnostic statistic is not proof that the allocation process was biased or improperly implemented.

This framework requires that the recorded \(\pi_i\) is the probability actually used for assignment, that it is based only on information available before assignment \(i\), and that assignments are generated according to the stated mechanism. Calendar-time bins should be defined independently of the observed assignment pattern. If the algorithm uses earlier outcomes or assignments, those inputs belong to \(\mathcal{F}_{i-1}\); adaptivity does not invalidate the definition, but it makes correct logging of the probabilities especially important.

A changing \(\pi_i\) is not itself a discrepancy. It describes a changing intended allocation. The diagnostic question is whether the realized \(A_i\)'s are unusually far from the sequential probabilities that governed them—not whether each time period has the same treatment proportion.

Residuals Across Time Bins

For a calendar-time bin \(b\), let \(I_{ib}=1\) if participant \(i\) belongs to that bin and \(0\) otherwise. Define the bin's assignment residual total \(S_b\) and its predictable variance \(V_b\) by

The expected assignment count under the recorded probabilities is \(\sum_i I_{ib}\pi_i\), so \(S_b\) is the observed count minus that expected count. \(V_b\) is called the predictable variance because each contribution \(\pi_i(1-\pi_i)\) is determined from information available before assignment \(i\). It is the accumulated conditional variance used in martingale-based inference; it is not, in general, a variance conditional on the complete realized sequence of probabilities.

When \(V_b\) is adequate and no single assignment dominates the sum, a rough standardized diagnostic is \(Z_b=S_b/\sqrt{V_b}\). Under suitable martingale central limit conditions, this may be approximately standard normal. It is not an exact finite-sample test by definition. In particular, a probability sequence produced adaptively can reveal information about earlier assignments. Conditioning on that full realized sequence may therefore change the distribution of the residuals; treating it as fixed and independent of those assignments is generally unjustified.

Why the Sequential Variance Is Appropriate

The justification begins with the information available just before each allocation. Under the specified assignment mechanism,

The first equality follows because \(E(A_i\mid\mathcal{F}_{i-1})=\pi_i\). For the second, \(A_i\) is conditionally Bernoulli with probability \(\pi_i\), so its conditional variance—and thus the conditional second moment of \(A_i-\pi_i\)—is \(\pi_i(1-\pi_i)\).

For \(S=\sum_i D_i\), the residuals at different times have zero cross-products in expectation. If \(j<i\), then \(D_j\) is known by time \(i-1\), and

Consequently, under these assumptions,

The realized \(V\) is the predictable variance accumulated along the trial, not generally the exact variance of \(S\) conditional on the full probability path. Standardizing by its realized value is useful in martingale asymptotics, with appropriate regularity conditions; it does not license conditioning on that path. When those conditions are doubtful or the sample is small, calibration should respect the actual adaptive mechanism rather than assume a fixed probability sequence.

Distinguishing a Probability Trend From a Residual Trend

The assignment-probability timeline and the residual timeline answer different questions. The first describes what the design intended; the second describes departures from that intention. A rising treatment proportion in the table may simply follow a rising \(\pi_i\). Conversely, a roughly stable overall treatment proportion can conceal a sequence of residuals that are mostly positive early and mostly negative later.

Assessing Persistence Without Overreading a Signal

A sustained pattern is more informative than an isolated large residual, but visual persistence is not automatically statistically surprising. If many bins are inspected, some will appear unusual by chance. The analyst should define bins and any overall contrast before looking for a pattern when feasible. A global sum \(S=\sum_i(A_i-\pi_i)\) can detect overall excess or deficit, but positive and negative periods can cancel. Bin-specific residuals help reveal such cancellation, while repeated inspection increases the risk of false alarms.

For a formal assessment, one can use martingale-based procedures under their stated conditions or calibrate a statistic under the complete assignment mechanism. For an adaptive rule, valid calibration must preserve the way probabilities depend on available history. It is not enough to hold the observed probability path fixed if that path was itself shaped by earlier assignments. If the rule depends on outcomes, a simulation-based calibration must also represent the relevant outcome-feedback mechanism or use a valid conditional design argument; simply resampling assignments independently from a fixed list of observed probabilities can be wrong.

Compare this sequential approach with two tempting summaries. A raw chi-square comparison of treatment proportions across periods may use the wrong expected proportions when \(\pi_i\) varies, and it may ignore adaptive dependence. The overall arm total, by contrast, can look balanced while temporal residuals run in opposite directions in different periods. Neither summary replaces comparison with the probabilities actually used. Outcome residual plots address whether outcomes show temporal structure; assignment residuals address whether assignments departed from the randomization probabilities. These are related diagnostics, not interchangeable ones.

Diagnostics, Limitations, and a Common Mistake

Before interpreting a pattern, verify that the time bins were not chosen after inspecting the assignments, that every probability corresponds to the correct participant and decision time, and that the recorded value is the probability actually applied—not a target, rounded value, or later recomputation. Check for missing assignments, duplicate records, changes in eligibility or assignment rules, and periods with probabilities near zero or one. When \(\pi_i\) is near an endpoint, \(\pi_i(1-\pi_i)\) is small: few assignments in that arm may be entirely expected, and standardized diagnostics can become unstable if the total variance is small.

A departure can arise from ordinary random variation, a data or logging error, implementation differing from the protocol, a misspecified account of the algorithm, or another operational change. A statistical signal does not identify which explanation is correct. Nor does a lack of signal prove that allocation was unbiased in every relevant sense. These diagnostics assess a specified assignment mechanism and a specified pattern; they do not establish that calendar-time differences in outcomes are absent or that an estimated treatment effect is free of bias.

Common mistake: “The treatment share rose over time, so randomization failed.” This reasoning treats equal treatment proportions in every period as the benchmark. The correction is to compare each assignment with its pre-assignment probability. A rising share can follow a deliberate, correctly implemented change in \(\pi_i\), as in the second worked example. A persistent positive residual relative to \(\pi_i\) is more concerning, but remains a diagnostic signal rather than proof of bias.