The Null Case as a Stress Test

Why Examine the Null Case?

A response-adaptive design is often motivated by the possibility of directing more patients to a treatment that appears to perform better. But when treatments have equal outcomes, apparent differences arise from random variation. The allocation rule can react to those chance differences, changing who is assigned to each arm and how later estimates behave. The null case is therefore a stress test: it asks whether the full design and analysis procedure behave properly when there is no treatment advantage to discover.

A null evaluation is not a claim that treatments are equal in practice. It is a controlled operating-characteristic assessment. It should preserve the proposed assignment rule, outcome model, enrollment size, and analysis procedure, and repeat the entire trial many times. Looking at one simulated trial, or at average allocation alone, cannot establish how often the analysis rejects a true null or how variable its estimates are.

For two treatments, let \(n_A\) and \(n_B\) be their final sample sizes, with total \(N=n_A+n_B\). One simple measure of allocation imbalance is \(|n_A-n_B|\). A related descriptive quantity is the equal-allocation effective sample size, \(N_{\mathrm{eff}}=4n_An_B/N\). It equals \(N\) for equal allocation and decreases as the allocation becomes more uneven. It describes the allocation's balance; it is not, by itself, a correction for adaptive sampling or a guarantee about estimator variance.

Specify the Null and the Whole Procedure

A useful null stress test starts by specifying exactly what is null. In a global null, each patient's potential outcomes have the same distribution under A and B. In the examples below, outcomes are independent Bernoulli variables with common success probability \(p=0.30\), regardless of assigned treatment. Assignment may depend on outcomes already observed, but the next patient's outcome is independent of that history. We also assume a fixed enrollment total, complete outcome ascertainment, no interference between patients, and a fully specified assignment rule.

These assumptions make the simulated trials reproducible and give a clear target: the null distribution of the complete procedure. If the real design includes delayed outcomes, varying enrollment, clustering, or a different null (for example, equality of means but not of outcome distributions), those features must be represented in its own evaluation. A result under one null model does not establish operating characteristics under every plausible data-generating process.

Each replicate must rerun the adaptive decisions from the beginning. For each trial, record the final allocation, the treatment-effect estimate, its uncertainty measure, and whether the prespecified test rejects \(H_0\). Across replicates, summarize the rejection fraction, the mean estimate, its standard deviation, and allocation summaries. This separates four related but distinct questions: Was allocation uneven? Was the estimate centered at the null value? How much did estimates vary? Did the test reject too often?

An Analytic Illustration of Adaptive Bias

The following deliberately small example isolates one mechanism by which an adaptive rule can affect a familiar estimator. It is not a recommended trial design: with only three patients, conventional inference would not be meaningful. Under the global null, let all outcomes be independent \(N(0,1)\). Randomly assign patient 1 to A or B with equal probability, then assign patient 2 to the opposite arm. Let \(X_A\) denote the outcome of the patient assigned to A. Assign patient 3 to A if \(X_A>0\), and to B otherwise. This specifies the rule for every possible observed history.

Repeated-Trial Evaluation of Type I Error

For a larger illustrative evaluation, consider this fully specified response-adaptive rule with \(N=100\) binary outcomes. Randomize the first 20 patients independently, with probability \(0.5\) to each arm. After that, let \(n_j\) and \(s_j\) be the number assigned and the number of successes observed in arm \(j\), for \(j\in\{A,B\}\). Calculate the smoothed response rate \(r_j=(s_j+1)/(n_j+2)\), and assign the next patient to A with probability \(q_A=0.1+0.8r_A/(r_A+r_B)\); assign to B with probability \(1-q_A\). Under the stated Bernoulli null, each new outcome is generated independently with success probability \(0.30\), after its assignment. This keeps both assignment probabilities between 0.1 and 0.9.

For illustration, suppose a hypothetical simulation of 20,000 complete trials gives the summary below. These are hypothetical Monte Carlo results for this specified scenario, not empirical findings or universal properties of response-adaptive randomization. The prespecified analysis is the ordinary two-proportion pooled score test: calculate the pooled proportion \(\hat{p}\), compute \(Z=(\hat{p}_A-\hat{p}_B)/\sqrt{\hat{p}(1-\hat{p})(1/n_A+1/n_B)}\), and reject at the two-sided 0.05 level when \(|Z|>1.96\). The null outcomes are binary; if either arm has zero patients or the pooled proportion is 0 or 1, define the test decision as non-rejection, and otherwise use the stated rule.

Why Arm Sizes Alone Do Not Determine Variability

For independent outcomes with common variance \(\sigma^2\), the familiar variance formula for a difference in sample means is \(\sigma^2(1/n_A+1/n_B)\) when arm sizes are fixed, or are otherwise noninformative about the outcomes entering the means. In response-adaptive allocation, final arm sizes can depend on previously observed outcomes. Conditioning on final sizes can therefore select outcomes and change the conditional distribution of the estimator. One must not infer its conditional variance from the two final counts alone.

The appropriate null stress test estimates variability across complete repeated trials under the specified rule. In the hypothetical evaluation, the reported standard deviation of 0.101 is the standard deviation of the 20,000 trial-specific estimates. It summarizes their across-trial spread under that simulation model; it is not obtained by substituting a typical pair of arm sizes into a fixed-allocation formula. If a conditional variance for a particular allocation history is needed, it must be derived for that assignment rule or estimated by simulation conditional on that history.

Diagnostics, Comparisons, and Limitations

A null stress test is most useful when it reports more than a single rejection fraction. Examine allocation distributions, not just their mean: a mean of 50 patients per arm can coexist with substantial trial-to-trial imbalance. Check the frequency of very small arm sizes, the estimator's mean and spread, and whether standard errors track empirical variability. Also inspect the assignment probabilities and the timing of updates. Unexpectedly extreme allocations, unstable estimates, or rejection rates that change markedly with simulation settings are reasons to investigate the implementation and assumptions.

The main comparison is with fixed allocation under the same null outcome model and analysis. Fixed allocation gives a useful benchmark because it does not respond to observed outcomes; it is not automatically a perfect standard if the chosen test has poor small-sample calibration. A second comparison is between the usual analysis and an analysis calibrated to the complete adaptive design. For example, a simulation-based null calibration can reproduce the assignment and analysis procedures under the stated null. Any calibration must preserve the dependence created by the actual adaptive rule rather than treating the final arm counts as if they had been fixed in advance.

The null case does not answer every design question. It does not show how often a genuinely effective treatment will be identified, nor how many patients will benefit when treatments differ. It also cannot establish robustness to alternative outcome distributions, missingness, or implementation errors unless those features are included. A favorable null result is evidence about one specified operating condition, not a general guarantee. Conversely, a poor result identifies a problem with the combination of assignment rule and analysis under the tested conditions; it does not by itself identify which component caused it.

Common Mistakes

• Calling balanced average allocation proof of good behavior. A mean of 50 per arm does not rule out large imbalances in individual trials. Inspect the distribution of allocations and estimator variability.
• Using the fixed-size variance formula after outcome-dependent assignment. Final arm sizes may select observed outcomes. Estimate variability across full simulated trials or derive it under the complete rule.
• Interpreting a Monte Carlo rejection fraction as exact. The fraction has simulation error. Report the number of replicates and its Monte Carlo standard error.
• Leaving branches of the allocation rule unspecified. Conditional expectations and overall bias cannot be derived reliably until the rule is defined for every reachable history.
• Treating the null as the only operating condition that matters. It tests false-positive behavior under a stated model, but says nothing by itself about performance when treatment outcomes differ.