Why Test Adaptive Decisions With Small Cases?
An adaptive trial’s operating characteristics depend on more than its statistical model. They also depend on whether the implemented rules correctly interpret eligibility, thresholds, stopping conditions, and selection priorities. A one-character boundary error can change a decision for a trial state exactly at a cutoff. A unit test makes such errors visible by passing a deliberately chosen input to a decision function and comparing its output with a result worked out in advance.
A unit test targets a small component, such as the function that processes an interim look. Its inputs are fixed rather than randomly generated, so the expected decision can be checked directly. This is different from simulating many trials to estimate a design’s operating characteristics: a unit test checks whether the rule was implemented as specified, not whether the design has desirable power or error rates.
Specify the Rule Before Testing It
The examples use a hypothetical trial with experimental arms A and B. At a decision look, each arm has an evaluable sample size \(n_j\), a count \(e_j\) of safety events, and an efficacy score \(z_j\). The score is simply a supplied decision statistic in these examples; the tests do not make a claim about its sampling distribution. The trial uses a fixed arm order, A before B, to resolve exact ties.
The hypothetical rule is deliberately explicit. An arm is eligible if it has at least 20 evaluable participants and its safety-event proportion is no greater than 0.20. An arm with no evaluable participants is ineligible; its event proportion is undefined. At an interim look, the rule first determines eligibility. If no arm is eligible, it stops with no selection for lack of an eligible arm. Otherwise, it stops for futility with no selection if the largest eligible-arm score is less than 0.5. If the largest score is at least 2.0, it stops early and selects the eligible arm with the largest score. In all other cases, it continues.
At the final analysis, the same eligibility rule applies. If no arm is eligible, the trial selects none. Otherwise, it selects the eligible arm with the largest score if that score is at least 1.5; if the largest score is less than 1.5, it selects none. Ties at either look are resolved in favor of A. A real protocol may use different rules; the testing principle is to encode the protocol’s actual rules, including its equality conventions and precedence.
The rule also implies basic input preconditions: counts are nonnegative integers, \(e_j \leq n_j\), and scores used for eligible arms are finite numbers. The function should return a consistent, explicit result for every permitted input. Defining these details prevents a test from relying on accidental behavior for missing, malformed, or ambiguous data.
Worked Tests for Eligibility and Boundary Conditions
Worked Example: Testing Eligibility at the Safety Boundary
At an interim look, arm A has \(n_A=19\), \(e_A=0\), and \(z_A=2.4\). Arm B has \(n_B=20\), \(e_B=4\), and \(z_B=1.7\). Arm C is not part of this trial; it is included here only to illustrate a second hypothetical arm state: suppose the actual comparison arm B instead had \(e_B=5\), with all other B inputs unchanged. Work the eligibility decisions and resulting interim actions for the original A/B case and for the modified B case.
For A, \(n_A=19<20\), so A is ineligible regardless of its high score. For B in the original case, \(n_B=20\) meets the sample-size requirement and \(\frac{e_B}{n_B}=\frac{4}{20}=0.20\). Equality meets the rule’s “no greater than” condition, so B is eligible. Its only eligible score is 1.7. Since \(0.5 \leq 1.7<2.0\), the expected interim output is continue, with no selected arm.
In the modified case, B has \(\frac{5}{20}=0.25>0.20\), so it is ineligible. A is also ineligible, leaving no eligible arms. The expected output is stop with no selection: no eligible arm. The score of 2.4 for A must not override its sample-size ineligibility.
For the safety boundary, an implementation can avoid relying on rounded proportions. For \(n_j>0\), multiplying both sides of \(e_j/n_j \leq 1/5\) by the positive quantity \(5n_j\) gives an equivalent integer comparison:
For example, the original B state passes because \(5(4)=20\leq20\); the modified state fails because \(5(5)=25>20\). This equivalence is valid only when \(n_j>0\), which is why the zero-denominator case needs its own eligibility handling. It also explains why tests should include equality: an implementation using a strict inequality would incorrectly reject the boundary case.
Worked Example: Testing the Futility Boundary
Suppose the inputs establish that both arms are eligible, and their interim scores are \(z_A=0.49\) and \(z_B=0.30\). The largest eligible score is \(\max(0.49,0.30)=0.49\). Because \(0.49<0.5\), the expected action is to stop for futility with no selection.
Now change only A’s score to \(z_A=0.50\). The maximum is exactly 0.50, which is not less than 0.5. It is also less than 2.0, so the expected action is to continue. This pair of tests checks the distinction between “less than 0.5” and “less than or equal to 0.5.” The equality case is especially informative because a mistaken operator changes the output.
Test Interim and Final Selections
Worked Example: Testing Early Stopping and a Tie
Assume A and B are eligible at an interim look, with \(z_A=2.4\) and \(z_B=2.4\). Their maximum score is 2.4, so the rule stops early because \(2.4\geq2.0\). The scores are tied; the specified priority order selects A. The expected output is stop early and select A.
A useful companion test changes B’s score to 2.5 while leaving A at 2.4. Now the maximum is 2.5 and B has the larger score, so the expected output is stop early and select B. Together, these cases check both the tie-break rule and whether the code selects the actual maximum rather than always returning the first arm.
Worked Example: Testing the Final Selection Cutoff
At the final analysis, suppose both arms are eligible and \(z_A=1.49\), \(z_B=1.20\). The largest score is \(1.49<1.5\), so the expected result is select no arm.
Change only \(z_A\) to 1.50. The maximum is now exactly the cutoff, and the rule says to select when the maximum is at least 1.5. Therefore the expected result is select A. As at interim, testing values on both sides of the cutoff and exactly at it catches errors in the comparison operator.
Organizing Tests and Diagnosing Failures
A test should isolate the condition it is intended to check. Hold other inputs safely away from their thresholds when testing one boundary, and change one feature at a time when comparing two cases. For example, the futility-boundary pair changes only one score; this makes a difference in output interpretable as a consequence of the 0.5 comparison.
It is also useful to check the order of operations. In the specified interim rule, eligibility is determined before comparing scores. Scores from ineligible arms cannot trigger early stopping or determine the selected arm. A test with an ineligible arm carrying the largest score, as in the first example, checks this directly. Tests should also cover no eligible arms, a single eligible arm, ties, and an eligible arm whose score lies between the futility and early-stopping thresholds.
Write down input fields, valid values, boundary inclusions, decision precedence, tie-breaking, and the required output.
Choose counts and scores that trigger one rule, including values exactly at its boundary where appropriate.
Apply the written conditions by hand and record the action, selected arm, and reason.
When a test fails, inspect inputs, condition order, eligibility filtering, and tie-breaking before changing the implementation.
Failures can reveal different defects. A boundary failure often points to an incorrect strict versus inclusive comparison. A selection failure may indicate that an ineligible arm was not filtered out, that the maximum was computed over the wrong set, or that tie-breaking is unspecified in the implementation. A “no eligible arms” failure can expose division by zero or an unintended default selection. Invalid counts should be rejected or handled according to an explicit input policy, rather than silently producing a plausible-looking decision.
What Unit Tests Establish—and What They Do Not
Small deterministic tests are valuable because each has a transparent expected answer and can be rerun after a code change. Compared with a broad simulation, they are inexpensive and can target rare but consequential states, such as exact equality at a stopping boundary. Compared with informal code review, they provide a repeatable check that the implemented function returns the specified output for the tested inputs.
Their limitation is equally important: passing a set of unit tests does not establish that the rule has the intended operating characteristics, that all relevant inputs have been tested, or that the statistical quantities supplied to the rule were computed correctly. Unit tests check selected input-output mappings. They complement, rather than replace, broader simulation and other forms of software validation. A practical test suite should be expanded when a rule changes, and should keep the expected outputs tied to an explicit written specification.
Common mistake: “The code passed several simulated trials, so the decision rule is correct.” A finite set of stochastic runs may never encounter an equality case, a tie, or an unusual eligibility pattern. The correction is to create small deterministic tests that deliberately place inputs at those conditions and state the expected result in advance. Conversely, passing those tests is not proof that every possible input is handled correctly; coverage is only as broad as the cases and properties checked.
Check Your Understanding
Answer each question using the hypothetical rules in this tutorial. Solutions follow the questions.
- At an interim look, an arm has \(n=20\), \(e=4\), and score \(z=2.1\). Is it eligible, and what action follows if it is the only arm?
- Two eligible arms have interim scores 0.5 and 1.2. Does the rule stop for futility, stop early, or continue?
- At the final analysis, both arms are eligible with scores 1.5 and 1.5. Which arm is selected?
- Why should a test include an arm with a score above 2.0 that fails the sample-size eligibility requirement?
Solutions. (1) The arm is eligible because \(n=20\) meets the minimum and \(4/20=0.20\) is within the inclusive safety limit. Since \(2.1\geq2.0\), stop early and select that arm. (2) The maximum is 1.2, which is not below 0.5 and is below 2.0, so continue. (3) Both meet the inclusive 1.5 final cutoff; the tie-break priority selects A. (4) That test checks that eligibility is applied before the score comparison: the ineligible arm must not trigger early stopping or be selected.