Unit Tests for Interim Decision Logic

Why Test Interim Decision Logic Directly?

A MAMS simulation repeatedly applies interim rules to statistics calculated for several treatment arms. A small coding error in a comparison, such as using the wrong inequality at a boundary, can alter which arms continue and whether the trial stops. If that error is embedded in a large simulation, its consequences may be hard to locate among random variation and other code. Unit tests address a narrower question: given a deliberately chosen input, does the decision function return the result required by its stated rule?

These tests use hand-constructed values rather than simulated trial data. They are deterministic: the same inputs should always produce the same outputs. The purpose is to verify the software contract for the interim logic, not to estimate error rates, power, or expected sample size. Those operating characteristics depend on the full design and on the data-generating assumptions, not just on whether a function implements a rule correctly.

State the Decision Contract First

For a concrete example, suppose an interim function receives a statistic \(Z_j\) for each currently active arm \(j\). Assume larger values indicate stronger evidence in favour of that treatment. Let \(c_F\) be a futility boundary and \(c_E\) an efficacy boundary, with \(c_F<c_E\). This direction and ordering are part of the example’s contract; another design may use a different statistic or different decision rules.

To keep the example precise, take \(c_F=0.50\) and \(c_E=2.50\). The decision function should reject invalid configurations such as \(c_F\geq c_E\), and it should receive a statistic for every active arm. For these unit tests, assume the arm identifiers are distinct, the statistics are finite numbers, and the input lists active arms only. These are preconditions of this example’s function, not statistical assumptions about the distribution of the statistics.

The function can return three pieces of information: the set of arms dropped for futility, the set meeting efficacy, and the trial status. Making those outputs explicit helps a test pinpoint a failure. For example, a trial status of “stop for efficacy” does not by itself verify that the correct arm was identified as meeting the boundary.

A Boundary Partition Justifies the Test Cases

The threshold comparisons divide every possible real-valued statistic into three non-overlapping regions. This gives a short justification for why tests must include values on both sides of each boundary and the boundary values themselves.

Because \(c_F<c_E\), every finite \(Z_j\) belongs to exactly one of these regions. The first region means drop for futility; the middle region means continue; the final region means efficacy. At \(Z_j=c_F\), the strict futility comparison is false, so the arm is not dropped. At \(Z_j=c_E\), the inclusive efficacy comparison is true, so the arm meets efficacy. This partition also shows why a test only with values far from the boundaries is insufficient: it can pass even if the implementation uses the wrong equality convention.

A practical unit test pairs each input with an explicit expected output. The expected output should be worked out from the written rule before running the function; otherwise, a test can simply repeat the same mistaken logic as the implementation.

Build Tests Around Expected Outputs

A useful test set is small enough to inspect by hand but varied enough to expose common logical errors. Include separate tests for each decision category and for trial-level combinations. A simple decision table can act as a checklist.

The first four rows test the arm-level classification. The last two test the trial-level rule, which depends on the combined decisions. Tests should assert the returned arm sets as well as the status. A function that returns the right status for the wrong reason is not implementing the complete contract.

Useful Properties and Failure Diagnostics

Alongside specific input-output tests, check simple logical properties implied by the contract. These do not replace hand-worked examples; they broaden the ways an inconsistency can be detected.

• Mutual exclusivity: no arm can be both dropped for futility and meet efficacy, because \(c_F<c_E\).
• Completeness: every valid statistic is classified as dropped, continuing, or meeting efficacy.
• Boundary equality: \(Z_j=c_F\) is not a futility drop, while \(Z_j=c_E\) meets efficacy.
• Arm-order invariance: reordering the input arms should reorder corresponding outputs, not change which named arms are selected or the trial status.
• Monotonicity within this rule: increasing one arm’s statistic cannot turn an efficacy result into a non-efficacy result; decreasing it cannot turn a futility result into a non-futility result.

When a test fails, inspect the contract and the smallest failing input first. A wrong result at \(Z_j=0.50\) often indicates that the code used \(Z_j\leq c_F\) instead of \(Z_j<c_F\). A wrong result at \(Z_j=2.50\) often indicates that it used \(Z_j>c_E\) rather than \(Z_j\geq c_E\). If arm-level decisions are correct but the trial status is wrong, examine the aggregation rule: “any efficacy” and “no active arms remain” are distinct conditions in this example.

Use exactly representable boundary values such as 0.50 and 2.50 for direct equality tests. Do not add a tolerance to the decision comparison unless the design’s rule explicitly defines one: a tolerance changes the rule being tested. If statistics are calculated by preceding numerical code, separate tests can check those calculations; the unit tests here supply the statistic directly so that the decision function is tested in isolation.

What Unit Tests Do—and Do Not—Establish

A unit test checks a small function against a specified input-output contract. It is especially effective for branch logic, boundary conventions, and combinations of arm-level decisions. It is not evidence that the chosen boundaries are scientifically appropriate, that the test statistic has the intended distribution, or that the complete simulation produces correct operating characteristics.

Compared with running a large simulation, hand-built unit tests are easier to diagnose because the expected answer is known without Monte Carlo uncertainty. Compared with testing only an end-to-end simulation, they localize an error to a function and a particular decision branch. But they cover only the cases chosen. A function can pass several examples and still fail on an untested input, so use the decision partition and stated properties to design a compact but purposeful test set.

These checks also depend on a clearly specified contract. Some MAMS designs use different stopping conventions, such as stopping only when a particular set of hypotheses meets criteria. The illustrative “any arm meets efficacy” rule above must not be silently substituted for a protocol-specific rule. Translate the actual design’s logic into explicit conditions first, then construct tests for those conditions.