Why Test Design Code?
Group sequential design software can produce plausible-looking results even when a boundary comparison is reversed, a look is skipped, or an output summary uses the wrong denominator. Tests make these behaviors explicit: they compare what the software does with what a clearly specified input contract requires. The earlier material on implementing stopping rules and independently verifying software results provides the context; here the focus is on turning expected behavior into repeatable tests.
A test does not establish that a design is appropriate, nor does a passing test demonstrate that its type I error or power is acceptable. It checks a narrower claim: for specified inputs and conditions, the implementation returns the specified result. Statistical operating characteristics still require evaluation under an appropriate model and scenario matrix.
State the Contract Before Writing Tests
A test needs a precise expected behavior. For a design function, that begins with an input contract: what values are allowed, what relationships among inputs must hold, and how invalid values are reported. For example, a function that applies a stopping rule might require a positive number of looks, strictly increasing information times, one boundary per look, and finite observed statistics. If the function accepts a significance level as an input, it should require \(0 < \alpha < 1\). These are contract choices that must match the function’s specification; tests cannot decide them on its behalf.
A useful test suite covers ordinary valid inputs, invalid inputs, and boundary cases. Check not only that an invalid input fails, but also that it fails in the intended way, such as returning a clear validation error rather than proceeding to an unrelated calculation. For valid inputs, include simple cases where the expected result can be worked out by hand. Those cases can expose off-by-one errors and incorrect handling of equality at a boundary.
Assume for the stopping examples below that the function receives a finite sequence of test statistics \(Z_j\) and fixed upper efficacy boundaries \(b_j\), one for each scheduled look. The specified rule rejects \(H_0\) at the first look \(j\) for which \(Z_j \ge b_j\); if no such look occurs, it reports no rejection by the final look. The tests are checks of this specified rule, not a claim that any particular set of boundaries controls the overall type I error rate. That error rate depends on the design and the joint distribution of the test statistics across looks.
Worked Example: Testing Input Validation
Suppose a hypothetical function accepts information times \(t=(0.4,0.7,1.0)\), three upper boundaries, and a significance level \(\alpha\). Its contract requires that information times be finite and strictly increasing, that each lie in \((0,1]\), that the final information time equal 1, and that \(0 < \alpha < 1\). Design tests for one valid input and two invalid inputs.
Valid case. Use \(t=(0.4,0.7,1.0)\), three finite boundaries, and \(\alpha=0.05\). The times satisfy \(0<0.4<0.7<1.0\), each is at most 1, and the final value is 1. The test should confirm that validation succeeds and that the function continues to produce its specified output.
Non-increasing times. Change the times to \(t=(0.4,0.7,0.7)\). The strict-increase requirement fails because \(0.7\) is not less than \(0.7\). The test should expect the documented validation error. It should not accept a later failure caused by an indexing operation or calculation.
Invalid significance level. Use \(\alpha=1.2\), holding the other inputs valid. Since \(1.2\notin(0,1)\), validation should reject the input. This test isolates the alpha check: changing one input at a time makes it easier to identify which contract rule was violated.
The expected outcomes follow directly from the stated contract. In an actual implementation, tests should also cover a non-finite time, a missing boundary, and a final time below 1 if those cases are possible at the function interface.
Test Stopping Decisions With Small Paths
A stopping-rule unit test should separate the rule’s logic from random-number generation. Supply a fixed path of statistics and verify the decision and look. Include paths that cross at the first look, cross at a later look, never cross, and equal a boundary exactly. The equality case is important because \(Z_j \ge b_j\) and \(Z_j > b_j\) are different rules.
For the rule defined above, the first crossing can be found by checking the looks in order. If the crossing set is \(C=\{j:Z_j\ge b_j\}\), then the stopping look is the smallest member of \(C\), when \(C\) is nonempty. This directly justifies checking that later crossings do not overwrite an earlier stopping look.
Worked Example: Checking the First Boundary Crossing
Consider a hypothetical three-look design with fixed upper boundaries \(b=(2.5,2.2,2.0)\) and a supplied path \(Z=(1.1,2.2,2.6)\). Assume the function implements the rule \(Z_j\ge b_j\) and returns the first crossing look.
At look 1, \(1.1<2.5\), so the rule does not reject. At look 2, \(2.2=2.2\), so the equality condition is met and the first crossing is look 2. Look 3 also crosses because \(2.6\ge2.0\), but it cannot change the first stopping look.
A unit test should therefore assert rejection at look 2, not look 3. A second test with \(Z=(1.1,2.1,1.9)\) should assert no rejection by look 3, because \(1.1<2.5\), \(2.1<2.2\), and \(1.9<2.0\). These checks use the supplied path as fixed input; they do not require simulation.
A robust set of stopping tests checks both the decision and its associated outputs: stopping look, information time, and sample size if the function returns them. Verify that a stopped trial is not assigned a later look’s sample size. Also check the final-look boundary and any defined rule for ties or missing values. If a function combines efficacy and futility rules, tests should cover each decision and any priority rule for simultaneous conditions; do not assume a priority that is absent from the specification.
Check Summary Functions With Hand-Calculated Results
Summary functions are well suited to unit tests because a small fixed input can produce an exact expected result. First state what each summary means: for example, rejection proportion is the number of simulated trials rejecting \(H_0\), divided by the number of included trials; mean stopping look is the arithmetic average of the recorded look indices. Confirm how the function handles missing, failed, or incomplete simulation runs. An unintended change in the denominator can affect all reported operating characteristics.
Worked Example: Verifying a Summary of Trial Outcomes
Suppose a hypothetical summary function receives eight completed trial records. Their rejection indicators are \(1,0,1,0,0,1,0,0\), their stopping looks are \(1,2,2,3,1,3,2,3\), and the scheduled sample sizes at looks 1, 2, and 3 are 40, 80, and 120 participants. Calculate the expected summaries for a test.
There are three rejections among eight trials, so the rejection proportion is
The stopping-look indices sum to \(1+2+2+3+1+3+2+3=17\), so their mean is \(17/8=2.125\) looks. Mapping each recorded look to its sample size gives \(40,80,80,120,40,120,80,120\). These sum to 680 participants, so the mean sample size is \(680/8=85.00\) participants.
A test can compare the returned values with \(0.375\), \(2.125\), and 85.00, and verify that the units and labels correspond to the documented outputs. If the function also reports the number of trials, it should return 8. Keeping this fixture small makes every expected value auditable.
Regression Tests: Detect Changes, Not Just Failures
A regression test preserves evidence about behavior that has already been checked. For deterministic functions, a compact fixture can store inputs and expected outputs, including discrete decisions and numerical summaries. When code changes, the test compares current output with the established result. A changed result is a signal to investigate, not automatic proof that the new code is wrong: the change might be a defect, a deliberate correction, or an intentional specification update.
Use exact comparisons for discrete outputs when the contract specifies them, such as the selected stopping look or an error category. For floating-point results, define a tolerance appropriate to the quantity and calculation. A common comparison is
where \(\varepsilon_{\mathrm{abs}}\) is an absolute tolerance and \(\varepsilon_{\mathrm{rel}}\) is a relative tolerance chosen in advance. A tolerance should reflect numerical precision and the output’s required accuracy, not be widened merely to make a failing test pass. For outputs near zero, the absolute tolerance matters; for larger outputs, the relative tolerance can be more informative.
Simulation outputs require particular care. If random-number generation is involved, a fixed seed may make a test reproducible within a specified environment, but results can still vary with changes to random-number libraries, parallel execution, or software versions. Prefer deterministic unit tests for logic and use regression comparisons for simulation summaries only when the expected variability and reproducibility conditions are understood. Do not treat two Monte Carlo estimates as identical merely because they came from the same design; simulation error is part of their variability.
Write down valid inputs, invalid inputs, boundary conventions, return values, and summary definitions before encoding expectations in tests.
Use hand-checkable inputs that isolate one behavior, such as equality at a boundary or a single invalid argument.
Assert decisions, indices, units, counts, and documented errors rather than checking only that the function ran without crashing.
When a regression test fails, determine whether the change is a defect or an intentional, documented revision before updating the expected result.
Diagnostics, Alternatives, and Limitations
Warning signs include tests that pass only for one ordinary input, tests that check only whether an output is nonempty, and regression baselines that are regenerated automatically after every failure. Tests can also be misleading when a fixture’s expected value was copied from the same function under test: that checks repeatability, not correctness. Hand calculations, a separate implementation, or an independently verified reference value provide stronger expectations.
Unit tests, integration tests, and reference comparisons answer related but different questions. Unit tests isolate a function’s contract and make failures easy to locate. Integration tests check that components work together, such as a simulator passing trial paths to a stopping-rule function and then to a summary function. Comparison with an independently implemented reference design can reveal broader discrepancies, but it may be more costly and can share assumptions or errors with the implementation. Regression tests complement these approaches by checking that established behavior remains stable.
No single test suite can prove a complex design implementation correct for every possible input. Tests cover selected cases; numerical edge cases, platform differences, undocumented assumptions, and untested branches may remain. Passing tests also cannot establish that the statistical model is suitable, that boundaries achieve intended operating characteristics, or that an output is correctly interpreted for a clinical question. Tests should therefore be maintained alongside specifications and supplemented by independent statistical review.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A rule rejects at the first look where \(Z_j\ge b_j\). With \(Z=(2.0,2.4)\) and \(b=(2.0,2.3)\), what stopping look should a unit test expect, and why?
- A summary contains 4 rejections among 10 completed trials. What rejection proportion should a test expect? What would be wrong with using 4 as the proportion?
- Why should a test include an input with information times that are equal, if the contract requires strictly increasing times?
- A floating-point regression result differs from its expected value by a small amount. What should guide the tolerance used to compare them?
- What does a passing stopping-rule unit test establish, and what does it not establish about type I error?
Solutions. (1) The expected stopping look is 1: \(2.0\ge2.0\), so the first look already crosses, regardless of the second look. (2) The expected rejection proportion is \(4/10=0.4\). The count 4 is not a proportion and can exceed 1 when there are more than one rejection per trial record or is otherwise on the wrong scale; the function’s specified output is a fraction of completed trials. (3) Equal times violate the strict-increase contract, so the test checks that this invalid input is rejected clearly rather than being silently accepted. (4) The tolerance should reflect the numerical precision and required accuracy of the output, using pre-specified absolute and relative tolerances where appropriate. It should not be enlarged simply to hide an unexplained discrepancy. (5) It establishes that the tested input follows the specified stopping logic. It does not establish type I error control, which depends on the boundaries and the joint distribution of the statistics under \(H_0\), and must be evaluated separately.