Stress-Testing Boundary Cases

Why Boundary Cases Need Direct Tests

A simulation can run for many replicates without ever reaching an unusual decision state. That does not show that the decision code handles such states correctly. A rare exact tie, an empty set of continuing arms, or a stage at its minimum allowed sample size may expose a defect that ordinary random draws do not reveal. Boundary testing supplies these states deliberately and checks the design’s specified behavior.

The focus here is not to estimate operating characteristics. It is to verify that a MAMS decision rule and its software behave coherently at the edges of their inputs. The central requirement is to translate the protocol into explicit rules before testing: which comparisons are strict or inclusive, what happens if no experimental arm continues, and what sample sizes make each calculation defined.

Decision States and Assumptions

For a generic interim decision, suppose there are \(K\) experimental arms and a shared control. At a stage, the code calculates a statistic \(Z_k\) for each experimental arm \(k\), oriented so larger values favor that arm over control. Let \(c_f\) be a futility boundary and \(c_e\) an efficacy boundary, with \(c_f<c_e\). This tutorial uses the following illustrative rule: stop an arm for futility when \(Z_k\le c_f\), declare efficacy when \(Z_k\ge c_e\), and continue the arm otherwise. A real protocol may use different statistics, boundaries, or actions; tests must encode that protocol rather than assume this example’s rule is universal.

For the numerical illustrations, assume the interim outcomes are independent and normally distributed within arms, with a known common standard deviation \(\sigma\), and that each arm and control has \(n\) observations at the interim. For an arm-versus-control contrast, use \(Z_k=(\bar{x}_k-\bar{x}_0)/SE\), where \(SE=\sigma\sqrt{1/n+1/n}\). These assumptions make the examples’ calculations transparent; they are not requirements for every MAMS design. If a protocol uses a different model or statistic, its own assumptions and valid sample-size range govern the tests.

A decision implementation should keep distinct statuses distinct. An arm stopped for futility is not the same as an arm declared efficacious, even though neither may recruit further participants in the same way. Record the action for each arm, and separately record whether any experimental arms remain eligible to continue. In particular, “no arms continue” does not by itself say whether all arms stopped for futility, all crossed an efficacy boundary, or a mixture occurred.

Worked Tests for Empty and Full Continuation Sets

Ties, Inclusive Boundaries, and Ordering

Comparisons at a boundary must match the written rule exactly. With the rule above, \(Z_k=c_f\) means futility, while \(Z_k=c_e\) means efficacy. A test using values just below and just above each boundary can check the transition; a test using the exact value checks whether the implementation uses \(\le\) and \(\ge\), rather than accidentally replacing them with strict comparisons.

Equal statistics for different arms are another useful case. If each arm is judged independently against fixed boundaries, a tie between arms does not itself require choosing a winner: both receive the action implied by their shared statistic. If a protocol instead ranks arms or selects a fixed number for continuation, it must specify how a tie in the ranking is resolved. A stable arm label can make the software deterministic, but that programming convention must not silently change the trial’s statistical rule.

Boundary ordering itself should be validated. For the three-way rule used here, \(c_f<c_e\) leaves a nonempty interval of values that continue. If \(c_f=c_e\), a statistic at that common value satisfies both inclusive stopping conditions; if \(c_f>c_e\), some values satisfy both as well. Such a configuration is ambiguous unless a protocol explicitly gives precedence or defines another rule. A configuration check should reject it when the intended rule requires separated boundaries.

Minimum Sample Sizes and a Short Justification

A minimum sample-size test must be tied to the statistic actually calculated. With the known-\(\sigma\) normal statistic used here, the standard error is defined for \(n\ge1\) in both groups. If instead the code estimates each group’s variance from its observations, a group sample standard deviation is not defined at \(n=1\); that calculation requires at least two observations per group. A design’s permitted minimum and the statistic’s mathematical requirements therefore need to agree.

For the known-\(\sigma\) illustration, take the minimum \(n=1\), \(\sigma=10\), and a one-observation outcome of 12 in an experimental arm and 10 in control. The observed means are those values, and \(SE=10\sqrt{1/1+1/1}=14.142\) outcome units. The statistic is \(Z=(12-10)/14.142=0.1414\), which lies strictly between 0 and 2.5. Under the stated rule the arm continues. This is a valid calculation under the known-variance model, although the estimate is imprecise.

The underlying check follows directly from the formula: \(SE=\sigma\sqrt{2/n}\) when both group sizes equal \(n\). For \(\sigma>0\) and \(n\ge1\), this is positive and finite, so division is defined. At \(n=0\), the expression contains division by zero and no group mean is available. Thus zero is not a valid sample size for this statistic; it is not a low-information observation that can be passed through the same formula.

How to Build and Diagnose Boundary Tests

A useful test specifies its inputs and expected outputs before the code runs. For each case, include the number of arms, stage, sample sizes, statistics or outcomes, boundaries, per-arm status, continuing-arm set, and any trial-level action. When possible, use hand-calculated values so the expected result is independent of the implementation being checked.

Symptoms of a boundary defect include an arm changing status when a statistic is exactly on a threshold, a continuation count that disagrees with the per-arm statuses, or a later-stage calculation using an empty arm set. A test that passes only for ordinary interior values is incomplete. Conversely, a deliberate boundary input is not automatically an invalid replicate: it is a valid test if it satisfies the design’s assumptions and input requirements.

Deterministic edge tests complement, rather than replace, simulation. Randomized simulation estimates how often events occur under specified scenarios; deterministic tests check what the code does when a selected event occurs. Analytic special cases can also serve as checks when a known answer is available, while boundary tests target exact logic and state transitions that such aggregate comparisons may not expose. None of these checks establishes the correctness of every aspect of a design.

Common Pitfalls and Limits

• Using an unspecified tie rule. Treating equal arm statistics as a reason to select one arm is unjustified unless the protocol calls for ranking and defines a tie-break.
• Ignoring equality. A test only at values well inside decision regions cannot reveal whether the code implements inclusive or exclusive thresholds as specified.
• Equating no continuation with one particular outcome. An empty continuation set can arise from different arm-level decisions. Preserve those statuses and apply the stated trial-level policy.
• Testing an unsupported sample size as if it were valid. If a statistic needs an estimated variance, one observation per group may be insufficient even though a known-variance statistic can be computed there.
• Assuming edge tests prove operating characteristics. They verify selected logic paths; they do not estimate familywise error, power, expected sample size, or performance across all scenarios.

The main alternative is to rely on random simulation alone, which is useful for operating characteristics but may rarely generate exact ties or particular empty-set states. Direct deterministic tests are more targeted and interpretable, but have limited coverage unless the important cases are chosen systematically. The strongest practice is to combine explicit boundary cases with broader simulation and independent checks appropriate to the design.