Why Small Coding Errors Change Operating Characteristics
A group sequential simulation applies a planned sequence of interim analyses to many simulated trials. Its purpose is not just to generate test statistics: each trial must follow the design’s decision process in the correct order. A boundary attached to the wrong look, a comparison that omits equality, or a denominator that excludes inconvenient simulation runs can change estimated rejection rates, power, or stopping distributions.
The focus here is implementation fidelity. Assume a design has \(K\) planned looks, ordered information times \(I_1 < I_2 < \cdots < I_K\), and a specified decision rule at each look. For illustration, let \(Z_k\) be the test statistic at look \(k\), \(u_k\) its upper efficacy boundary, and \(\ell_k\) its lower boundary. A generic two-sided stopping rule rejects \(H_0\) at look \(k\) if \(Z_k \ge u_k\), and stops for the lower-boundary decision if \(Z_k \le \ell_k\). If neither condition holds, the trial continues unless it is at the final look. Some designs use only an upper efficacy boundary or have other specified actions; code must implement the actual rule, not assume this example applies to every design.
The implementation assumes that the data-generating model, analysis statistic, planned information times, boundaries, and decision rules have been specified consistently. It also assumes the simulation records enough information to determine whether each trial was rejected, stopped without rejection, reached the final look, or encountered a computational failure. Correct code cannot compensate for an incorrectly specified design, and a correct design can be misrepresented by faulty code.
A Reliable Evaluation Pattern
A safe implementation makes the order of operations explicit: identify the planned look, calculate its statistic, apply its boundary, record the first stopping decision, and proceed only if the trial continues. Keep the planned look number separate from any array position. In a zero-based programming language, planned look \(k\) may be stored at position \(k-1\); the boundary and information time must use the same mapping.
Set the stopping status to “continuing,” and initialize the stopping look and rejection indicator as unset. This prevents an uninitialized value from being mistaken for a valid final decision.
At look \(k\), use the statistic based on information available by \(I_k\), and retrieve the boundary specified for that same look.
Record the decision and look as soon as a stopping condition is met. Do not allow later looks to overwrite that result.
If no earlier stopping condition applies, record the final-look outcome. Separately flag a trial whose outcome could not be computed.
One simple check follows from the first-crossing definition. If \(A_k\) is the event that a trial first rejects at look \(k\), then the events \(A_1,\ldots,A_K\) are disjoint: a trial cannot have two first rejection looks. Thus the overall rejection indicator is the sum of the first-rejection indicators across looks. A code result that counts the same trial as rejecting at multiple looks has not implemented a stopping rule.
Worked Examples: Tracing Decisions and Indexing
Worked Example: Stop at the First Efficacy Crossing
Consider a hypothetical three-look design with upper boundaries \(u_1=2.40\), \(u_2=2.10\), and \(u_3=2.00\). A simulated trial produces \(Z_1=2.05\), \(Z_2=2.18\), and \(Z_3=2.70\). The rule is to reject at the first look where \(Z_k \ge u_k\).
At look 1, \(2.05 < 2.40\), so the trial continues. At look 2, \(2.18 \ge 2.10\), so the first crossing occurs at look 2. The trial is recorded as a rejection at look 2. Although \(2.70 \ge 2.00\) at look 3, that statistic is not used to revise the decision: the trial would not have continued to look 3 under the stated rule.
A loop that checks all looks and reports the last crossing could still mark this trial as a rejection, but would give the wrong stopping look and may distort sample-size summaries. The decision record should be frozen at the first crossing.
Worked Example: Use the Intended Boundary Comparison
Suppose a hypothetical upper-boundary rule is \(Z_k \ge u_k\), with \(u_1=2.00\). A computed statistic is \(Z_1=2.00\). Under the stated rule, \(2.00 \ge 2.00\), so this trial rejects at the first look.
Replacing the rule with \(Z_k > u_k\) changes the result: \(2.00 > 2.00\) is false. For an exactly continuous statistic, equality to a fixed boundary has probability zero under an ideal mathematical model, but software operates with finite precision, and some test statistics or simulated data can be discrete. The code should therefore express the design’s comparison explicitly. Do not round a statistic for the decision: for example, a value displayed as 2.00 might be 1.996 before rounding and may not cross a boundary of 2.00.
Worked Example: Map a Planned Look to Its Boundary
Suppose a hypothetical design has planned information times \(I_1=50\), \(I_2=75\), and \(I_3=100\), with upper boundaries \(u_1=2.40\), \(u_2=2.10\), and \(u_3=2.00\). In a programming language using zero-based arrays, the boundaries are stored at positions 0, 1, and 2. The second planned look is still look \(k=2\), so its boundary is stored at position \(k-1=1\), namely 2.10.
If the statistic at the second planned look is \(Z_2=2.15\), the correct comparison is \(2.15 \ge 2.10\), and the trial rejects. A common indexing error retrieves position 2 when processing look \(k=2\), thereby using \(u_3=2.00\). This trial still rejects, concealing the bug. But if \(Z_2=2.05\), the correct result is continuation because \(2.05 < 2.10\), whereas the misindexed comparison \(2.05 \ge 2.00\) incorrectly rejects. Tests should include values between adjacent boundaries, not just values that lead to the same outcome under both.
Information times also need a defined mapping. If an operational deviation means a look occurs at information \(I=76\), the code should not silently treat it as the planned \(I_2=75\) or \(I_3=100\). The design must specify how such a deviation is handled; the implementation should then follow that rule and retain the actual information value for diagnostics.
Missing Outcomes and Performance Denominators
A simulation run can fail to yield a decision because of a programming error, invalid input, numerical failure, or a deliberately modelled event. These situations must not be treated as interchangeable. For example, a simulated participant’s missing outcome may be part of the data-generating process, in which case the analysis rule should process the resulting trial as designed. A software exception that prevents calculation of the test statistic is a simulation failure, not automatically a non-rejection.
For \(R\) simulated trials under \(H_0\), the usual Monte Carlo estimate of Type I error is the number of trials that reject divided by \(R\), provided the outcome is defined for every simulated trial under the specified design. Under an alternative scenario, the analogous rejection proportion estimates power. If some runs have no computable outcome, silently dropping them changes the denominator and estimates a conditional quantity among successful runs, not necessarily the intended operating characteristic for a simulated trial.
Worked Example: Keep Failures Out of the Denominator by Accident
A hypothetical null simulation starts \(R=1{,}000\) trials. Of these, 20 end in a software failure before a valid decision, and 80 of the remaining 980 valid trials reject. Among valid runs, the rejection proportion is
This is an estimate conditional on a run producing a valid outcome. It is not automatically the intended Type I error estimate for all 1,000 started trials. Reporting \(80/1{,}000=0.08000\) would treat every failed run as a non-rejection, which is not justified merely because the code failed. The appropriate response is to diagnose and, where possible, fix and rerun the failures; report the failure count and state how failures were handled. If failures cannot be resolved, the valid-run estimate must be labelled as conditional, and its limitations made clear.
The denominator depends on the estimand and the simulation plan. If every trial has a well-defined outcome, use all \(R\) trials. If a subset truly has an undefined outcome under the intended scenario, specify whether the target is conditional on an evaluable outcome or is another clearly defined quantity. Do not choose the denominator after seeing which choice gives a more favorable estimate.
Missingness can also corrupt stopping summaries. If a trial has no valid stopping look because the statistic failed to compute, it should not be coded as “reached the final look.” Report valid stopping-look counts and failures separately. Check that counts across all recorded stopping categories and failure categories reconcile with the number of started trials.
Diagnostics, Alternatives, and Limitations
Useful diagnostics target errors that can remain invisible in an overall rejection percentage. Trace a small number of trials by recording, for each look, the planned look number, information value, statistic, boundary, comparison result, and whether continuation occurred. Then verify that the recorded decision is the first applicable one. Include edge cases: a statistic just below a boundary, exactly equal to it, just above it, crossing at the first look, crossing only at the final look, and never crossing.
Also check structural properties of the results. Each valid trial should have one terminal status and no more than one first stopping look. The sum of valid stopping-category counts plus failure counts should equal the number of started runs. Look indices and boundary arrays should have compatible lengths. If the design has increasing information, the recorded information values should be in the planned order. These checks do not prove the entire simulation is correct, but they can expose common implementation defects early.
A sequential loop that stops immediately on a crossing is often easiest to audit because it mirrors the decision rule. A vectorized implementation can be faster, but must explicitly select the earliest crossing rather than any or the last crossing. Either approach is acceptable if it returns identical decisions on carefully constructed test paths. Unit tests check specific inputs and expected outputs; independent comparisons and larger simulation diagnostics provide additional evidence, but agreement alone cannot rule out a shared mistake.
- Look-order error: checking later looks after stopping can overwrite the first stopping look or inflate look-specific counts.
- Boundary error: using a strict comparison, a rounded statistic, or a boundary from another look can change decisions near a threshold.
- Information-index error: confusing a planned look label, an array position, and an information value can pair the wrong statistic and boundary.
- Outcome error: treating a computational failure as a non-rejection or a final-look result disguises the failure mechanism.
- Denominator error: excluding failures without explanation reports performance only among successful runs, which may not be the intended target.
These checks are implementation safeguards, not a substitute for validating the design specification, data-generating model, or analysis method. They also cannot establish that simulated assumptions represent future trial operations. Their value is narrower and important: they make it less likely that code evaluates a different stopping procedure from the one intended.
Check Your Understanding
Answer each question by applying the stated rule, not by relying on a familiar coding convention.
- A trial has \(Z_1=2.30\) with \(u_1=2.50\), then \(Z_2=2.20\) with \(u_2=2.10\), then \(Z_3=2.80\) with \(u_3=2.00\). At which look does it first reject?
- A design specifies rejection when \(Z_k \ge u_k\). What should happen when the unrounded statistic equals the boundary? Why is equality still worth testing?
- There are four planned looks in a zero-based array. Which array position stores the boundary for planned look \(k=3\)?
- In 500 null-simulation runs, 10 have software failures and 45 of the 490 valid runs reject. Calculate the valid-run rejection proportion and state what it estimates.
- Why is a count of valid stopping outcomes plus failures that is smaller than the number of started runs a warning?
Solutions. (1) Look 1 does not cross because \(2.30 < 2.50\). Look 2 crosses because \(2.20 \ge 2.10\), so the first rejection is at look 2; look 3 must not overwrite it. (2) It rejects, because equality satisfies \(\ge\). Equality tests catch the wrong strict comparison and matter with finite precision or discrete statistics. (3) Position \(k-1=2\). (4) \(45/490=0.09184\), or about 9.184%; this estimates rejection among runs that produced valid outcomes, not automatically the rejection probability over all started runs. (5) At least one started run has not been accounted for as a valid terminal outcome or a failure, suggesting lost records, incomplete processing, or an untracked status.