Why Verify Independently?
A simulation can produce plausible operating-characteristic estimates even when its implementation contains a mistake. A boundary may be applied at the wrong stage, an arm that should stop may continue enrolling, or the final summary may count the wrong event. Unit tests can check known cases, but their value depends on which cases were anticipated and whether the tests use the same logic as the simulation. Independent verification adds a different check: a second implementation evaluates the same replicate inputs by a separately written route.
The purpose is not to generate a second set of random trials and compare its averages. Randomly generated datasets will generally differ, so their replicate-level decisions need not match. Instead, save or otherwise make available the inputs for each replicate, then pass those identical inputs to both implementations. Compare the resulting decisions and resource use replicate by replicate, before comparing summaries across replicates.
The implementations are most useful when they do not share the decision code being checked. They may use the same stated design specification and the same input data, but should reach their outputs through independently written calculations. If both implementations call the same boundary-evaluation function, agreement between them cannot detect an error in that shared function.
A verification record should identify the design settings and the input data for each replicate. At minimum, retain stage- and arm-specific outcome summaries or the underlying observations, together with any randomization or allocation information needed to reproduce the analysis. Record outputs in a structured form: for each replicate, include the decision for each experimental arm at each decision point, whether it continues, stops for futility, or meets the efficacy rule, and the number of participants enrolled. Use the design's specified labels and conventions consistently.
A Replay-Based Verification Procedure
The verifier should follow the design specification, not infer what the primary program intended to do. Before comparing outputs, write down the rule being checked, including the stage at which it applies, the comparison direction, and what each decision implies for later enrollment. Discrete outputs such as decisions and participant counts should ordinarily match exactly. For continuous outputs, define a numerical tolerance in advance and make it appropriate to the units and numerical scale.
Use the same saved outcomes, allocation, design settings, and any other inputs in both implementations. Do not generate fresh outcomes for the second run.
Apply the specified interim and final rules in separately written code, without importing the primary decision function into the verifier.
Check every arm-stage decision and the enrollment consequences for that replicate. Investigate the first disagreement rather than comparing only final averages.
Recalculate the reported summaries from the verified replicate-level records. Check that the summary code uses the intended events and denominators.
This procedure assumes that the verifier receives the intended inputs and implements the intended design rules. It does not establish that the design specification itself is appropriate, that the outcome model is scientifically realistic, or that a simulation is free from every possible software fault. It provides evidence about whether the implementation carries out specified calculations correctly on the inputs checked.
Worked Example: Replaying One Two-Stage Replicate
Worked Example: Replaying One Two-Stage Replicate
Consider a hypothetical two-stage design with experimental arms A and B and a shared control. At stage 1, 16 participants per arm are observed. Assume a known outcome standard deviation of 4 units. For this illustration only, define each stage-1 statistic as the difference in arm and control means divided by \(4\sqrt{1/n_E+1/n_C}\). An arm meets the interim efficacy rule if its statistic is at least 2.5, stops for futility if its statistic is below 0.5, and otherwise continues. Arm A's stage-1 mean is 14 units, arm B's is 11 units, and the control mean is 10 units.
For each arm, the standard error is
Thus \(Z_A=(14-10)/1.4142=2.828\), which meets the efficacy rule. For B, \(Z_B=(11-10)/1.4142=0.7071\), which is neither at least 2.5 nor below 0.5, so B continues. The independently written verifier should reproduce both decisions from these same inputs.
Suppose stage 2 enrolls 12 more participants in each continuing arm and in the shared control. Because A has stopped, the stage-2 allocation is B and control only. Enrollment is therefore 48 participants at stage 1 plus 24 at stage 2, or 72 in total.
At the final analysis, suppose the stage-2 means are 13 units for B and 11 units for control. Pooling the two stages gives B's mean \((16(11)+12(13))/28=11.857\) units and the control mean \((16(10)+12(11))/28=10.429\) units. The difference is \(1.429\) units. Its standard error is \(4\sqrt{1/28+1/28}=1.069\) units, so the final statistic is \(1.429/1.069=1.336\). If the hypothetical final efficacy boundary is 1.30, B meets it. A complete verification record for this replicate therefore contains A's stage-1 efficacy stop, B's continuation and final efficacy decision, and total enrollment of 72.
These numerical rules are deliberately specified for the example; they are not universal MAMS boundaries. In an actual verification, both implementations must follow the particular design's stated test statistic, thresholds, and stopping conventions.
Check Decisions and Enrollment Before Summaries
Enrollment is a consequence of decisions, not just another number to check. A stopped arm should not accrue participants at later stages unless the design explicitly requires otherwise. Shared control enrollment, if applicable, must be counted according to the allocation rule: a control participant supporting comparisons with multiple active arms is still one participant, not several.
Worked Example: Finding an Enrollment Discrepancy
Use the hypothetical replicate above. The stage-1 enrollment is \(16\) per arm across A, B, and control: \(3(16)=48\). Since A stops and B continues, stage 2 enrolls 12 in B and 12 in control: \(2(12)=24\). The reference total is \(48+24=72\).
Suppose the primary implementation reports 84 participants. The difference is \(84-72=12\), exactly one stage-2 arm allocation. Inspecting the arm-by-stage record reveals that the primary implementation enrolled 12 participants in A at stage 2 even though A had stopped for efficacy. The overall count identifies a discrepancy; the detailed record identifies its source. Correcting the summary alone would not fix the underlying decision-to-enrollment error.
To verify a batch of replicates, compare participant counts by stage and arm as well as totals. A total can agree by coincidence if one arm is over-counted and another under-counted. The design's allocation rules provide a check on whether each reported count is possible given the preceding decisions.
Why Matching Averages Is Not Enough
A summary can conceal a mismatch because aggregation discards which replicate produced each result. For example, suppose the reference implementation records success indicators \(1,1,0,0\), while the verifier records \(1,0,1,0\). Both estimated success rates are
Yet the implementations disagree on replicates 2 and 3. This matters because a decision error can be paired with a different decision error elsewhere and leave the overall success rate unchanged. The same issue applies to average sample size and other estimates: matching summaries are useful as a reconciliation check, but do not replace replicate-level comparisons.
Worked Example: A Summary Can Hide Decision Errors
Suppose five hypothetical replicates have reference success indicators \(0,1,1,0,1\), while the independent implementation gives \(0,1,0,0,1\). The reference estimate is \((0+1+1+0+1)/5=3/5=0.60\); the verifier's estimate is \((0+1+0+0+1)/5=2/5=0.40\). The mismatch is localized to replicate 3, where the reference declares success and the verifier does not.
Now consider a separate illustration of how agreement can hide errors: if two implementations report success indicators \(1,1,0,0\) and \(1,0,1,0\), respectively, each rate is \(2/4=0.50\), but two of four replicate-level decisions differ. Thus, compare the full ordered set of replicate results, not only the rate.
Interpreting Differences and Choosing Checks
For discrete outputs, report the number and identity of mismatching replicates, and identify the earliest decision point at which each mismatch occurs. For continuous outputs, report the difference between implementations and the tolerance used. A tolerance should not be chosen after seeing the disagreement: doing so risks treating a real error as acceptable. Exact agreement is usually appropriate for integer counts and categorical decisions; a small, prespecified tolerance may be needed for floating-point calculations.
When a mismatch appears, trace it in order: input values, calculated statistic, boundary comparison, decision label, subsequent allocation, and final summary contribution. This sequence helps distinguish a calculation difference from a downstream bookkeeping difference. Also inspect missing or duplicated replicate records and confirm that both summaries use the same denominator and event definition.
| Check | What it establishes | Main limitation |
|---|---|---|
| Unit tests on selected cases | Known inputs produce expected outputs for the tested rules. | Untested cases and shared errors in test and production logic can remain. |
| Independent replay | Two separate implementations agree on decisions and outputs for identical inputs. | Agreement is only as informative as the independence and coverage of the verifier. |
| Comparison of simulation summaries | Aggregate estimates are consistent with one another at the chosen level of detail. | Different replicate-level errors can cancel in the summaries. |
Independent replay is therefore complementary to, not a replacement for, unit tests and careful review. It can reveal errors in the executed decision logic, but agreement between two implementations does not prove either is correct if they share the same mistaken interpretation of the design or the same underlying assumption.
Common Pitfall
A related mistake is to run both programs with the same random-number seed and assume that this guarantees comparable replicate inputs. Changes in random-number consumption, stream handling, or execution order can cause the programs to receive different simulated data. Verification is clearer when the replicate inputs themselves are saved and explicitly replayed. If only independent random datasets are available, differences in outputs may reflect Monte Carlo variation rather than a coding discrepancy.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why should an independent verifier receive the same replicate inputs as the primary implementation?
- In a replicate, the primary implementation reports 60 participants and the verifier reports 72. Name two detailed records to inspect before changing any summary code.
- Two implementations have success indicators \(1,0,1,0\) and \(0,1,1,0\). Calculate each success rate and state whether the replicate-level decisions agree.
- Which output generally needs an exact match: an arm's stop/continue label, or a floating-point statistic? Explain how to handle a small numerical discrepancy in the statistic.
Solutions. (1) Identical inputs make output differences attributable to implementation or interpretation rather than different simulated data. (2) Compare arm-by-stage enrollment and the decisions that determine which arms continue; also inspect the first decision point where the records diverge. (3) Both rates are \(2/4=0.50\), but the first two replicate labels are reversed, so two replicate-level decisions disagree. (4) The stop/continue label should generally match exactly. For a floating-point statistic, compare the values using a tolerance specified before examining the results, and investigate differences that exceed it.