Why Compare Independent Implementations?
An adaptive trial simulation combines several linked components: a model for generating trial data, rules that use accumulating data to change the design, and calculations that summarize the resulting trial outcomes. A mistake in any component can alter reported operating characteristics such as the probability of stopping early or the probability of selecting a treatment. Testing individual rules and checking results against known cases are valuable, but they do not necessarily reveal errors in the way components work together.
A second implementation offers a different kind of check. If two separately coded versions follow the same precise specification, disagreements can reveal an error that was not apparent from tests of either version alone. The comparison is most informative when it is designed to locate the disagreement: compare generated data first, then adaptation decisions, trial-level outcomes, and finally summaries. A difference in a final percentage is a signal to investigate, not an explanation of what went wrong.
The written specification must be precise enough for two coders to make the same choices. It should define the trial population and data-generating assumptions, the timing and content of interim analyses, the exact adaptation rules and boundary conventions, the final outcome definitions, and the denominators used in summaries. For example, “stop if the response rate is high” is not precise; “at the interim analysis, stop for efficacy if the observed response proportion is at least 0.30” is testable.
There are two useful comparison settings. With aligned inputs, both implementations receive the same prespecified random variates or fixed test data. If the specification requires the same deterministic calculation from those inputs, exact agreement is expected. With separately generated random inputs, individual trials will ordinarily differ. Then compare appropriate distributions or estimates, accounting for Monte Carlo variation; do not expect trial-by-trial equality.
What to Compare and in What Order
A validation comparison should preserve enough intermediate information to trace an end-to-end difference back to its source. At a minimum, retain a trial identifier, the input scenario, generated participant-level or aggregate data, interim statistics, adaptation decisions, final outcomes, and the numerator and denominator used in each reported summary. The two implementations should use the same definitions and precision rules when writing these records.
Write down the data model, analysis schedule, exact decision inequalities, outcome definitions, and reporting denominators before comparing code.
For component tests, provide both versions the same fixed inputs or canonical random variates. For distributional comparisons, record how random inputs and scenarios were generated.
Check generated data and interim summaries before checking adaptations, final outcomes, or aggregate rates. Find the first point at which the records diverge.
Classify the discrepancy, correct the implementation or specification as appropriate, and rerun the same comparison plus the relevant existing tests.
The comparison has a simple logic. If the first discrepancy is already in generated data, later differences in decisions may be consequences rather than separate decision-rule bugs. If generated data and interim statistics agree but a decision differs, inspect the decision rule, its threshold, and the analysis timing. If trial-level outcomes agree but a reported rate differs, inspect aggregation, exclusions, and denominators.
Worked Example: A Boundary Difference in Generated Data
Suppose the specification generates a binary response using a fixed uniform input \(U\) and response probability \(p=0.30\): set the response to 1 when \(U<p\), and to 0 otherwise. Two implementations are tested on the same inputs, \(0.12,\ 0.30,\ 0.81\). Implementation A uses the specified strict inequality. Implementation B uses \(U\leq p\).
For A, the responses are \(1,0,0\): \(0.12<0.30\), while \(0.30<0.30\) is false and \(0.81<0.30\) is false. For B, the responses are \(1,1,0\), because its second comparison is \(0.30\leq0.30\).
The records first disagree on the second generated response, before any adaptation or outcome calculation. The specification therefore identifies B’s comparison operator as inconsistent with the required rule. In a simulation using a truly continuous uniform random variable, the probability of drawing exactly \(0.30\) is zero. Even so, a fixed boundary test is useful: software may use finite-precision values, and the written rule must be implemented consistently at its boundary.
Separating Decisions from Their Inputs
For an adaptive design, the decision is a function of the information available at a specified time. The validation record should make that relationship inspectable. Store the interim data, the statistic computed from those data, the exact rule applied, and the resulting action. A decision mismatch should not be “fixed” by changing a threshold until the two outputs happen to agree; first verify which implementation matches the frozen specification.
Particular attention is warranted at thresholds and at the boundaries of allowable design changes. The specification should state whether equality triggers a rule, which analysis time is used, and how ties or missing interim quantities are handled. Otherwise, two implementations may each be internally consistent while representing different interpretations of an incomplete rule.
Worked Example: An Interim Stopping Rule
A hypothetical trial has an interim analysis after 20 participants. Its specification says to stop for futility if the observed response proportion is less than or equal to 0.20. At the interim look, 4 of 20 participants have responded. Implementation A uses the specified rule; implementation B stops only when the proportion is strictly less than 0.20.
The observed proportion is
Under the specified rule, \(0.20\leq0.20\), so the trial stops for futility. Under B’s rule, \(0.20<0.20\) is false, so it continues. The input data and interim statistic agree, and the first divergence is the action. This localizes the discrepancy to the boundary convention rather than the data generator.
The two versions should be reconciled against the written specification, not against whichever behavior seems preferable after seeing the result. If the intended rule had instead been “less than 0.20,” the specification would need to say so and both implementations would need to follow that revised rule.
Comparing Outcomes and Reported Operating Characteristics
Once trial-level records agree, the summaries should follow from those records. For a binary outcome, the simulated rate is the number of trials with the outcome divided by the number of eligible simulated trials. The numerator and denominator should be reported or retained separately. This makes it possible to distinguish a changed outcome classification from a changed inclusion rule or an arithmetic error.
When the same replicate inputs are supplied to both versions and the outcome definitions are identical, the trial-level outcomes and their aggregate counts should agree exactly. If the implementations use independent random streams, their estimated rates need not match exactly, even when both are correct. In that case, a difference can reflect simulation variation as well as coding differences. When replicates are aligned, let \(I_{Ar}\) and \(I_{Br}\) indicate whether replicate \(r\) meets a binary outcome definition in implementations A and B. The paired difference in estimated rates is
where \(R\) is the number of aligned replicates. Its Monte Carlo standard error, treating the paired replicate differences as independent across replicates, can be estimated by
where \(s_D\) is the sample standard deviation of the \(R\) paired differences. This quantifies Monte Carlo variation in the paired comparison; it does not certify that a discrepancy is harmless. In an aligned deterministic test where both versions are required to produce identical results from identical inputs, any mismatch should be investigated even if it is small relative to a standard error.
Worked Example: Tracing a Difference in a Reported Rate
In a hypothetical comparison of 1,000 aligned simulated trials, both implementations record 620 trials meeting a prespecified success definition. The trial-level success indicators match for all 1,000 trials. Implementation A reports \(620/1{,}000\). Implementation B reports \(620/1{,}013\), because its summary routine has included 13 duplicated records in the denominator.
The reported values are
The discrepancy is about \(0.620-0.612=0.008\), or 0.8 percentage points. Because every trial-level outcome agrees, the evidence points to aggregation rather than data generation or adaptation. The correct rate under the stated definition is \(620/1{,}000=0.620\): there are 1,000 eligible trials, not 1,013. The remedy is to correct the denominator logic and then verify that the reported numerator and denominator are both derived from the eligible trial records.
Justifying the Paired Comparison
Pairing is useful because each implementation is evaluated on the same replicate. Define \(D_r=I_{Ar}-I_{Br}\), so \(D_r\) is 1 when only A records the outcome, \(-1\) when only B records it, and 0 when the two agree. The difference between the two reported rates is exactly the average paired difference:
If the aligned replicates are independent draws from the comparison setup and have finite variance, the estimated standard error of \(\bar{D}\) is \(s_D/\sqrt{R}\). This follows from the variance of a mean of independent, identically distributed replicate differences: \(\operatorname{Var}(\bar{D})=\operatorname{Var}(D_r)/R\). Pairing matters because it uses the replicate-level differences; comparing two rates as though they came from unrelated runs discards information about which trials disagree.
This calculation is a way to describe uncertainty when comparing stochastic runs. It is not a substitute for tracing discrepancies. A small estimated difference in aggregate rates can conceal many offsetting trial-level mismatches, so inspect the number and pattern of discordant records as well as the net difference.
Diagnostics, Alternatives, and Limitations
Useful diagnostics include the number of mismatched records at each stage, the first stage of divergence, and whether mismatches cluster at particular thresholds, analysis times, or scenario values. A comparison table can keep the investigation concrete.
| First disagreement | Likely areas to inspect | Next check |
|---|---|---|
| Generated trial data | Probability parameter, distribution, boundary rule, or random-input mapping | Compare fixed inputs and intermediate generated values |
| Interim statistic | Analysis set, denominator, missing-data handling, or calculation | Recalculate the statistic by hand on a small fixed data set |
| Adaptation decision | Threshold inequality, timing, tie handling, or action mapping | Test values just below, at, and just above each boundary |
| Reported summary only | Outcome coding, eligible trials, duplicates, or aggregation | Rebuild the numerator and denominator from trial-level records |
Independent implementations complement, rather than replace, other checks. Unit tests focus on specified inputs and expected outputs for a component. Checks against known results test behavior in cases with established answers. Independent implementations can expose shared assumptions that were not covered by those checks. None of these approaches alone establishes correctness: both versions may implement the same mistaken interpretation, share a flawed specification, or reproduce an unnoticed conceptual error.
There are practical limits. A full duplicate simulation may require substantial development and maintenance, so a second implementation can be targeted to high-consequence components: data generation, adaptation logic, and final outcome calculations. Different programming languages do not guarantee independence if one version is copied from the other or both are derived from the same unexamined code. Conversely, differences in random-number generators can make pathwise equality inappropriate when the versions are run with separate streams. In that setting, compare the distributions and operating-characteristic estimates under a common scenario, and retain enough records to diagnose unexpected differences.
A useful governance record states the specification version, code versions, inputs or randomization setup, comparison scope, results, unresolved discrepancies, and the disposition of each discrepancy. Resolve differences before relying on the affected operating-characteristic estimates. Record whether a correction changed the specification, the code, or both; these are distinct decisions and should not be conflated.
Common Mistake
Wrong reasoning: “The two implementations reported nearly the same success rate, so the simulation is validated.” This conclusion can fail even when both rates are numerically close: one version may classify some trials incorrectly while another set of errors offsets the difference, or both versions may share the same defect.
Correction: Agreement in a summary is evidence of agreement for that summary under the tested conditions, not proof of correctness. Compare intermediate and trial-level records where aligned inputs make exact agreement appropriate, test boundary cases, and verify both implementations against the written specification and independent known results.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Two implementations use independent random streams and produce different trial-level outcomes. Does this alone show that one is wrong?
- At an interim look, the generated data and interim statistic match, but the actions differ at exactly the futility threshold. What should be checked first?
- Both implementations report the same success rate, but no trial-level records were compared. What can be concluded, and what cannot?
- In \(R=500\) aligned replicates, A records 18 successes that B does not, and B records 8 successes that A does not. What is the paired difference in estimated rates?
Solutions. (1) No. With independent random inputs, trial-level outcomes are expected to differ. Compare the implementations under a common scenario and assess their output distributions and estimates, while checking that each follows the specification. (2) Check the exact inequality and boundary convention in the frozen specification, then test values below, at, and above the threshold. Since upstream values match, the decision rule is the first point to investigate. (3) Only that the reported rates agree for the comparison performed. It does not establish that the trial-level classifications, denominators, or underlying implementation are correct; inspect the records and validate against the specification. (4) There are 10 net successes in favor of A, so \(\hat{d}=(18-8)/500=0.020\), a paired rate difference of 2 percentage points. This net difference does not reveal the full discordance pattern without also reporting both counts.