Why Simulation Bugs Can Look Like Design Results
A MAMS simulation repeatedly generates trial outcomes, applies interim decisions, and records final operating characteristics. A coding error can change that decision process while leaving the program running smoothly. The resulting estimates may look plausible: an error that makes standard errors too small, for example, can appear as an unexpectedly high probability of success rather than as a software defect.
This tutorial focuses on four recurring risks: reusing interim outcomes when later-stage data are added, counting shared-control patients more than once, dropping the wrong treatment arms, and testing arms that are no longer eligible. The examples use a hypothetical design with a control arm and treatment arms A, B, and C. Higher continuous outcomes are better. Interim decisions follow pre-specified rules, and each replicate has a clearly recorded set of patients, arms, and stages.
A simulation should implement the design as specified, not infer it from whichever data happen to be available. In particular, the design rules must say which comparisons use a shared control, how interim decisions determine eligibility, and which analysis is used at the final stage. For the arithmetic examples below, assume patients are independent, each patient belongs to one randomized arm, and outcome summaries refer to the same outcome and estimand across stages. These assumptions make the bookkeeping transparent; a real simulation must also represent its specified allocation, outcome, missingness, and analysis models.
Keep Stage Data and Cumulative Data Distinct
For a treatment arm \(j\), let \(n_{j,s}\) be the number of new observations in stage \(s\), and let \(\bar{x}_{j,s}\) be their mean. If the target is the mean across all accrued observations, the cumulative mean after stage \(S\) is the weighted mean
The weights are stage sample sizes because stage means need not represent equal numbers of patients. Equivalently, the cumulative sum and count are updated by adding only the new-stage sum and count. This update is valid when the stage summaries refer to non-overlapping observations of the same quantity. If the design instead calls for a particular combination of stage-specific statistics, that rule must be implemented; an ordinary pooled mean is not a substitute for a specified combination test.
Worked Example: Detecting Reused Interim Outcomes
Suppose a hypothetical replicate has 10 patients per arm in each of two stages. In treatment A, the stage 1 mean is 8 and the new stage 2 mean is 12. In control, the stage 1 mean is 7 and the new stage 2 mean is 8. All four summaries are based on distinct patients. Find the cumulative treatment-control difference after stage 2.
Treatment A. The cumulative mean uses both stage cohorts once:
Control. The corresponding cumulative mean is
Thus the cumulative treatment-control contrast is \(10-7.5=2.5\) outcome units. A faulty update might append the stage 1 data a second time when stage 2 arrives. Its incorrect means would be \(\{10(8)+10(12)+10(8)\}/30=280/30\approx9.333\) for A and \(\{10(7)+10(8)+10(7)\}/30=220/30\approx7.333\) for control. The resulting contrast, \(60/30=2.0\), is not the cumulative contrast from the 20 distinct patients per arm.
A useful diagnostic is to compare cumulative counts with the number of unique patient identifiers in each arm. Here, each count must be 20, not 30. Also verify that the first-stage sum is included exactly once in the cumulative sum.
This calculation gives a short justification for a robust update rule. If \(T_{j,S}\) is the sum of outcomes and \(N_{j,S}\) the count through stage \(S\), then adding a disjoint new cohort gives \(T_{j,S}=T_{j,S-1}+T_{j,S}^{\mathrm{new}}\) and \(N_{j,S}=N_{j,S-1}+n_{j,S}\). Therefore \(\bar{x}_{j,1:S}=T_{j,S}/N_{j,S}\). Re-adding earlier records violates the disjoint-cohort update and inflates the count as well as the sum.
Shared Controls: Reuse Comparisons, Not Patients
When several treatment arms share a control, the same control patients can legitimately contribute to multiple treatment-control comparisons. They remain one set of patients in the simulated trial. Repeating a control dataset in a separate comparison is not the same as generating extra control patients or treating copies as independent observations. Pairwise estimates are also statistically dependent because they use the same control outcomes.
Worked Example: Checking a Shared-Control Standard Error
Consider a hypothetical single-stage replicate with 20 control patients, 20 patients in A, and 20 in B. Suppose the observed means are 0, 1, and 2 outcome units, respectively. For illustration only, assume the known outcome variance is 4 in every arm and use the usual variance calculation for a difference in independent arm means. The two comparisons may each use the same 20 control observations.
For A versus control, the estimated difference is \(1-0=1\) outcome unit, and its standard error under this illustrative known-variance model is
For B versus control, the difference is \(2-0=2\) outcome units, and the same variance calculation gives \(SE=\sqrt{4/20+4/20}\approx0.6325\) outcome units. The control data are used in both comparisons, but the replicate still contains 20 control patients, not 40.
If a program mistakenly doubles the control count to 40 while retaining its variance of 4, it would calculate \(SE=\sqrt{4/20+4/40}=\sqrt{0.3}\approx0.5477\) for each comparison. This is artificially smaller because the software has acted as if there were 40 independent control patients. This example is a bookkeeping check, not a general final-test formula: the simulation must use the variance estimator and testing procedure specified by its design.
A practical audit should check both the stored data and the comparison construction. Each unique control identifier should occur once in the replicate dataset. For each contrast, the control summary should refer to the intended eligible control cohort, and code should not add the same control rows into a pooled analysis as though they were new independent observations. When reporting joint behavior across arms, also remember that comparisons sharing controls are dependent; treating their results as independent can misrepresent their joint operating characteristics.
Drop Arms by Label and Gate Final Tests by Eligibility
Interim rules change which treatment arms can proceed. A reliable implementation keeps arm identity separate from position in an array or table. If arms are reordered, filtered, or stored in a different order, a positional instruction such as “remove the second arm” may no longer mean “remove B.” Decisions should be joined to outcomes by explicit arm labels, and the eligible-arm set should be updated according to the rule.
Worked Example: A Drop Rule With Reordered Arms
Suppose the original arm labels are A, B, and C. At an interim analysis, the pre-specified rule drops B and continues A and C. The outcome table is then sorted by interim mean, giving the order C, A, B. If software removes the second row because it assumes that B is always in position 2, it drops A instead.
The correct operation is to apply the decision by label: remove rows whose arm label is B, leaving the eligible set \(\{A,C\}\), regardless of row order. A check after every interim decision is that the actual eligible labels equal the labels prescribed by the rule. In this example, comparing sets gives \(\{A,C\}=\{A,C\}\); a positional error would instead leave \(\{B,C\}\), which fails the check.
This example assumes that the interim decision is already correctly calculated. The coding issue is the mapping from the decision to the arm records, not whether the decision threshold itself is appropriate.
Dropping an arm also affects which final analyses may be reported as confirmatory. Being able to calculate a statistic from stored outcomes does not make an arm eligible for a final test. A program should maintain an explicit eligibility indicator or set, updated by the design's interim rules, and use it to gate final testing. The decision rule can be represented as: report a final rejection for arm \(j\) only if \(j\) is eligible at the final stage and its specified final test meets the design's rejection criterion.
Worked Example: Preventing a Final Test on a Dropped Arm
Suppose a hypothetical interim rule drops B for futility, while A and C continue. At the end of the simulation, a stored calculation gives B a nominal final \(p\)-value of 0.01. Suppose the final rejection threshold for eligible arms in this illustrative example is 0.025.
A faulty program may report B as a success because \(0.01<0.025\). The correct logic also checks eligibility: B was dropped, so it is not eligible for the final test under this design. Its final rejection indicator is 0, regardless of the nominal \(p\)-value that could be calculated from retained data. For an eligible arm such as A, the program may apply the specified final test and threshold. This distinction concerns implementation of the design; it does not claim that a particular threshold is appropriate for every MAMS design.
A diagnostic is to record, for every replicate and arm, the interim decision, final eligibility, whether a final test was run, and whether a rejection was recorded. There should be no final rejection for an arm marked ineligible. If the design calls for a different rule after a particular interim event, the eligibility logic must reflect that rule exactly.
Auditing Logic Before Trusting Operating Characteristics
These errors are often easier to find by checking invariants than by inspecting a final estimate across thousands of replicates. A program can produce a smooth rejection-rate estimate even when its underlying records have duplicated patients or misapplied decisions. For each replicate, useful checks include:
- Each patient identifier appears once in the trial dataset and is assigned to one arm and one first-accrual stage.
- Cumulative arm counts equal the number of unique eligible patients accrued through that stage.
- For each arm and stage, the cumulative sum equals the prior cumulative sum plus the new-stage sum.
- After each interim decision, the eligible arm labels match the pre-specified rule, independent of row order.
- A final test or rejection is recorded only for an arm eligible under the design.
- Shared-control comparisons refer to the intended control observations without creating duplicated independent records.
When a check fails, do not silently discard the replicate or replace the result with a default value. Identify whether the cause is an invalid replicate, a numerical failure, or a logic error; record enough information to reproduce it, and resolve the cause before summarizing operating characteristics. A large number of replicates reduces Monte Carlo uncertainty around the program's output, but it cannot correct a systematic implementation error.
| Implementation choice | What it does | What to check |
|---|---|---|
| Incremental stage update | Adds only newly accrued records to cumulative totals | Counts and sums agree with unique patient records |
| Shared-control comparison | Uses the same control cohort in several intended pairwise comparisons | Control patients are not duplicated as independent trial records |
| Label-based arm update | Applies interim decisions to named arms | Eligible label set matches the decision rule after reordering |
| Eligibility-gated final analysis | Runs or reports final tests only when permitted | No ineligible arm has a final rejection recorded |
These checks complement, rather than replace, unit tests, independent verification, analytic special cases, and scenario-level summaries. Unit tests can target a specific interim decision; a replicate audit can expose a mismatch between labels and stored outcomes; and analytic benchmarks can reveal a systematic discrepancy in operating characteristics. No one check proves the whole simulation correct. In particular, a hand-checked mean update does not validate the random-number generation, allocation process, final test, or other design components.
Common Pitfalls
- Using a cumulative dataset as if it were new-stage data. This counts earlier patients again. Keep separate stage-specific and cumulative records, or update cumulative sums and counts using only new records.
- Interpreting shared-control use as additional control accrual. The same control cohort may inform several contrasts, but that does not multiply the number of control patients. Track unique patient identifiers and preserve the dependence among comparisons.
- Removing an arm by row position. Sorting or filtering can change positions. Apply decisions by stable arm labels and verify the resulting eligible set.
- Testing every arm with an available final statistic. Availability is not eligibility. Apply the design's interim rules before deciding which final tests and rejections can be recorded.
- Assuming a plausible summary validates the code. A plausible rejection rate can arise from biased logic. Check replicate-level invariants and compare results with independent or analytic checks where appropriate.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In stage 1 an arm has 8 patients with mean 5. In stage 2 it has 12 new patients with mean 7. What is the cumulative mean, and why is averaging the two stage means without weights incorrect?
- A shared control has 30 unique patients and is used in three treatment-control comparisons. How many control patients are in the replicate, and what coding mistake would create false extra information?
- Arms A, B, and C are stored in the order C, B, A. The interim rule drops C. What should the eligible set be, and why is deleting the first row safe only under a particular implementation assumption?
- An arm is dropped at interim, but its stored final \(p\)-value is below the nominal threshold. Should the simulation record a final rejection for that arm under a rule that makes dropped arms ineligible?
Solutions. (1) The cumulative mean is \(\{8(5)+12(7)\}/(8+12)=(40+84)/20=124/20=6.2\). The stage means should be weighted by their different sample sizes; their unweighted average, \((5+7)/2=6\), gives equal weight to stages despite having 8 and 12 patients. (2) There are 30 unique control patients. The error is copying or concatenating the same control records and treating the copies as independent observations, which can inflate the apparent control sample size and distort uncertainty. (3) The eligible set is \(\{A,B\}\). Deleting the first row works only if the program has established that the first row is C and applies the rule to that verified ordering; label-based removal is safer when order can change. (4) No. Under the stated rule, the arm is ineligible, so a computable nominal \(p\)-value does not authorize a final rejection.