Benchmarking Against Analytic Special Cases

Why Benchmark a Simulation?

A simulation can run without errors and still return the wrong operating characteristics. A decision boundary may be applied at the wrong stage, a standard error may use the wrong sample size, or a reported error rate may count the wrong rejection event. Benchmarking checks the output against a case whose answer is known independently of the simulation code.

The useful question is not whether a simulated estimate equals its analytic target exactly. With a finite number of replicates, random variation is expected. Instead, check that the target follows from the same design and assumptions, then judge the discrepancy in relation to its Monte Carlo standard error. A simple case that fails this check gives a concrete reason to investigate before interpreting more complicated MAMS results.

A One-Stage, One-Comparison Benchmark

Begin with the simplest treatment comparison: one experimental arm, one control arm, and one final analysis. Assume \(n\) independent outcomes are observed in each arm, with normally distributed outcomes and a common known variance \(\sigma^2\). Let \(\delta\) be the difference in population means, experimental arm minus control. The estimated difference has sampling distribution

For a one-sided test of \(H_0:\delta\leq 0\) against \(H_a:\delta>0\), reject when \(Z=(\bar{x}_1-\bar{x}_0)/SE>c\), where \(c\) is the critical value chosen for the test. Under the boundary null \(\delta=0\), \(Z\sim N(0,1)\), so the type I error is \(P(Z>c)=1-\Phi(c)\). Under a specified alternative \(\delta>0\), the standardized statistic has mean \(\delta/SE\) and variance 1. Its power is therefore

This formula is a benchmark only when the simulated analysis uses the same endpoint model, known variance, allocation, sample size, test direction, and critical value. If the simulation estimates the variance and uses a \(t\)-test, or includes interim decisions, this particular normal calculation is not its exact target.

Multiple Comparisons Without Arm Selection

A second benchmark removes interim selection while retaining more than one treatment comparison. “No selection” here means that all prespecified comparisons are carried through to the final analysis; it does not mean that multiplicity disappears. For a particularly transparent analytic check, suppose there are \(K\) independent one-sided tests under the global null and each uses significance level \(\alpha/K\). Independence is an explicit assumption for the calculation below. It can hold, for example, if comparisons use disjoint groups. It does not generally hold when treatment arms share a control group.

Let \(A_k\) be the event that comparison \(k\) rejects its null hypothesis. Under the global null, \(P(A_k)=\alpha/K\). Familywise error is the probability of at least one false rejection. For independent tests, the probability of no rejections is the product of the probabilities of no rejection in each test. Therefore,

This is the exact familywise error for this independent-test configuration with Bonferroni-adjusted individual levels. It is generally below \(\alpha\), since Bonferroni controls familywise error at no more than \(\alpha\). For small \(\alpha\), the value is close to, but not exactly, \(\alpha\).

The independence condition matters. In a usual multi-arm trial with a common control, comparisons are correlated because they use the same control observations. The product formula above is then not the exact familywise error. For a shared-control benchmark, derive or numerically evaluate the joint distribution of the test statistics under the specified model, including their correlation, and compare the simulation with that target. Do not label an independence-based calculation as exact for a shared-control design.

Benchmarking Expected Sample Size

Not every useful benchmark is a rejection probability. A fixed design with no early stopping or arm dropping has a deterministic total sample size, so its expected sample size is known exactly. This checks whether the simulation counts participants correctly and whether its output uses the intended unit: participants per arm, total participants, or participants accrued by a given stage.

How to Use a Benchmark as a Diagnostic

A useful benchmark is designed so that the simulated estimand and the analytic answer refer to the same event and the same design. Before comparing numbers, write down the assumptions and the exact event being counted. For example, “at least one rejection under the global null” is a familywise error event; “rejection for a particular experimental arm under its alternative” is that arm’s power. They are not interchangeable.

The Monte Carlo standard error describes random error from using a finite number of replicates; it does not measure bias from incorrect code or a misspecified model. A discrepancy of several Monte Carlo standard errors is a reason to investigate, not a proof of a specific bug. Conversely, a close match in one special case does not establish that every part of a complex simulation is correct. It provides evidence about the components exercised by that benchmark.

Common Mistakes and Limitations

• Using a benchmark with different assumptions. The known-variance normal formula is not the exact answer for a simulated \(t\)-test with estimated variance. Match the analysis procedure before comparing results.
• Assuming shared-control tests are independent. Their statistics are correlated. The independent familywise-error product is not valid unless independence is justified.
• Comparing unlike events. A per-comparison type I error is not the same as familywise error, and power under an alternative is not a null error rate.
• Expecting exact equality from random replicates. A finite simulation varies. Quantify that variation with a Monte Carlo standard error and assess whether the observed difference is plausible.
• Treating a passing benchmark as full validation. A one-stage case cannot exercise interim stopping, arm dropping, or other MAMS decision logic that is absent from that case.

Analytic formulas are often the clearest benchmark when they apply. For more complicated dependence structures, numerical integration or exact enumeration of a small discrete model can provide a separate reference. Simulation remains valuable where the design is too complex for a convenient analytic answer, but special cases test foundational pieces at low cost. Independent verification of simulation code complements benchmarking: the two approaches offer different checks, and neither makes the other unnecessary.