Why Benchmark a Planning Workflow?
A planning workflow can run without errors and still implement the wrong design. It might use the wrong efficacy boundary, omit a stopping rule, or translate information into sample size incorrectly. Benchmarking addresses a focused question: when the design and assumptions are specified in a standard, checkable case, does the workflow reproduce an operating characteristic that is already known or has been independently established?
A benchmark is not a general proof that the software is correct. It is a comparison against a reference for particular inputs and outputs. The comparison is informative only if both sides describe the same design, parameter scenario, endpoint, and convention. This tutorial develops that comparison, including cases where a reference is analytic and cases where an independently obtained numerical value is used.
Specify the Comparison Before Running It
A reference value is meaningful only in relation to a precise specification. Record the test direction, type I error level, information or sample sizes, interim looks, boundaries, stopping rules, treatment of nuisance parameters, and effect scenario. Also identify the output being checked: for example, probability of rejection by the final look, probability of early stopping, or expected sample size. A matching label such as “power” is not enough if the underlying definitions differ.
For an output \(\theta\), let \(\hat{\theta}\) be the value from the workflow and \(\theta_{\mathrm{ref}}\) the reference. A common absolute-error check is
where \(\tau\) is a prespecified tolerance in the units of the output. For probabilities, absolute error is measured in probability points: a difference of 0.001 is one tenth of one percentage point. A relative-error tolerance may be more appropriate for some positive outputs, but it behaves poorly when the reference is near zero. State which metric is used rather than selecting one after seeing the result.
For simulation-based outputs, the tolerance and Monte Carlo uncertainty answer different questions. The tolerance is an engineering or scientific criterion for acceptable agreement; Monte Carlo uncertainty describes the random variation in the simulation estimate. Under independent replications with a binary outcome, if \(\hat{p}\) is the estimated probability from \(R\) replications, its estimated standard error is
A difference larger than a few estimated standard errors may indicate a discrepancy, but a small difference does not establish correctness. Conversely, a result can be statistically compatible with the reference yet fail a strict prespecified tolerance. Report both when the workflow uses simulation.
Worked Benchmarks
Worked Example: One-Look Normal-Theory Power
Suppose a hypothetical workflow estimates power for a single upper-tail normal test. Under the alternative, the standardized test statistic has distribution \(Z\sim N(2.8,1)\), and the design rejects when \(Z>1.959964\). An exact reference can be calculated from the normal distribution. Assume the workflow reports simulated power 0.8002 and the prespecified absolute tolerance is 0.001.
Check the assumptions. This reference applies to a fixed, single-look test with a standard normal statistic of variance 1 under the stated alternative and the stated fixed boundary. It does not describe a test with repeated interim looks or a variance estimated from the data.
The exact power is the probability that the statistic crosses the upper boundary:
Thus the reference power is approximately 0.7996. The workflow’s absolute difference is
Since \(0.0006441<0.001\), the result passes this benchmark. The interpretation is that the workflow’s estimated rejection probability is within 0.001 of the normal-theory reference for this specified alternative—not that the benchmark establishes performance for every design.
Worked Example: Two Independent Tests Under the Null
Consider a hypothetical benchmark consisting of two independent standard normal test statistics under \(H_0\). The design rejects if the first exceeds 2.8 or the second exceeds 2.0. Suppose a workflow estimates the overall type I error as 0.0239, and the tolerance is 0.0003.
Check the assumptions. Both statistics are standard normal under \(H_0\), and they are independent. The design’s rejection event is the union of the two upper-tail events. This is a simple independent-test benchmark; it is not a conventional group-sequential design with correlated cumulative test statistics across interim looks.
Under the null, the probability of no rejection is the probability that both statistics remain below their boundaries. Independence gives
The absolute discrepancy is
The difference exceeds 0.0003, so this benchmark fails. The workflow’s estimate is below the reference, but that direction alone does not identify the cause. Investigate whether it used both rejection opportunities, implemented the strictness of the boundaries consistently, and generated independent standard normal statistics as specified.
Worked Example: Checking a Normal-Theory Sample Size
A hypothetical planning workflow targets 80% power for a one-sided normal test at \(\alpha=0.025\). Assume a known standard deviation and a standardized mean difference of 0.3 per square root of sample size, so the test statistic under the alternative has mean \(0.3\sqrt{n}\) and variance 1. Check the workflow’s proposed sample size against the normal-theory reference.
Check the assumptions. This calculation assumes a fixed design, a known standard deviation, a normal test statistic with variance 1, and the specified effect. It does not include interim monitoring, attrition, or an estimated-variance adjustment.
For upper-tail rejection at \(z_{1-\alpha}=1.959964\), target power 0.80 requires the alternative mean to exceed the boundary by \(z_{0.80}=0.841621\). Solving for the continuous sample size gives
The smallest integer meeting the target is \(n=88\). At \(n=88\), the reference power is
As a check on the integer choice, at \(n=87\) the power is \(\Phi(0.8383)\approx0.7990\), below 0.80. Thus 88 is consistent with the target and the integer constraint. If a workflow reports 90, it may still meet a planning requirement, but it does not reproduce the minimum reference sample size; the benchmark criterion should say whether it checks exact agreement or merely attainment of target power.
What Makes a Benchmark Trustworthy?
An analytic reference is especially useful when its assumptions are simple enough to verify directly, as in the normal-theory examples. For designs without a convenient closed form, the reference may come from an independently written implementation, numerical integration, or a published standard design. “Independent” should mean more than rerunning the same code with a different seed: ideally the reference uses a different computational route and is checked against the design specification.
For group-sequential designs, the reference specification should include the number and timing of looks, the joint distribution or increments used to form the test statistics, boundaries, and exact stopping and rejection rules. A fixed-sample critical value cannot be substituted for an interim boundary while retaining the same type I error interpretation. If an operating characteristic is computed by numerical integration, retain the integration method, accuracy settings, and convergence evidence alongside the reference value.
A practical comparison proceeds in a controlled order:
Write down all inputs and define precisely what output is being compared.
Use an exact calculation when available, or document an independent numerical calculation and its accuracy.
Set the error metric and tolerance before examining the workflow result; for simulation, also report Monte Carlo uncertainty.
Report both values and their difference. If the check fails, first verify matching definitions and inputs, then inspect implementation and numerical precision.
Diagnostics, Alternatives, and Limitations
A single benchmark can miss errors that happen not to affect that case. Use a small, deliberate set of reference designs that exercise distinct features: for example, null rejection probability, power under a nonzero effect, stopping behavior, and maximum sample size. Include boundary cases where small changes to a rule could alter the output. The earlier material on simulation diagnostics is useful when a benchmark result is noisy, but benchmarking adds the separate requirement that the target reference itself be defensible.
When a comparison fails, useful checks include whether the same alternative and null scenario were used; whether boundaries are applied with \(>\) or \(\geq\) consistently; whether an early stop is treated as final; whether the reported quantity is unconditional or conditional; and whether random-number generation or numerical integration has adequate precision. A discrepancy that persists when the simulation replication count increases is less plausibly explained by Monte Carlo noise alone.
Benchmarking differs from several related checks. Simulation validation asks whether a simulation estimates an operating characteristic reliably. Sensitivity analysis asks how results change under alternative assumptions. A benchmark asks whether a specified implementation reproduces a trusted value in a specified case. These approaches complement one another: an exact benchmark may be narrow, while simulation and sensitivity analysis cover broader but less exactly known scenarios.
The main limitation is scope. Passing all chosen benchmarks does not prove the implementation correct for every combination of boundaries, nuisance assumptions, or operational deviations. A tolerance also requires judgment: one suitable for a probability near 0.5 may be inappropriate for a rare event, and rounding before comparing can conceal a real difference. Keep adequate precision for the comparison, then round values for presentation.
- Wrong reasoning: “The simulated result is close to the reference, so the workflow is correct.”
- Correction: Agreement supports the implementation for that specified case and tolerance only. Check additional designs and outputs, and verify the reference and assumptions independently.
- Wrong reasoning: “A failed tolerance means the design itself is invalid.”
- Correction: A failed benchmark means the workflow and reference disagree more than the stated allowance. First determine whether inputs, definitions, simulation error, numerical accuracy, or code explain the difference.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A simulated power is 0.612 and the reference is 0.610, with an absolute tolerance of 0.003. Does it pass, and what does that conclusion establish?
- Why is a simulation estimate’s standard error not the same thing as the benchmark tolerance?
- Under the assumptions of the independent-test example, calculate the no-rejection probability before taking its complement.
- A two-look workflow is compared with a one-look normal reference. Name one design feature that could make the comparison invalid.
- A benchmark fails by more than its tolerance. Give two checks to make before changing the design code.
Solutions. (1) Yes: \(|0.612-0.610|=0.002\leq0.003\). It establishes agreement within tolerance for that case, not universal correctness. (2) The standard error quantifies random variation in a simulation estimate; the tolerance is the prespecified acceptable discrepancy. (3) The no-rejection probability is \(\Phi(2.8)\Phi(2.0)\approx0.9747529\), so the rejection probability is about 0.0252471. (4) Interim boundaries, stopping rules, or the dependence between cumulative statistics can differ from the one-look reference. (5) Verify that both calculations use the same scenario and output definition, and check simulation uncertainty or numerical precision before attributing the discrepancy to code.