From Design Goals to Acceptance Rules
Simulation produces estimates of how a proposed design behaves under specified conditions. Design acceptance is the separate decision about whether those results meet requirements set in advance. A rule such as “the design should have adequate power” is not yet quantitative: it does not say how much power is adequate, which scenarios must meet the target, or how simulation error will be handled.
A defensible acceptance rule specifies the operating characteristic, its direction and threshold, the scenarios to which the rule applies, and how uncertainty in the simulation estimate is treated. The threshold should reflect the trial’s scientific, ethical, operational, and governance requirements. There is no universal power, sample-size, or safety threshold that is appropriate for every trial.
An operating characteristic is a performance quantity defined for a particular design and data-generating scenario. Examples include the probability of rejecting a null hypothesis when it is true, the probability of declaring efficacy under an alternative, or the expected number of participants enrolled. The scenario is part of the definition: power is not one fixed property of an adaptive design, but varies with the treatment effect and other scenario inputs.
For a probability-valued characteristic, let \(p\) be the probability of the event of interest under one specified scenario. If \(R\) independent simulated trials are run under that scenario and the event occurs in \(X\) of them, the estimated probability is \(\hat{p}=X/R\). For an expected sample size, the estimate is the mean sample size across the simulated trials. The acceptance threshold applies to the characteristic, not merely to the number printed by the simulation software.
Specify the Criterion Before Evaluating Results
Write each criterion so that another analyst can decide whether it passes without having to guess what “acceptable” means. A maximum criterion has the form “the false-positive rate must not exceed 0.05.” A minimum criterion has the form “power must be at least 0.80.” A two-sided range may be appropriate when both very low and very high values are undesirable, though the reason for both limits should be explicit.
The acceptance specification should also identify the scenarios and the aggregation rule. For example, a requirement can apply to every prespecified null scenario, to a particular worst-case value, or to a weighted average. These are different requirements: an average can meet a threshold while an individual scenario does not. The earlier material on worst-case operating characteristics and scenario weighting distinguishes these summaries; an acceptance decision must use the one that was actually specified.
Where a requirement is intended to hold separately across a class of scenarios, use the same threshold and pass rule throughout that class unless a scientific or governance rationale justifies prespecified differences. It is not defensible to relax the threshold for a scenario after seeing that it fails. Nor should thresholds be chosen by inspecting the simulation results and then presented as if they had been fixed in advance.
Monte Carlo Uncertainty at the Threshold
Simulation estimates have Monte Carlo error because only a finite number of trial replicates are run. For an event probability estimated from \(R\) independent replicates, the estimated Monte Carlo standard error is
This standard error describes the simulation error in estimating \(p\); it is not uncertainty about the clinical inputs or uncertainty about how well the model represents a future trial. The earlier material on Monte Carlo error versus design uncertainty treats that distinction. When the estimated probability is not near 0 or 1 and \(R\) is sufficiently large, an approximate one-sided 95% upper bound is \(\hat{p}+1.645SE(\hat{p})\), and an approximate one-sided 95% lower bound is \(\hat{p}-1.645SE(\hat{p})\). Exact binomial bounds are preferable when counts are small or probabilities are near the boundaries.
For a maximum requirement, a conservative acceptance rule is to require the upper bound to be no greater than the threshold. For a minimum requirement, require the lower bound to be no less than the threshold. This avoids treating an estimate just on the desired side of a cutoff as conclusive when its simulation error could readily put the underlying operating characteristic on the wrong side.
These are approximate, one-sided 95% rules, not universal governance requirements. A protocol or design committee may specify a different confidence level or a different prespecified decision rule. If many scenario-specific bounds are used, their individual confidence levels do not automatically provide a simultaneous confidence guarantee across all scenarios. The required level of joint protection should be settled in the analysis plan rather than improvised after seeing results.
Worked Example: A Maximum False-Positive Rate
Suppose the prespecified requirement is that the false-positive rate must be no greater than 0.05 in a particular null scenario. In \(R=10{,}000\) simulated trials, 472 reject the null hypothesis. The point estimate is
The estimated Monte Carlo standard error is
The approximate one-sided 95% upper bound is
The point estimate is below 0.05, but the upper bound is above 0.05. Under the stated uncertainty-aware rule, this result does not demonstrate that the criterion is met. It is not evidence that the true false-positive rate is necessarily above 0.05; the simulation is too imprecise to establish that it is at or below the limit. A larger prespecified simulation could reduce Monte Carlo error. For comparison, with \(R=20{,}000\) and 944 rejections, \(\hat{p}=0.0472\), \(SE\approx0.001500\), and the upper bound is approximately \(0.04967\), which meets this rule.
Worked Example: A Minimum Power Requirement Across Scenarios
A design is required to achieve power of at least 0.80 in each of two prespecified alternative scenarios. Use \(R=10{,}000\) simulated trials per scenario and the approximate one-sided 95% lower-bound rule.
In Scenario A, 8,230 trials meet the efficacy decision rule. Thus \(\hat{p}=8{,}230/10{,}000=0.823\), and
Scenario A passes because its lower bound, approximately 0.8167, exceeds 0.80. In Scenario B, 8,060 of 10,000 trials meet the efficacy rule, so \(\hat{p}=0.806\). Its standard error and lower bound are
Scenario B does not pass: its estimated power exceeds 0.80, but its lower bound is just below the threshold. Because the requirement applies to each scenario, the design has not met the full acceptance criterion. Averaging the two power estimates would answer a different question and could conceal this failure.
Match the Metric to the Requirement
A threshold is useful only if the operating characteristic measures the concern it is meant to control. For example, a limit on expected sample size is not a limit on the probability of exceeding a recruitment cap. If the concern is that a trial might enroll more than 500 participants, the probability of exceeding 500 or a direct cap-violation measure is relevant; the mean alone cannot establish that such events are rare.
Likewise, an acceptable average duration does not imply acceptable duration in every scenario. A threshold on probability of selecting the correct treatment is not interchangeable with a threshold on power to reject a particular null hypothesis. Define the event or summary precisely, and use its units: probabilities are unitless, expected sample size is in participants, and expected duration is in the specified time unit.
Worked Example: An Expected Sample-Size Limit
Suppose the requirement is that expected enrollment be no more than 500 participants in a specified scenario. Across \(R=1{,}600\) simulated trials, the mean enrollment is 486 participants and the sample standard deviation of enrollment is 60 participants. The estimated Monte Carlo standard error of the mean is
Using an approximate one-sided 95% upper bound, the result is
This is below 500, so the result meets the expected-enrollment criterion under this rule. The conclusion is limited to the mean: it does not establish that every simulated trial, or 95% of simulated trials, enrolls fewer than 500 participants. Those claims would require criteria based on the distribution of trial-level enrollment, not just its expectation.
Consistent Application, Diagnostics, and Limitations
A practical acceptance process can be organized as a short sequence. First, identify the criterion and scenario from the prespecified plan. Next, calculate the operating-characteristic estimate and its Monte Carlo uncertainty using the chosen method. Then apply the correct inequality: upper bound for a maximum, lower bound for a minimum. Finally, record whether the criterion passes, fails, or is not resolved by the simulation precision required by the rule.
Write the metric, threshold, units, applicable scenarios, and aggregation rule before reviewing the results.
Use the simulated trials generated under the specified design and scenario, and calculate the corresponding Monte Carlo error.
For an upper limit, compare the upper bound with the limit; for a lower limit, compare the lower bound with the minimum.
Evaluate every required scenario and use the prespecified aggregation rule rather than substituting a more favorable summary.
Several diagnostics help identify an unreliable acceptance decision. Check that replicate trials are independent for the Monte Carlo calculation, that the simulated decision rule matches the proposed trial design, and that each reported estimate is paired with the correct scenario. Examine whether a bound is close to the threshold: a near-boundary result can be sensitive to simulation precision. If it is, use the prespecified precision plan or an appropriate larger simulation, rather than changing the threshold.
A point-estimate rule is simpler and may be explicitly chosen for a particular purpose, but it does not account for the chance that Monte Carlo error moved the estimate across the cutoff. A bound-based rule is more cautious, but can require more simulation and can leave a result unresolved near the boundary. Neither rule removes uncertainty in the scenario inputs or protects against a misspecified simulation model. Those questions require separate evaluation.
| Approach | Decision basis | Main caution |
|---|---|---|
| Point-estimate rule | Compare \(\hat{p}\) or another estimate directly with its threshold | Can pass or fail because of Monte Carlo fluctuation near the cutoff |
| Monte Carlo-bound rule | Compare an appropriate one-sided bound with the threshold | Requires a stated confidence level and can be inconclusive near the boundary |
| Scenario-average rule | Compare a prespecified weighted or unweighted average with its threshold | Can hide poor performance in an individual scenario |
| Every-scenario rule | Require the criterion in each scenario in the stated set | Can be demanding; the scenario set and uncertainty treatment must be prespecified |
A common mistake is to say, “The estimated false-positive rate is 0.047, so the design passes a 0.05 limit.” That reasoning compares only the point estimate and ignores the acceptance rule’s treatment of simulation error. The correction is to apply the specified rule consistently: if the rule requires the one-sided upper bound to be at most 0.05, calculate that bound and compare it with 0.05. Another mistake is to average across scenarios when the requirement is that each scenario pass. The correction is to respect the stated aggregation rule and report the scenario-level results needed to verify it.
Check Your Understanding
Answer each question using the stated acceptance rule. Solutions are included after the questions.
- A maximum false-positive rate is 0.05. An estimate is 0.048 with an upper one-sided bound of 0.052. Does it pass a rule requiring the upper bound to be at most 0.05?
- A minimum power requirement is 0.80 in every one of three scenarios. Two scenarios pass and one fails. Can the design be declared to meet the stated requirement by averaging the three estimates?
- Why does an expected sample size below a recruitment cap not show that the trial will never exceed the cap?
- In one sentence, distinguish Monte Carlo error from uncertainty about the scenario inputs.
Solutions. (1) No. The upper bound, 0.052, exceeds 0.05. (2) No. The requirement applies to every scenario, so one failing scenario means the full criterion is not met. (3) The expectation summarizes an average and does not rule out individual trials with enrollment above the cap. (4) Monte Carlo error comes from using a finite number of simulated replicates, whereas input uncertainty concerns which scenario values are appropriate for the real design problem.