Why Precision Must Be Planned Across a Family
A simulation study may report type I error, power, selection accuracy, failure probability, and expected trial duration under several scenarios. A Monte Carlo standard error or interval for each quantity describes uncertainty in that individual estimate. But if the design decision depends on the whole collection, it is also useful to ask whether the estimates are precise together. A set of individually narrow intervals can still have a substantial chance that at least one misses its target.
The family must be specified before choosing a multiplicity adjustment: it is the complete set of scenario-by-operating-characteristic estimates for which a joint precision statement is intended. For example, four operating characteristics in three scenarios make a family of 12 estimates if every combination is included. A different scientific question may define a smaller family, but estimates cannot be quietly omitted after seeing their results while retaining the original familywise guarantee.
This guarantee concerns Monte Carlo sampling error: the variation that would arise if the simulation were repeated with new random replicates. It does not quantify uncertainty about whether the scenarios represent the real world, whether the trial algorithm was implemented correctly, or whether the assumed data-generating model is adequate.
Bonferroni Precision for Simulated Rates
Suppose there are \(K\) prespecified rate estimates, possibly drawn from different scenarios. For rate \(j\), let \(I_{jr}\) equal 1 when simulation replicate \(r\) meets the event definition and 0 otherwise. Assume that within each scenario the replicates are independent and identically generated under a fixed scenario and trial algorithm. Then \(I_{jr}\) is Bernoulli with event probability \(p_j\), and the estimate from \(R\) replicates is \(\hat{p}_j\), the observed event proportion.
For a sufficiently large \(R\), a normal-approximation interval for one rate has estimated standard error \(\sqrt{\hat{p}_j(1-\hat{p}_j)/R}\). To target a simultaneous familywise level \(1-\alpha\), a simple approach assigns error probability \(\alpha/K\) to each two-sided interval. Its normal critical value is \(z_{1-\alpha/(2K)}\). Bonferroni's inequality ensures that the chance of any interval failing is no more than the sum of the individual failure probabilities.
This justification does not require the estimates to be independent of one another. Outcomes from the same simulated trial may contribute to several operating characteristics, and estimates across scenarios may also be dependent; the union bound still applies if each individual interval has its stated coverage. The individual coverage condition itself must be reasonable, including its approximation assumptions.
If the desired half-width for rate \(j\) is \(d_j\), planning with a value \(p_j\) gives the approximate requirement below. Because \(p_j\) is generally not known, it may be replaced by a scientifically justified anticipated value for a planning calculation, or by the conservative bound \(p_j(1-p_j)\leq 0.25\). Round the required replication count up to an integer.
For a common replication count across all rates, calculate the requirement for each one and use the largest. A conservative plan substitutes 0.25 for each variance term. An anticipated rate can reduce the planned count, but it does not guarantee the target if the true rate is farther from that value.
Worked Example: Planning Precision for Ten Rates
Suppose a hypothetical simulation will report \(K=10\) rates across its prespecified scenario-characteristic family. The goal is simultaneous 95% precision, with a half-width of 0.02 for each rate. For planning only, suppose a particular rate is expected to be about 0.20. The replicates are generated independently under a fixed scenario, and the normal approximation is used for this planning calculation.
Here \(\alpha=0.05\), so the Bonferroni two-sided critical value is approximately \(z_{1-0.05/(2\times10)}=z_{0.9975}=2.807\). The anticipated Bernoulli variance is \(0.20(1-0.20)=0.16\). Thus:
The planning count is therefore at least 3,152 replicates per rate-producing scenario, after rounding upward. This calculation relies on the anticipated rate of 0.20; it does not guarantee the half-width if the rate differs materially. Using the conservative variance bound gives \(R\geq(2.807)^2(0.25)/(0.02)^2=4924.5306\), requiring 4,925 replicates. If all ten rates use the same replication count and the conservative guarantee is wanted for each, the larger count is the appropriate plan.
Different Targets and Non-Rate Summaries
Precision goals need not be identical across the family. A clinically consequential type I error rate may need a tighter half-width than an exploratory measure of average sample size. Give each estimate a target \(d_j\), apply the family adjustment to the complete prespecified set, and plan to meet every target. If one common \(R\) is used, the most demanding requirement determines it. If scenarios have different computational costs, different replication counts are possible, but the coverage and precision plan must account for each scenario's own \(R_j\).
For a continuous replicate-level outcome \(Y_{jr}\), such as trial duration measured in weeks, suppose replicates are independent and identically distributed with finite mean and variance. The estimated mean is \(\bar{Y}_j\), and its estimated Monte Carlo standard error is \(s_j/\sqrt{R}\). A large-sample simultaneous interval uses the same Bonferroni critical value, multiplying this standard error by \(z_{1-\alpha/(2K)}\). This approximation is most credible when the number of replicates is large and the replicate-level distribution is not so skewed or heavy-tailed that the sampling distribution of the mean remains poorly approximated by a normal distribution.
Worked Example: A Shared Count for Four Rate Targets
A hypothetical assurance report has four prespecified rate estimates and seeks familywise 90% precision. Two rates require half-widths no larger than 0.03, and the other two require half-widths no larger than 0.015. For a conservative plan, use \(p_j(1-p_j)\leq0.25\) for every rate. Assume independent replicates within each scenario and use the normal approximation.
With \(K=4\) and \(\alpha=0.10\), the Bonferroni critical value is \(z_{1-0.10/8}=z_{0.9875}\), approximately 2.2414. For a half-width of 0.03:
This target requires at least 1,396 replicates. For a half-width of 0.015:
The tighter target requires 5,583 replicates. If all four estimates use a common replication count, use at least 5,583; that count also meets the looser targets. The familywise guarantee is based on the individual interval procedure having adequate coverage, not merely on reaching these nominal planning counts.
Worked Example: Reporting Simultaneous Precision for Mean Duration
In a hypothetical simulation, six prespecified scenario-level mean trial durations form the family. For one scenario, \(R=4{,}000\) independently generated replicates have a mean duration of 1,200.0 patient-days and sample standard deviation \(s=12.0\) patient-days. A large-sample normal approximation is reasonable for this illustration. For simultaneous 95% intervals, the Bonferroni critical value is approximately \(z_{1-0.05/(2\times6)}=2.638\).
The estimated Monte Carlo standard error is \(12.0/\sqrt{4{,}000}=12.0/63.2456=0.1897\) patient-days. The simultaneous half-width is approximately \(2.638(0.1897)=0.5005\) patient-days, giving the interval:
This interval is one member of the six-interval family. Under the stated large-sample conditions and Bonferroni construction, the six intervals together have approximate familywise coverage of at least 95%. It is not a 95% probability statement about this fixed interval alone, nor does its narrowness establish that the scenario assumptions are correct.
Choosing an Uncertainty Summary and Checking It
Bonferroni-adjusted intervals are transparent and robust to dependence among the estimates, but they can be conservative, especially for a large family whose estimates are strongly related. Alternatives include a simultaneous procedure based on the joint distribution of the estimates or, under appropriate independence assumptions, a Šidák adjustment. Such methods can be narrower, but require additional assumptions and a defensible way to estimate the joint behavior. A collection of unadjusted intervals is not a simultaneous alternative: it only provides the stated coverage separately for each interval.
For rates near zero or one, the normal approximation can be poor, particularly when few events are observed. A normal interval may have inadequate coverage or extend outside the possible range from 0 to 1. Inspect event counts and interval endpoints; if counts are sparse, use a suitable binomial interval procedure and adjust its individual error level to \(\alpha/K\), or plan using a method with a valid finite-sample guarantee. A target half-width by itself does not repair a poor interval approximation.
After simulation, report for each estimate its scenario, point estimate, uncertainty summary, replication count, and whether its prespecified target was met. Compare realized half-widths with targets rather than assuming the planning calculation guarantees them. If the rate or variability is larger than anticipated, the realized interval may be wider. If simulations are extended after inspecting interim results, ordinary fixed-\(R\) intervals do not automatically retain their nominal coverage; the stopping or extension rule must be accounted for.
A common mistake is to calculate a 95% interval for every scenario and conclude that the full table has 95% coverage. Those are marginal, not familywise, statements: the probability that all intervals cover can be below 95%. The correction is to define the family and use a simultaneous procedure, such as Bonferroni, with adjusted individual error probabilities. Another mistake is to treat a very precise Monte Carlo estimate as evidence that the underlying design is robust. Monte Carlo precision describes simulation error conditional on the specified model; sensitivity to scenario assumptions is a separate question.
Practical Workflow
List every scenario and operating characteristic that must be covered by the joint statement, and set the familywise level \(1-\alpha\).
Choose a half-width \(d_j\) for each estimate in its own units, based on how precisely it must be known for the design decision.
Use the Bonferroni critical value and an anticipated or conservative variance to calculate the count for each estimate. Use the largest requirement when a common count is planned.
Verify the replicate-generation assumptions, inspect counts and distributions, and report adjusted uncertainty summaries alongside the achieved precision.
The main value of this workflow is not simply a larger simulation. It makes explicit which collection of conclusions the simulation is intended to support, how much Monte Carlo uncertainty is tolerable for each, and where the reported precision does not extend.
Check Your Understanding
Answer each question, then compare with the solutions below.
- There are five rate estimates in a family with 95% simultaneous coverage. What individual two-sided error probability does the Bonferroni construction assign to each interval?
- For one rate, use the conservative variance bound, a 95% individual confidence level, and target half-width 0.025. What replication count is required by the normal planning formula?
- Why can Bonferroni provide a familywise guarantee when the estimated operating characteristics are dependent?
- Four mean-duration estimates form a family with 95% simultaneous coverage. For one estimate, \(s=20\) days and \(R=2{,}500\). Using critical value 2.498, calculate its simultaneous half-width in days.
- Does meeting every Monte Carlo precision target show that the chosen data-generating scenarios are realistic? Explain.
Solutions. (1) The family has \(\alpha=0.05\), so each interval is assigned error probability \(\alpha/K=0.05/5=0.01\); its individual coverage target is 99%. (2) The calculation is \(R\geq(1.96)^2(0.25)/(0.025)^2=0.9604/0.000625=1536.64\), so use at least 1,537 replicates. (3) The probability of at least one failure is bounded by the sum of the individual failure probabilities; this union bound does not require independence. (4) The standard error is \(20/\sqrt{2500}=0.4\) days, so the half-width is \(2.498(0.4)=0.9992\) days. (5) No. The targets address Monte Carlo sampling error conditional on the simulation model; realism of the scenarios requires separate scientific justification.