Why Simulate the Effect of Adding Looks?
A test conducted once at level \(\alpha\) has a specified null rejection probability. Repeating that test at several planned looks with the same per-look threshold can increase the chance of rejecting at least once. A simulation offers a direct diagnostic: generate data under \(H_0\), apply the complete monitoring rule to each simulated data set, and estimate the proportion of data sets that trigger rejection.
The comparison is informative only when the simulation represents the procedure being studied. In particular, interim statistics based on accumulating data are dependent. Simulating independent test statistics at each look would represent a different procedure and could give a misleading answer. The focus here is on comparing simulated null rejection rates while holding the per-look threshold fixed; the simulation is a diagnostic, not a method for selecting a calibrated threshold.
Specify the Simulation Before Running It
A simulation check should start with a precise description of the null model and the decision rule. For a simple illustration, suppose observations are independent with \(X_i\sim N(0,1)\) under \(H_0\). At each planned sample size \(n_j\), calculate \(Z_j=\sqrt{n_j}\bar{x}_j\), where \(\bar{x}_j\) is the mean of the first \(n_j\) observations. Reject at a look if \(Z_j\geq 1.6449\), the approximate upper-tail 5% standard-normal cutoff. This is a one-sided rule; the direction and cutoff must be specified before simulating.
For each simulated data set, generate one sequence of observations and calculate the statistics at all scheduled looks from that sequence. This preserves the shared data across looks. Record whether at least one statistic crosses the cutoff. Repeat this process \(B\) times. To compare schedules, keep the null model, cutoff, and simulation count fixed, changing only the number of planned looks being compared.
The assumptions in this illustration are independent standard normal observations, a correctly specified known variance, a true null mean of zero, and a prespecified schedule. Other applications require a null data-generating model and test statistic appropriate to the study. The simulation must also implement the stopping rule faithfully: after a rejection, the procedure stops if that is the rule, although later statistics need not be generated to determine the at-least-once rejection indicator.
State the data-generating model, the number and timing of looks, the per-look cutoff, and the stopping rule. Decide which feature will change in the comparison.
For each replicate, generate the data needed through the latest look. Compute each interim statistic from the appropriate accumulated data, not from a newly independent data set at each look.
Mark a replicate as a rejection if any planned look meets the rejection rule. Otherwise mark it as a nonrejection.
Divide the number of rejecting paths by \(B\), quantify Monte Carlo uncertainty, and interpret the difference in the context of the simulated procedure.
Worked Example: Comparing One and Two Looks
Worked Example: Comparing One and Two Looks
Suppose a hypothetical simulation uses \(B=100{,}000\) independent null data sets from the model above. Compare a one-look procedure at \(n=100\) with a two-look procedure at \(n=50\) and \(n=100\). Both use the same upper-tail cutoff, 1.6449.
For the one-look procedure, 5,020 of the 100,000 paths reject. Its estimated null rejection rate is
For the two-look procedure, 8,310 paths reject at one or both looks:
Thus the simulated rate is 8.310% with two looks and 5.020% with one, a difference of 3.290 percentage points. Under this null model and these schedules, the added look is associated with a higher estimated probability of at least one false rejection. This comparison does not say that every individual path that rejected with one look also rejected with two in every possible design; here they do, because the two-look rule retains the final look and adds an earlier opportunity using the same cutoff.
For an estimated rate \(\hat{r}\) based on \(B\) independent simulation replicates, the Monte Carlo standard error is estimated by \(\sqrt{\hat{r}(1-\hat{r})/B}\). For the two-look result,
The one-look estimate has Monte Carlo standard error \(\sqrt{0.05020(1-0.05020)/100{,}000} =\sqrt{0.04767996/100{,}000}\approx 0.0006905\). These standard errors describe simulation variability, not uncertainty about the model assumptions or about an actual treatment effect.
Why a Matched Comparison Helps
When the one-look schedule is contained in the two-look schedule and the final look is unchanged, use the same simulated data paths for both procedures. On each path, the two-look procedure can reject at the final look or at the added earlier look. Therefore its rejection indicator cannot be smaller than the one-look indicator. The difference between the two rejection-rate estimates is then the proportion of paths that newly reject at the added look.
In the example, \(8{,}310-5{,}020=3{,}290\) paths reject only under the two-look rule. The estimated added rejection probability is \(3{,}290/100{,}000=0.03290\). Since this is a binary outcome across independent paths, its Monte Carlo standard error is
This matched estimate directly quantifies the extra rejection probability in the specified simulation. It is generally preferable to treating the two estimated rates as independent, because the rates came from the same paths and are consequently related. A rough 95% Monte Carlo interval for the added rejection probability is \(0.03290\pm1.96(0.0005640)\), or approximately 0.03179 to 0.03401. This interval reflects simulation noise only.
Worked Example: Adding a Third Planned Look
Worked Example: Adding a Third Planned Look
Continue the same hypothetical model and cutoff, with looks at \(n=50\), \(n=100\), and \(n=150\). Suppose 10,940 of 100,000 simulated paths reject at one or more of these three looks. The estimated null rejection rate is \(10{,}940/100{,}000=0.1094\), or 10.94%.
Compared with the two-look estimate of 0.08310, the estimated increase is \(0.1094-0.08310=0.02630\), or 2.630 percentage points. If the same paths were used and the earlier two looks are unchanged, this difference corresponds to paths that reject for the first time at the third look. The estimated rate has Monte Carlo standard error
The simulation suggests a further increase in false-rejection probability when the third opportunity is added. It does not establish that the exact increase in every model or schedule will be 2.630 percentage points. That conclusion is limited to the stated null model, look timings, and cutoff.
Worked Example: Reading Monte Carlo Uncertainty
Worked Example: Reading Monte Carlo Uncertainty
In a separate hypothetical simulation study, a fixed monitoring procedure produces 6,220 rejecting paths out of 100,000. The estimated null rejection rate is \(\hat{r}=0.06220\), or 6.220%. Its estimated Monte Carlo standard error is
A rough 95% Monte Carlo interval is \(0.06220\pm1.96(0.0007637)\), approximately 0.06070 to 0.06370. This quantifies the precision of the simulation estimate conditional on the simulated procedure. It is not a confidence interval for the actual false-positive probability under every plausible data-generating model, and it does not account for errors in programming or misspecification of the null model.
Diagnostics, Alternatives, and Limitations
A useful diagnostic begins with checks of the simulation itself. Verify that the null model generates data with the intended properties, each statistic is calculated at the correct look, the same cutoff is applied at every look, and rejection is recorded if any look crosses the boundary. Check that counts sum to \(B\), and that rates are calculated using all simulated paths, including paths that stopped after a rejection. Repeating the simulation with a larger \(B\) should reduce Monte Carlo standard errors at approximately the rate \(1/\sqrt{B}\); it should not systematically change the underlying rejection probability.
For schedules where one design adds a look while retaining all the others, a matched-path comparison is especially clear. If the schedules differ in more than the number of looks—for example, if look timing or the final sample size also changes—the difference cannot be attributed solely to adding a look. Describe all changes explicitly. Comparing independent simulation runs is possible, but it usually gives a less precise comparison than applying both rules to the same null paths.
The main alternative to simulation is an analytic calculation under a fully specified joint model for the test statistics. Such calculations can be more precise and can help explain how the schedule affects the overall rejection probability, but they require appropriate probability assumptions and dependence calculations. Simulation is flexible and easy to adapt to complex data-generating mechanisms, but only approximates the probability and only for the model programmed. The later material on calibrated repeated-testing thresholds addresses how thresholds can be chosen to control the overall error rate; this diagnostic does not perform that calibration.
- Too few replicates: Monte Carlo noise may obscure modest differences. Report \(B\) and a Monte Carlo standard error, rather than relying on a rate alone.
- Independent statistics at each look: This discards the shared-data dependence in interim testing. Simulate the accumulating observations or otherwise use the correct joint structure.
- Changing multiple design features: If threshold, look timing, and sample size all change, the comparison does not isolate the effect of the number of looks.
- Confusing simulation and study uncertainty: A small Monte Carlo standard error does not validate the null model, the code, or a clinical interpretation.
A simulation check matters because it makes the consequence of repeated testing visible on the scale of the overall null rejection probability. Its value comes from a transparent, controlled comparison—not from treating a simulated percentage as universally true.
Check Your Understanding
Answer each question using the stated simulation principles.
- A simulated procedure rejects in 4,800 of 100,000 null replicates. What is the estimated null rejection rate?
- Why should statistics at successive looks be computed from accumulating data rather than generated independently?
- With matched paths, a one-look schedule rejects 5,100 times and a schedule adding an earlier look rejects 8,000 times, both out of 100,000. What is the estimated added rejection probability?
- A simulation reports a rejection rate of 0.08 with a very small Monte Carlo standard error. Does this prove the true study procedure has an 8% false-positive rate? Explain.
Solutions. (1) The estimated rate is \(4{,}800/100{,}000=0.048\), or 4.8%. (2) Repeated looks usually reuse observations, so their statistics are dependent; independent simulation would represent a different monitoring process. (3) The added rejection estimate is \((8{,}000-5{,}100)/100{,}000=0.029\), or 2.9 percentage points, because the earlier opportunity newly rejects 2,900 matched paths. (4) No. A small Monte Carlo standard error indicates that random simulation error is small for the programmed model and procedure; it does not establish that the model, code, or real study assumptions are correct.