Why Read Complete Paths?
A repeated-testing simulation produces more than a collection of separate p-values. Each simulated run has an ordered sequence of results, one for each scheduled look. Reading the whole sequence shows whether a run never crossed a rejection threshold, crossed it before the final look, or crossed it for the first time at the final look. These are different paths through the monitoring process, even when two runs end with the same final-look result.
This distinction matters when a protocol allows a study to stop after a rejection. A final-look-only summary cannot show how often the protocol would have reached an earlier rejection. Conversely, counting every significant p-value in every run can count one run more than once. The path-based summary assigns each run to one category, according to its first rejection.
Set Up the Path Classification
Suppose there are \(K\) scheduled looks. Let \(p_{rj}\) denote the p-value at look \(j\) in simulated run \(r\), where \(j=1,\ldots,K\). Let \(\alpha_j\) be the prespecified rejection threshold at look \(j\). Define the per-look rejection indicator \(I_{rj}=1\) if \(p_{rj}\leq\alpha_j\), and \(I_{rj}=0\) otherwise. The threshold may be the same at every look or may vary; the path must be assessed using the thresholds actually specified for the monitoring procedure.
For the simple two-look case, look 1 is the interim look and look 2 is the final look. The three mutually exclusive classifications are:
- No rejection: neither look rejects \(H_0\).
- First interim rejection: look 1 rejects \(H_0\). This remains the path category even if a simulated look 2 result would also reject.
- Final-only rejection: look 1 does not reject, but look 2 does.
These categories describe first-rejection timing. If the actual procedure stops at the first rejection, a later test is not conducted on a path that rejects at the interim. A simulation may nevertheless calculate later results to inspect the sequence; that does not change the operational classification. For more than two looks, a path can be labelled by its exact first rejection look, or the interim categories can be combined as “first rejection before the final look.” The final-only category still means no earlier rejection and a rejection at the final look.
The simulation needs to represent the procedure being studied. In particular, each run must be generated under the null model of interest, the tests and thresholds must match the planned analysis, and looks must occur according to the specified schedule. Separate simulated runs should be independent replications. Results within a run do not need to be independent: repeated looks commonly use overlapping data, and the simulation should preserve that dependence rather than replace the sequence with unrelated tests.
Worked Example: Classifying Individual Two-Look Paths
Consider hypothetical null simulations with two looks and a threshold of \(0.05\) at each look. The table gives p-values for four runs. Classify each path by its first rejection.
| Run | Interim p-value | Final p-value | Path category |
|---|---|---|---|
| A | 0.18 | 0.62 | No rejection |
| B | 0.041 | 0.27 | First interim rejection |
| C | 0.30 | 0.048 | Final-only rejection |
| D | 0.02 | 0.01 | First interim rejection |
Run A has both p-values above \(0.05\), so it has no rejection. Run B crosses the threshold at the interim look, so its first rejection is interim, regardless of the later non-significant result. Run C does not reject at the interim but does reject at the final look, making it a final-only rejection. Run D rejects at both looks in the simulated sequence, but the first rejection occurs at the interim. If monitoring stops at rejection, its operational path stops there.
The classification is based on the ordered sequence, not on the final p-value alone and not on the number of significant p-values within a run.
From Paths to Estimated Proportions
Let \(R\) be the number of simulated runs. Count how many runs have no rejection, how many first reject at an interim look, and how many reject only at the final look. Divide each count by \(R\) to obtain the corresponding simulated proportion. Because every run belongs to exactly one category, the three counts must sum to \(R\), and their proportions must sum to 1, apart from rounding.
The any-look rejection event is the union of the interim-first and final-only categories. Thus, its simulated proportion is their combined proportion. A final-look rejection proportion answers a different question: it counts runs whose final-look test rejects, including runs that may already have rejected at an interim look in a continued simulation. The final-only proportion excludes those earlier paths. Always label which of these quantities is being reported.
This identity is a partition of the any-look event, not an assumption that results at different looks are independent. It remains valid when the tests within a run are dependent, provided the path categories are assigned by first rejection and every simulated run is classified once.
Worked Example: Summarizing Ten Simulated Paths
Suppose a hypothetical null simulation uses two looks, both with threshold \(0.05\). Its ten p-value sequences are listed below. Apply the first-rejection rule and count the path categories.
| Run | Interim p-value | Final p-value |
|---|---|---|
| 1 | 0.31 | 0.44 |
| 2 | 0.08 | 0.12 |
| 3 | 0.049 | 0.03 |
| 4 | 0.20 | 0.62 |
| 5 | 0.11 | 0.047 |
| 6 | 0.70 | 0.55 |
| 7 | 0.02 | 0.01 |
| 8 | 0.33 | 0.049 |
| 9 | 0.051 | 0.40 |
| 10 | 0.14 | 0.18 |
Runs 3 and 7 reject at the interim, so there are 2 first interim rejections. Runs 5 and 8 do not reject at the interim but reject at the final look, so there are 2 final-only rejections. The other 6 runs have no rejection. The counts check: \(2+2+6=10\).
The estimated proportion with any rejection is
Thus, 40% of these ten simulated null paths rejected at some point in the two-look procedure. This is a simulation proportion based on only ten runs, not a precise estimate of the procedure’s long-run false-positive probability. The final-only proportion is \(2/10=0.20\); the interim-first proportion is also \(2/10=0.20\). Notice that run 3 contributes to the interim-first category even though its final p-value is also below \(0.05\).
Why the Category Identity Holds
For each run \(r\), define \(A_r=1\) if there is any rejection and 0 otherwise. Define \(B_r=1\) if the first rejection is at an interim look, and \(C_r=1\) if the path has no earlier rejection and rejects at the final look. Under the stated first-rejection classification, \(B_r\) and \(C_r\) cannot both equal 1. If a path has any rejection, its first rejection is either before the final look or at the final look. Therefore, for every run,
Summing over the \(R\) runs and dividing by \(R\) gives
The left side is the simulated any-look rejection proportion; the two terms on the right are the interim-first and final-only proportions. The argument is a run-by-run partition. It does not require independent tests across looks or any special joint distribution for the p-values.
Monte Carlo Uncertainty and Diagnostics
A simulated proportion varies from one batch of runs to another. If the independent null-simulation runs have a common probability \(\pi\) of any rejection, then each \(A_r\) is a Bernoulli indicator with variance \(\pi(1-\pi)\). The sample proportion \(\hat{\pi}\) has approximate Monte Carlo standard error
This standard error describes simulation noise in estimating \(\pi\); it is not the standard error of a treatment effect or an uncertainty measure for the data in one trial. It also does not correct a misspecified null model or an incorrectly implemented monitoring rule.
Worked Example: Assessing Monte Carlo Precision
In a hypothetical simulation of \(R=2{,}000\) independent null paths, 84 paths first reject at an interim look, 55 reject only at the final look, and 1,861 have no rejection. The counts sum to \(2{,}000\). The estimated any-look rejection proportion is
The Monte Carlo standard error is approximately
The estimated any-look rejection proportion is 6.95%, with Monte Carlo standard error about 0.569 percentage points. This quantifies the variability attributable to using a finite number of simulations. It does not mean there is a 6.95% probability that the null hypothesis is true, nor does it make the estimate exact. Increasing the number of independent runs generally reduces this simulation error.
Useful checks include verifying that all runs use the intended null-generating model, the prescribed looks, and the correct thresholds; confirming that the three path counts add to \(R\); and checking that the any-look count equals the sum of interim-first and final-only counts. If purportedly independent runs reuse random outcomes or share simulated data, the usual Monte Carlo standard error calculation may not apply. If a run stops at rejection, record later looks as not conducted rather than treating them as non-significant.
What Path Summaries Show—and What They Do Not
A path summary preserves timing information that a single any-rejection indicator loses. It can show whether observed simulated rejections tend to occur at interim or final looks and can expose a classification or implementation error, such as counting one run as several first rejections. By comparison, a final-look-only summary ignores interim-first stopping, while a list of per-look rejection proportions does not by itself say how many distinct paths rejected at least once.
Path counts are descriptive of the simulation design and null model used. They are not evidence that an effect exists in a real study, and they do not guarantee the same rejection behavior under a different null model, testing procedure, threshold scheme, or monitoring schedule. A small number of runs can also make category proportions unstable, especially for uncommon paths. The introductory material on null simulation gives the broader role of simulation; here the essential discipline is to preserve the sequence and classify each run once.
A common mistake is to count every significant look as a separate false positive, then call that count the proportion of studies with any false positive. That reasoning confuses tests with simulated runs. The correction is to form one first-rejection category per run: interim-first, final-only, or no rejection. Another mistake is to say that a significant final result means the path was final-only. That is false whenever an earlier look already rejected. Read the full sequence before assigning the category.
Check Your Understanding
Answer each question using the first-rejection rule. Brief solutions follow the questions.
- With a threshold of \(0.05\) at both looks, classify the path with p-values 0.07 at interim and 0.04 at final.
- With the same thresholds, classify the path with p-values 0.03 at interim and 0.02 at final. Why is it not counted twice?
- In 500 simulated runs, 18 paths first reject at an interim look and 22 reject only at the final look. What is the estimated any-look rejection proportion, and how many runs have no rejection?
- Why can tests within one simulated path be dependent without invalidating the identity that partitions any rejection into interim-first and final-only paths?
- A simulation records 40 significant p-values across 100 runs, but it does not report which p-values came from the same run. Can this alone determine the proportion of runs with any rejection? Explain.
Solutions. 1. Final-only rejection: the interim p-value exceeds \(0.05\), and the final p-value is at or below it. 2. First interim rejection: the earliest crossing is interim, so the run enters one category even if the final result also rejects. 3. The any-look estimate is \((18+22)/500=0.08\), or 8%; the no-rejection count is \(500-18-22=460\). 4. The identity follows separately for each run from mutually exclusive first-rejection categories; it does not use independence across looks. 5. No. A count of significant tests does not reveal whether several came from the same run; the ordered results must be grouped by run before counting paths.