Why Calibrate Boundaries by Null Simulation?
A repeated-testing design rejects \(H_0\) if its test statistic crosses a boundary at any planned interim or final look. Therefore, the relevant type I error is not the error rate at an individual look. It is the probability of at least one crossing over the full monitoring plan. For a weighted logrank process, this probability can depend on the planned weights, event-time pattern, censoring, and the way looks are triggered. Null simulation lets the calibration reflect those features directly.
The aim is not to simulate treatment benefit. Instead, the trial is generated under \(H_0\), and the complete monitoring rule is applied to each simulated trial. The fraction of trials that cross a boundary estimates the design’s sequential type I error. A boundary can then be adjusted until this estimated probability is close to its target.
Specify the Null Process Before Simulating
A weighted logrank analysis compares event experience between randomized groups over time. At event time \(t_j\), let \(Y_{1j}\) and \(Y_{0j}\) denote the numbers at risk in the treatment and control groups, and let \(d_{1j}\) and \(d_j\) denote the treatment-group and total event counts. One common score convention is $$ U(t)=\sum_{t_j\leq t} w(t_j)\left(d_{1j}-\frac{Y_{1j}}{Y_{1j}+Y_{0j}}d_j\right), \qquad V(t)=\sum_{t_j\leq t} w(t_j)^2 \frac{Y_{1j}Y_{0j}d_j}{(Y_{1j}+Y_{0j})^2}. $$ Here \(w(t)\) is the prespecified weight function and \(V(t)\) is the usual variance estimate under the null. This score convention makes positive \(U(t)\) indicate more treatment-group events than expected. If the efficacy alternative is that treatment reduces events, use the oriented statistic \(Z_k=-U(t_k)/\sqrt{V(t_k)}\), so larger values favor that alternative. The sign convention must be fixed before calibration.
A null simulation must represent the protocol and analysis, not merely generate independent standard normal values. The usual assumptions include randomized treatment assignment as planned, no treatment effect under \(H_0\), independent censoring conditional on modeled factors, and use of the same prespecified weight function and statistic in every simulated trial. The simulation should also reproduce accrual, event and censoring distributions, planned look triggers, and any enrollment or follow-up limits that affect which data are available at each look. If the test uses a fixed number of events per look, each replicate should reach its looks by that rule; if looks occur at specified information levels, those levels should be implemented as specified.
Weights and monitoring rules should not be changed after looking at simulated or trial treatment effects. If a weight is estimated or adapted using data, that adaptation is part of the procedure and must be reproduced in every replicate; otherwise, the calibration does not describe the actual test. The earlier material on alpha-spending boundaries gives ways to specify cumulative error targets. Simulation can assess or calibrate the resulting boundary under a more detailed null model.
How the Calibration Works
For a candidate boundary vector \((b_1,\ldots,b_K)\), generate \(B\) independent trial data sets under \(H_0\). In each replicate, calculate the weighted logrank statistic at every planned look and record whether the trial crosses at least one boundary. Let \(I_r=1\) if replicate \(r\) crosses and \(I_r=0\) otherwise. Then the estimated sequential type I error is $$ \widehat{\alpha}_{\mathrm{seq}}=\frac{1}{B}\sum_{r=1}^{B}I_r=\frac{C}{B}, $$ where \(C\) is the number of simulated trials with a crossing.
A short justification explains why this estimates the desired quantity. Under the specified null model, each \(I_r\) is a Bernoulli random variable with success probability \(\alpha_{\mathrm{seq}}\). The independent simulation replicates therefore give $$ E(\widehat{\alpha}_{\mathrm{seq}})=\alpha_{\mathrm{seq}}, \qquad \operatorname{Var}(\widehat{\alpha}_{\mathrm{seq}}) =\frac{\alpha_{\mathrm{seq}}(1-\alpha_{\mathrm{seq}})}{B}. $$ Thus, the estimator is unbiased, and its Monte Carlo standard error is estimated by $$ \widehat{SE}_{MC} =\sqrt{\frac{\widehat{\alpha}_{\mathrm{seq}}(1-\widehat{\alpha}_{\mathrm{seq}})}{B}}. $$ This standard error measures simulation noise, not uncertainty about whether the null model or trial assumptions are correct.
For a common one-sided boundary \(c\), set \(b_k=c\) at every look and estimate \(P_{H_0}(\max_k Z_k\geq c)\). This probability decreases as \(c\) increases, so a search over candidate values can locate a boundary giving the target error rate. For look-specific boundaries, adjust the vector and evaluate the same whole-trial crossing event each time. Marginal error rates at the separate looks are not substitutes for this calculation because crossings across looks are dependent.
Worked Example: Quantifying Monte Carlo Error
Suppose 100,000 hypothetical trials are simulated under \(H_0\) using the planned weighted logrank analysis and boundary vector. In 2,487 trials, at least one boundary is crossed. The estimated sequential type I error is $$ \widehat{\alpha}_{\mathrm{seq}}=\frac{2{,}487}{100{,}000}=0.02487. $$
The estimated Monte Carlo standard error is $$ \widehat{SE}_{MC} =\sqrt{\frac{0.02487(1-0.02487)}{100{,}000}} =\sqrt{\frac{0.02487(0.97513)}{100{,}000}} \approx 0.000492. $$
An approximate 95% Monte Carlo interval is \(0.02487\pm1.96(0.000492)\), or about \(0.02391\) to \(0.02583\). This describes the simulation precision around the estimated crossing probability under the assumed null model. The estimate is close to a 0.025 target; the small difference from 0.025 is not, by itself, evidence of a calibration failure.
Choosing a Boundary From Simulated Crossings
When a common boundary is appropriate, the simulated maximum statistic \(M_r=\max_k Z_{rk}\) reduces the calibration to estimating an upper-tail probability. The boundary is selected so that approximately the target fraction of null maxima exceed it. In finite simulation, however, the estimate changes in discrete steps of \(1/B\), and the selected value has Monte Carlo uncertainty. It is generally better to report the simulation size, crossing count, estimated error, and Monte Carlo standard error than to imply that the boundary is exact.
Worked Example: Selecting a Common Boundary
Consider a hypothetical one-sided design with three looks and target sequential type I error 0.025. In \(B=20{,}000\) null simulations, the maximum \(M_r\) is recorded for each trial. At candidate boundary \(c=2.28\), 521 maxima are at least \(c\); at \(c=2.30\), 493 are at least \(c\).
The estimated crossing probabilities are $$ \widehat{\alpha}_{\mathrm{seq}}(2.28)=\frac{521}{20{,}000}=0.02605, \qquad \widehat{\alpha}_{\mathrm{seq}}(2.30)=\frac{493}{20{,}000}=0.02465. $$
The target crossing count is \(20{,}000(0.025)=500\). The \(2.30\) boundary is closer in estimated probability to the target: its estimate differs by \(0.00035\), compared with \(0.00105\) for \(2.28\). If the design requires estimated error not to exceed 0.025, \(2.30\) is the appropriate choice from these candidates. A finer search could examine additional values, but it cannot remove Monte Carlo uncertainty; increasing \(B\) can reduce that uncertainty.
A vector of boundaries may be needed when the design has different planned stringency at different looks. A direct approach is to choose a candidate vector, count a trial as a crossing only if it crosses at least one boundary, and adjust the vector until the total crossing probability meets the target. If cumulative error allocations are specified in advance, the simulation should also check the cumulative probability of crossing by each look against those allocations. Matching the final total alone does not guarantee that the intended pattern of error spending has been achieved.
Worked Example: Checking a Three-Look Boundary Vector
Suppose a hypothetical design uses one-sided boundaries \(b_1=2.55\), \(b_2=2.30\), and \(b_3=2.10\). In 100,000 null simulations, 550 trials cross at look 1, 1,050 first cross at look 2, and 900 first cross at look 3. The categories are mutually exclusive because a trial that crossed earlier is not counted again.
The total number of trials with any crossing is \(550+1{,}050+900=2{,}500\). Hence $$ \widehat{\alpha}_{\mathrm{seq}}=\frac{2{,}500}{100{,}000}=0.02500. $$
The estimated cumulative crossing probabilities are \(550/100{,}000=0.00550\) by look 1, \(1{,}600/100{,}000=0.01600\) by look 2, and \(2{,}500/100{,}000=0.02500\) by look 3. These values should be compared with any prespecified cumulative targets as well as the final target. The final Monte Carlo standard error is $$ \sqrt{\frac{0.02500(0.97500)}{100{,}000}}\approx 0.000494. $$ The estimated final error matches 0.025, but the vector is suitable only if its crossing pattern and all other protocol constraints are also acceptable.
Diagnostics, Alternatives, and Limitations
A trustworthy calibration requires checks on both the simulation and the analysis. Inspect the following before relying on the reported error rate:
- Null behavior: confirm that treatment assignment has no effect in the generating mechanism and that any stratification or randomization restrictions are represented.
- Analysis match: verify that each simulated trial uses the same weight function, statistic, direction of testing, variance estimate, and look definitions as the intended analysis.
- Data-process match: examine event and censoring generation, accrual, ties, and administrative follow-up. A mismatch can change the dependence between statistics at different looks.
- Crossing implementation: test the code on small simulated data sets and confirm that a trial is counted once if it crosses at any look, not once per crossing.
- Monte Carlo precision: use enough replicates for the desired accuracy. If few or no crossings occur, the estimated rate alone is not evidence that the true rate is zero.
There are two useful alternatives or complements. A multivariate-normal calculation approximates the joint distribution of the statistics at the planned looks and can be efficient when its covariance model is adequate. Alpha-spending methods specify how much type I error may have accumulated by each look, often with boundaries derived from a canonical joint distribution. Null simulation can check those boundaries under a realistic event and censoring model, or calibrate boundaries when the normal approximation is not sufficiently accurate. It is more computationally intensive and only as credible as its data-generating assumptions.
| Approach | Strength | Main caution |
|---|---|---|
| Null simulation | Can reproduce the planned event process and monitoring rules directly | Requires a credible null model and enough replicates |
| Multivariate-normal calibration | Fast and convenient for a specified joint distribution | May not represent finite-sample or detailed event-process behavior adequately |
| Alpha-spending calibration | Provides a structured way to allocate cumulative error across looks | Its validity depends on the assumptions and implementation used to obtain boundaries |
Simulation does not prove that the trial’s actual type I error equals the estimate. It evaluates the procedure under the chosen null model. Misspecified censoring, a changed weight rule, altered look timing, or incorrect handling of ties can make the simulated result irrelevant to the implemented design. Moreover, calibration under \(H_0\) does not establish power under alternatives; that is a separate design question.
Common Pitfall
A frequent error is to set each look’s nominal significance level to the trial-wide target, such as 0.025, and then conclude that the overall error is 0.025. The events of crossing at different looks overlap and are dependent; the probability of their union is not generally the individual-look probability. The correction is to simulate the full path of statistics and count each null trial once if it crosses at least one boundary. Another error is to report an empirical rate such as 0.02487 as exact. It is an estimate with Monte Carlo error, conditional on the null model and implementation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a null simulation of 50,000 trials, 1,240 trials cross at least one boundary. What is the estimated sequential type I error?
- Why should a trial that crosses at both the second and third looks count only once in the final crossing count?
- What must change in the simulation if the protocol defines looks by reaching specified event counts rather than by calendar dates?
- A simulation estimate is close to the target, but the censoring distribution used in the simulation does not match the planned trial. What can be concluded?
Solutions. (1) The estimate is \(1{,}240/50{,}000=0.0248\), or 2.48%. (2) The estimand is the probability of at least one crossing, a union of crossing events; counting a trial more than once would estimate the expected number of crossings per trial instead. (3) Each simulated replicate must determine its looks by the event-count rule, including generating events and censoring until those counts are reached or the protocol’s stopping limit applies. (4) The estimate describes the simulated, misspecified null model and cannot establish calibration for the planned trial; the censoring and monitoring process should be represented appropriately and the calibration rerun.