Diagnosing Error Inflation by Null Simulation

Why Simulate Repeated Testing Under the Null?

A test conducted once at significance level \(\alpha\) has a nominal probability \(\alpha\) of rejecting \(H_0\) when \(H_0\) is true, provided its assumptions hold. When the same trial is tested at several scheduled looks, the probability of at least one false rejection can differ from \(\alpha\). Null simulation gives a direct way to estimate that probability while preserving the dependence among the tests, if the simulation represents the monitoring process correctly.

The goal here is diagnostic: simulate trial outcomes when the trial-wide null hypothesis is true, apply the planned tests, and count how often any test rejects. The result estimates the probability of any rejection under that null and that particular testing plan. It is not the probability that the null hypothesis is true, nor does it show that every observed rejection in a real trial is false.

Definition: For one simulated trial, let \(R_k\) equal 1 if the test at look \(k\) rejects \(H_0\), and 0 otherwise. The trial has an any-rejection event if at least one look rejects: \(A=1\) when \(\max_k R_k=1\), and \(A=0\) otherwise. With \(B\) independently simulated null trials, the estimated null rejection probability is \(\hat{q}=B^{-1}\sum_{b=1}^{B} A_b\).

The per-look significance level is the threshold used at an individual look. The simulation estimate \(\hat{q}\) instead concerns the whole schedule: whether at least one rejection occurs across its looks. These quantities answer different questions. The earlier material on dependence among cumulative tests is essential when constructing the simulated outcomes: the tests must have the same dependence as the planned analysis.

Specify the Null Model Before Simulating

A simulation is only as useful as the probability model it represents. Before generating outcomes, specify the trial-wide null, the number and timing of the looks, the test statistic at each look, the rejection threshold, and the rule for deciding whether a look rejects. For a two-sided test at per-look level \(\alpha=0.05\), for example, a standard-normal statistic rejects when its absolute value exceeds approximately 1.96. A one-sided upper-tail test at that level rejects when the statistic exceeds approximately 1.645. The direction and threshold must match the planned tests.

For the examples below, the null model uses independent standard-normal increments \(Y_1,\ldots,Y_K\). At look \(k\), the standardized cumulative statistic is \(Z_k=(Y_1+\cdots+Y_k)/\sqrt{k}\). Under this model, each \(Z_k\) has a standard normal distribution under \(H_0\), but the statistics across looks are dependent because they share earlier increments. This is a simplified model of accumulating information; real trial simulations should use a data-generating model that matches the outcome, design, and analysis as closely as needed.

Y_i \sim N(0,1), \qquad Z_k=\frac{\sum_{i=1}^{k}Y_i}{\sqrt{k}}, \qquad R_k=I(Z_k>c)

Here \(c\) is the one-sided rejection threshold and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The null model is appropriate for these examples if the increments are independent and standard normal, the statistics are formed as stated, and the same prespecified looks and threshold are used in every simulated trial. The interpretation depends on these assumptions; simulation does not make a misspecified model correct.

What the Simulation Estimates

Each replicate produces one binary result \(A_b\): 1 if any look rejects and 0 otherwise. If the true probability of an any-rejection event under the specified null model is \(q\), then \(A_b\) is Bernoulli with probability \(q\). Because the simulated trials are independent, the sample proportion \(\hat{q}\) estimates \(q\), with approximate Monte Carlo standard error

SE_{\mathrm{MC}}=\sqrt{\frac{\hat{q}(1-\hat{q})}{B}}.

This standard error measures the random variation from using a finite number of simulated trials. It does not measure uncertainty about whether the chosen null model is appropriate. A rough 95% Monte Carlo interval is \(\hat{q}\pm 1.96SE_{\mathrm{MC}}\), when the simulation count is large enough for a normal approximation and the estimate is not extremely close to 0 or 1. It describes numerical uncertainty in estimating \(q\) under the simulation model; it is not a confidence interval for a treatment effect.

Worked Example: Checking a Single Look as a Baseline

Suppose a hypothetical null simulation applies one upper-tail standard-normal test at level \(\alpha=0.05\) in each of \(B=20{,}000\) independent replicates. It records 1,023 rejections. Estimate the null rejection probability and assess its Monte Carlo uncertainty.

The estimated rejection probability is

\hat{q}=\frac{1{,}023}{20{,}000}=0.05115.

Its Monte Carlo standard error is

SE_{\mathrm{MC}} =\sqrt{\frac{0.05115(1-0.05115)}{20{,}000}} =\sqrt{\frac{0.0485336775}{20{,}000}} \approx 0.001558.

The approximate 95% Monte Carlo interval is \(0.05115\pm1.96(0.001558)\), or about \([0.04810,0.05420]\). The nominal 0.05 level is inside this interval. The small difference between 0.05115 and 0.05 is therefore quite compatible with ordinary simulation variation; it does not by itself indicate a faulty test. This one-look check is a useful baseline before investigating repeated looks.

Simulating Cumulative Looks

For repeated testing, one replicate must represent a whole hypothetical trial, not a separately generated test result at each look. Generate the null data or increments for that trial, calculate every planned test statistic from those shared outcomes, apply the rejection rule at each look, and record whether any test rejects. Repeat this process independently \(B\) times. This preserves the dependence induced by cumulative data.

Fix the analysis plan.
Specify the null model, looks, statistic at each look, sidedness, threshold, and any-rejection event before generating simulations.

Generate one null trial.
Generate outcomes under \(H_0\), retaining shared observations or increments across looks as the actual design requires.

Apply every planned test.
Compute the look-specific statistics and compare each with its prespecified rejection threshold.

Record one trial-level indicator.
Set \(A_b=1\) if any test rejects and \(A_b=0\) otherwise. Count indicators, not individual look-level rejections.

Summarize and check.
Calculate \(\hat{q}\) and \(SE_{\mathrm{MC}}\), then examine whether the model and implementation match the intended design.

Worked Example: Estimating Any-Rejection Probability at Two Looks

Consider a hypothetical trial with two cumulative looks. Under the null model, generate independent \(Y_1,Y_2\sim N(0,1)\), and calculate \(Z_1=Y_1\) and \(Z_2=(Y_1+Y_2)/\sqrt{2}\). At each look, use a one-sided upper-tail test with per-look level 0.05 and threshold \(c=1.645\). In \(B=50{,}000\) simulated trials, 4,280 have at least one statistic above the threshold.

The simulated any-rejection estimate is

\hat{q}=\frac{4{,}280}{50{,}000}=0.08560.

The Monte Carlo standard error is

SE_{\mathrm{MC}} =\sqrt{\frac{0.08560(1-0.08560)}{50{,}000}} =\sqrt{\frac{0.07827264}{50{,}000}} \approx 0.001251.

The approximate 95% Monte Carlo interval is \(0.08560\pm1.96(0.001251)\), or \([0.08315,0.08805]\). Under this model and testing plan, the estimated probability of at least one null rejection is about 8.56%, above the 5% per-look level. That comparison is the point: the per-look level does not describe the probability of any rejection across both looks. The interval reflects simulation error only; it does not account for uncertainty in the model or design assumptions.

Justification and a Model-Diagnostic Example

The estimate and its standard error follow directly from the trial-level indicators. Since \(A_b\) is 1 with probability \(q\) and 0 otherwise, \(E(A_b)=q\) and \(\operatorname{Var}(A_b)=q(1-q)\). Independence across the \(B\) simulated trials gives

E(\hat{q})=q,\qquad \operatorname{Var}(\hat{q})= \operatorname{Var}\left(\frac{1}{B}\sum_{b=1}^{B}A_b\right) =\frac{q(1-q)}{B}.

Thus \(\hat{q}\) is unbiased for the null rejection probability under the specified model, and replacing the unknown \(q\) by \(\hat{q}\) gives the stated estimated Monte Carlo standard error. This justification assumes independent simulated trials. Reusing a simulated trial as if it were several independent replicates, or generating dependent replicates without accounting for that dependence, invalidates this standard-error calculation.

Worked Example: Detecting an Incorrect Independence Assumption

Suppose a hypothetical three-look simulation uses the same cumulative-normal model: generate three independent standard-normal increments and calculate \(Z_1=Y_1\), \(Z_2=(Y_1+Y_2)/\sqrt{2}\), and \(Z_3=(Y_1+Y_2+Y_3)/\sqrt{3}\). Apply the one-sided 0.05 threshold \(c=1.645\) at each look. Of \(100{,}000\) simulated trials, 10,640 have at least one rejection.

The estimate is \(\hat{q}=10{,}640/100{,}000=0.1064\). Its estimated Monte Carlo standard error is

SE_{\mathrm{MC}} =\sqrt{\frac{0.1064(1-0.1064)}{100{,}000}} =\sqrt{\frac{0.09507904}{100{,}000}} \approx 0.0009751.

The approximate 95% Monte Carlo interval is \(0.1064\pm1.96(0.0009751)\), or \([0.1045,0.1083]\). As a diagnostic comparison, if the three tests were instead independent and each had rejection probability 0.05, the probability of at least one rejection would be \(1-(1-0.05)^3=0.1426\). That value is not the right benchmark for the cumulative model because its tests share increments. The simulated estimate is compatible with the cumulative model, but the comparison alone does not prove that the model is correct; the code and assumptions still require checking.

One useful implementation check is to inspect a few simulated trial paths by hand: confirm that the later cumulative statistics include earlier increments and that the recorded indicator is 1 exactly when at least one threshold is crossed. Another check is to run the one-look version of the simulation. If it does not estimate the nominal per-look level within plausible Monte Carlo variation, investigate the threshold, tail direction, null generator, and coding before interpreting a repeated-look result.

What Can Go Wrong, and What Simulation Can Tell You

Simulation is particularly helpful when the repeated tests are dependent or when a simple analytic calculation is inconvenient. It also makes the event being estimated explicit. However, it is not a substitute for careful design specification. The simulated result applies to the model and analysis implemented, not automatically to every trial that might be described as having repeated tests.

Simulating independent tests by accident. Generating a fresh, unrelated test statistic at each look removes the dependence created by cumulative data. Use the same observations or increments across looks when the design requires it.
Counting look-level rejections rather than trials. A trial with two rejections should contribute one any-rejection indicator, not two. The target is a proportion of simulated trials.
Using the wrong threshold. A one-sided cutoff does not match a two-sided test, and a threshold for a different per-look \(\alpha\) changes the target probability. Check the rejection rule against the planned analysis.
Using too few replicates. A noisy estimate can differ from its target by chance. Report \(B\) and the Monte Carlo standard error rather than treating the simulated proportion as exact.
Confusing numerical precision with model validity. A very large \(B\) can make Monte Carlo error small, but it cannot repair an incorrect null distribution, look schedule, or test statistic.

An analytic calculation is preferable when the joint distribution and rejection event are tractable, because it may provide an exact or highly accurate benchmark without Monte Carlo noise. Null simulation is often more flexible for complicated data-generating models, but introduces finite-simulation error and requires validated code. The two approaches can complement one another: where an analytic benchmark is available, use it to test the simulation implementation before relying on more complex simulations.

Key takeaway: A null simulation estimates the probability of any rejection only for the null model, testing schedule, dependence structure, and rejection rule that it actually simulates. The per-look level is a reference for each individual test, not the estimated probability for the full schedule.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions below.

A simulation produces 600 any-rejection events in 10,000 independent null trials. What is \(\hat{q}\)?
For the estimate in Question 1, calculate the approximate Monte Carlo standard error.
Why should cumulative statistics at different looks generally not be generated independently when the trial uses accumulating data?
A two-sided test at level 0.05 is simulated using the one-sided upper-tail threshold 1.645. Identify the problem.

Solutions. (1) \(\hat{q}=600/10{,}000=0.0600\), or 6.00%. (2) \(SE_{\mathrm{MC}}=\sqrt{0.0600(0.9400)/10{,}000}=\sqrt{0.00000564}\approx0.002375\). (3) The looks share earlier observations or increments, so their statistics are dependent; generating them independently changes the null rejection probability being estimated. (4) The threshold is for a one-sided upper-tail test, not a two-sided test. A standard-normal two-sided test at 0.05 uses approximately \(\lvert Z\rvert>1.96\), so the simulated rejection rule does not match the stated test.