Why Count the Looks?
A test performed once at significance level \(\alpha\) has a specified false-rejection probability under its null hypothesis. Repeating the same test creates more opportunities to reject when the null is true. The relevant quantity for a monitoring plan is often the probability of at least one false rejection over all planned looks, rather than the false-rejection probability at any one look.
This tutorial isolates the role of the number of looks. We first calculate an independent-test benchmark, then use null simulation to estimate the same kind of probability. The benchmark makes the increasing pattern precise, while simulation shows how to examine a specified repeated-testing process. Neither calculation implies that every design with the same number of looks has the same error rate.
An Independent-Look Benchmark
Suppose each look is a valid level-\(\alpha\) test under \(H_0\), and suppose the rejection events at different looks are independent under \(H_0\). Each look then has probability \(1-\alpha\) of not rejecting, and independence gives probability \((1-\alpha)^K\) of no rejection at every look. Taking the complement gives the any-look probability:
The assumptions matter: the formula requires both a level-\(\alpha\) test at each look and independence of the rejection events under the null. It does not apply just because there are \(K\) looks. For example, cumulative interim analyses generally use overlapping data, so their test statistics are dependent. The independent calculation is a useful reference point, not a default model for every monitoring plan.
Worked Example: Comparing Several Independent-Look Counts
Suppose each look is a two-sided test at \(\alpha=0.05\), and assume for this calculation that the rejection events are independent under \(H_0\). Find the probability of at least one false rejection for \(K=1, 2, 5,\) and \(10\).
The model is that each look has false-rejection probability \(0.05\), and the outcomes of the tests are independent. Thus the probability of no false rejection across \(K\) looks is \(0.95^K\).
For one look, \(1-0.95^1=0.0500\). For two looks, \(1-0.95^2=1-0.9025=0.0975\). For five looks, \(1-0.95^5=1-0.7738=0.2262\). For ten looks, \(1-0.95^{10}=1-0.5987=0.4013\).
Under these assumptions, the chance of at least one false rejection rises from 5.00% at one look to about 40.1% at ten looks. These are probabilities under \(H_0\), not probabilities that \(H_0\) is true after observing data.
The change from \(K\) to \(K+1\) independent looks can be derived directly. The new opportunity matters only on paths with no rejection in the first \(K\) looks. The probability of such a path is \((1-\alpha)^K\), and the chance of rejection at the added independent look is \(\alpha\). Therefore:
For \(0<\alpha<1\), this increment is positive, so adding a look increases the independent-look probability. The increment gets smaller as \(K\) grows because fewer paths remain unrejected at all earlier looks. This does not mean the total probability stops increasing; it continues to rise toward 1 as \(K\) becomes large.
Worked Example: Choosing an Independent Per-Look Level
Suppose the target any-look false-rejection probability is 0.05 across \(K=10\) independent tests. What per-look significance level \(\alpha_{\text{look}}\) would give that overall probability?
Under the independent-test model, set the chance of at least one rejection equal to 0.05:
Rearranging gives \((1-\alpha_{\text{look}})^{10}=0.95\), so \(1-\alpha_{\text{look}}=0.95^{1/10}\) and \(\alpha_{\text{look}}=1-0.95^{1/10}\approx0.005116\). Thus each independent test would use a per-look level of about 0.00512, or 0.512%.
This value is conditional on independence and equal per-look levels. It is not a universal level for ten looks. Under a different dependence structure, a different calibration may be needed to attain the same any-look probability.
Estimating the Probability by Null Simulation
A null simulation estimates the frequency of the any-look event by repeatedly generating a complete sequence of test outcomes under \(H_0\). Each simulated replicate represents one hypothetical trial or monitoring sequence. For replicate \(b\), record an indicator \(I_b\) equal to 1 if any look rejects and 0 otherwise. With \(B\) replicates, the estimated probability is the proportion of sequences with at least one rejection:
The simulation must represent the null model and the dependence among the looks in the design being studied. One can simulate independent tests when independence is the assumption of interest. For a cumulative analysis, one must instead generate data and calculate all looks from the same simulated path, so the overlap in information is retained. The earlier material on null simulation explains this general diagnostic; here the emphasis is how the estimated rate behaves as the number of looks changes.
Worked Example: Simulating Independent Looks
Consider a hypothetical null simulation with \(B=100{,}000\) replicate sequences. At each look, a two-sided test has exact null rejection probability 0.05, and tests are generated independently. For each replicate, count a false rejection if one or more of its \(K\) tests rejects.
One simple way to generate each test outcome is to draw an independent \(U_{b,k}\sim\operatorname{Uniform}(0,1)\) for every replicate \(b\) and look \(k\), and reject when \(U_{b,k}\leq0.05\). This is an exact representation of the rejection indicators under the stated independent-test model. Suppose a hypothetical run gives these numbers of replicates with at least one rejection:
| Looks \(K\) | Replicates with a rejection | Estimated any-look probability | Independent-model value |
|---|---|---|---|
| 1 | 4,994 | 0.04994 | 0.05000 |
| 2 | 9,752 | 0.09752 | 0.09750 |
| 5 | 22,637 | 0.22637 | 0.22622 |
| 10 | 40,008 | 0.40008 | 0.40126 |
| 20 | 64,191 | 0.64191 | 0.64151 |
For example, at \(K=10\), the estimate is \(40{,}008/100{,}000=0.40008\). The exact model value is \(1-0.95^{10}=0.40126\). Their difference is small relative to ordinary simulation fluctuation. The estimated rate is a Monte Carlo estimate, not a new exact probability.
For an estimated probability \(\hat q\), its approximate Monte Carlo standard error is \(\sqrt{\hat q(1-\hat q)/B}\). At \(K=10\), this is \(\sqrt{0.40008(0.59992)/100{,}000}\approx0.00155\), or about 0.155 percentage points. Small discrepancies between the simulated and exact values are expected.
Why the Number of Looks Is Not Enough
The independent formula gives a clean increasing curve, but dependence can change its size. With strongly overlapping analyses, a rejection at one look may make rejection at nearby looks more likely. The tests then provide less than \(K\) independent opportunities. At the opposite extreme, identical tests repeat the same decision: no matter how many times the identical decision is recorded, the any-look probability remains the single-test level.
Worked Example: Perfectly Dependent Looks
Suppose every look uses exactly the same null test statistic \(Z\sim N(0,1)\), and rejects \(H_0\) when \(|Z|>1.96\). Each look has approximately a 0.05 two-sided null rejection probability, but the decisions are perfectly dependent because the same \(Z\) is reused.
For any \(K\geq1\), the union of the rejection events is just the single event \(\{|Z|>1.96\}\): if it occurs, all looks reject; if it does not, none reject. Therefore \(P(\bigcup_{k=1}^{K}R_k\mid H_0)\approx0.05\) for every \(K\), unlike the independent-test values.
As a simulation check, generate \(B=100{,}000\) independent standard-normal values, and use each simulated value at every look in its replicate. If 4,987 values exceed 1.96 in absolute value, then the estimated any-look probability is \(4{,}987/100{,}000=0.04987\) for \(K=1,2,5,\) or \(10\). The unchanged estimate is correct for this deliberately extreme dependence model. It is not a realistic model for all monitoring plans; it demonstrates why the look count alone cannot determine the answer.
When looks are added while preserving all earlier tests and the same rejection rule, the any-look event can only stay the same or expand. Thus its probability cannot decrease. In simulation, however, separately generated estimates for different values of \(K\) may appear slightly out of order because of Monte Carlo error. Generating nested sequences—using the first \(K\) looks of the same simulated paths for each \(K\)—makes comparisons clearer and helps distinguish random simulation noise from a change in the modeled design.
Diagnostics, Alternatives, and Limitations
A useful comparison starts with three quantities: the single-look level, the number of looks, and the joint behavior of the tests under \(H_0\). The independent formula is exact only when its assumptions hold. The union bound from the earlier material gives an upper bound of \(K\alpha\) for \(K\) tests, capped at 1; it does not require independence but can be conservative. Direct null simulation can represent a specified dependence structure, but it is only as trustworthy as the data-generating model and implementation.
Several checks help diagnose a calculation or simulation that is going wrong:
- Check the event being counted. Count a replicate once if any look rejects. Counting every rejection instead estimates the average number of rejections, a different quantity.
- Check the null calibration at each look. If a nominal 0.05 test rejects far more or less than 5% in a one-look null simulation, investigate the test implementation or the simulated null model before interpreting a multi-look result.
- Check that dependence matches the design. Simulating independent test outcomes for analyses based on overlapping data can give the wrong any-look probability.
- Check Monte Carlo precision. With few replicates, estimates can be noisy. Report \(B\) and, when useful, the approximate standard error rather than treating the estimate as exact.
- Check monotonicity in a nested comparison. If each larger-look design retains every earlier look with the same rule, a true decrease in the any-look probability signals either simulation noise, a coding error, or a change in the design being compared.
The main alternative to estimating error by simulation is a calculation based on a justified probability model. When independence is credible, \(1-(1-\alpha)^K\) is direct and exact. When independence is not justified, the union bound is a simple conservative benchmark, while simulation can estimate the probability under an explicitly specified joint null model. These approaches answer different questions; none supplies a universal inflation factor based on \(K\) alone.
The practical lesson is not simply that more looks always produce a particular amount of inflation. Under a fixed rule, adding opportunities cannot reduce the any-look chance, but the size of the increase depends on how the tests relate to one another. A sound report identifies the per-look level, the number of looks, the null model and dependence assumptions, and whether the overall error rate was calculated or estimated.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Under independent tests at \(\alpha=0.05\), calculate the probability of at least one false rejection across three looks.
- For independent tests at \(\alpha=0.05\), what is the probability increment from adding the sixth look after five looks?
- A null simulation has 2,000 replicate sequences, of which 186 have at least one rejection. What is the estimated any-look probability?
- Why can two procedures with the same \(K\) and per-look level have different any-look false-rejection probabilities?
Solutions. (1) \(1-0.95^3=1-0.857375=0.142625\), about 0.1426 or 14.3%. (2) The increment is \(\alpha(1-\alpha)^5=0.05(0.95^5)\approx0.03869\), about 3.87 percentage points. (3) The estimate is \(186/2{,}000=0.0930\), or 9.30%. (4) The dependence among the rejection events can differ; independence gives \(1-(1-\alpha)^K\), while other joint null distributions can yield a different probability of at least one rejection.