Why Repeated-Testing Thresholds Need Calibration

Why One-Look Control Does Not Set Trial-Wise Control

At a single analysis, a test conducted at significance level $\alpha$ has a null rejection probability no greater than $\alpha$. That statement concerns one decision. In a trial with multiple planned analyses, the relevant false-positive event is different: at least one analysis rejects the null when the null is true. Repeating the same single-test threshold does not automatically keep that across-the-trial probability at $\alpha$.

The distinction matters because the trial can produce more than one opportunity for a false rejection. If the trial stops as soon as a boundary is crossed, the trial-wise Type I error is the probability of crossing at any scheduled look—not the probability of rejection at the final look considered in isolation. The earlier material on repeated-testing pitfalls and trial-wise error motivates this distinction; here we quantify why the single-test threshold alone is not a guarantee.

Definition: For a planned set of $K$ looks, let $R_k$ denote the event that look $k$ rejects $H_0$. The trial-wise Type I error is $$ P\left(\bigcup_{k=1}^{K} R_k \,\middle|\, H_0\right), $$ the probability, under the null hypothesis, of at least one rejection across the trial. A test at look $k$ has null rejection probability $p_k=P(R_k\mid H_0)$; level-$\alpha$ control means $p_k\leq\alpha$, not necessarily $p_k=\alpha$.

This definition presumes that the looks and their rejection rules have been specified for the trial. It also presumes that each individual test is valid under $H_0$, so its null rejection probability is at most its stated level. Neither validity at each look nor a fixed schedule implies that the union of rejection events has probability at most $\alpha$. Dependence among the events matters, and it must not be silently replaced by independence.

A Direct Calculation Under Independence

First consider a deliberately simple case: the tests at different looks are independent under $H_0$. If the null rejection probability at look $k$ is $p_k$, the probability of no rejection at that look is $1-p_k$. Independence means the probability of no rejection at every look is the product of those probabilities. Taking the complement gives the exact trial-wise error:

P\left(\bigcup_{k=1}^{K}R_k\mid H_0\right) =1-\prod_{k=1}^{K}(1-p_k).

If each test has exact null rejection probability $\alpha$, so $p_k=\alpha$ for every $k$, this becomes

1-(1-\alpha)^K.

That expression is not the general answer whenever tests are merely level $\alpha$. A valid test can be conservative, with $p_k<\alpha$. Under independence, the exact answer then uses the actual $p_k$'s. Since $p_k\leq\alpha$, it is at most $1-(1-\alpha)^K$, but it need not equal that quantity.

Worked Example: Four Independent Exact Tests at the Same Threshold

Suppose a hypothetical trial has four scheduled looks. For illustration, assume the four tests are independent under $H_0$ and each is an exact level-$0.05$ test. Thus $p_k=0.05$ at every look. Find the trial-wise Type I error.

The probability of no false rejection at one look is $1-0.05=0.95$. By the stated independence assumption, the probability of no false rejection at all four looks is

$$ 0.95^4=0.81450625. $$

Therefore, the probability of at least one false rejection is

1-0.81450625=0.18549375\approx 0.1855.

So the trial-wise Type I error is about $18.55\%$, not $5\%$, under these assumptions. This is an illustration of the risk from reusing the threshold, not a claim that interim statistics in an ordinary trial are independent. The independence assumption is essential to this exact calculation.

Conservative Tests and the Actual Rejection Probabilities

The calculation above uses exact tests. A threshold labelled $0.05$ usually guarantees that the null rejection probability is at most $0.05$; it does not promise that the probability equals $0.05$. For example, discreteness or other features of a test can make its rejection probability smaller. Repeated-testing calculations must respect that distinction.

Worked Example: Unequal, Conservative Per-Look Error Probabilities

Consider three hypothetical independent tests, each using a threshold intended to provide level-$0.05$ control. Suppose their actual null rejection probabilities are $p_1=0.04$, $p_2=0.05$, and $p_3=0.03$. Each test is valid because its probability is no greater than $0.05$. Calculate the trial-wise error.

The chance of no rejection at the three looks is, respectively, $0.96$, $0.95$, and $0.97$. Under the stated independence assumption,

P(\text{no rejection}\mid H_0) =0.96(0.95)(0.97) =0.88464.

Thus,

P(\text{at least one rejection}\mid H_0) =1-0.88464 =0.11536.

The exact trial-wise error is $11.536\%$. It is not $1-(1-0.05)^3=0.142625$, because not every test has null rejection probability exactly $0.05$. The latter value is an upper bound for these independent tests, not the exact answer. It is also no larger than the simpler union bound $0.04+0.05+0.03=0.12$.

What Can Be Said Without Independence?

In group sequential monitoring, analyses often use data that overlap across looks. Their rejection events therefore need not be independent. Marginal information alone—the values of $p_k$—does not determine the exact probability of at least one rejection. Without a joint model, the general bound that is always available is the union bound:

P\left(\bigcup_{k=1}^{K}R_k\mid H_0\right) \leq \sum_{k=1}^{K} P(R_k\mid H_0) =\sum_{k=1}^{K}p_k \leq K\alpha.

The first inequality follows because an outcome in the union belongs to at least one rejection event; adding the probabilities of the individual events can count some outcomes more than once, so it cannot undercount the union. This bound does not require independence. It may be loose, especially when rejection events overlap substantially. In particular, if $K\alpha>1$, it is still a mathematically valid bound but may provide little useful information.

Worked Example: Same Marginal Error, Different Dependence

Take four tests, each with $P(R_k\mid H_0)=0.05$. Compare two possible dependence structures. In the first, all four rejection events are identical: whenever one test rejects, all four do. In the second, the four events are independent. These are hypothetical probability structures used to show why marginal probabilities do not settle the trial-wise error.

In the identical-events case, the union is just one event, so

P\left(\bigcup_{k=1}^{4}R_k\mid H_0\right)=0.05.

In the independent case, the chance of no rejection at all four tests is $0.95^4=0.81450625$. Hence the union probability is

1-0.95^4=0.18549375\approx 0.1855.

Both structures have the same $5\%$ null rejection probability at each look, yet their trial-wise errors differ. Thus, knowing that every look uses a valid $0.05$ test is not enough to calculate the exact trial-wise error. Dependence must be accounted for to obtain an exact value; without it, the union bound gives $0.05+0.05+0.05+0.05=0.20$.

Why Calibration Is Needed

The numerical examples establish the central point: repeating a single-test threshold is not a general procedure for controlling the trial-wise Type I error at that same level. Under one particular structure—independent exact tests—the error increases according to $1-(1-\alpha)^K$. Under other structures it can be smaller or larger than that value, subject to general bounds. For ordinary repeated analyses of accumulating data, an independence calculation should not be used as if it described the trial unless independence has actually been justified.

A monitoring plan therefore needs a rejection rule whose error control is defined for the whole sequence of looks. One simple conservative approach for a fixed number of tests is to assign each look a level no greater than $\alpha/K$. By the union bound, the probability of any rejection is then at most $K(\alpha/K)=\alpha$, regardless of dependence. This can be conservative, and it is not the only possible approach. Group sequential methods instead account for the planned sequence and its joint behavior when setting thresholds. The detailed calculation of those boundaries requires more than reusing a single-test cutoff.

The practical comparison is straightforward. An unadjusted per-look threshold is easy to state, but does not by itself provide the desired trial-wise guarantee. A simple union-bound adjustment gives a guarantee under arbitrary dependence but may sacrifice power. A method that uses the joint distribution of the planned statistics can target the trial-wise error more closely, provided its model and assumptions are appropriate. The right comparison is not between a “correct” and “incorrect” single test; it is between procedures with different guarantees for the entire monitoring plan.

Diagnostics begin with the design rather than a calculation. Check how many looks can trigger rejection, whether the schedule is fixed, what null rejection probability each rule actually has, and whether the chosen error statement applies to one look or to the full trial. If an exact trial-wise number is reported, check that the dependence assumptions or joint model supporting it are stated. A simulation can help assess a proposed design, but it does not turn an unjustified independence assumption into a valid calculation.

Common mistake: “Each look is a $5\%$ test, so the trial has a $5\%$ Type I error.” This confuses per-look control with the probability of rejection at any look.
Correction: Define the trial-wise event as the union of the look-specific rejection events. Calculate its probability under a stated joint model, or use a valid bound such as the union bound.
Common mistake: “The trial-wise error is $1-(1-\alpha)^K$ whenever every test is level $\alpha$.” This treats both exact marginal size and independence as guaranteed.
Correction: Under independence, the exact expression is $1-\prod_k(1-p_k)$, where $p_k$ is the actual null rejection probability. It reduces to $1-(1-\alpha)^K$ only when every $p_k=\alpha$.

Key takeaway: Single-look validity means $P(R_k\mid H_0)\leq\alpha$ at each look. Trial-wise control concerns $P(\bigcup_k R_k\mid H_0)$. The latter depends on the number of looks and the joint behavior of their rejection events, so it requires a rule calibrated for the whole monitoring plan.

Check Your Understanding

Answer each question using the distinction between per-look and trial-wise error.

Three independent exact tests each have null rejection probability $0.04$. What is the exact probability of at least one rejection?
Two independent valid tests have actual null rejection probabilities $0.02$ and $0.04$. Is $1-(1-0.05)^2$ their exact trial-wise error? Calculate the exact value.
Four tests each have null rejection probability at most $0.025$. Give a trial-wise upper bound that does not require independence.
Why can two monitoring plans with the same per-look rejection probabilities have different trial-wise errors?
What additional condition is needed to use $1-(1-\alpha)^K$ as the exact trial-wise error?

Solutions. (1) By independence, $1-(1-0.04)^3=1-0.96^3=1-0.884736=0.115264$, or about $11.53\%$. (2) No. The exact value is $1-(1-0.02)(1-0.04)=1-(0.98)(0.96)=1-0.9408=0.0592$, or $5.92\%$. (3) The union bound gives at most $4(0.025)=0.10$, or $10\%$. (4) The dependence among rejection events changes the probability of their union even when each event has the same marginal probability. (5) The tests must be independent under $H_0$, and every test must have exact null rejection probability $\alpha$, rather than merely probability at most $\alpha$.