The First-Rejection Rule and Its Error Event

Why Specify the Stopping Rule?

In a monitoring study, results may be examined at several scheduled looks. If the procedure says to stop once a result is significant, then a significant result at look 3 is relevant only if the study reached look 3 without stopping at looks 1 or 2. The rule must therefore describe not just the test at each look, but also how the sequence of results determines whether the next test is performed.

This tutorial formalizes that rule and identifies the false-rejection event at a particular look when the global null hypothesis is true. The key distinction is between a raw rejection at look \(k\), considered by itself, and the first rejection occurring at look \(k\). These events are not generally the same.

Definition: Suppose a study has \(K\) prespecified looks, indexed \(k=1,\ldots,K\). At look \(k\), a test of \(H_0\) produces a p-value \(p_k\), and the prespecified rejection cutoff is \(\alpha_k\). The first-rejection rule stops at the first look for which \(p_k\leq\alpha_k\). If no look meets its cutoff, the study reaches look \(K\) and does not reject \(H_0\).

For the error event, assume the global null is true: the null hypothesis being tested remains true throughout the monitoring process. Also assume the schedule, the cutoffs, the test directions, and the stopping rule have been specified before the results are examined. Each look's test should be valid for its stated null under the study's data-generating assumptions. These conditions let us interpret a rejection under the global null as a false rejection. They do not require the tests at different looks to be independent.

The usual one-look condition is that, under \(H_0\), a test's p-value is valid: for any \(u\) between 0 and 1, \(P(p_k\leq u\mid H_0)\leq u\). A continuous test may have equality; a discrete test can be conservative. This is a marginal statement about a look's p-value, not a statement that the p-value remains valid after conditioning on earlier non-rejections.

Writing the First-Rejection Event

Let \(A_k\) denote the raw rejection event at look \(k\): \(A_k=\{p_k\leq\alpha_k\}\). It describes whether the result at that look crosses its cutoff, without yet accounting for whether the study would have reached that look. Define the stopping time \(\tau\) as the first index \(k\) for which \(A_k\) occurs. If no such index exists, set \(\tau=K+1\).

\tau=\min\{k:p_k\leq\alpha_k\}, \qquad \min(\varnothing)=K+1.

The event that the procedure first rejects at look \(k\) requires two things: no rejection at any earlier look, and rejection at look \(k\). Thus, for \(k=1,\ldots,K\), the first-rejection event is

R_k=\{\tau=k\} =A_k\cap\bigcap_{j=1}^{k-1}A_j^{\,c} =\{p_k\leq\alpha_k,\ p_j>\alpha_j\text{ for every }j<k\}.

At look 1 there are no earlier looks, so \(R_1=A_1\). At later looks, the earlier non-rejection conditions are essential. Under the global null, \(R_k\) is the event of a false first rejection at look \(k\). It is the trial-wise Type I error event for that stopping stage. A raw crossing \(A_k\) alone does not establish that the trial-wise event occurred: the study might already have stopped.

Key distinction: \(A_k\) means that the look-\(k\) result crosses its cutoff. \(R_k\) means that it crosses its cutoff and every earlier look failed to cross its cutoff. The first-rejection rule makes the \(R_k\) events mutually exclusive.

Justification: Why the Events Are Mutually Exclusive

The first-rejection events cannot occur at two different looks on the same monitoring path. If \(R_k\) occurs, then \(A_k\) occurs. For any later look \(l>k\), the definition of \(R_l\) requires \(A_k^c\), because look \(k\) is an earlier look than \(l\). These requirements conflict. Therefore \(R_k\cap R_l=\varnothing\) whenever \(k\ne l\).

The events also account for every possible stopping outcome. Either the first crossing occurs at one of the \(K\) looks, or no crossing occurs and \(\tau=K+1\). This follows directly from taking the first index in the finite sequence \(A_1,\ldots,A_K\) at which a crossing occurs, if any. Under the global null, each event \(R_k\) is a possible false-rejection path, while \(\{\tau=K+1\}\) is the path with no rejection. This event partition clarifies which look produced the first false rejection; calculating a combined error probability is a separate question.

Worked Examples

Worked Example: Locating the First Rejection

Consider a hypothetical study with four scheduled looks, a common cutoff of \(\alpha_k=0.05\), and the following potential sequence of p-values: \(0.21,\ 0.047,\ 0.012,\ 0.030\). Suppose the global null is true. Which event occurs under the first-rejection rule?

At look 1, \(p_1=0.21>0.05\), so \(A_1\) does not occur and the study continues. At look 2, \(p_2=0.047\leq0.05\), so \(A_2\) occurs. Because the first look did not reject, the event is

R_2=A_2\cap A_1^c =\{p_2\leq0.05,\ p_1>0.05\}.

The procedure stops at look 2, and under the stated global null this is a false first rejection at look 2. The listed values at looks 3 and 4 are not used by the procedure: those analyses would not be conducted after stopping at look 2. They are included only to illustrate that later hypothetical values do not change the first-rejection event.

Worked Example: Different Prespecified Cutoffs

A hypothetical monitoring plan has three looks with cutoffs \(0.01,\ 0.025,\) and \(0.05\), respectively. A potential sequence of p-values is \(0.014,\ 0.031,\ 0.018\). Assume the global null is true. Identify the first-rejection event.

At look 1, \(0.014>0.01\), so there is no rejection. At look 2, \(0.031>0.025\), so there is still no rejection. At look 3, \(0.018\leq0.05\), so the rule rejects for the first time at look 3:

R_3=\{p_3\leq0.05,\ p_1>0.01,\ p_2>0.025\}.

The comparison must use the cutoff assigned to each look. It would be incorrect to compare every p-value with \(0.05\): that would label the first look significant even though its prespecified cutoff is \(0.01\). Under the global null, the identified \(R_3\) path is a false first rejection at look 3.

Worked Example: Conditional Error Among Trials Reaching a Look

Suppose a hypothetical joint null distribution for two scheduled tests gives \(P(A_1)=0.04\), \(P(A_2)=0.04\), and \(P(A_1\cap A_2)=0.0208\). Find the probability of a first rejection at look 2 among trials that reach look 2 without a rejection at look 1.

The event of a first rejection at look 2 is \(R_2=A_2\cap A_1^c\). Subtract the overlap from the raw look-2 rejection probability:

P(R_2)=P(A_2)-P(A_1\cap A_2) =0.04-0.0208=0.0192.

The probability of reaching look 2 is \(P(A_1^c)=1-0.04=0.96\). Therefore the conditional trial-wise error probability among trials reaching that look is

P(A_2\mid A_1^c) =\frac{P(A_2\cap A_1^c)}{P(A_1^c)} =\frac{0.0192}{0.96}=0.0200.

Thus, in this hypothetical null distribution, 2.00% of the trials that reach look 2 reject there. The marginal raw rejection probability at look 2 is 4.00%, but it includes paths that already rejected at look 1 and would not reach look 2 under the rule. The supplied probabilities describe the joint distribution; they should not be inferred from the cutoffs alone.

Assumptions, Diagnostics, and Alternatives

A first-rejection rule is meaningful only if the data and analysis at each look match the prespecified plan. The following checks help determine whether the event definition correctly represents the monitoring process.

Check the schedule and cutoffs. Confirm that the number and timing of looks, the test at each look, and each \(\alpha_k\) were specified. If the analyst changes a cutoff or adds a look in response to results, the actual rule is not the one described by the original plan.
Check the null and test validity. Confirm that the global null applies to the scientific claim being tested and that each test's assumptions are suitable for the data available at that look. A formula for \(R_k\) does not repair a misspecified model or an invalid p-value.
Do not assume independence. Tests based on accumulating data often share observations and are dependent. The event definition remains correct under dependence, but probabilities such as \(P(A_k\mid A_1^c,\ldots,A_{k-1}^c)\) must reflect the joint distribution of the tests.
Distinguish planned and unplanned results. A result that crosses a cutoff after an unplanned analysis is not automatically covered by the prespecified event definition. Record which analyses were actually conducted and when, rather than reconstructing a rule after seeing the results.
Respect stopping in the observed data. Once a trial stops, later p-values are not observed as actual analyses. The event \(R_k\) depends only on the results through look \(k\); values imagined for later looks cannot change where the procedure first stopped.

The first-rejection rule differs from a fixed one-look test, which makes a decision at only one prespecified analysis, and from a plan that conducts several tests but does not stop at the first crossing. Under the rule taught here, crossing a cutoff determines both the rejection and the end of monitoring. The distinction matters when describing a trial's decision path, even before asking how the rule's error probabilities compare with those of another procedure.

A common mistake is to say, “The look-2 p-value was below its cutoff, so look 2 produced a trial-wise Type I error.” That reasoning ignores whether the trial reached look 2. The correction is to check every earlier look: the event is \(A_2\cap A_1^c\), not just \(A_2\). A related error is to infer the conditional probability of rejection at look 2 from its nominal cutoff alone. Marginal p-value validity does not generally imply the same validity conditional on surviving earlier looks, especially when the tests are dependent.

Key takeaway: Under the global null, the trial-wise Type I error event at look \(k\) is a first crossing: \(p_k\leq\alpha_k\) together with non-rejection at every earlier look. State the earlier-survival conditions explicitly whenever describing a false rejection under a stop-at-first-significance rule.

Check Your Understanding

Answer each question using the first-rejection rule and the event definitions above. Brief solutions follow the questions.

With three looks and cutoffs \(0.05\) at every look, a potential p-value sequence is \(0.08,\ 0.02,\ 0.01\). At which look does the rule stop, and what is the event?
Write the event \(R_4\) for four looks with cutoffs \(\alpha_1,\alpha_2,\alpha_3,\alpha_4\).
Why is \(A_3\) alone not the trial-wise error event at look 3?
If \(P(A_1)=0.10\), \(P(A_2)=0.05\), and \(P(A_1\cap A_2)=0.02\), calculate \(P(R_2)\) and \(P(A_2\mid A_1^c)\).
Does dependence between looks change the definition of \(R_k\)? What does it change?

Solutions. (1) Look 1 does not reject because \(0.08>0.05\); look 2 rejects because \(0.02\leq0.05\). The rule stops at look 2, and the event is \(R_2=A_2\cap A_1^c\). (2) \(R_4=\{p_4\leq\alpha_4,\ p_1>\alpha_1,\ p_2>\alpha_2,\ p_3>\alpha_3\}\). (3) \(A_3\) does not require that the study reached look 3; the trial-wise event also requires non-rejection at looks 1 and 2. (4) \(P(R_2)=P(A_2)-P(A_1\cap A_2)=0.05-0.02=0.03\). Also, \(P(A_1^c)=0.90\), so \(P(A_2\mid A_1^c)=0.03/0.90=0.0333\), or 3.33%. (5) No. The event is still a rejection at look \(k\) preceded by non-rejections at all earlier looks. Dependence changes the probabilities of these events and their conditional probabilities, not their definitions.