The Trial-Wise Null Hypothesis in Monitoring

Why Monitoring Requires a Trial-Wide Definition

A one-look test has one opportunity to reject $H_0$. A monitoring plan can have several planned analyses, each of which may lead to rejection. The relevant error event is therefore not just rejection at one selected analysis: it is rejection at at least one analysis in the plan. The probability of that event under the null is the trial-wise Type I error.

The introductory material on one-look testing provides a reference point: its Type I error is the probability of rejection at that single analysis when $H_0$ is true. For monitoring, the probability must describe the complete procedure. That includes the planned looks, the rule used at each look, and whether the trial stops after a rejection. It is a property of the procedure under repeated sampling, not a probability that the null hypothesis is true for a particular trial.

Definition: Suppose a trial has $K$ planned analyses. Let $R_k$ be the event that the analysis at look $k$ rejects $H_0$, under the trial's specified monitoring and stopping rules. The trial-wise Type I error at a null parameter value $\theta$ is $$ \alpha_{\mathrm{trial}}(\theta) = P_{\theta}\!\left(\bigcup_{k=1}^{K} R_k\right), \qquad \theta\in\Theta_0, $$ where $\Theta_0$ is the set of parameter values for which $H_0$ is true. Thus the event is rejection at one or more looks. If $H_0$ is simple, this probability is evaluated at its specified null value. For a composite null, the error-control requirement is usually stated as a bound on the largest such probability over $\Theta_0$.

Specify the Procedure Before Defining Its Error

The probability in the definition is meaningful only after the monitoring procedure is specified. At minimum, this requires a null and alternative hypothesis, a fixed schedule of $K$ planned looks, the data or test statistic used at each look, and a rejection rule for each look. The stopping rule also matters: in a design that stops at the first rejection, later analyses are not conducted after that rejection. The event remains “at least one rejection”; the stopping rule determines which analyses can occur.

For example, let $Z_k$ be the test statistic at look $k$, and suppose rejection at that look occurs when $Z_k$ belongs to a specified rejection set $A_k$. Then $R_k=\{Z_k\in A_k\}$. In a common one-sided setup, $A_k$ might be values above a boundary $c_k$, so that $R_k=\{Z_k\ge c_k\}$. The trial-wise error is the probability, under $H_0$, that the observed sequence of statistics enters at least one of the rejection sets. No independence assumption is required for this definition.

For a valid calculation, the probability model must match the design: it must describe the statistic at every planned look under the null, with the actual analysis schedule and stopping rule represented. If an analysis is added, omitted, or changed, the procedure whose error is being described has changed too. The definition itself does not say how to calculate the probability; that depends on the joint null distribution of the sequence of statistics and the specified rejection rules.

Worked Example: Counting Rejection Somewhere in Three Looks

Consider a hypothetical three-look monitoring plan testing a true simple null hypothesis. For this example, suppose the joint null model assigns the following probabilities to the mutually exclusive outcomes classified by the first look at which rejection occurs:

No rejection at any look: probability $0.90$.
First rejection at look 1: probability $0.04$.
First rejection at look 2: probability $0.03$.
First rejection at look 3: probability $0.03$.

These probabilities sum to $0.90+0.04+0.03+0.03=1.00$, so they form a complete partition of the possible outcomes. The event of a trial-wise Type I error is the union of the three first-rejection events. Since those events are mutually exclusive, their probabilities add:

\alpha_{\mathrm{trial}} =0.04+0.03+0.03 =0.10.

Under the stated null model, the probability that this monitoring plan rejects the true null at least once is $0.10$, or 10%. The calculation uses the supplied joint outcome probabilities; it does not assume that the looks are independent. It also shows why the “first rejection” categories are useful for describing the event: every path with a rejection belongs to exactly one category.

A Useful Decomposition: The Look of First Rejection

Let $F_k$ be the event that the first rejection occurs at look $k$. In terms of the rejection events $R_1,\ldots,R_K$,

F_k = R_k\cap\left(\bigcap_{j=1}^{k-1}R_j^c\right).

The events $F_1,\ldots,F_K$ are disjoint: a trial cannot have its first rejection at two different looks. Their union is exactly the event of at least one rejection. Therefore, by additivity of probability for disjoint events,

P_{\theta}\!\left(\bigcup_{k=1}^{K}R_k\right) = \sum_{k=1}^{K}P_{\theta}(F_k).

This identity is a decomposition of the trial-wise error, not an assumption that the tests at different looks are independent. Each probability $P_{\theta}(F_k)$ is a probability under the joint null model for the monitoring sequence, because it requires both rejection at look $k$ and no rejection at earlier looks. The decomposition is useful for stating the event precisely and organizing its calculation.

Worked Example: Two Looks With Joint Outcomes Supplied

Suppose a hypothetical trial has two looks, and under a true simple null the joint probabilities of the two rejection indicators are given directly. The first digit below records whether look 1 rejects; the second records whether look 2 rejects:

$(0,0)$: probability $0.91$.
$(1,0)$: probability $0.03$.
$(0,1)$: probability $0.04$.
$(1,1)$: probability $0.02$.

The probabilities sum to $0.91+0.03+0.04+0.02=1.00$. A trial-wise false rejection occurs in every outcome except $(0,0)$. Equivalently, the first rejection is at look 1 in outcome $(1,0)$ or $(1,1)$, and at look 2 in outcome $(0,1)$. The outcome $(1,1)$ is assigned to look 1 when classifying by first rejection. Thus

\alpha_{\mathrm{trial}} = P(F_1)+P(F_2) = (0.03+0.02)+0.04 = 0.09.

The trial-wise Type I error is $0.09$, or 9%. The joint probabilities were supplied, so no independence model was needed. In particular, the calculation is based on the probability of the complete rejection pattern, not on treating the looks as separate experiments.

Composite Null Hypotheses and Error Control

A composite null contains more than one parameter value. For example, $H_0:\theta\le 0$ is true for every $\theta$ in $\Theta_0=\{\theta:\theta\le0\}$. The chance of a false rejection can differ across these null values, so reporting a single probability without saying which value it describes can be incomplete.

A standard error-control statement for a composite null requires the trial-wise error to be no greater than the target level for every null value. In notation, the requirement is

\sup_{\theta\in\Theta_0} P_{\theta}\!\left(\bigcup_{k=1}^{K}R_k\right) \le \alpha.

The supremum is the largest trial-wise error over the null parameter space. If the null is simple, there is only one value to evaluate. If it is composite, checking a few selected values can be informative but does not by itself prove the required bound over all of $\Theta_0$. Establishing the bound requires a justified analysis of the null model across that space.

Worked Example: Reporting Error Across a Composite Null

Suppose a hypothetical one-sided trial tests $H_0:\theta\le0$. For illustration, assume a specified analysis has already provided these trial-wise rejection probabilities under three null values:

At $\theta=-1$, $P_{\theta}(\text{at least one rejection})=0.03$.
At $\theta=-0.2$, $P_{\theta}(\text{at least one rejection})=0.06$.
At $\theta=0$, $P_{\theta}(\text{at least one rejection})=0.08$.

Among these three evaluated values, the largest probability is $0.08$. If the target is $\alpha=0.05$, the reported values show that the procedure exceeds the target at $\theta=0$, since $0.08>0.05$. They do not, on their own, establish the supremum over every $\theta\le0$; other null values have not been assessed. If a justified analysis established that the maximum over the full null space is $0.08$, the procedure would fail to meet the $0.05$ error-control requirement. The conclusion must distinguish the observed finite set of evaluations from a proven maximum.

Interpretation, Comparisons, and Failure Modes

Trial-wise Type I error is a repeated-sampling probability conditional on a null parameter value. If the null is true and the complete trial procedure were repeated many times under the same model, this probability describes the long-run proportion of trials in which at least one false rejection occurs. It is not the probability that $H_0$ is true, nor the probability that a particular observed rejection is false.

A fixed single-look test and a monitoring plan define different rejection events. The single-look event concerns rejection at one analysis; the trial-wise event concerns rejection anywhere in the specified schedule. The monitoring definition is also distinct from a probability referring only to a particular endpoint analysis. These are different events, so their probabilities need not be the same.

The main diagnostic question is whether the reported event matches what the trial actually did. Check that every planned analysis is represented, that the rejection rule at each look is explicit, and that the stopping rule is reflected in the probability model. A calculation based on an incomplete schedule, or on a joint null distribution inconsistent with the design, does not describe the trial-wise error of the implemented procedure.

Mistake: “The Type I error is the chance of rejection at each look.” That describes a look-specific event, not the probability of rejection somewhere in the trial. Define the event first, then calculate its probability.
Mistake: Adding look-specific probabilities without justification. The first-rejection events are disjoint and can be added. The rejection events themselves may overlap, so their probabilities cannot generally be added as if they were disjoint.
Mistake: Assuming independence because there are multiple looks. Independence is not part of the definition. The required probability is determined by the joint null behavior of the monitoring sequence.
Mistake: Reporting one null value for a composite null without qualification. State the value at which the probability was evaluated, or state and justify a bound over the entire null parameter space.

A concise report should identify the null hypothesis, the monitoring plan, and the event being controlled. For a composite null, it should also state whether the reported probability is at a specified null value or is a bound over the null space. That precision makes clear what the error rate does—and does not—say.

Check Your Understanding

Answer each question by identifying the relevant event or probability. Solutions follow the questions.

A trial has four planned looks. Write the event whose probability defines its trial-wise Type I error.
Why can the probability of first rejection at look 1, 2, or 3 be added to obtain the probability of any rejection by look 3?
Under a simple null, a two-look procedure has joint outcome probabilities $P(0,0)=0.94$, $P(1,0)=0.02$, $P(0,1)=0.03$, and $P(1,1)=0.01$. Find the trial-wise Type I error.
For a composite null, what must be bounded to claim trial-wise error control at level $\alpha$?

Solutions. (1) If $R_k$ denotes rejection at look $k$, the event is $R_1\cup R_2\cup R_3\cup R_4$. (2) The first-rejection events are mutually exclusive and their union is the event of at least one rejection, so additivity for disjoint events applies. (3) The only outcome with no rejection is $(0,0)$; thus the probability of any rejection is $0.02+0.03+0.01=0.06$, or 6%. (4) The supremum over all null parameter values of the probability of at least one rejection must be no greater than $\alpha$.