Error Events in Sequential Multi-Arm Testing

Why the Timing of a False Rejection Matters

In a multi-arm sequential trial, several treatment comparisons may be examined at more than one stage. A trial can stop early, drop arms, or carry some comparisons forward. These features make it useful to describe not only whether any true null hypothesis is rejected, but also when the first such rejection occurs. The first quantity is the familywise error event; the second gives a stage-specific description of that event.

These are related but distinct summaries. The event of any false rejection is a single trial-level event: it occurs if at least one true null is rejected at any stage. The first false rejection time records the earliest stage at which that happens. It does not count how many true nulls are rejected, and a false rejection at a later stage does not change the time of the first one.

Stagewise Rejection Indicators

Suppose a trial has \(J\) treatment-arm hypotheses, indexed by \(j=1,\ldots,J\), and at most \(K\) stages, indexed by \(k=1,\ldots,K\). Let \(H_j\) denote the null hypothesis for arm \(j\). For a fixed parameter configuration, define the set of true nulls as \(T=\{j:H_j\text{ is true}\}\). Whether a null is true is determined by the underlying parameter, not by the observed trial results.

Definition: Let \(D_{jk}=1\) if the testing procedure formally rejects \(H_j\) for the first time at stage \(k\), and let \(D_{jk}=0\) otherwise. An arm that has already been rejected, dropped, or is not eligible for testing at a stage cannot receive a new first-rejection indicator at that stage. The indicators therefore record formal decisions made by the procedure, not merely promising interim results or selection for further study.

The definition accommodates data-dependent selection: the procedure determines which arms are eligible for a formal test at each stage, and \(D_{jk}\) records its actual rejection decision. If no rejection is made for arm \(j\), then \(D_{jk}=0\) for every stage \(k\). For each arm, at most one of its stagewise indicators can equal 1 under this first-rejection convention.

Write \(R_j=1\) if \(H_j\) is rejected at some stage and \(R_j=0\) otherwise. Then \(R_j=\sum_{k=1}^{K}D_{jk}\), and \(R_j\) is either 0 or 1. These definitions do not assume independence between arms or between stages.

R_j=\sum_{k=1}^{K}D_{jk}, \qquad R_j\in\{0,1\}.

The Any-False-Rejection Event and the First-Error Time

A false rejection occurs when the procedure rejects a true null. The familywise error event is therefore the event that at least one arm in \(T\) has a rejection indicator equal to 1 at some stage. Let \(F\) denote that event. Its probability under the parameter configuration is the familywise error probability for that configuration.

F = \left\{\sum_{j\in T}\sum_{k=1}^{K}D_{jk}\geq 1\right\} = \left\{\text{at least one true }H_j\text{ is rejected at some stage}\right\}.

The first false rejection time, denoted by \(\tau\), is the earliest stage at which at least one true null is rejected. If no true null is rejected, set \(\tau=\infty\). A trial with two true nulls rejected at the same earliest stage still has one first-error time; the set of arms involved can contain more than one member.

\tau= \begin{cases} \min\left\{k:\displaystyle\sum_{j\in T}D_{jk}\geq 1\right\}, &\text{if a true null is rejected},\\[6pt] \infty,&\text{otherwise.} \end{cases}

For each stage \(k\), the event \(\{\tau=k\}\) means that at least one true null is rejected at stage \(k\), and none was rejected at any earlier stage. These first-error events are disjoint: a trial cannot have its first false rejection at two different stages. Their union is exactly \(F\). Consequently, adding their probabilities gives the probability of any false rejection. This partition is a bookkeeping identity; it does not require independence or a particular testing boundary.

F=\bigcup_{k=1}^{K}\{\tau=k\}, \qquad P(F)=\sum_{k=1}^{K}P(\tau=k).

Worked Examples

Worked Example: Identifying the First Error in a Three-Arm Trial

Consider a hypothetical two-stage trial with three arms. The null hypotheses for arms 1 and 3 are true, so \(T=\{1,3\}\); the null for arm 2 is false. Suppose the observed stagewise rejection indicators are:

\begin{array}{c|ccc} & D_{1k}&D_{2k}&D_{3k}\\ \hline k=1&0&0&0\\ k=2&1&1&0 \end{array}

At stage 1, neither true-null arm is rejected: \(D_{11}=0\) and \(D_{31}=0\). Thus no false rejection occurs at that stage. At stage 2, arm 1 is rejected and its null is true, because \(D_{12}=1\) and \(1\in T\). The first false rejection therefore occurs at stage 2, so \(\tau=2\) and the trial is in \(F\). Arm 2 is also rejected at stage 2, but its null is false, so that rejection is not a false rejection.

The example shows why rejection status must be interpreted together with null truth. A rejection indicator alone does not establish that an error occurred; the arm must also belong to the true-null set.

Worked Example: Calculating Familywise Error From Trial Paths

Suppose a hypothetical procedure has the following mutually exclusive possible trial paths under a specified parameter configuration. The probabilities are assumed to be the procedure’s joint path probabilities; no independence assumption is being made. There are two true-null arms.

Path	Probability	True-null rejections	First false rejection
A	0.50	None	None
B	0.15	One at stage 1	Stage 1
C	0.20	One at stage 2, none at stage 1	Stage 2
D	0.15	None; only a false null is rejected	None

The path probabilities sum to \(0.50+0.15+0.20+0.15=1.00\). Paths B and C, and only those paths, contain at least one false rejection. Hence

$$ P(F)=0.15+0.20=0.35. $$

The first-error-time probabilities are \(P(\tau=1)=0.15\) and \(P(\tau=2)=0.20\); the probability of no false rejection is \(P(\tau=\infty)=0.50+0.15=0.65\). Checking the partition gives \(0.15+0.20=0.35=P(F)\). The \(0.15\) probability for path D contributes to no false rejection because rejecting a false null is not a familywise error.

Worked Example: Avoiding Double-Counting Across Arms

Suppose a hypothetical sequential trial has two true-null arms. Let \(A\) be the event that the first arm is falsely rejected at some stage, and \(B\) the corresponding event for the second arm. Assume the joint probabilities under the configuration are \(P(A)=0.12\), \(P(B)=0.10\), and \(P(A\cap B)=0.04\). The familywise error event is \(A\cup B\), so inclusion-exclusion gives

P(F)=P(A\cup B) =P(A)+P(B)-P(A\cap B) =0.12+0.10-0.04 =0.18.

Adding the two armwise probabilities without adjustment would give \(0.22\), which counts the \(0.04\) probability of rejecting both true nulls twice. The first false rejection still occurs somewhere on every path in \(A\cup B\), so \(P(\tau<\infty)=0.18\). If both arms are first rejected at the same stage on some paths, those paths contribute to that stage’s first-error event once, even though two false rejections occur there.

Why the Partition Works

For each possible stage \(k\), define \(F_k\) to be the event that at least one true null is rejected at stage \(k\), and let \(F_{<k}\) be the event that at least one true null was rejected at a stage before \(k\). Then the event that the first false rejection occurs at stage \(k\) is \(F_k\cap F_{<k}^{\,c}\). The superscript \(c\) denotes the complement of an event.

If a trial has any false rejection, there is a smallest stage index at which it occurs, because the set of stages is finite and ordered. That trial belongs to exactly one event \(F_k\cap F_{<k}^{\,c}\). Conversely, every trial in one of these events has at least one false rejection. Thus the events partition \(F\), which justifies summing their probabilities:

F=\bigcup_{k=1}^{K}\left(F_k\cap F_{<k}^{\,c}\right), \qquad \left(F_k\cap F_{<k}^{\,c}\right) \cap \left(F_\ell\cap F_{<\ell}^{\,c}\right)=\varnothing \quad(k\ne \ell).

This argument concerns events and their ordering only. It does not derive a boundary, provide a method for controlling error, or say that stagewise events are independent. In particular, selection and continuation rules can make the stagewise rejection indicators dependent in complicated ways without changing the partition identity.

Interpretation, Diagnostics, and Common Pitfalls

The familywise error event answers a trial-level question: did at least one true null get rejected anywhere in the procedure? The first-error time adds the stage at which that event first became true. Reporting \(P(\tau=k)\) across stages can help describe when false rejections arise, while \(P(F)\) summarizes their overall probability. A first-error-time distribution is not a count of all false rejections: if several true nulls are rejected on one trial, that trial still has only one value of \(\tau\).

Check null truth separately from rejection. A rejection is false only if its null is true under the parameter configuration being considered.
Track the full sequence. A true-null rejection at a later stage cannot be called the first false rejection if another true null was rejected earlier.
Do not double-count simultaneous errors. Several true nulls rejected at the same earliest stage still contribute to one first-error event for that stage.
Include every tested stage. Restricting the event to the final analysis misses false rejections declared earlier.
Keep selection distinct from rejection. Retaining an arm for further study is not itself a false rejection unless the procedure formally rejects its null at that point.

A common mistake is to add the probabilities of armwise false-rejection events and call the result the familywise error probability. That sum can count trials more than once when multiple true nulls are rejected. The correction is to calculate the probability of the union of armwise events, accounting for overlap, or to partition the union by the stage of the first false rejection. Likewise, the first-error stage must be assigned from the observed sequence, not merely from the stage at which the largest number of errors occurred.

These definitions are useful whether a trial uses a fixed arm set or allows arms to be dropped and selected along the way. Their limitation is that they describe the error event, not whether a particular procedure controls its probability. That assessment requires evaluating the procedure under the relevant parameter configurations and accounting for its complete decision rule.

Check Your Understanding

Answer each question using the event definitions above. Brief solutions follow the questions.

In a two-stage trial, arms 1 and 2 have true nulls. At stage 1, only arm 3 is rejected. At stage 2, arm 2 is rejected. What are \(F\) and \(\tau\) for this trial?
Can two true null hypotheses be rejected while the trial has only one first false rejection time? Explain.
For two true-null arms, suppose \(P(A)=0.08\), \(P(B)=0.11\), and \(P(A\cap B)=0.03\), where \(A\) and \(B\) denote false rejection of the respective null at any stage. Find the familywise error probability.
Why is it generally incorrect to obtain the probability of any false rejection by summing stage-specific probabilities of a false rejection without defining “first”?

Solutions. (1) Arm 3 has a false null, so its rejection at stage 1 is not a false rejection. Arm 2 has a true null and is rejected at stage 2; therefore \(F\) occurs and \(\tau=2\). (2) Yes. If both are rejected at the same earliest stage, that stage is the single value of \(\tau\); if one is rejected later, the earlier rejection determines \(\tau\). (3) By inclusion-exclusion, \(P(F)=P(A\cup B)=0.08+0.11-0.03=0.16\). (4) Trials with false rejections at more than one stage can be counted repeatedly in an unqualified sum. The disjoint first-error events \(\{\tau=k\}\) avoid that overlap, and their probabilities sum to \(P(F)\).