Defining Success Events in Multi-Arm Trials

From a Trial Objective to a Success Event

A multi-arm trial may compare several treatments with a common control, yet the trial’s objective is not always simply “get a significant result.” Investigators might want evidence that at least one treatment works, evidence that every treatment in a specified group works, or evidence for a more focused prespecified objective. These statements correspond to different events, even when they use the same treatment comparisons and the same multiple-testing procedure. Since power is the probability of success under a specified treatment-effect configuration, the success event must be stated before power can be defined.

Consider \(m\) treatment comparisons, indexed by \(i=1,\ldots,m\). For a one-sided superiority question, let \(\Delta_i\) denote the treatment effect for comparison \(i\), with positive values favoring treatment over control. The hypotheses are \(H_{0i}:\Delta_i\leq 0\) against \(H_{ai}:\Delta_i>0\). A different clinical question may require a different null and alternative, but the event-based definitions below apply in the same way.

Definition: For each comparison \(i\), let \(R_i=1\) if the prespecified testing procedure rejects \(H_{0i}\), and \(R_i=0\) otherwise. The vector \(R=(R_1,\ldots,R_m)\) records the trial’s rejection decisions. A success event is a prespecified set of possible rejection vectors that count as success for the trial objective. At a fixed parameter configuration \(\theta\), the power for that objective is the probability that the rejection vector falls in the success set: \(\pi(\theta)=P_\theta\{R\text{ is in the success set}\}\).

Here \(\theta\) denotes the collection of true treatment effects and any other model parameters needed to describe the sampling distribution. The probability is calculated under that fixed configuration, with the planned design and decision procedure in place. It is not the probability that the hypotheses are true, nor the probability that a particular observed conclusion is correct.

Writing the Event Precisely

Every possible rejection vector belongs to \(\{0,1\}^m\). A decision rule divides these vectors into success and non-success outcomes. The event can be written in words, with logical symbols, or as a set of vectors. For example, “reject at least one of comparisons 1, 2, or 3” is the event \(R_1=1\) or \(R_2=1\) or \(R_3=1\). “Reject every one of the three” is the event that all three indicators equal 1. These are distinct criteria: a trial can satisfy the first while failing the second.

It is important to say which hypotheses the objective refers to. Let \(E(\theta)\) be the set of comparisons whose alternatives are true under configuration \(\theta\). If the objective is to identify at least one treatment that truly benefits patients, the relevant event is that at least one comparison in \(E(\theta)\) is rejected. Merely rejecting any null is not equivalent: that event could be satisfied by rejecting a null that is actually true. If the objective is to establish benefit for every comparison in a prespecified target set \(S\), the event is that every comparison in \(S\) is rejected. The set \(S\) must come from the objective, not from whichever comparisons happen to look favorable in the data.

\begin{aligned} \text{At least one target succeeds:}\quad &\bigcup_{i\in S}\{R_i=1\},\\ \text{Every target succeeds:}\quad &\bigcap_{i\in S}\{R_i=1\}. \end{aligned}

These expressions define events; they do not yet calculate their probabilities. If \(A_S\) denotes the event that matches the trial objective, its power at \(\theta\) is \(P_\theta(A_S)\). A short justification follows directly from the indicator of the event. The indicator \(1\{R\in A_S\}\) equals 1 on success and 0 otherwise, so its expectation is the probability of success:

\pi_S(\theta) =E_\theta[1\{R\in A_S\}] =P_\theta(A_S).

This definition requires a specified sampling model (or design-based probability distribution), a fixed parameter configuration, a fully specified testing procedure, and an unambiguous success event. It does not require the comparisons to be independent. The dependence among their statistics affects the probability of a joint event, but it does not change what the event means.

Worked Example: Success Means at Least One Truly Beneficial Treatment Is Rejected

Hypothetical trial. Three experimental treatments are compared with a common control using one-sided superiority tests. A prespecified familywise-error procedure is used at \(\alpha=0.05\). Suppose the adjusted p-values from one hypothetical data set are 0.012 for treatment 1, 0.041 for treatment 2, and 0.083 for treatment 3. For illustration, suppose the fixed configuration being considered has \(\Delta_1>0\), \(\Delta_2\leq0\), and \(\Delta_3>0\), so \(E(\theta)=\{1,3\}\).

The rejection rule compares each adjusted p-value with 0.05. Since \(0.012\leq0.05\), \(0.041\leq0.05\), and \(0.083>0.05\), the observed rejection vector is \(R=(1,1,0)\). The trial objective is to reject at least one null corresponding to a truly beneficial treatment, so the success event is \(\{R_1=1\}\cup\{R_3=1\}\). In this observed outcome, \(R_1=1\), so the event occurs. Treatment 2 is also rejected, but under the stated configuration its null is true; that rejection does not itself count as evidence of a true benefit.

The power for this objective is \(P_\theta(R_1=1\text{ or }R_3=1)\). The observed vector shows whether this particular trial outcome meets the criterion; it does not by itself give the probability. Power concerns the chance of meeting the criterion across repetitions under the specified configuration.

Worked Example: Success Means Rejecting Every Specified Comparison

Hypothetical objective. A trial is designed to support a claim that all three prespecified treatments are superior to control. The success event is therefore \(R_1=1\) and \(R_2=1\) and \(R_3=1\). Consider a fixed configuration in which all three treatment effects are positive. Suppose the prespecified procedure gives adjusted p-values 0.018, 0.032, and 0.047 in a hypothetical data set, with \(\alpha=0.05\).

Each comparison is rejected because \(0.018\leq0.05\), \(0.032\leq0.05\), and \(0.047\leq0.05\). Thus \(R=(1,1,1)\), and the all-comparisons event occurs. The power for the objective is \(P_\theta(R_1=1,R_2=1,R_3=1)\), evaluated under the stated configuration. It is not the product of three separate rejection probabilities unless an appropriate independence assumption is justified; no such assumption is needed to define the event.

If the same procedure instead produced \(R=(1,1,0)\), the trial would have rejected two nulls but would not meet the specified all-three success criterion. Calling that outcome “success” because two comparisons were significant would change the objective after seeing the results.

Worked Example: A Prespecified Combined Objective

Hypothetical objective. Suppose a trial’s scientific claim requires evidence that treatment 1 is superior and evidence that at least one of treatments 2 or 3 is superior. The specified success event is \(R_1=1\) and \((R_2=1\text{ or }R_3=1)\). This rule makes the primary requirement explicit while allowing either of two additional comparisons to satisfy the second requirement.

For a hypothetical outcome \(R=(1,0,1)\), substitute the indicators into the rule: \(1=1\) and \((0=1\text{ or }1=1)\). The first condition holds, and the second holds because \(R_3=1\). The success event therefore occurs. For \(R=(0,1,1)\), the second condition holds but the first does not, so the event does not occur. The corresponding power is \(P_\theta\{R_1=1\text{ and }(R_2=1\text{ or }R_3=1)\}\).

This example illustrates why a written objective should be translated into a logical rule before any power calculation. A statement such as “the trial should demonstrate the primary treatment and another option” is not precise enough until the required comparisons and the meaning of “another option” are identified.

Keep the Success Criterion Separate From the Testing Procedure

The testing procedure determines the rejection vector \(R\). It may use a familywise-error procedure chosen for the prespecified family, such as one of the procedures considered in the earlier material on selecting an FWER procedure for a multi-arm trial. The success criterion then says which values of \(R\) count as success for the trial’s objective. These are related but separate choices: changing the testing procedure can change the probability of each rejection vector, while changing the objective changes which vectors are counted as success.

Familywise error rate (FWER) and power answer different questions. FWER is the probability of rejecting at least one true null in the family; it is an error-rate guarantee under configurations containing true nulls. Power is the probability of the prespecified success event under a stated configuration. If the success event were simply “reject any null,” then under the global null its probability would be an error probability, not evidence of efficacy. That is why the scientific target and the truth configuration matter when defining power.

Other summaries may be useful but cannot replace a clear event definition. Pairwise power describes a particular comparison, while a simultaneous confidence procedure or adjusted p-values characterize evidence across a family. Neither one automatically specifies what counts as overall trial success. The event must still match the actual claim the trial is meant to support.

Diagnostics, Pitfalls, and Limits

Before interpreting a reported power value, check that its event can be written as a rule on the rejection indicators. Then check that the target set and the assumed truth configuration are explicit. A useful diagnostic is to test the rule against example vectors: would \(R=(1,0,0)\) count? Would \(R=(0,1,1)\)? If the answer is unclear, the objective has not yet been translated precisely enough.

Confusing any rejection with efficacy success. Rejecting a true null is possible. For an efficacy objective, identify the target alternatives and state whether success requires rejection of one or all of them.
Changing the criterion after seeing results. Defining the target set using the most favorable observed comparisons makes the reported success rule data-dependent. Prespecify the family and criterion.
Equating a rejection with a guaranteed benefit. A rejection is a decision made by a statistical procedure, not certainty that the treatment effect is positive or clinically important.
Assuming that marginal results settle a joint objective. Knowing that each comparison has a certain chance of rejection does not, by itself, identify the probability that a multi-comparison event occurs. The joint event and its distribution matter.

The definitions are general, but a power value is only as useful as its assumptions and objective. It depends on the specified treatment effects, the sampling model, the planned allocation and analysis, and the exact decision rule. A single power value can also conceal that different configurations may produce different probabilities of success. State the configuration alongside the event rather than presenting power as a property of the procedure alone.

Key takeaway: First write the trial’s scientific success rule as an event involving the treatment-level rejection indicators. Then define power as the probability of that event under a clearly stated parameter configuration. “At least one rejection” and “every specified comparison rejected” are different objectives, not interchangeable descriptions of power.

Check Your Understanding

For each question, identify the event or explain what must be specified to define it. Brief solutions follow the questions.

Three comparisons are in the family, and success means rejecting comparison 1 or comparison 3. Write the success event.
All four treatments are targets, and success requires evidence for every target. What rejection vector represents success?
Under a configuration where comparisons 1 and 2 have true alternatives but comparison 3 has a true null, does rejecting comparison 3 alone meet an objective of identifying at least one truly beneficial treatment?
What does \(P_\theta(R_1=1,R_2=1)\) represent if the stated objective requires rejecting both comparisons 1 and 2?

Solutions. (1) The event is \(\{R_1=1\}\cup\{R_3=1\}\), with no requirement on \(R_2\). (2) Success is \(R=(1,1,1,1)\), so the event is \(\bigcap_{i=1}^{4}\{R_i=1\}\). (3) No. That rejection is of a true null and does not establish rejection of either of the two true alternatives. (4) It is the probability, under configuration \(\theta\), that both specified null hypotheses are rejected; it is the power for that all-comparisons success criterion.