Power Under Mixed Treatment-Effect Configurations

Why the True-Effect Configuration Matters

A multi-arm trial does not have just one possible state of nature. Some treatments may have a true beneficial effect while others have none. Power therefore depends not only on the design and the testing procedure, but also on which treatment effects are truly present. The vector of true effects across the prespecified comparisons is called a treatment-effect configuration.

This matters because the probability of rejecting a null hypothesis differs according to whether that null is true. For a comparison whose effect is truly positive, the rejection probability is its pairwise power. For a comparison whose effect is zero, the rejection probability is a type I error probability. When several comparisons are combined into an “any” or “all” event, both kinds of probabilities can enter the result.

Definition: Suppose there are $J$ prespecified treatment comparisons, with true effects $\theta_1,\ldots,\theta_J$, where a positive value denotes benefit. For a one-sided test of $H_{0j}:\theta_j\leq 0$ against $H_{aj}:\theta_j>0$, an arm is active in the configuration if $\theta_j>0$, and inactive if $\theta_j=0$. Write $A=\{j:\theta_j>0\}$ for the active comparisons. Let $R_j=1$ indicate rejection of $H_{0j}$, and $R_j=0$ otherwise. The pairwise rejection probability is $\pi_j=P(R_j=1)$ at the specified configuration; it is power for an active comparison and a type I error probability when that comparison is inactive.

This definition uses “inactive” to mean an exactly zero effect for clarity. A design may instead define inactivity using a clinically relevant threshold, but that threshold and the corresponding hypotheses must be specified before applying these calculations. A treatment’s status here is determined by its true parameter, not by whether its observed test happens to be significant.

Mixed Configurations and Success Events

Let the prespecified success event be rejection of a comparison’s null hypothesis. The disjunctive event across all $J$ comparisons is at least one rejection, $\bigcup_{j=1}^{J}\{R_j=1\}$; the conjunctive event is rejection of every comparison, $\bigcap_{j=1}^{J}\{R_j=1\}$. Their probabilities depend on the full configuration. In particular, the all-comparisons disjunctive event can occur because an inactive-arm null is rejected. That is a rejection, but it is not evidence that a truly active treatment was detected.

It is useful to distinguish the probability of any rejection from the probability of detecting at least one active treatment. The latter is $P(\bigcup_{j\in A}\{R_j=1\})$, when $A$ is nonempty. Similarly, the all-comparisons conjunctive event requires rejecting inactive-arm nulls as well as active-arm nulls. Thus it may have a very different interpretation from an all-success criterion defined only over a selected set of treatments. Always state which event a reported probability refers to.

Key result under independence: If the rejection indicators $R_1,\ldots,R_J$ are mutually independent at the configuration, then $$ P\left(\bigcup_{j=1}^{J}\{R_j=1\}\right) =1-\prod_{j=1}^{J}(1-\pi_j), \qquad P\left(\bigcap_{j=1}^{J}\{R_j=1\}\right) =\prod_{j=1}^{J}\pi_j. $$ For a nonempty active set $A$, the probability of rejecting at least one active comparison is $1-\prod_{j\in A}(1-\pi_j)$ under the same independence assumption.

Worked Example: Labeling a Mixed Configuration

Suppose four hypothetical treatments are compared with control, and the true beneficial effects, in milligrams per decilitre of a prespecified outcome reduction, are $\theta_1=4$, $\theta_2=0$, $\theta_3=2$, and $\theta_4=0$. The tests are one-sided tests for a positive benefit.

The configuration is $(4,0,2,0)$ mg/dL. Comparisons 1 and 3 are active because their true effects are positive, so $A=\{1,3\}$. Comparisons 2 and 4 are inactive because their true effects are exactly zero. No observed data are needed to make this classification: it describes the assumed parameter values under which operating characteristics are evaluated.

If the pairwise rejection probabilities under this configuration were, for example, $(0.82,0.02,0.70,0.02)$, the first and third numbers would be pairwise powers, while the second and fourth would be null rejection probabilities. Calling all four numbers “power” would obscure the mixed configuration.

Calculating Any-Rejection and All-Rejection Probabilities

The following calculation assumes independent rejection indicators and uses stipulated marginal probabilities from a hypothetical design calculation. Independence must be justified by the design and test statistics; it is not guaranteed merely because different treatments are being compared. The example isolates how a mixed configuration enters the calculation.

Worked Example: One Active and Two Inactive Comparisons

Consider three prespecified comparisons. Comparison 1 is active, with pairwise rejection probability $\pi_1=0.808366$. Comparisons 2 and 3 are inactive, each with null rejection probability $\pi_2=\pi_3=0.0167$. Suppose their rejection indicators are independent.

The probability of at least one rejection among all three comparisons is the disjunctive probability:

1-(1-0.808366)(1-0.0167)^2 =1-(0.191634)(0.9833)^2 =1-(0.191634)(0.96687889) =0.81471313079374\approx 0.814713.

Thus the all-comparison “any rejection” probability is about 81.4713%. The probability of rejecting the active comparison is 80.8366%. The difference is

100(0.81471313079374-0.808366) =0.634713079374\text{ percentage points} \approx 0.6347\text{ percentage points}.

That difference consists of outcomes in which the active comparison is not rejected but at least one inactive comparison is rejected. It is not extra probability of detecting a truly active treatment. In this example, the probability of detecting at least one active treatment is simply 0.808366, because there is only one active comparison.

For the all-three-comparisons conjunctive event, all three rejections are required:

0.808366(0.0167)(0.0167) =0.808366(0.00027889) =0.000225445\approx 0.0002254.

The all-comparisons conjunctive probability is therefore about 0.02254%. Its small value is driven by the requirement to reject two null hypotheses that are in fact true. This is a probability for the stated all-three event, not a probability that all three treatments are effective.

The formulas follow directly from the event definitions. Under independence, the probability that no comparison rejects is the product of the probabilities of no rejection in each comparison, $\prod_j(1-\pi_j)$. Taking the complement gives the probability of at least one rejection. For the all-rejection event, independence makes the probability of the intersection the product of the marginal rejection probabilities. Restricting the union to active comparisons gives the corresponding probability of detecting at least one active treatment.

Worked Example: Two Active Arms and One Inactive Arm

Now suppose comparisons 1 and 2 are active, with pairwise powers $\pi_1=0.80$ and $\pi_2=0.65$, while comparison 3 is inactive, with null rejection probability $\pi_3=0.02$. Assume independent rejection indicators.

The probability of detecting at least one active treatment is

$$ 1-(1-0.80)(1-0.65) =1-(0.20)(0.35) =0.93. $$

So the active-treatment detection probability is 93%. If the disjunctive event instead includes all three comparisons, its probability is

$$ 1-(1-0.80)(1-0.65)(1-0.02) =1-(0.20)(0.35)(0.98) =0.9314. $$

The additional 0.14 percentage points arise when neither active comparison rejects but the inactive comparison does. For the all-three-rejection event, the calculation is

$$ (0.80)(0.65)(0.02)=0.0104. $$

This is 1.04%: both active comparisons and the inactive comparison must reject. Each result answers a different event question, despite being calculated from the same three marginal probabilities.

Assumptions, Diagnostics, and Alternatives

Before using the product formulas, check that the probabilities correspond to the same prespecified testing procedure and configuration, and that the rejection indicators really are independent. In a multi-arm trial, comparisons may share a control group, which can induce dependence. A familywise procedure may also make rejection indicators dependent even if the underlying test statistics are independent. Marginal powers alone do not justify multiplying probabilities.

When independence is not defensible, use a joint model for the rejection indicators or test statistics, an appropriate inclusion–exclusion calculation, or simulation of the complete prespecified procedure under the configuration. For three comparisons, inclusion–exclusion expresses the any-rejection probability as the sum of the three marginal rejection probabilities, minus the three pairwise intersection probabilities, plus the triple intersection probability. The needed intersection probabilities must be supplied by a valid joint model; they cannot be recovered from marginal powers alone.

Practical diagnostics include checking that the active set agrees with the stated true effects, confirming that inactive comparisons are assigned null rejection probabilities rather than active-arm power, and verifying the dependence assumption against the trial’s allocation and test procedure. If the event is defined as “any success,” check whether it includes inactive comparisons. If the event is “all success,” check whether the intended requirement truly includes every prespecified comparison. A change in the success event changes the estimand, not just the arithmetic.

The independent-products calculation is attractive because it is transparent and easy to audit. Its limitation is that it can give an incorrect joint probability under dependence. Joint-model or simulation approaches can accommodate dependence and more complex procedures, but require a correctly specified joint distribution or a faithful implementation of the testing algorithm. None of these calculations establishes which treatments are effective from trial outcomes; they describe operating probabilities under assumed true effects.

Common Mistakes

Using “power” for every marginal rejection probability. For an inactive comparison with $\theta_j=0$, its rejection probability is a type I error probability, not power to detect a true effect.
Equating any rejection with detection of an active treatment. An inactive comparison can reject by chance. If the scientific event is detection of an active treatment, restrict the event to the active set.
Multiplying marginal probabilities without checking independence. Products give the stated joint probabilities only under independence. Shared controls or a multiple-testing procedure can invalidate that assumption.
Treating an observed nonsignificant result as proof of inactivity. The active/inactive labels describe true parameters in the configuration; a test outcome does not reveal those labels with certainty.

Key takeaway: State the true-effect configuration and the exact success event before calculating multi-arm power. In a mixed configuration, active-arm power and inactive-arm type I error probabilities can both contribute to an any-rejection or all-rejection probability.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For three comparisons with true effects $(0,3,0)$, which comparison is active, and which pairwise rejection probabilities are power rather than null rejection probabilities?
Two independent comparisons have rejection probabilities 0.70 and 0.05. Calculate the probability of at least one rejection.
In a configuration with one active and one inactive comparison, why can the probability of any rejection exceed the probability of detecting the active treatment?
Three independent comparisons have rejection probabilities 0.60, 0.50, and 0.10. Calculate the probability that all three reject.
What additional information is needed to calculate a joint any-rejection probability when comparisons are dependent?

Solutions. (1) Comparison 2 is active because its true effect is positive. Its rejection probability is pairwise power; the probabilities for comparisons 1 and 3 are null rejection probabilities because their true effects are zero. (2) By the complement rule, $1-(1-0.70)(1-0.05)=1-(0.30)(0.95)=0.715$, or 71.5%. (3) The inactive comparison can reject even when the active comparison does not, so the any-rejection event includes outcomes that are not detections of the active treatment. (4) Under independence, $0.60(0.50)(0.10)=0.030$, or 3.0%. (5) The joint distribution, or enough intersection probabilities to apply inclusion–exclusion, is needed; marginal rejection probabilities alone do not determine the result.