Estimating Pairwise, Disjunctive, and Conjunctive Power

Why Power Has More Than One Meaning in a Multi-Arm Trial

In a multi-arm, multi-stage (MAMS) trial, the design may be intended to identify one effective treatment, identify every effective treatment, or assess the performance of each treatment separately. Those goals correspond to different probabilities. A design can have a reasonable chance of detecting each treatment individually yet a substantially smaller chance of detecting all of them in the same trial. Conversely, its chance of detecting at least one can be high even when some treatments are often missed.

Simulation estimates these probabilities by repeatedly generating a trial under a specified treatment-effect scenario and applying the complete protocol: interim analyses, arm-dropping rules, stopping rules, and final efficacy decisions. The target is the probability of a decision event under that scenario, not simply the proportion of simulated participants who respond or the probability that an interim statistic crosses a boundary.

Definition: Fix a treatment-effect scenario \(\theta\), including which treatments are truly effective, and let \(R_j=1\) if the complete MAMS decision procedure declares treatment \(j\) effective and \(R_j=0\) otherwise. The pairwise power for an effective treatment \(j\) is \(P(R_j=1\mid\theta)\). If \(A(\theta)\) is the set of truly effective treatments, the disjunctive power is the probability of declaring at least one treatment in \(A(\theta)\) effective, and the conjunctive power is the probability of declaring every treatment in \(A(\theta)\) effective.

\begin{aligned} \pi_j(\theta) &= P(R_j=1\mid\theta), \qquad j\in A(\theta),\\ \pi_{\mathrm{dis}}(\theta) &= P\left(\bigcup_{j\in A(\theta)}\{R_j=1\}\mid\theta\right),\\ \pi_{\mathrm{con}}(\theta) &= P\left(\bigcap_{j\in A(\theta)}\{R_j=1\}\mid\theta\right). \end{aligned}

The set \(A(\theta)\) matters. Under a scenario in which only treatment A is effective, disjunctive and conjunctive power are both the probability of rejecting A’s null hypothesis. Under a scenario in which A and B are effective, disjunctive power counts a rejection of A, B, or both; conjunctive power requires both rejections. A rejection of a truly ineffective treatment is not a successful detection for either definition. Such rejections are relevant to error control, but they do not turn a missed effective treatment into power.

Estimating the Probabilities From Replicates

Suppose \(M\) simulated trials are generated independently under the same scenario and run through the same protocol. For replicate \(r\), record \(R_{rj}=1\) if treatment \(j\) is declared effective and \(0\) otherwise. Then count each event of interest across the \(M\) replicates. The same simulated trial can contribute to several pairwise rejection counts, and the joint count must be formed from the treatment decisions in that trial.

\begin{aligned} \widehat{\pi}_j &= \frac{1}{M}\sum_{r=1}^{M} R_{rj},\\ \widehat{\pi}_{\mathrm{dis}} &= \frac{1}{M}\sum_{r=1}^{M} I\left(\sum_{j\in A(\theta)}R_{rj}\geq 1\right),\\ \widehat{\pi}_{\mathrm{con}} &= \frac{1}{M}\sum_{r=1}^{M} I\left(\sum_{j\in A(\theta)}R_{rj}=|A(\theta)|\right). \end{aligned}

Here \(I(\cdot)\) is 1 when its condition is true and 0 otherwise, and \(|A(\theta)|\) is the number of effective treatments in the scenario. These estimates are proportions of simulated trials satisfying the corresponding events. For example, if 735 of 1,000 replicates reject a particular effective treatment, its estimated pairwise power is \(735/1{,}000=0.735\), or 73.5%.

Justification: For an event with probability \(\pi\), let \(I_r\) equal 1 when replicate \(r\) satisfies the event and 0 otherwise. Under independent, identically generated replicates, \(E(I_r)=\pi\). Therefore the average indicator has expectation \(E(M^{-1}\sum_{r=1}^M I_r)=M^{-1}\sum_{r=1}^M E(I_r)=\pi\). The replicate proportion estimates the event probability directly; the event must first be defined correctly.

The independence assumption is about the simulated trial replicates used for this estimate. Outcomes for different arms within one replicate do not need to be independent. In particular, treatments sharing a control can have dependent test statistics and dependent rejection decisions. Computing all indicators from each replicate’s complete set of decisions preserves that dependence in the joint estimates.

Worked Example: Two Effective Treatments and Their Joint Power

Suppose a hypothetical MAMS design is simulated for 1,000 independent trials under a scenario where treatments A and B are both effective. Applying the full decision rules gives these counts: both A and B are declared effective in 410 trials; only A is declared effective in 210; only B in 160; and neither in 220. The counts sum to \(410+210+160+220=1{,}000\).

A is rejected in the “both” and “only A” groups, so its pairwise power estimate is \( (410+210)/1{,}000=0.620\). B’s estimate is \( (410+160)/1{,}000=0.570\).

At least one effective treatment is rejected in every group except “neither.” Thus disjunctive power is estimated as \( (410+210+160)/1{,}000=0.780\). Both effective treatments are rejected only in the “both” group, so conjunctive power is \(410/1{,}000=0.410\).

These estimates answer different questions: in this scenario, A is detected in 62.0% of simulated trials, B in 57.0%, at least one in 78.0%, and both in 41.0%. The pairwise estimates alone do not determine the joint estimates; the overlap between the rejection events is needed.

Dependence Determines the Joint Probabilities

For two effective treatments, the probability of rejecting at least one is found by adding the two pairwise probabilities and subtracting the probability of rejecting both. Subtraction is necessary because trials in which both are rejected appear in each pairwise probability and would otherwise be counted twice. The probability of rejecting all effective treatments is the intersection probability itself.

\begin{aligned} P(R_A=1\ \text{or}\ R_B=1\mid\theta) &=P(R_A=1\mid\theta)+P(R_B=1\mid\theta)\\ &\quad-P(R_A=1,\ R_B=1\mid\theta),\\ P(R_A=1\ \text{and}\ R_B=1\mid\theta) &=P(R_A=1,\ R_B=1\mid\theta). \end{aligned}

In the example, the corresponding calculation for disjunctive power is \(0.620+0.570-0.410=0.780\). Multiplying the pairwise estimates instead would give \(0.620\times0.570=0.3534\), but that would estimate the joint probability only under an independence assumption. Shared-control data, common interim decisions, and common stopping rules can all make rejection indicators dependent. The replicate-level joint count requires no independence assumption between treatments.

Worked Example: A Partial-Null Scenario

Now consider a hypothetical scenario with three treatments: A and B are effective, while C is not. In 2,000 simulated trials, A is declared effective in 1,030 and B in 950. Both A and B are declared effective in 760; neither is declared effective in 780. The remaining trials contain exactly one effective-treatment rejection: 270 contain A but not B, and 190 contain B but not A. These four categories sum to 2,000.

The pairwise power estimates for the effective treatments are \( (760+270)/2{,}000=0.515\) for A and \( (760+190)/2{,}000=0.475\) for B. Disjunctive power is \( (2{,}000-780)/2{,}000=0.610\), because every trial except those with neither effective treatment rejected contributes to the event. Conjunctive power is \(760/2{,}000=0.380\).

Suppose C is also declared effective in 130 trials. This is a false rejection under the scenario, so it should be reported as an error-control outcome, not counted as a successful detection of an effective treatment. It does not change the disjunctive or conjunctive power just calculated: those events are defined using A and B, the effective-treatment set. If the scientific target instead explicitly required a decision involving any arm, that would be a different event and should be named and estimated separately.

Applying the Definitions to a Multi-Stage Decision Rule

A rejection indicator must represent the protocol’s declared outcome, not an intermediate statistic considered in isolation. A treatment may cross an interim efficacy boundary and be declared effective early; that should count as \(R_j=1\) if the protocol defines it as an efficacy declaration. A treatment dropped for futility and never declared effective has \(R_j=0\). A treatment that continues to the final analysis is classified according to the final rule. The same logic applies whether there are two stages or more.

Worked Example: Three Effective Treatments

Consider 500 hypothetical simulated trials under a scenario where A, B, and C are all effective. The full staged protocol declares A effective in 290 trials, B in 260, and C in 240. At least one of the three is declared effective in 360 trials, and all three are declared effective in 175 trials. The estimates are therefore \(290/500=0.580\) for A, \(260/500=0.520\) for B, \(240/500=0.480\) for C, \(360/500=0.720\) for disjunctive power, and \(175/500=0.350\) for conjunctive power.

The reported joint counts are consistent with the definitions: the 175 trials in which all three are rejected must be among the 360 trials in which at least one is rejected. Also, each pairwise rejection count must be at least 175, because every “all three” trial includes a rejection of each individual treatment. The given counts satisfy both checks. A trial where an effective arm was dropped without an efficacy declaration does not count as a rejection of that arm, even if another active arm was successfully identified.

Checks, Comparisons, and Limitations

A reliable estimate depends on both the event definition and the simulated trial mechanism. The data generator must represent the chosen scenario, including the specified treatment effects and relevant shared-control dependence. Each replicate must then follow the same allocation, interim information, arm-dropping, stopping, and terminal decision rules as the intended design. The earlier material on simulation targets and protocol logic provides the framework for specifying these ingredients; for power, the additional requirement is to define the success event using the effective-treatment set.

Several simple checks can reveal implementation or reporting errors:

Every pairwise, disjunctive, and conjunctive estimate must lie between 0 and 1.
For a nonempty effective-treatment set, conjunctive power cannot exceed disjunctive power, since rejecting all effective treatments necessarily means rejecting at least one.
For each effective treatment, conjunctive power cannot exceed its pairwise power.
Disjunctive power must be at least as large as every active treatment’s pairwise power.
The total of mutually exclusive patterns of active-treatment rejections must equal the number of simulated replicates.

These checks are necessary, not sufficient: a plausible-looking estimate can still come from an incorrectly coded stopping rule or an inaccurate outcome generator. It is also important to report the scenario, the number of replicates, the decision event, and whether estimates are conditional on a particular treatment being effective. A single power value without that context can be ambiguous.

Simulation is especially useful when the MAMS decision procedure is complex and the joint distribution of the interim and final statistics is difficult to handle analytically. An analytic calculation can be more direct when its assumptions and joint distribution are tractable. Either way, marginal power and joint power remain different targets. Estimating pairwise power in separate simulations and multiplying the results is not a substitute for estimating the joint event from shared replicate-level decisions. Simulation estimates are also subject to Monte Carlo variation, so they should not be interpreted as exact probabilities.

Common pitfall: “A has pairwise power 0.70 and B has pairwise power 0.65, so the chance of detecting both is \(0.70\times0.65=0.455\).” This multiplication assumes independent rejection events, which a shared-control, multi-stage design generally does not guarantee. The correction is to count replicates in which both A and B are declared effective and divide that count by the total number of replicates.

Check Your Understanding

For each problem, use the scenario-specific definitions above. Solutions are included after the questions.

In 800 trials with two effective treatments, A is rejected in 500, B in 460, and both are rejected in 300. Calculate the estimated disjunctive and conjunctive power.
In 600 trials, three treatments are effective. All three are rejected in 180 trials, while at least one is rejected in 420. Which estimate is conjunctive power, and which is disjunctive power?
A treatment is truly ineffective but is declared effective in 35 of 1,000 simulated trials. Should those 35 rejections count toward power when two other treatments are effective? Explain.
Why can the product of two pairwise power estimates fail to estimate the probability of rejecting both treatments?

Solutions. (1) By inclusion-exclusion, disjunctive power is \((500+460-300)/800=660/800=0.825\). Conjunctive power is \(300/800=0.375\). (2) Conjunctive power is \(180/600=0.300\), and disjunctive power is \(420/600=0.700\). (3) No. The treatment is outside the effective set, so its rejection is a false rejection, not a successful detection for power. (4) The product gives the joint probability only if the two rejection events are independent. Their decisions may be dependent, so count trials in which both are rejected and divide by the total number of trials.