Power Definitions After Intermediate Selection

Why power needs a precise definition after selection

In a multi-arm, multi-stage design, an arm may be dropped because of its intermediate outcome and never reach the final analysis. Consequently, the chance of a final rejection among arms that reach that analysis does not, by itself, describe the chance that the design detects an effective treatment. Power must include both stages: whether an effective arm is retained and whether its final test rejects the null hypothesis.

There is a second choice to make: what counts as successful detection when there is more than one effective arm? A design may be considered successful if it detects at least one effective arm, or only if it detects every effective arm. These are different operating characteristics. The distinction matters when comparing designs and interpreting power under a specified pattern of treatment effects. The earlier material on operating characteristics of intermediate-based rules motivates evaluating the complete selection-and-testing procedure; here, the focus is on defining its power targets precisely.

“Effective” refers to the specified true state of an arm under the scenario being evaluated, not to whether its observed result is statistically significant. The set \(E\) must be stated for the power calculation. In practice, its meaning should match the scientific target—for example, a pre-specified clinically relevant treatment effect or a particular alternative hypothesis. An arm that is retained but does not reject its final null hypothesis has not been detected under this definition.

Disjunctive and conjunctive power

When \(E\) contains one or more arms, disjunctive power is the probability that the design detects at least one effective arm. Conjunctive power is the probability that it detects every effective arm. Both probabilities include the intermediate selection rule and the final testing rule.

For a single effective arm, the two definitions coincide with that arm’s power. With multiple effective arms, conjunctive power cannot exceed disjunctive power: detecting all effective arms implies detecting at least one. Neither definition asks whether an ineffective arm is rejected. The probability of rejecting any arm, effective or not, is a different quantity and is not a substitute for either power definition.

This product identity is the probability chain rule, not an independence assumption. It is valid whenever the conditional probability is defined. It also shows why a high final-test power among retained arms can coexist with lower overall power: some effective arms may be removed before final testing.

Worked calculations

Dependence changes joint power

For more than one effective arm, arm-specific powers alone do not determine disjunctive or conjunctive power. The joint distribution of the detection events is also needed. Arms may have dependent outcomes because they share a control group, share information in a selection rule, or experience common trial-level influences. Intermediate outcomes can also induce dependence in which arms are retained.

The following identities hold without an independence assumption for two effective arms:

The second expression simply emphasizes that conjunctive power is the joint detection probability; it cannot be recovered from the marginal powers unless the dependence is known or an assumption is made. For multiple arms, the same principle applies: probabilities of intersections or unions depend on the joint distribution.

Justifying the independence shortcut

If the detection events for effective arms are mutually independent, then the probability of detecting none is the product of their non-detection probabilities. This gives a convenient formula for disjunctive power. Likewise, the probability of detecting all is the product of the arm-specific powers. The derivation is direct:

The first equality in the disjunctive expression is the complement rule; the product steps require mutual independence. Independence of final test statistics alone is not enough if selection events are dependent, and independence of selection events alone is not enough if final-test outcomes remain dependent. A shared control or a common selection constraint can make the shortcut inappropriate. When independence is not justified, use joint probabilities from the specified joint model or a valid operating-characteristic calculation.

Keep the success criterion tied to the effective set

Suppose one arm is effective and another is ineffective. The disjunctive power for detecting an effective arm is just the detection probability of the effective arm. It is not the probability that either arm is rejected. A rejection of the ineffective arm does not count as detecting the effective treatment, even though it may matter for error-rate assessment.

For example, if the effective arm has detection probability \(0.600\) and an ineffective arm has probability \(0.050\) of being retained and rejected, and the two detection events are independent, then the probability of rejecting at least one arm is

But disjunctive power for the set containing only the effective arm remains \(0.600\). This distinction prevents false discoveries from being counted as successful detection. If no arms are effective, disjunctive power for detecting an effective arm is zero; conjunctive power is generally not reported for an empty effective set because “detecting every effective arm” would be a vacuous event rather than a useful design target.

Assumptions, diagnostics, and limitations

A power value is meaningful only relative to a fully specified scenario and design rule. At a minimum, state which arms are effective, the assumptions about intermediate and final outcomes, the selection threshold and retention rule, and the final testing rule. If the final test uses a critical value that depends on which arms survive, that rule belongs in the definition of \(F_i\). Power is then evaluated under the actual rule, not under an imagined test that ignores selection.

Useful checks include verifying that \(P(D_i)\leq P(A_i)\), that conjunctive power does not exceed any effective arm’s marginal power, and that disjunctive power is at least as large as each effective arm’s power. For two arms, the proposed joint probability must satisfy

A calculation that reports only final-test power conditional on selection, or that multiplies marginal powers without a defensible independence assumption, should be treated as a warning sign. Another frequent mistake is to call the chance of any rejection “power,” even when an ineffective arm could account for that rejection. State the event explicitly before interpreting the probability.

Disjunctive and conjunctive power answer different scientific questions; neither is universally preferable. Disjunctive power is relevant when finding at least one effective option is sufficient, while conjunctive power is relevant when the objective requires success across all effective arms in the scenario. The values also depend on the chosen effect configuration: a design can have high power when only one arm is effective and much lower power when several arms must all be detected. They are operating characteristics, not guarantees for an individual trial or evidence that an arm is effective.