Reading Pairwise Power Profiles Across Arms

Reading a Power Profile Across Arms

In a multi-arm trial, each treatment-versus-control comparison can have a different probability of success. The treatment effects may differ, and the comparisons may also differ in precision. A list of pairwise powers makes those differences visible: it shows which comparisons are expected to be easy or difficult to detect under a specified set of true effects.

A profile is not, by itself, the probability that the trial will meet a multi-arm success criterion. That probability depends on how success is defined and, for a criterion involving several comparisons, on their joint behavior. The profile is still useful: it highlights a potentially weak comparison and helps explain why a trial designed to detect at least one effective treatment can look adequate while a trial requiring success for every treatment may not.

The conditioning scenario matters. Power is calculated for specified true effects, sample sizes, variability, test direction, and significance threshold. A different effect scenario or a different analysis threshold can produce a different profile. In particular, a power value belongs to a comparison under a particular assumed effect; it is not a permanent property of the treatment.

Calculating a Profile With a Normal Test

For a transparent calculation, suppose each treatment is compared with control using a one-sided normal test for a positive treatment effect. Assume the test statistic for comparison \(i\), under its specified true effect, follows a normal distribution with variance 1 and mean \(\lambda_i=\theta_i/SE_i\). Here \(\theta_i\) is the treatment-minus-control effect in outcome units, and \(SE_i\) is its standard error in the same units. The test rejects when the statistic exceeds \(z_{1-\alpha}\), the \(1-\alpha\) standard-normal quantile.

Under these assumptions, the marginal power is the probability that a normal statistic with mean \(\lambda_i\) exceeds the rejection threshold. If \(\Phi\) denotes the standard normal cumulative distribution function, then

This calculation describes each comparison separately; it does not assume that the comparison statistics are independent. Dependence is not needed to calculate these marginal powers. It does matter when combining the comparisons into an “any succeeds” or “all succeed” probability.

When standard errors and critical values are common across arms, larger positive effects produce larger noncentrality parameters and therefore larger powers. In general, however, raw effects alone do not rank powers: a smaller effect can have greater power if it is estimated more precisely. The relevant comparison is through the standardized effect \(\lambda_i=\theta_i/SE_i\), together with the applicable rejection threshold.

What a Weak Comparison Means for Trial Success

A pairwise profile helps distinguish two common success criteria. Disjunctive success means that at least one specified comparison rejects; conjunctive success means that every specified comparison rejects. For three comparisons, these events are respectively \(R_1\cup R_2\cup R_3\) and \(R_1\cap R_2\cap R_3\).

A weak comparison is especially important for conjunctive success because the event that all comparisons succeed is contained in the event that any one particular comparison succeeds. Consequently, its probability cannot exceed the power of the weakest comparison. The logic does not require independence:

Taking the smallest marginal power gives \(P(\text{all succeed})\leq\min_i\pi_i\). This is an upper bound, not a formula for conjunctive power. A low marginal power warns that an all-comparisons criterion cannot have high power, but the profile alone does not say exactly how often all comparisons succeed.

For disjunctive success, any one strong comparison can make success likely, but a high marginal power for one arm does not by itself specify the probability of at least one success. The joint behavior matters. Without assumptions about dependence, the marginal probabilities give bounds:

The lower bound for the union follows because the union contains each individual event; the upper bound follows from adding event probabilities and capping the result at 1. For the intersection, the upper bound follows from event containment. The lower bound follows by applying the union bound to the complements: the probability that at least one comparison fails is at most \(\sum_i(1-\pi_i)\). These bounds can be wide, which is another way to see why a marginal profile does not determine joint power.

The influence of a weak arm can also be seen by comparing profiles. Consider two hypothetical independent-comparison profiles, \((0.90,0.90,0.30)\) and \((0.70,0.70,0.70)\). For the first, disjunctive power is \(1-(0.10)(0.10)(0.70)=0.993\), while conjunctive power is \((0.90)(0.90)(0.30)=0.243\). For the second, disjunctive power is \(1-(0.30)^3=0.973\), while conjunctive power is \((0.70)^3=0.343\). The first profile gives greater power to detect at least one successful comparison, but the second gives greater power to succeed on all three. The profile shape is relevant to the criterion; a single summary number can hide that distinction.

A Short Justification: Why Adding an Arm Has Opposite Effects

Suppose a success criterion already involves events \(R_1,\ldots,R_K\), and a further required comparison has event \(R_{K+1}\). The old disjunctive event is contained in the new one, whereas the new conjunctive event is contained in the old one:

Probability preserves event inclusion, so adding a comparison cannot reduce the probability of “at least one succeeds” and cannot increase the probability of “all succeed.” This statement concerns the event definition. If adding a treatment also changes the design, sample sizes, or testing thresholds, the marginal powers themselves may change as well.

Diagnostics, Comparisons, and Limitations

Before interpreting a profile, check that each entry corresponds to the intended arm and treatment-versus-control comparison. Then verify that the assumed effects, standard errors, test direction, and significance thresholds match the analysis being planned. If the calculations use multiplicity-adjusted thresholds, those thresholds should be used consistently; a profile based on unadjusted tests does not automatically describe power under a different testing procedure.

Several patterns are informative. A single low entry may point to a small assumed treatment effect, low precision, or both. Similar effects but markedly different powers call for checking standard errors and critical values. A profile in which power decreases as standardized effects increase suggests a calculation, indexing, or direction-of-test error under the normal model above. A power above 1 or below 0 is impossible and signals a reporting or computation mistake.

Do not interpret the weakest power as evidence that the treatment is ineffective. Power is calculated under an assumed effect and design; it is not an estimate of the effect observed in the trial. Nor should the largest entry be reported as “the power of the multi-arm trial” unless the stated success criterion is specifically tied to that comparison.

The main alternative to reading the full profile is reporting a single joint success probability for a chosen criterion. That can be more directly aligned with a design objective, but it requires a defensible joint model or other basis for calculating the probability. Under independence, the joint probability can be obtained from the marginal powers by multiplication or the complement rule. When comparisons are dependent, those independence calculations need not apply. The marginal profile and the joint success probability answer different questions, so neither can generally replace the other.

A profile also depends on the scenario used to calculate it. If treatment effects are uncertain, one profile cannot represent every plausible configuration. It is useful to examine the profile under the scenarios that matter for the scientific question, while keeping clear which scenario each vector describes. This tutorial’s normal-test calculation assumes a known standard error and a positive one-sided alternative; other tests require their own power calculations, even though the distinction between marginal and joint probabilities remains.