Why Define Power for One Comparison?
A multi-arm trial can contain several treatment-versus-control hypotheses, but each comparison has its own chance of being declared successful. Before combining those chances into a trial-level criterion, it is useful to define the probability for one particular comparison. This probability is called pairwise power. It describes how often the specified testing procedure would reject that comparison’s null hypothesis if the assumed treatment effect and other design conditions were true.
Pairwise power is not a property of the treatment effect alone. It is an operating characteristic of a fully specified comparison: the test, its rejection rule, the significance threshold, the sample sizes, the outcome variability, and the parameter values under which the probability is evaluated all matter. In a multi-arm design, the rule for one comparison may also be affected by the procedure used to control error across the family of hypotheses.
What Must Be Specified?
Write the treatment effect as \(\Delta_j=\mu_j-\mu_0\), where \(\mu_j\) and \(\mu_0\) are the population mean outcomes in treatment \(j\) and control. A test may use a one-sided null such as \(H_{0j}:\Delta_j\leq 0\), or a two-sided null such as \(H_{0j}:\Delta_j=0\). The alternative and the rejection rule must agree: for instance, a one-sided test directed toward positive effects rejects only for sufficiently large evidence in that direction, whereas a two-sided test can reject for evidence of a difference in either direction.
A useful model description states how outcomes are generated and what quantities are treated as known or specified. For example, participants may be randomized independently to treatment and control, and outcomes within each arm may be independent draws from distributions with means \(\mu_j,\mu_0\) and variances \(\sigma_j^2,\sigma_0^2\). The variances need not be equal unless the chosen test assumes equality. Sample sizes in both arms, any planned allocation, the target effect \(\Delta_j\), and the chosen test must also be specified. If the design uses a shared control, the comparison’s sampling distribution reflects its control sample; comparisons that reuse this control may therefore be dependent, even though each comparison still has a well-defined marginal power.
The significance level \(\alpha\) describes a bound on the probability of rejection under a null hypothesis, when the test is valid under the stated assumptions. In a multi-arm trial, the rejection threshold for one comparison may be adjusted to help control the family-wise error rate. Thus, the relevant input is not merely a number called alpha: it is the actual threshold or rejection rule assigned to this comparison by the prespecified procedure.
The parameter configuration \(\eta\) can include more than \(\Delta_j\). It may include the outcome variances, the effects in other treatment arms, and other nuisance parameters. If a multiplicity procedure changes the comparison’s rejection rule according to other test results, its pairwise power is still the marginal probability of rejecting \(H_{0j}\), but evaluating that probability may require specifying the joint model for the trial.
How the Probability Is Obtained
The essential calculation is to identify every possible data outcome that leads to rejection and add the probabilities of those outcomes under the assumed parameter configuration. If a test statistic \(T_j\) rejects when \(T_j\) belongs to a rejection region \(C_j\), then
For a discrete statistic, this probability is the sum of the probabilities of the statistic values in \(C_j\). For a continuous statistic, it is the probability assigned to that region by the statistic’s sampling distribution under \(\eta\). This identity follows directly from the definition of the rejection event: \(R_j=\{T_j\in C_j\}\), so \(P_\eta(R_j)=P_\eta(T_j\in C_j)\). The calculation is specific to the sampling distribution and rejection rule; there is no single power value determined by \(\Delta_j\) and \(\alpha\) alone.
The probability of not rejecting under a specified alternative is often denoted \(\beta_j(\eta)=P_\eta(R_j^c)\). Since rejection and non-rejection are complementary events, pairwise power and this alternative-specific non-rejection probability satisfy
When a design is planned for a particular effect, its power is evaluated at that effect. A different effect, variance, sample size, or rejection threshold generally gives a different probability. Power is therefore a function over parameter values, not a permanent label attached to a treatment.
Worked Example: Summing Rejection Probabilities for One Arm
Consider a hypothetical two-arm trial comparing treatment A with control. Suppose 40 participants are assigned to each arm, the outcome variance is 16 in both arms, and the true mean difference is \(\Delta_A=3\) outcome units. A prespecified one-sided procedure rejects \(H_{0A}\) when its test statistic \(T_A\) is at least 2. Suppose a design calculation under these assumptions gives the following sampling distribution under this effect:
| Value of \(T_A\) | Probability under \(\Delta_A=3\) |
|---|---|
| 0 | 0.10 |
| 1 | 0.20 |
| 2 | 0.30 |
| 3 | 0.40 |
The probabilities sum to \(0.10+0.20+0.30+0.40=1.00\). Rejection occurs for \(T_A=2\) or \(T_A=3\), so the pairwise power at the specified effect, variance, sample sizes, and rule is
Under the stated design conditions, the procedure rejects the treatment-A null in 70% of repeated trials if the true mean difference is 3 outcome units. This is not a statement that there is a 70% probability the effect is true; it is a probability about the procedure’s repeated-sampling behavior at the specified effect.
Pairwise Power Depends on the Actual Rule
The same data-generating assumptions can produce different power values if the rejection region changes. A stricter threshold typically makes rejection less likely under an alternative, although its purpose may be to limit false rejections across several hypotheses. This is one reason to distinguish the pairwise power of an unadjusted test from the pairwise power of a comparison embedded in an FWER-controlling procedure. The latter uses the rejection rule that the procedure actually assigns to the comparison.
Worked Example: A Stricter Threshold Lowers Pairwise Power
Suppose a hypothetical multi-arm trial has three treatment-versus-control comparisons. For one specified arm, assume 50 participants are allocated to treatment and 50 to control, both outcome variances are 25, and the true effect is 2 outcome units. Let \(T_j\) be the statistic for this comparison. Under this specified alternative, suppose its distribution is:
| Value of \(T_j\) | Probability |
|---|---|
| 0 | 0.08 |
| 1 | 0.22 |
| 2 | 0.31 |
| 3 | 0.39 |
For illustration, suppose the unadjusted rule rejects at \(T_j\geq 2\). Its pairwise power is
Now suppose a multiplicity procedure assigns this comparison a stricter rule, \(T_j\geq 3\). Its power under the same alternative distribution is
The change from 0.70 to 0.39 is due to the changed rejection region, not a changed treatment effect, variance, or sample size. This example isolates the marginal probability for one comparison; it does not calculate the probability that any or all of the three comparisons reject.
Effect Size, Variability, and Sample Size
For a fixed test, increasing the magnitude of an effect in the direction of the alternative will often move the sampling distribution toward the rejection region and raise power. Greater outcome variability often makes treatment differences harder to distinguish from random variation, while larger sample sizes often make estimates more precise. These are useful design expectations, not universal monotonicity guarantees for every discrete or adaptive procedure. The exact power follows from the sampling distribution and the rejection rule.
The effect used for planning should be stated in interpretable units and justified as scientifically relevant. A power value at a large, optimistic effect may not describe performance at a smaller but still important effect. Similarly, a variance estimate from an inappropriate population or measurement process can make the calculated power misleading. A power analysis should therefore report the parameter values and assumptions, not only a percentage.
Worked Example: Comparing Power at Two Specified Effects
Suppose a hypothetical comparison has 30 participants per arm, outcome variance 9 in each arm, and the same prespecified one-sided rejection rule in both calculations. Assume the design calculation provides the following alternative distributions for \(T_j\). Under an effect of 1 outcome unit, the probabilities at \(T_j=0,1,2\) are \(0.50,0.30,0.20\); under an effect of 2 outcome units, they are \(0.25,0.35,0.40\). The rule rejects when \(T_j=2\).
At effect 1, only the \(T_j=2\) outcome leads to rejection, so
At effect 2, the same rule gives
Under these supplied distributions, power is 20% at an effect of 1 outcome unit and 40% at an effect of 2 outcome units. The sample sizes, variances, and rule are held fixed, so the difference illustrates how power is indexed by the true effect. The numbers are hypothetical sampling probabilities, not observed trial results.
Assumptions, Diagnostics, and Limits
A reported pairwise power is meaningful only relative to its assumptions. At minimum, the analysis should make clear which test is used, the null and alternative, the rejection threshold, the target effect, the outcome variance or other nuisance parameters, and the allocation and sample sizes. The sampling distribution used to calculate the probability must correspond to the design and analysis. For example, a calculation based on independent outcomes within arms may be unreliable if clustering, repeated measurements, or substantial departures from the planned allocation are present.
Several checks help detect a calculation that is not representing the intended comparison:
- Check null calibration. Under parameter values in \(H_{0j}\), the test should satisfy its stated type I error requirement, subject to the multiplicity procedure’s design. A threshold that is more permissive than intended can inflate the calculated power while also increasing false rejection risk.
- Check the alternative distribution. Confirm that it uses the target effect and plausible variability, sample sizes, and allocation. A mismatch in any of these inputs changes the probability being calculated.
- Check the rejection event. Make sure the calculation uses the rule that will actually be applied, including any multiplicity adjustment or direction of testing.
- Check for design features omitted by the calculation. Shared controls, missing outcomes, interim decisions, or dependence can change a test’s sampling distribution or rejection event. A calculation that ignores a relevant feature may not describe the planned procedure.
Pairwise power is a marginal quantity: it answers how often one specified null is rejected. It does not by itself give the probability of at least one rejection, of rejecting every required comparison, or of meeting another trial-level success criterion. Those questions require probabilities for combinations of rejection events, including their dependence. Nor does pairwise power guarantee that the effect estimate in a realized trial will be close to its assumed value.
A common alternative planning approach is to choose a target power, such as 80%, and solve for the sample size that attains it at a prespecified effect and variance. This is another use of the same pairwise operating characteristic; it does not remove the need to state the test and assumptions. In a multi-arm trial, the target may instead concern trial-level success. That is a different criterion from the power of one pairwise comparison, and the marginal probabilities alone are generally insufficient to determine it.
Check Your Understanding
Use the definition of pairwise power and the rejection-event calculations to answer these questions. Short solutions follow each question.
- A statistic takes values 0, 1, 2, and 3 with probabilities 0.15, 0.25, 0.35, and 0.25 under a specified treatment effect. If the rule rejects when the statistic is at least 2, what is the pairwise power? Solution: The rejection values are 2 and 3, so power is \(0.35+0.25=0.60\), or 60%.
- In one sentence, distinguish pairwise power from the probability that at least one of three treatment comparisons rejects. Solution: Pairwise power is the rejection probability for one specified comparison, whereas the probability that at least one rejects concerns the union of three rejection events and depends on their joint behavior.
- Why can two tests with the same target effect and sample sizes have different pairwise powers? Solution: They may have different sampling distributions or rejection regions, for example because they use different tests or different significance thresholds.
- If the non-rejection probability under a specified alternative is 0.27, what is pairwise power under that alternative? Solution: Rejection and non-rejection are complements, so power is \(1-0.27=0.73\), or 73%.
- State two assumptions or design inputs that should accompany a reported pairwise power value. Solution: Valid answers include the target effect, outcome variance, sample sizes and allocation, test and rejection threshold, or the model used for the sampling distribution.