Calculating Pairwise Power With a Normal Test

From a Treatment Effect to Pairwise Power

A pairwise power calculation asks a focused question: if a particular treatment truly differs from control by a specified amount, how often would the planned test reject its null hypothesis? For a normal test, the calculation reduces to locating the alternative distribution relative to the test’s rejection boundary. A larger effect or smaller standard error moves that distribution farther into the rejection region; a more stringent critical value moves the boundary farther away.

This tutorial concerns one treatment-versus-control test at a time. Its result is the marginal power for that comparison, not the probability that at least one treatment succeeds or that all treatments succeed. Those trial-wide probabilities require information about multiple success events.

Assumptions and the Normal-Test Setup

For a concrete treatment-versus-control setup, suppose outcomes are independent between participants and normally distributed in each arm, with a common known standard deviation \(\sigma\). Let \(n_T\) and \(n_C\) be the treatment and control sample sizes, and let the true difference in population means be \(\Delta=\mu_T-\mu_C\). The estimated difference in sample means has standard error \(SE=\sigma\sqrt{1/n_T+1/n_C}\). The standardized statistic is \(Z=(\bar{x}_T-\bar{x}_C)/SE\).

Under the null hypothesis \(H_0:\Delta=0\), this statistic has a standard normal distribution. Under an alternative with a fixed difference \(\Delta\), it has distribution \(N(\lambda,1)\), where \(\lambda=\Delta/SE\). The formula is exact under the stated normal, independence, and known-variance assumptions. If the variance is estimated, a t-test may be appropriate for finite samples; a normal calculation can also be a large-sample approximation. In either case, the chosen test and its assumptions should match the planned analysis.

For an upper-tail test, the direction must be prespecified so that positive values of \(Z\) represent evidence for the treatment effect of interest. At a one-sided significance level \(\alpha\), the critical value is \(c=z_{1-\alpha}\), where \(z_q\) is the \(q\) quantile of the standard normal distribution. If a multiplicity procedure assigns this comparison a particular local significance level, that level determines its critical value; the calculation below takes the relevant critical value as given.

Why the Power Formula Works

The formula follows by centering the statistic at its alternative mean. Under the alternative, \(Z-\lambda\sim N(0,1)\). For an upper-tail test, subtracting \(\lambda\) from both sides of the rejection inequality gives \(Z>c\) exactly when \(Z-\lambda>c-\lambda\). The probability of that event is the upper tail of a standard normal variable. For a two-sided test, apply the same shift to each tail separately.

This also clarifies the roles of the inputs. Holding \(c\) fixed, increasing a positive \(\lambda\) raises upper-tail power. Holding \(\lambda\) fixed, increasing \(c\) lowers it. The NCP is dimensionless: it is the treatment effect measured in standard-error units. The effect \(\Delta\) and its standard error must use the same units, such as both being measured in points.

Two-Sided Power and Direction

A two-sided test rejects for sufficiently large evidence in either direction. At two-sided level \(\alpha=0.05\), the standard normal critical value is \(c=z_{0.975}=1.960\). A nonzero alternative puts some probability in both rejection tails. The tail opposite the expected treatment effect may be small, but it belongs in the power calculation.

Calculation Checks and Common Pitfalls

Before interpreting a computed power, verify the direction, units, and rejection rule. A critical value must correspond to the test’s actual significance level and tail structure. If a comparison has a locally adjusted significance level, substituting the unadjusted nominal level gives the wrong boundary and therefore the wrong power. Likewise, one-sided and two-sided tests generally do not share the same critical value at the same nominal \(\alpha\).

The sign of \(\lambda\) matters. With an upper-tail test designed for positive effects, a negative \(\lambda\) shifts probability away from rejection and yields low power. If the clinically favorable direction is negative, define the statistic or contrast consistently with that direction before calculating. Do not change the direction after seeing the data.

• Wrong reasoning: “The effect is 0.5 points, so I use 0.5 as the NCP.” Correction: Divide the effect by its standard error; here \(0.5/1.000=0.500\).
• Wrong reasoning: “A two-sided test has power \(1-\Phi(c-\lambda)\).” Correction: Add the lower rejection-tail probability \(\Phi(-c-\lambda)\).
• Wrong reasoning: “Power of 0.323 means a 32.3% probability that the null hypothesis is true.” Correction: It is the probability of rejection under a specified alternative and model; it is not a probability assigned to either hypothesis.
• Wrong reasoning: “The pairwise power is the probability the whole multi-arm trial declares success.” Correction: It describes one comparison only. A trial-wide success event is a different probability.

Power calculations can also fail when their inputs do not represent the planned analysis. For example, the standard error may be wrong if allocation sizes or outcome variability are misrepresented, and a normal approximation may be poor with small samples or highly non-normal outcomes. If variance is estimated and samples are modest, use the appropriate t-based calculation; for complex designs, a design-matched simulation may be needed. The normal calculation is valuable because it is transparent, but it is only as credible as its model and inputs.