Comparing Binary-Outcome Power

Why Compare Binary-Outcome Power?

When an outcome is binary, each participant either experiences the event or does not. A trial’s power depends not only on the total sample size and the difference in event probabilities, but also on how that total is divided between the two arms. This tutorial compares approximate power for two fixed allocations with the same total sample size. The comparison is about the precision of a difference in event proportions, not about patient benefit or an allocation rule that changes in response to observed outcomes.

The calculations below use a two-sided test of equal event probabilities. They are prospective planning approximations: the specified event probabilities and allocation sizes determine the standard errors and rejection threshold used in each calculation. Matching the total sample size and the assumed event probabilities makes the allocation comparison interpretable.

Definition: Let \(n_1\) and \(n_0\) be the fixed sample sizes in arms 1 and 0, with total sample size \(N=n_1+n_0\). Let \(X_{1i}\) and \(X_{0j}\) be independent binary outcomes, where 1 denotes the event and 0 denotes no event. Assume \(X_{1i}\sim\mathrm{Bernoulli}(p_1)\) and \(X_{0j}\sim\mathrm{Bernoulli}(p_0)\). The estimated difference in event probabilities is \(\hat{\Delta}=\hat{p}_1-\hat{p}_0\), and the target difference under the alternative is \(\Delta=p_1-p_0\).

Model, Test, and Power Approximation

The test considered is \(H_0:p_1=p_0\) against \(H_a:p_1\ne p_0\), at significance level \(\alpha\). Under \(H_0\), specify a common reference event probability \(p_{\mathrm{ref}}\). For a prospective approximation, treat the null standard error as fixed at that reference probability. This is a planning approximation to the usual two-proportion test, whose pooled standard error is estimated from the data.

The assumptions are independent participants’ outcomes within and between arms, fixed arm sizes, a common event probability under the null, and a normal approximation for the difference in sample proportions. The normal approximation is most credible when expected event and non-event counts are not small in either arm. The standard error under the alternative uses the two specified probabilities separately; it is not the null standard error.

SE_0=\sqrt{p_{\mathrm{ref}}(1-p_{\mathrm{ref}}) \left(\frac{1}{n_1}+\frac{1}{n_0}\right)} \qquad SE_A=\sqrt{\frac{p_1(1-p_1)}{n_1}+ \frac{p_0(1-p_0)}{n_0}}

Write \(z_{1-\alpha/2}\) for the \(1-\alpha/2\) standard normal quantile. The approximate two-sided test rejects when \(\hat{\Delta}\) is below \(-c\) or above \(c\), where \(c=z_{1-\alpha/2}SE_0\). Under the alternative, \(\hat{\Delta}\) is approximated by a normal random variable with mean \(\Delta\) and standard deviation \(SE_A\). Thus power is the probability that this alternative distribution falls outside the null acceptance interval.

\mathrm{Power}\ \approx\ \Phi\left(\frac{-c-\Delta}{SE_A}\right) + 1-\Phi\left(\frac{c-\Delta}{SE_A}\right)

The lower-tail boundary is \(-c\), so its standardized value is \((-c-\Delta)/SE_A\). Keeping that boundary distinct from the alternative mean is essential: when \(\Delta\) is negative, for example, the lower-tail probability can be substantial. The formula adds the probability of rejecting in either direction.

Worked Comparisons at a Common Total Sample Size

Worked Example: Equal Versus 3-to-1 Allocation

Suppose a hypothetical trial has \(N=200\), with \(p_0=0.30\) and \(p_1=0.50\). Compare equal allocation, \(n_1=n_0=100\), with \(n_1=150\) and \(n_0=50\). Use a two-sided test with \(\alpha=0.05\) and \(p_{\mathrm{ref}}=0.40\). The outcomes are assumed independent Bernoulli observations, and the expected event and non-event counts are adequate for this normal approximation.

Here \(\Delta=0.50-0.30=0.20\), \(z_{0.975}=1.96\), and \(p_{\mathrm{ref}}(1-p_{\mathrm{ref}})=0.24\). With equal allocation,

SE_0=\sqrt{0.24\left(\frac{1}{100}+\frac{1}{100}\right)} =\sqrt{0.0048}=0.06928,\qquad c=1.96(0.06928)=0.1358.

The alternative standard error is

SE_A=\sqrt{\frac{0.50(0.50)}{100}+ \frac{0.30(0.70)}{100}} =\sqrt{0.004600}=0.06782.

Therefore the approximate power is

\Phi\left(\frac{-0.1358-0.20}{0.06782}\right) +1-\Phi\left(\frac{0.1358-0.20}{0.06782}\right) \approx \Phi(-4.950)+1-\Phi(-0.9465) \approx 0.8279.

For the 150-to-50 allocation, the null standard error and cutoff are

SE_0=\sqrt{0.24\left(\frac{1}{150}+\frac{1}{50}\right)} =\sqrt{0.006400}=0.08000,\qquad c=1.96(0.08000)=0.1568.

The alternative standard error is

SE_A=\sqrt{\frac{0.25}{150}+\frac{0.21}{50}} =\sqrt{0.0058667}=0.07659.

The corresponding power is

\Phi\left(\frac{-0.1568-0.20}{0.07659}\right) +1-\Phi\left(\frac{0.1568-0.20}{0.07659}\right) \approx \Phi(-4.658)+1-\Phi(-0.5643) \approx 0.7136.

Under these assumptions, equal allocation has about 82.8% approximate power and the 150-to-50 allocation about 71.4%. Both calculations use the same total sample size and event probabilities; the unequal allocation has less information for estimating the difference.

Worked Example: A Smaller Difference and a Different Total

For a second hypothetical comparison, let \(N=240\), \(p_0=0.30\), and \(p_1=0.45\). Compare 120 participants per arm with \(n_1=180\), \(n_0=60\). Use \(\alpha=0.05\) and \(p_{\mathrm{ref}}=0.375\). Assume independent binary outcomes and sufficient expected counts for the normal approximation. Now \(\Delta=0.15\), and \(p_{\mathrm{ref}}(1-p_{\mathrm{ref}})=0.234375\).

For equal allocation,

SE_0=\sqrt{0.234375\left(\frac{1}{120}+\frac{1}{120}\right)} =\sqrt{0.00390625}=0.06250,\qquad c=1.96(0.06250)=0.1225.

Since \(p_1(1-p_1)=0.2475\) and \(p_0(1-p_0)=0.21\),

SE_A=\sqrt{\frac{0.2475}{120}+\frac{0.21}{120}} =\sqrt{0.0038125}=0.06175.

The power is

\Phi\left(\frac{-0.1225-0.15}{0.06175}\right) +1-\Phi\left(\frac{0.1225-0.15}{0.06175}\right) \approx \Phi(-4.413)+1-\Phi(-0.4454) \approx 0.6719.

With 180 participants in arm 1 and 60 in arm 0,

SE_0=\sqrt{0.234375\left(\frac{1}{180}+\frac{1}{60}\right)} =\sqrt{0.0052083}=0.07217,\qquad c=1.96(0.07217)=0.1414,

SE_A=\sqrt{\frac{0.2475}{180}+\frac{0.21}{60}} =\sqrt{0.004875}=0.06982.

Thus,

\Phi\left(\frac{-0.1414-0.15}{0.06982}\right) +1-\Phi\left(\frac{0.1414-0.15}{0.06982}\right) \approx \Phi(-4.173)+1-\Phi(-0.1224) \approx 0.5487.

The unequal allocation has lower approximate power here as well. It also gives a larger null cutoff because \(1/n_1+1/n_0\) is larger. A power comparison therefore needs both the alternative variability and the null rejection threshold, not just one standard error.

Worked Example: A Case Where the Power Difference Is Small

Consider a hypothetical trial with \(N=200\), \(p_0=0.10\), and \(p_1=0.30\), comparing equal allocation with \(n_1=120\), \(n_0=80\). Use \(\alpha=0.05\) and \(p_{\mathrm{ref}}=0.20\). Independent Bernoulli outcomes are assumed; the expected event counts are 10 and 30 under equal allocation and 36 and 8 under the unequal allocation, so a normal approximation is plausible but should still be treated as approximate.

The effect is \(\Delta=0.20\), and \(p_{\mathrm{ref}}(1-p_{\mathrm{ref}})=0.16\). With \(n_1=n_0=100\),

SE_0=\sqrt{0.16(0.01+0.01)}=0.05657,\qquad c=1.96(0.05657)=0.1109, $$ $$ SE_A=\sqrt{\frac{0.30(0.70)}{100}+\frac{0.10(0.90)}{100}} =\sqrt{0.003000}=0.05477.

The resulting power is

\Phi\left(\frac{-0.1109-0.20}{0.05477}\right) +1-\Phi\left(\frac{0.1109-0.20}{0.05477}\right) \approx \Phi(-5.675)+1-\Phi(-1.627) \approx 0.9481.

For \(n_1=120\), \(n_0=80\), the null and alternative standard errors are

SE_0=\sqrt{0.16\left(\frac{1}{120}+\frac{1}{80}\right)} =\sqrt{0.0033333}=0.05774,\qquad c=1.96(0.05774)=0.1132, $$ $$ SE_A=\sqrt{\frac{0.21}{120}+\frac{0.09}{80}} =\sqrt{0.002875}=0.05362.

The power is approximately

\Phi\left(\frac{-0.1132-0.20}{0.05362}\right) +1-\Phi\left(\frac{0.1132-0.20}{0.05362}\right) \approx \Phi(-5.84)+1-\Phi(-1.619) \approx 0.9473.

These powers are close: approximately 94.8% and 94.7%. In this example the modest imbalance reduces the alternative standard error, but it also raises the null cutoff slightly. The example illustrates why the power difference must be calculated rather than inferred from the allocation ratio alone.

Why Allocation Changes Precision

For fixed \(N\), the alternative variance of the difference is a sum of two terms, one from each arm. If \(v_1=p_1(1-p_1)\) and \(v_0=p_0(1-p_0)\), then

\operatorname{Var}(\hat{\Delta})\approx \frac{v_1}{n_1}+\frac{v_0}{n_0}, \qquad n_0=N-n_1.

A short derivation also explains why equal allocation is not automatically the variance-minimizing choice when the arm variances differ. Treat \(n_1\) as continuous between 0 and \(N\), and differentiate the variance with respect to \(n_1\):

\frac{d}{dn_1}\left(\frac{v_1}{n_1}+ \frac{v_0}{N-n_1}\right) =-\frac{v_1}{n_1^2}+\frac{v_0}{(N-n_1)^2}.

Setting this derivative to zero gives \(v_1/n_1^2=v_0/n_0^2\), or \(n_1/n_0=\sqrt{v_1/v_0}\). This is the continuous allocation that minimizes the alternative variance when the probabilities are treated as known and there are no additional design constraints. It does not by itself guarantee maximum power for every test approximation: the null cutoff also depends on allocation, and practical designs may have other objectives or constraints.

Checks, Comparisons, and Limits

Before trusting a comparison, check that both designs use the same total sample size, arm-specific event probabilities, significance level, and two-sided testing convention. Confirm that \(n_1\) is paired with \(p_1\) throughout and \(n_0\) with \(p_0\). Then calculate \(SE_0\), \(c\), and \(SE_A\) separately for each allocation. A numerical tool can evaluate \(\Phi\), but it cannot correct a mismatched arm size or an incorrectly signed boundary.

The principal alternative to this planning approximation is to calculate power for the actual test statistic and its actual variance estimator, often by enumerating or simulating binary outcomes. Such approaches can better represent a pooled test’s random denominator and discreteness, but require specifying the exact test and add computational or modeling choices. The normal approximation is useful for transparent comparisons under matched conditions; it is not an exact finite-sample result.

Check arm-specific expected counts. Small expected event or non-event counts can undermine the normal approximation, even if the total sample size seems substantial.
Check the reference probability. The null standard error depends on \(p_{\mathrm{ref}}\). If that planning value is not appropriate, the calculated cutoff and power may be misleading.
Check two-sided tails. Both rejection regions contribute to power. When \(\Delta\) is positive, the lower-tail term may be tiny, but it still has the boundary \(-c\), not \(\Delta-c\).
Check the scope. These formulas concern independent binary outcomes with fixed arm sizes. They do not account for clustering, loss to follow-up, covariate adjustment, or allocation that changes during the trial.

Equal allocation is a useful benchmark and often performs well for comparing two arms, but a power comparison is conditional on the assumed event probabilities and test. Unequal allocation can reduce precision, leave power nearly unchanged, or—in settings with sufficiently different arm variances—reduce the alternative variance. The calculated rejection boundary and both tail probabilities determine the approximate power.

Common mistake: “The unequal design has fewer participants in one arm, so its power must be exactly proportional to that arm’s sample size.” This is not correct. The variance depends on both arm-specific probabilities and both sample sizes, while the null cutoff also changes with allocation. Compute both standard errors and the two-sided rejection probability for each design.

Check Your Understanding

Use the stated normal-approximation framework. For each question, check the arm labels and distinguish the null standard error from the alternative standard error.

With \(n_1=80\), \(n_0=120\), \(p_1=0.40\), and \(p_0=0.25\), calculate \(\Delta\) and \(SE_A\).
At fixed \(N\), explain why \(1/n_1+1/n_0\) is larger for unequal than equal allocation.
For a two-sided test with cutoff \(c>0\) and alternative difference \(\Delta\), write the correct standardized lower-tail boundary.
Suppose a design comparison changes the total sample size as well as the allocation ratio. Why is it not a matched comparison of allocation alone?

Solutions. (1) \(\Delta=0.40-0.25=0.15\). The alternative variance is \(0.40(0.60)/80+0.25(0.75)/120=0.003000+0.0015625=0.0045625\), so \(SE_A=\sqrt{0.0045625}=0.06755\). (2) Since \(n_0=N-n_1\), the sum is \(1/n_1+1/(N-n_1)\), minimized at \(n_1=N/2\); moving away from equal allocation increases it. (3) The lower boundary is \((-c-\Delta)/SE_A\), because the rejection region is \(\hat{\Delta}<-c\). (4) The total sample size itself affects both standard errors and power, so the result would combine the effect of changing total size with the effect of changing allocation. To isolate allocation, hold \(N\) and the assumed probabilities fixed.

What You'll Learn