Why Rare Outcomes Complicate Power Comparisons
When an outcome is binary, each participant either experiences the event or does not. If events are rare, a trial can have many participants but still observe very few events in one or both arms. That matters for power comparisons: the usual normal approximation may return a numerical power estimate even when the event counts are too sparse for that estimate to be trustworthy.
Allocation changes both the precision of a difference in event proportions and the number of events expected in each arm. A design may therefore look preferable under a variance calculation yet offer little dependable power if its event counts are sparse. The focus here is on comparisons with fixed arm sizes; it is not a general analysis of the additional uncertainty introduced when adaptive allocation makes arm sizes random.
Under these assumptions, the variance of the estimated risk difference is
This variance identifies how allocation affects precision, but power also depends on the test’s rejection threshold. For a familiar large-sample approximation, define the pooled event rate used under the null as \(\bar{p}_0=(n_0p_0+n_1p_1)/(n_0+n_1)\). The null standard error is approximated by \(SE_0=\sqrt{\bar{p}_0(1-\bar{p}_0)(1/n_0+1/n_1)}\). Under the alternative, approximate the risk-difference estimator by a normal distribution with mean \(\delta\) and standard error \(SE_1=\sqrt{p_1(1-p_1)/n_1+p_0(1-p_0)/n_0}\). The resulting approximate power is
where \(z_{1-\alpha/2}\) is the corresponding standard normal quantile. This formula is a planning approximation, not a guarantee that the finite-sample test has that power. In particular, an expected event count \(np\) is a diagnostic, not a universal pass-or-fail rule. Counts near zero make the sampling distribution discrete and potentially skewed even when the total sample size is large.
Worked Power Comparison
Worked Example: A Few Expected Events in One Arm
Suppose a hypothetical fixed-allocation trial has \(n_0=100\), \(n_1=200\), \(p_0=0.02\), and \(p_1=0.05\). The target risk difference is \(\delta=0.05-0.02=0.03\), or 3 percentage points. The expected event counts are \(100(0.02)=2\) for control and \(200(0.05)=10\) for treatment. Expected non-event counts are 98 and 190.
For the normal approximation, the weighted pooled rate is \(\bar{p}_0=(100(0.02)+200(0.05))/300=12/300=0.04\). Thus
Under the alternative,
For a two-sided 5% test, \(z_{0.975}=1.96\), so the approximate rejection cutoffs for the estimated risk difference are \(\pm1.96(0.02400)=\pm0.04704\). The upper-tail contribution to power is approximately \(1-\Phi((0.04704-0.03)/0.02082)=1-\Phi(0.818)=0.207\). The lower-tail contribution is approximately \(\Phi((-0.04704-0.03)/0.02082)=\Phi(-3.70)\), which is about 0.0001. Approximate total power is therefore about 0.207.
That number should not be treated as a dependable finite-sample power calculation. The control arm has only two expected events, and the treatment arm has ten. For example, the exact probability of observing no control events is \(P(X=0)=0.98^{100}\approx0.1326\), where \(X\) is the control event count. A continuity-corrected normal approximation gives about \(\Phi((0.5-2)/\sqrt{1.96})=\Phi(-1.071)\approx0.142\). The discrepancy is a useful warning: the count distribution is discrete, and a normal approximation may misrepresent relevant probabilities even when the overall sample size is 300.
How Allocation Changes the Comparison
For a fixed total sample size, an unequal allocation can increase precision when it assigns more participants to the arm with the larger outcome variance \(p(1-p)\). But the power comparison must account for both the alternative variance and the null rejection threshold. The example below uses the same normal-approximation procedure in each design; its results illustrate the calculation, not a claim that the sparse-event approximation is reliable.
Worked Example: Equal Versus Unequal Allocation With Rare Events
Consider hypothetical event probabilities \(p_0=0.01\) and \(p_1=0.03\), with \(N=600\) participants and a two-sided 5% test. Compare equal allocation, \((n_0,n_1)=(300,300)\), with unequal allocation, \((n_0,n_1)=(200,400)\). The risk difference is \(\delta=0.02\).
With equal allocation, expected events are \(300(0.01)=3\) in control and \(300(0.03)=9\) in treatment, for 12 expected events in total. The null pooled rate is \(\bar{p}_0=(3+9)/600=0.02\). Therefore,
The rejection cutoff is \(1.96(0.01143)=0.02240\). The upper-tail power contribution is approximately \(1-\Phi((0.02240-0.02)/0.01140)=1-\Phi(0.211)=0.416\); the lower-tail contribution is about \(\Phi((-0.02240-0.02)/0.01140)=\Phi(-3.72)\), or approximately 0.0001. Approximate power is about 0.416.
With unequal allocation, expected events are \(200(0.01)=2\) in control and \(400(0.03)=12\) in treatment. The total expected event count is \(2+12=14\). The null pooled rate used in the approximation is \(\bar{p}_0=(2+12)/600=0.02333\). Thus
The cutoff is \(1.96(0.01307)=0.02562\). The upper-tail contribution is approximately \(1-\Phi((0.02562-0.02)/0.01106)=1-\Phi(0.509)=0.305\), and the lower-tail contribution is negligible, about \(\Phi(-4.12)\). The approximate power is about 0.305.
In this calculation, the unequal design has a slightly smaller alternative standard error but a larger null standard error and a higher rejection cutoff; its approximate power is lower. Yet both designs have sparse expected events, and the unequal design has only two expected control events. The numerical difference should prompt an exact or simulation-based check rather than be read as a precise prediction.
Variance-Optimal Allocation Is Not the Whole Answer
A useful benchmark is the allocation that minimizes the variance of the risk-difference estimator when \(p_0\) and \(p_1\) are treated as known planning values. Write \(v_0=p_0(1-p_0)\) and \(v_1=p_1(1-p_1)\). With fixed \(N=n_0+n_1\), minimize \(v_0/n_0+v_1/n_1\). Differentiating after substituting \(n_1=N-n_0\) gives
The solution is a variance benchmark, not a universal power optimum. It does not by itself ensure adequate event counts, account for the null standard error used by a particular test, or guarantee that a normal approximation is accurate.
Worked Example: The Variance-Minimizing Ratio
Using \(p_0=0.01\) and \(p_1=0.03\), the arm-specific variances are \(v_0=0.01(0.99)=0.0099\) and \(v_1=0.03(0.97)=0.0291\). The variance-minimizing ratio is
For \(N=600\), this corresponds approximately to \(n_0=600/(1+1.714)\approx221\) and \(n_1=379\). The \(200/400\) allocation has a ratio of 2 and is in the same general direction, assigning more participants to the arm with higher \(p(1-p)\). However, its expected control event count is still \(200(0.01)=2\). A variance calculation cannot remove the sparse-count problem.
Diagnostics, Alternatives, and Limitations
Before relying on a normal-approximation power comparison, inspect the expected event and non-event counts separately in each arm. Also consider the probability of zero events and whether the test statistic can become unstable when an arm has no events. A large total sample does not resolve these issues if one arm’s event probability is very small. There is no single count threshold that proves an approximation valid in every setting; the intended test, tail probabilities, and allocation all matter.
When counts are sparse, exact binomial-based methods can avoid the normal approximation. For example, an exact test can calculate rejection probabilities using the discrete distribution of the arm counts under a specified testing procedure. Its power is then found under the alternative by summing probabilities of outcomes in the rejection region. The chosen exact test matters: conditional and unconditional procedures need not have identical power, and discreteness can make their achieved significance level smaller than the nominal level. Simulation can also estimate power by repeatedly generating outcomes under the stated arm probabilities and applying the planned test, provided the simulation matches the trial design and analysis.
| Approach | Strength | Important caution |
|---|---|---|
| Normal approximation | Fast and useful for initial comparisons when counts are adequate. | Can be inaccurate with sparse events, discreteness, or zero counts. |
| Exact binomial-based test | Uses the discrete outcome distribution rather than a normal approximation. | Results depend on the specified test; discreteness can affect achieved error rates and power. |
| Simulation of the planned test | Can represent the actual allocation and analysis procedure. | Requires correct model inputs and enough simulations for stable estimates. |
A common mistake is to compare allocation schemes using only \(SE_1\) and conclude that the smaller standard error necessarily gives greater power. The correction is to calculate the test’s rejection threshold as well: power depends on the alternative distribution relative to that threshold. A second mistake is to regard a normal-approximation power value as precise merely because the total \(N\) is large. The arm-specific expected event counts—and the test’s behavior when counts are zero or very small—must also be examined.
Check Your Understanding
Answer each question, then check the brief solutions.
- With \(n_0=150\), \(p_0=0.02\), and \(n_1=250\), \(p_1=0.04\), what are the expected event counts in the two arms?
- Why does a large total sample size alone not establish that a normal approximation is reliable for a rare binary outcome?
- For fixed total sample size, what ratio \(n_1/n_0\) minimizes the variance of \(\hat{p}_1-\hat{p}_0\), given \(p_0\) and \(p_1\)?
- Suppose one allocation has smaller \(SE_1\) but a larger null standard error \(SE_0\). Can you infer which has greater power from \(SE_1\) alone? Explain.
Solutions. (1) Control: \(150(0.02)=3\) expected events; treatment: \(250(0.04)=10\). (2) One arm may still have very few expected events, making its count distribution discrete and skewed; check arm-specific counts and the planned test. (3) \(n_1/n_0=\sqrt{p_1(1-p_1)/[p_0(1-p_0)]}\), the variance-minimizing ratio under the stated planning probabilities. (4) No. The rejection threshold depends on \(SE_0\) as well as the alternative distribution, so compare power using both quantities and check whether the approximation is credible.