Why Small Trials Need Finite-Sample Comparisons
A power comparison is only meaningful after the test procedure is specified. In a small two-arm trial, the possible data may be few and widely separated: one additional success can change a test statistic sharply, and the nominal significance level may not be attainable by any nonrandomized rejection rule. A large-sample calculation can smooth over these features and report a power that the actual test does not have.
For a fair comparison, hold the trial conditions fixed: the arm sizes, outcome model, treatment effect, significance target, sidedness, and test procedure must be stated. In this tutorial, binary outcomes are independent Bernoulli observations within and between arms. Continuous-outcome calculations use independent normal observations with a common variance when the pooled two-sample t test is used. These assumptions make the finite-sample calculations explicit; if they are not plausible, the resulting power calculations need not apply.
For a discrete outcome, power can be found by summing the probabilities of all data configurations in the rejection region. If \(X_1\) and \(X_0\) are the numbers of successes in arms 1 and 0, respectively, and the outcomes are independent, then under success probabilities \(p_1\) and \(p_0\),
Adding this probability over the rejection region gives power under the alternative; using \(p_1=p_0\) gives the actual size at that null value.
Binary Outcomes: The Test Determines the Power
The following small example shows why a test’s name and its rejection rule must agree. For a one-sided pooled score test of \(H_0:p_1=p_0\) against \(H_a:p_1>p_0\), the pooled estimate is \(\hat p=(X_1+X_0)/(n_1+n_0)\). With equal arm sizes \(n\), the score statistic is
The denominator is estimated from the observed pooled proportion. It is therefore data-dependent, even when the null hypothesis specifies equal success probabilities.
Worked Example: Enumerating a Pooled Score Test
Suppose \(n_1=n_0=3\), and use the one-sided pooled score rule \(Z_{\mathrm{score}}\geq1.645\). First find its rejection region. For \((X_1,X_0)=(3,0)\), \(\hat p=3/6=0.5\), so \(Z_{\mathrm{score}}=1/\sqrt{0.25(2/3)}=2.449\). For \((3,1)\), \(\hat p=4/6\), and the statistic is \( (2/3)/\sqrt{(2/9)(2/3)}=1.732\). For \((2,0)\), the pooled estimate is \(2/6\), giving the same statistic, 1.732. The other pairs with \(X_1>X_0\) do not reach 1.645. Thus the rejection region is exactly \((3,0),(3,1),(2,0)\).
Under the null value \(p_1=p_0=0.5\), each arm count has probabilities \(1/8,3/8,3/8,1/8\) for counts 0 through 3. The probability of \((3,0)\) is \(1/64\), that of \((3,1)\) is \(3/64\), and that of \((2,0)\) is \(3/64\). The actual rejection probability is therefore \(7/64=0.1094\), which exceeds 0.05. This nominal 0.05 score rule does not control Type I error at 0.05 in this small trial.
Now take the alternative \(p_1=0.8,\ p_0=0.2\). The three rejection-pair probabilities are \(0.8^3(0.8)^3=0.262144\), \(0.8^3[3(0.2)(0.8)^2]=0.196608\), and \([3(0.8)^2(0.2)](0.8)^3=0.196608\). The power of this particular score rule is their sum, \(0.65536\). It is not appropriate to describe this as power for a valid level-0.05 test: its actual null rejection probability is about 0.1094.
An exact conditional alternative is Fisher’s exact test. For a one-sided test, it conditions on the total number of successes and evaluates the probability, under equal success probabilities, of seeing at least as many treated-arm successes as observed. This yields a valid test with conditional Type I error no greater than its chosen level, but its rejection region can be quite conservative when the sample is tiny.
Worked Example: Fisher’s Exact Test With Three Participants Per Arm
Use the same arm sizes and test \(H_0:p_1=p_0\) against \(H_a:p_1>p_0\) at level 0.05. At the outcome \((X_1,X_0)=(3,0)\), the total number of successes is 3. Under the null conditional distribution, the probability that all three successes fall in the treated arm is \(1/{6\choose3}=1/20=0.05\). This outcome is included in the rejection region. The other possible outcomes do not have a one-sided conditional p-value at or below 0.05, so the rejection region consists only of \((3,0)\).
At the null value \(p_1=p_0=0.5\), the unconditional probability of this outcome is \((1/8)(1/8)=1/64=0.01563\). Thus the test’s unconditional size at this null value is below its nominal level. Under \(p_1=0.8,\ p_0=0.2\), its power is \(0.8^3(1-0.2)^3=0.8^6=0.2621\). This is much smaller than the score-rule power above, but the comparison is not simply “which test has more power”: the score rule had actual size 0.1094, whereas Fisher’s test is conservative here. Comparisons should account for actual size as well as power.
Discreteness and Normal Approximation
A normal approximation replaces a discrete sampling distribution by a continuous one. It can be useful, but it does not alter the exact rejection region of a test. The next example separates an explicitly defined count rule from a pooled score test and compares its exact operating characteristics with a continuity-corrected normal approximation.
Worked Example: Exact and Approximate Power for a Count Rule
With four participants per arm, consider the one-sided rule “reject if \(D=X_1-X_0\geq3\).” This is a fixed count-difference rule, not a pooled score test. Under \(p_1=p_0=0.5\), the independent binomial probabilities give rejection pairs \((3,0),(4,0),(4,1)\), with respective probabilities \(4/256,1/256,4/256\). The actual size is \(9/256=0.03516\).
Under \(p_1=0.8,\ p_0=0.2\), \(P(X_1=3)=0.4096\), \(P(X_1=4)=0.4096\), \(P(X_0=0)=0.4096\), and \(P(X_0\leq1)=0.8192\). Hence the exact power is
For comparison, under the alternative, \(E(D)=4(0.8-0.2)=2.4\) and \(\operatorname{Var}(D)=4(0.8)(0.2)+4(0.2)(0.8)=1.28\), so the approximate standard deviation is \(\sqrt{1.28}=1.131\). Applying a continuity correction gives \(P(D\geq3)\approx P\{N(2.4,1.28)>2.5\}=\Phi((2.4-2.5)/1.131)\approx0.4648\). This differs from the exact power, 0.5033. The approximation also does not tell us the exact size of the stated rule; enumeration does.
A normal approximation can mislead especially when expected counts are small, probabilities are near 0 or 1, the rejection boundary falls near a discrete jump, or a test statistic’s variance is estimated from sparse data. Continuity corrections may improve an approximation, but they cannot guarantee accurate size or power. For very small samples, enumerate the actual rejection region whenever feasible.
Continuous Outcomes: Exact Small-Sample Power
For independent normal outcomes with a common but unknown variance, the usual pooled two-sample t test has an exact finite-sample reference distribution under \(H_0:\mu_1=\mu_0\). For arm sizes \(n_1,n_0\), its degrees of freedom are \(df=n_1+n_0-2\). Under a specified alternative \(\mu_1-\mu_0=\delta\), the test statistic follows a noncentral t distribution with noncentrality parameter
This result assumes independent normal observations and equal population variances. If those assumptions hold, exact power for a two-sided level-\(\alpha\) test is the probability that this noncentral t statistic falls below \(-t_{1-\alpha/2,df}\) or above \(t_{1-\alpha/2,df}\). A normal approximation instead treats the standardized statistic as approximately normal and does not fully account for estimating the variance.
Worked Example: Small-Sample t-Test Power Calculation
Suppose a hypothetical trial plans four participants per arm, with normal outcomes, common standard deviation \(\sigma=1\) outcome unit, and a difference in means of \(\delta=1.5\) units under the alternative. For a two-sided test at \(\alpha=0.05\), \(df=4+4-2=6\), and
The exact critical value is \(t_{0.975,6}\approx2.447\). Therefore the exact power is \(P(T_{6,2.121}<-2.447)+P(T_{6,2.121}>2.447)\), where \(T_{6,2.121}\) denotes a noncentral t random variable with 6 degrees of freedom and noncentrality 2.121. This expression, evaluated from the noncentral t distribution, is the finite-sample power for the stated model and test.
A large-sample normal calculation substitutes the standard normal cutoff 1.96 and approximates power by \(\Phi(2.121-1.96)+\Phi(-2.121-1.96)\approx0.564\). That number is an approximation, not the exact t-test power: with only six degrees of freedom, estimating the common variance contributes appreciable uncertainty. The model-based noncentral-t calculation is the appropriate benchmark here; a normal approximation should not be presented as exact.
How to Compare Procedures Responsibly
For small trials, report the test, its sidedness, the nominal level, and—when possible—its actual size under the null. Then report power under clearly specified alternatives. Exact enumeration is especially transparent for binary outcomes because both quantities are sums over a finite set of count pairs. For continuous normal outcomes, use the relevant finite-sample t distribution rather than substituting a standard normal distribution without qualification.
| Approach | Strength in small samples | Important limitation |
|---|---|---|
| Exact enumeration for a specified rejection rule | Computes size and power under the stated outcome model without a large-sample approximation | Results depend on the chosen rule and assumed probabilities |
| Fisher’s exact test for a binary outcome | Controls conditional Type I error at or below the target level | May be conservative, leaving actual size and power below their nominal targets |
| Score or normal approximation | Often convenient for larger samples and quick planning | Estimated variances and sparse counts can produce poor finite-sample calibration |
| Noncentral t calculation for a pooled continuous-outcome test | Uses the finite-sample distribution under normal, equal-variance assumptions | Can be inappropriate if normality, independence, or equal variances is not credible |
A useful diagnostic is to check whether the nominal level is close to the actual rejection probability under the null. If it is not, a power comparison at the same nominal \(\alpha\) may be misleading: one procedure may appear more powerful partly because it rejects more often when the null is true. Another diagnostic is sensitivity to the effect and baseline probabilities. With few observations, modest changes in these assumptions can move probability mass across a rejection boundary and substantially change power.
No single exact procedure is uniformly best. Conditional exact tests can sacrifice power through conservatism; approximate tests can be miscalibrated; and all model-based calculations can fail when the assumed outcome distribution is wrong. The practical response is not to choose the procedure with the largest reported power in isolation, but to compare tests at defensible Type I error rates, under the same trial conditions, and with assumptions that fit the planned analysis.
Check Your Understanding
Use the stated assumptions in each question. Solutions follow the questions.
- In the three-per-arm pooled score example, what is the null rejection probability, and why is it not a valid 0.05-size test at the stated null value?
- For the four-per-arm rule \(D\geq3\), why must the normal approximation use a boundary of 2.5 rather than 3?
- For the continuous-outcome example, calculate \(df\) and the noncentrality parameter from the given values.
- Why can Fisher’s exact test have actual size below its nominal level in a very small trial?
Solutions. (1) The rejection-pair probabilities sum to \(1/64+3/64+3/64=7/64=0.1094\), which exceeds 0.05; the nominal cutoff therefore does not control the actual Type I error at that null value. (2) Since \(D\) is integer-valued, \(D\geq3\) corresponds to a continuous approximation exceeding 2.5, the midpoint between 2 and 3. (3) \(df=4+4-2=6\), and \(\lambda=1.5/\sqrt{1/4+1/4}=2.121\). (4) The attainable conditional tail probabilities are discrete. A nonrandomized rejection region may have to omit outcomes whose inclusion would exceed the target level, leaving the actual rejection probability below that level.