Fisher Test Power Across Alternative Patterns

Why the Pattern of Evidence Matters

Fisher’s combination test reduces stagewise p-values to a single statistic, \(T=-2\sum_i\log(P_i)\). For two stages, the observed statistic depends on the product \(P_1P_2\): a pair of moderately small p-values can produce the same statistic as a pair in which one p-value is very small and the other is larger. But the power of the test is not determined by one observed pair. It is the probability, under a specified alternative, that the random stagewise p-values will produce a statistic beyond the rejection threshold.

This distinction matters in adaptive trials because treatment effects, information, and resulting stagewise evidence may not be distributed evenly across stages. We will compare alternatives in which evidence is expected to arise moderately at both stages with alternatives in which it is concentrated in one. The comparison requires an explicit model for the stagewise p-values under the alternative; there is no universal power ordering that follows just from the words “moderate” and “concentrated.”

Test and Power Under a Specified Alternative

The reference distribution requires null-uniform stagewise p-values and their independence under the null. For power, we additionally need a specified joint distribution under the alternative. Independence under the alternative is a common working assumption when the stages use independent data and the stagewise p-values are determined separately. If the stages share data, or the design induces dependence between their statistics, the joint distribution must reflect that dependence. Using only the two marginal alternative distributions is not enough to determine power when \(P_1\) and \(P_2\) are dependent.

A useful illustrative model gives each stage a beta-distributed p-value: $$ P_i\sim\operatorname{Beta}(a_i,1),\qquad f_i(p)=a_i p^{a_i-1},\quad 0<p<1. $$ For \(a_i=1\), the p-value is uniform, as under a continuous null. For \(0<a_i<1\), the distribution puts more mass near zero, representing an alternative that tends to produce smaller p-values. This is a simple model for comparing patterns, not a general law for p-values under treatment effects.

Deriving Power for the Beta Model

Set \(Y_i=-2\log(P_i)\). If \(P_i\sim\operatorname{Beta}(a_i,1)\), then for \(y\geq0\), $$ P(Y_i>y)=P(P_i<e^{-y/2})=(e^{-y/2})^{a_i}=e^{-a_i y/2}. $$ Thus \(Y_i\) has an exponential distribution with rate \(\lambda_i=a_i/2\). Since \(T=Y_1+Y_2\), its alternative distribution is the distribution of a sum of two independent exponential random variables.

When both stages have the same \(a\), the sum has a gamma distribution with shape 2 and rate \(a/2\). Its survival probability gives the power directly. When the rates differ, integrating the density of the sum gives the following expression. For \(t\geq0\), the density is $$ f_T(t)=\int_0^t \lambda_1e^{-\lambda_1 y}\lambda_2e^{-\lambda_2(t-y)}\,dy =\frac{\lambda_1\lambda_2}{\lambda_2-\lambda_1} \left(e^{-\lambda_1t}-e^{-\lambda_2t}\right), $$ when \(\lambda_1\ne\lambda_2\). Integrating this density from \(t=c_\alpha\) to infinity yields the unequal-rate power formula: $$ P(T>c_\alpha\mid H_a) =\frac{\lambda_2e^{-\lambda_1c_\alpha}-\lambda_1e^{-\lambda_2c_\alpha}} {\lambda_2-\lambda_1}. $$ When the rates are equal, the limiting expression is the equal-rate gamma survival probability: $$ P(T>c_\alpha\mid H_a) =e^{-ac_\alpha/2}\left(1+\frac{ac_\alpha}{2}\right). $$ These formulas make the comparison precise for this model: a pattern changes the distribution of \(T\), not just its expected value.

Interpreting and Checking a Power Comparison

The examples illustrate why matching the mean of \(T\) does not match its tail probability beyond \(c_\alpha\). Rejection depends on the probability of crossing a particular threshold, so the full distribution of \(T\) under the alternative matters. Two alternatives with the same \(E(T\mid H_a)\) can have different variability and different upper tails. A concentrated pattern can also have greater or lower power under other choices of stagewise distributions, critical values, or dependence.

For alternatives that do not yield a tractable formula, power can be estimated by numerical integration or simulation. A simulation should draw a pair \((P_1,P_2)\) from the specified joint alternative distribution, calculate \(T\), and record whether \(T>c_\alpha\). The estimated power is the fraction of simulated pairs that reject. Its Monte Carlo standard error for \(B\) independent simulated pairs and estimated power \(\hat{\pi}\) is approximately \(\sqrt{\hat{\pi}(1-\hat{\pi})/B}\). More simulation draws reduce this numerical uncertainty, but they do not repair a poorly chosen alternative model.

Useful diagnostics include checking whether the proposed alternative distributions resemble stagewise p-value behavior implied by the planned effect sizes and stage information; checking simulated dependence rather than assuming independence automatically; and verifying that the null calibration of the test matches the p-value type and design. Discrete p-values may not be continuously uniform under the null, and dependence can invalidate the usual chi-square reference distribution. In those cases, the nominal critical value may not give the intended type I error rate, so a calibrated reference distribution or a direct design-based calculation is needed.

How Fisher Power Compares With Other Combination Approaches

Fisher’s method combines p-values through their product and is sensitive to small p-values. Inverse-normal combination tests instead transform stagewise p-values to normal scores and combine those scores using specified weights. Those approaches can have different power because they use different functions of the stagewise evidence and may encode different design priorities. Neither method is uniformly more powerful across all alternatives: comparison requires a common null calibration, a specified alternative model, and calculation of each test’s rejection probability under that model.

A useful design comparison therefore reports more than the expected contribution at each stage. It should state the stagewise alternative distributions or the underlying data model, the dependence assumptions, the significance level and critical value, and the resulting power. Sensitivity analyses across plausible patterns are important when it is uncertain whether treatment effects or information will be evenly distributed across stages.

Common Mistakes and Limitations

• Confusing an observed statistic with power. Two observed p-value pairs with the same product give the same Fisher statistic. Power instead averages the rejection indicator over all possible stagewise p-value pairs under an alternative.
• Comparing alternatives only by mean evidence. Equal expected values of \(T\) do not imply equal probabilities of exceeding the critical value. Compare tail probabilities, not just means.
• Treating a beta p-value model as universal. The beta model is a convenient illustration. A realistic clinical-trial power analysis should derive or simulate p-values from a defensible model for outcomes, treatment effects, information, and the design.
• Assuming independence without justification. The chi-square reference and the example power formulas rely on independent stagewise p-values. If that condition fails, the relevant joint distribution must be used and the test’s null calibration checked.
• Assuming the concentrated pattern must lose power. The numerical ordering in the worked examples is specific to their model and threshold. Different alternative distributions can reverse the ordering.