Why Dependence Matters
A combination test turns stagewise evidence into a single test statistic. The familiar reference distributions for Fisher's method and the inverse-normal method rely on a particular feature of the stagewise results: under the null hypothesis, their p-values must be independent, as well as individually valid. If participants contribute data to more than one stage, or if stages share a control group or other data, that independence can fail. Applying the usual reference distribution anyway can give the combined test the wrong type I error rate.
The issue is not that a p-value is inherently unusable when it is dependent on another p-value. Rather, the issue is using a reference distribution derived under independence when the joint null distribution is different. The relevant dependence condition concerns the stagewise p-values under the null being tested. Seeing different stage labels, different analysis times, or separately calculated p-values does not establish that condition.
For the standard exact reference distributions, the p-values must also be continuous and uniform under the null. A valid but conservative p-value, such as a discrete p-value, need not be exactly uniform. In that case, independence alone does not guarantee the usual exact reference distribution; the test's calibration must also account for the marginal distributions. The focus here is the additional problem caused by dependence.
Where the Standard Reference Distributions Come From
For \(K\) independent, continuous, uniform null p-values \(p_1,\ldots,p_K\), Fisher's statistic is \(F=-2\sum_{i=1}^K\log(p_i)\). Since \(-2\log(p_i)\) has a chi-square distribution with 2 degrees of freedom, their independent sum has a chi-square distribution with \(2K\) degrees of freedom.
For the inverse-normal method, define \(Z_i=\Phi^{-1}(1-p_i)\) for one-sided p-values, where \(\Phi\) is the standard normal cumulative distribution function. Under the null, each \(Z_i\) is standard normal if \(p_i\) is continuous uniform. With fixed weights \(w_i\) satisfying \(\sum_{i=1}^K w_i^2=1\), the combination statistic is \(Z_{\mathrm{comb}}=\sum_{i=1}^K w_iZ_i\). Independence makes its variance equal to one, so its null distribution is standard normal.
This variance expression follows by expanding the variance of a sum: each \(w_iZ_i\) contributes \(w_i^2\operatorname{Var}(Z_i)\), and each pair contributes twice its covariance. Under independence, the covariances are zero and the weights' squared sum is one. Under dependence, nonzero covariances can change the variance; the standard normal reference is then not justified by this argument. For more than two stages, knowing pairwise covariances may still be insufficient to determine the full distribution of the sum unless additional joint-distribution assumptions, such as joint normality, hold.
Worked Example: Identical P-Values Break Fisher Calibration
Suppose two stagewise analyses use exactly the same participants and produce identical p-values. Under the null, assume this shared p-value \(U\) is continuous uniform on \((0,1)\), so \(p_1=p_2=U\). Fisher's usual two-p-value test compares \(F=-2(\log p_1+\log p_2)\) with a chi-square distribution with 4 degrees of freedom. At a nominal 5% level, that reference distribution gives a critical value of approximately 9.488.
With identical p-values, \(F=-4\log U\). The probability of exceeding the usual critical value is therefore \(P(-4\log U>9.488)=P(U<\exp(-9.488/4))\).
Thus, under this hypothetical dependence structure, the nominal 5% test rejects about 9.32% of the time under the null. The chi-square reference is wrong because it treats the two identical contributions as independent. The result does not imply that Fisher's method is unusable for dependent p-values; it shows that the standard chi-square calibration is not valid for this dependence pattern.
Participant Overlap and Shared Data
Overlap is one direct route to dependence. For example, two stagewise treatment-effect estimates may both include observations from the same participants. A shared control group can also contribute to both comparisons, even when the treatment groups are distinct. Other sources include reused interim data, a common estimated nuisance parameter, or a shared source of random variation. Conversely, overlap does not by itself specify the magnitude or sign of dependence: that depends on the estimands, analysis, and data-generating model.
A simple model makes the overlap mechanism explicit. Suppose each stage's statistic is a standardized sum of \(n\) independent, mean-zero normal participant outcomes with variance \(\sigma^2\), and the stages share exactly \(m\) of those outcomes. Each statistic has variance one. Their covariance is the contribution of the shared outcomes, \(m\sigma^2\), divided by the two standard deviations, each \(\sigma\sqrt n\), so their correlation is \(m/n\). This calculation depends on the stated equal-size, equal-variance, normal-sum setup; it is not a universal formula for every trial analysis.
Worked Example: How Overlap Alters Inverse-Normal Size
Consider two hypothetical stages, each with 100 participant outcomes in a standardized normal-sum statistic. Forty outcomes are shared, and all outcomes are independent with the same variance except for that reuse. Under the null, the model above gives correlation \(40/100=0.40\) between the stagewise statistics. Because the inverse-normal transform is monotone, the corresponding one-sided transformed statistics also have this correlation in this bivariate normal setup.
Use fixed inverse-normal weights \(w_1=\sqrt{0.40}=0.6325\) and \(w_2=\sqrt{0.60}=0.7746\), so \(w_1^2+w_2^2=1\). The null variance is \(1+2w_1w_2(0.40)=1+2(0.6325)(0.7746)(0.40)=1.3919\), not 1. At the usual one-sided 2.5% critical value \(1.96\), the actual null rejection probability in this model is \(1-\Phi(1.96/\sqrt{1.3919})=1-\Phi(1.661)\approx0.0483\).
So a procedure using the standard-normal critical value would reject about 4.83% of the time in this hypothetical model, rather than 2.5%. The discrepancy arises even though each stagewise statistic is individually standard normal under the null and the weights were fixed in advance. Marginal validity and prespecified weights do not repair a dependence mismatch.
Checking a Combined Result
The practical question is not simply whether any participants appear more than once. It is whether the joint null distribution of the inputs matches the distribution assumed by the chosen combination reference. A useful review traces which observations and estimated quantities feed into every stagewise statistic, then asks how those shared components behave under the null. If a joint model is available, it can quantify the dependence. A design description alone may reveal a concern but not its exact calibration consequence.
Record which participants, control observations, interim outcomes, and nuisance estimates contribute to each stagewise p-value.
Specify the null hypothesis and the marginal p-value distributions on which the combination reference relies.
Determine whether the stagewise p-values are mutually independent under that null, or calculate the joint distribution using justified model assumptions.
Use the standard Fisher or inverse-normal reference only when its assumptions are met. Otherwise, use a justified calibration for the actual joint distribution or redesign the inputs to avoid the problematic dependence.
Worked Example: Recalibrating an Observed Inverse-Normal Statistic
Suppose two hypothetical one-sided stagewise p-values are \(p_1=0.040\) and \(p_2=0.080\), with fixed weights \(w_1=0.60\) and \(w_2=0.80\). The squared weights sum to one. The transformed values are \(Z_1=\Phi^{-1}(0.960)=1.751\) and \(Z_2=\Phi^{-1}(0.920)=1.405\), giving \(Z_{\mathrm{comb}}=(0.60)(1.751)+(0.80)(1.405)=2.175\).
If the stagewise transformed statistics are independent under the null, the one-sided combined p-value is \(1-\Phi(2.175)\approx0.0148\). Now suppose the null model instead gives their correlation as 0.50 and their joint distribution is bivariate normal. The combination statistic then has variance \(1+2(0.60)(0.80)(0.50)=1.48\).
Under the stated correlated normal model, the relevant upper-tail p-value is approximately 0.0369, not 0.0148. Both calculations use the same observed statistic; they differ because they use different null reference distributions. The correlated calculation is justified here only because the example specifies the bivariate normal null model and its correlation.
Alternatives, Diagnostics, and Limitations
There are several possible responses to dependence, and they answer different design needs. Using disjoint stagewise participant sets may support independence when the data-generating assumptions also make those sets independent. A shared-control or repeated-data design may be retained, but then the joint null distribution must be characterized and the combination test calibrated to it. Simulation under a fully specified null model can help assess calibration, provided the simulated design reproduces overlap, adaptation, and the stagewise analyses. A conservative method that does not require independence may also be considered, but it may sacrifice power and must be matched to the testing objective.
Diagnostics should focus on the source and consequences of dependence, not just on a sample correlation observed in one trial. Under the null, model-based calculations or carefully designed simulations can assess the joint distribution and the resulting rejection rate. In a Gaussian setting, covariance can determine the distribution of a linear inverse-normal combination; outside that setting, covariance alone may not suffice. For Fisher's statistic, the joint distribution of the p-values matters, and a correlation summary is not generally enough to recover its null distribution.
Dependence also affects power and interpretation: two similar-looking p-values may represent less independent evidence than two results from genuinely separate data. But a dependence adjustment cannot be chosen from the observed p-values alone. It requires a defensible account of the design and the null distribution. The distinction between unconditional independence and conditional validity is also important: a future stagewise p-value can have a specified conditional null distribution given earlier data without being independent in the way assumed by an unconditional combination reference.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- What joint-distribution condition supports the usual Fisher chi-square reference when the stagewise p-values are continuous and uniform under the null?
- Two stages use disjoint participant groups but reuse the same estimated variance from the pooled data. Does disjoint membership alone establish independence? Explain.
- For two standard-normal null statistics with correlation \(0.25\), combined with equal inverse-normal weights \(w_1=w_2=1/\sqrt{2}\), calculate the variance of their combination.
- Why can pairwise independence be insufficient when a combination test has three or more stages?
Solutions. (1) The p-values must be mutually independent under the null, in addition to being continuous uniform for the standard exact Fisher reference. (2) No. The shared estimated variance can link the stagewise p-values; its effect on their joint null distribution must be assessed. (3) The variance is \(1+2(1/\sqrt{2})(1/\sqrt{2})(0.25)=1.25\). (4) Pairwise independence does not in general determine the full joint distribution of three or more variables, while the combination statistic's null distribution can depend on that joint distribution.