Combining Discovery and Replication P-Values

Why Combine Discovery and Replication Evidence?

A discovery cohort may identify a genetic association, while an independent replication cohort tests whether the finding is supported in new data. Combining the two p-values can summarize evidence across cohorts, but it answers a different question from asking whether the replication result itself meets a prespecified criterion. A combined result can be small even when the replication p-value is not.

The distinction matters because a p-value combination method usually tests a global null: the hypothesis that none of the component studies contains an association. Rejecting that null does not show that both studies support the association, that their effects have the same direction, or that the replication effect meets a specified size threshold. The earlier material on replication criteria and replication failure distinguishes these questions; p-value combination does not replace those criteria.

Assumptions Before Calculation

For the standard Fisher and Stouffer reference distributions, each component p-value must be valid under its own null: under that null, it is uniformly distributed on \([0,1]\) for a continuous test, or at least super-uniform for a conservative or discrete test. The component tests must also be independent under the joint null for the usual independent-study formulae to have their stated calibration. Independent cohorts make this assumption more plausible, but independence can still fail if participants overlap, data are shared, or study results are otherwise dependent.

The studies should address compatible association questions. For example, a p-value for one variant and outcome should not be combined as though it tested the same question as a p-value for a different variant or phenotype. Alleles, effect direction, and outcome definitions need to be aligned when direction is used. Fisher’s method uses p-value magnitudes and does not require aligned directions; that feature is also a limitation. Stouffer’s signed version requires a consistent definition of positive direction.

A further concern is selection. If a discovery result was chosen because its p-value was unusually small among many tested variants, that selected p-value is not distributed like an unselected uniform p-value, conditional on selection. Applying an ordinary combination reference distribution to that selected discovery p-value can therefore overstate combined evidence. An independent, prespecified replication p-value remains directly informative for its replication test; the discovery selection process does not make the replication result significant or nonsignificant by itself.

Fisher’s Method

Fisher’s method combines p-value magnitudes. For \(k\) independent, continuous, valid p-values under the joint null, define the statistic \(X_F\) as follows.

Under the joint null, \(X_F\) has a chi-square distribution with \(2k\) degrees of freedom. Thus, the combined p-value is the upper-tail probability \(P(\chi^2_{2k}\geq X_F)\). Small component p-values contribute larger positive values to \(X_F\), making the combined result more significant.

The reference distribution follows directly from the uniform null distribution. If \(U\sim\text{Uniform}(0,1)\), then for \(t\geq0\),

This is the cumulative distribution function of a chi-square random variable with 2 degrees of freedom. Each independent term \(-2\ln(p_i)\) therefore has a \(\chi^2_2\) distribution; their sum has a \(\chi^2_{2k}\) distribution. Independence is essential to this sum result.

Fisher’s method is sensitive to very small p-values and treats all p-values as evidence in the same direction. It can be useful when the question is whether there is evidence against the global null in at least one study. It is not a directional consistency test. Even a very small Fisher p-value cannot establish that discovery and replication effects agree.

Stouffer’s Method and Direction

Stouffer’s method transforms p-values to standard-normal scores and combines those scores. For two-sided p-values, the unsigned transformation is \(z_i=\Phi^{-1}(1-p_i/2)\), where \(\Phi\) is the standard normal cumulative distribution function. This score is nonnegative and measures the magnitude of evidence, but it discards effect direction.

When effects have a meaningful, harmonized direction, use a signed score. Define \(s_i=1\) if the estimated effect in study \(i\) is in the prespecified positive direction, and \(s_i=-1\) if it is in the opposite direction. The signed score is \(z_i=s_i\Phi^{-1}(1-p_i/2)\). For two independent studies with equal weights, the Stouffer statistic is

Under the joint null, each signed score is standard normal when the underlying tests have the appropriate continuous null calibration and the signs are defined consistently. The sum of two independent standard normal scores has variance 2, so division by \(\sqrt{2}\) gives a standard normal statistic. A two-sided combined p-value is \(2[1-\Phi(|Z_S|)]\); for a prespecified one-direction alternative, the corresponding one-sided tail is used instead.

More generally, fixed weights \(w_i\) give \(Z_S=(\sum_i w_i z_i)/\sqrt{\sum_i w_i^2}\). Under independent standard normal scores, the numerator has mean zero and variance \(\sum_i w_i^2\) under the joint null, so the normalized statistic is standard normal. Weights and the direction of the test should be chosen before examining the results. Sample-size-based weights are not automatically appropriate for every test; the scores must have a compatible interpretation for the weighting scheme to represent the intended evidence.

What a Combined Result Establishes

A correctly calibrated combined p-value is evidence against the joint null that all component tests are null. The evidence can arise from one study, several studies, or a mixture, depending on the method and the observed results. It does not report how much evidence came from each cohort. Always show the component p-values and, when direction matters, the cohort-specific effect estimates and directions alongside the combined result.

For a claim specifically about replication, the replication test and its prespecified success rule remain central. For example, if replication was defined as a one-sided test in the discovery direction at a specified \(\alpha\), the replication p-value must be evaluated against that criterion. If success instead requires an effect at least as large as a meaningful threshold, a combined p-value does not answer that effect-size question. A small combined p-value cannot retroactively change the replication rule.

Fisher and Stouffer also differ in what they retain. Fisher uses p-value magnitudes and ignores signs. Stouffer can combine magnitudes without signs, or can use signed scores to evaluate evidence in a common direction. The signed version can lose evidence when estimates point in opposite directions, but that is appropriate if the target claim requires a shared direction. Neither method estimates a common effect size or quantifies between-study heterogeneity.

These methods are not replacements for methods that combine effect estimates and their sampling uncertainties. Such approaches address estimation questions that a combined p-value does not. Likewise, if cohorts are dependent, the independent-study calibration cannot simply be assumed; the covariance among contributions must be accounted for by an appropriate method.

Diagnostics and Common Pitfalls

Before reporting a combined result, check that the tests concern the same prespecified association, that p-values are valid for their stated tests, and that the independence assumption is plausible. For signed Stouffer combination, verify allele coding and effect direction, and state whether the combined test is one-sided or two-sided. Inspect the component p-values and estimates rather than presenting only the combined number. A dramatic combined result paired with a weak or opposite-direction replication result is a signal to be precise about what has and has not been established.

• Wrong reasoning: “The Fisher p-value is below 0.05, so the association replicated.” Correction: Fisher rejects the joint null under its assumptions; it does not require the replication p-value to pass a replication threshold.
• Wrong reasoning: “A non-significant replication p-value proves no replication effect exists.” Correction: Failure to reject is not proof of a zero effect; precision and power affect the result.
• Wrong reasoning: “The two studies are independent, so their p-values can always be combined with the ordinary formula.” Correction: Independence is only one requirement. P-values must also be valid and the tests must address compatible hypotheses; selection and dependence among tests can invalidate calibration.
• Wrong reasoning: “Fisher’s method confirms that both effects have the same sign.” Correction: Fisher’s statistic uses p-value magnitudes only. Use harmonized signed evidence if direction is part of the scientific claim.