Worked Example: Combining Two Stage P-Values

Why Combine Stagewise P-Values?

A two-stage trial produces evidence at two points in time. A combination test uses each stage’s p-value to form a prespecified final test, rather than treating the two p-values as if they could simply be averaged or compared separately with the significance level. The choice of combination function matters: inverse normal and Fisher tests can give different final p-values from the same stagewise evidence, while each can control the type I error rate under its own stated conditions.

This worked tutorial uses two stages, independent stagewise p-values, and combination rules specified before the results are observed. The examples use one-sided p-values in a common direction, so smaller p-values consistently represent evidence against the null in that direction. This is an important condition for interpreting the inverse normal result as a directional test.

Rules and Conditions for the Two Procedures

For the equal-weight inverse normal test, transform each one-sided stagewise p-value into a normal score and average the scores with weights \(1/\sqrt{2}\). Here \(\Phi\) is the standard normal cumulative distribution function and \(\Phi^{-1}\) is its quantile function.

For Fisher’s two-stage test, multiply the stagewise p-values inside a logarithmic statistic. With independent, continuous uniform p-values under the null, the reference distribution has four degrees of freedom.

The assumptions matter. For these reference distributions to be exact, each stagewise p-value must be valid under the null, and the two p-values must be independent under the null. In the examples, they are treated as continuous and uniform under \(H_0\). Stage independence is a design and analysis property to establish, not something that can be verified merely by looking at the two observed p-values. The equal inverse normal weights and Fisher rule must also be chosen in advance, not selected after comparing their results.

The reference distributions have short justifications. If \(p_i\) is uniform on \((0,1)\) under \(H_0\), then \(1-p_i\) is also uniform, and applying \(\Phi^{-1}\) gives a standard normal score. Independence of the two scores then makes their sum divided by \(\sqrt{2}\) standard normal. For Fisher’s rule, \(-2\ln(p_i)\) has a chi-square distribution with 2 degrees of freedom under a uniform null p-value; independence makes the sum of the two such terms chi-square with 4 degrees of freedom. These facts explain why each procedure has a calibrated null reference distribution under the stated assumptions.

Different Combination Rules Can Give Different Decisions

Both tests are calibrated to their own null distributions. They do not, however, give the same numerical summary of stagewise evidence. The inverse normal test combines transformed p-values, while Fisher’s test is driven by the logarithm of their product. It is therefore possible for them to fall on opposite sides of the same significance threshold.

How to Check a Calculation and Interpret Its Result

A careful calculation proceeds from the design assumptions to the final decision, rather than from a desired conclusion back to a chosen test. Keep enough precision in the stagewise p-values and intermediate calculations; round the reported results only after computing the final statistic and p-value. Small rounding differences can matter when a p-value is near \(\alpha\).

These procedures answer a testing question: whether the stagewise evidence, combined by a specified rule, is sufficiently inconsistent with \(H_0\). An overall p-value is not the probability that \(H_0\) is true, and a non-rejection does not establish that there is no treatment effect. Neither combination statistic is, by itself, a pooled estimate of treatment effect or a confidence interval.

Diagnostics, Alternatives, and Limitations

The central diagnostic is whether the assumptions and the prespecified analysis match the trial actually conducted. Independence can be compromised if stages reuse participants or observations, share data in a way that induces dependence, or use stagewise tests whose inputs overlap. A p-value that is not valid under the null also invalidates the stated reference calculation. In either case, simply applying the standard normal or chi-square reference distribution does not repair the problem.

The inverse normal formula here requires a consistent one-sided direction: a small stagewise p-value must correspond to evidence in the same favorable direction at both stages. If stagewise p-values are two-sided, or if effects point in different directions, this calculation cannot be interpreted as a simple directional combination without an explicitly specified rule for handling direction. Fisher’s statistic is based on p-value magnitude and does not encode the sign of an effect; it can combine small p-values even when effects point in opposite directions.

The main alternative in this tutorial is not “combine” versus “do not combine,” but one prespecified combination function versus another. Inverse normal expresses evidence on a normal-score scale and, in this example, uses equal fixed weights. Fisher’s rule aggregates through the product of p-values. Other valid combination rules may be appropriate for other design goals, but they require their own prespecified calibration. A pooled-data test is a different analysis and should not be assumed equivalent to either combination test.

A common mistake is to report whichever method gives the smaller p-value and describe it as the trial’s final test. That procedure makes the test depend on the observed data and can inflate the type I error rate unless the selection is itself accounted for in the design and calibration. The correction is to define the primary combination rule and its rejection threshold in advance. If alternative combination results are also reported, distinguish them clearly from the prespecified primary decision.