Why Different Stagewise Tests Can Be Combined
Across stages of a clinical trial, the endpoint or the appropriate analysis may differ. A binary outcome such as response can be tested with an exact binomial procedure, while a continuous outcome such as change in a symptom score may be tested with a \(t\)-test. Their raw statistics are not on a common scale: a count, a \(t\)-statistic, and a standardized normal statistic do not have a directly meaningful sum. Their calibrated p-values, however, are on a common probability scale. A combination test can use those p-values to test a global null hypothesis across stages.
This does not make the stagewise questions identical. The combined test addresses whether the planned evidence across stages is inconsistent with the specified global null. It does not, by itself, estimate one shared treatment effect, establish that every endpoint improved, or show that effects measured on different scales are clinically equivalent.
For a two-stage Fisher combination test, let \(P_1\) and \(P_2\) be valid p-values for the pre-specified tests at stages 1 and 2. The Fisher statistic is
Small stagewise p-values make \(F\) large. If, under the global null, the two p-values are independent and each is continuous uniform on \((0,1)\), then \(F\) has a chi-square distribution with 4 degrees of freedom. The combined p-value is the probability, under that reference distribution, of a value at least as large as the observed \(F\).
Why the Reference Distribution Works
The calibration relies on the p-values, not on the stagewise statistics having the same distribution or units. To see the result, let \(U\) be uniform on \((0,1)\) and define \(Y=-2\log(U)\). For \(y\geq 0\),
This is the cumulative distribution function of a chi-square random variable with 2 degrees of freedom. Thus, for two independent uniform p-values, each term \(-2\log(P_i)\) has a chi-square distribution with 2 degrees of freedom. Independence means their sum has a chi-square distribution with the degrees of freedom added:
For two stages, the upper-tail probability can also be calculated directly. If \(F_{\mathrm{obs}}\) is the observed statistic, then
If the independent p-values are valid but discrete and conservative rather than exactly uniform, the usual chi-square calibration is conservative under the global null: replacing a uniform p-value by an independent p-value that is stochastically no smaller cannot increase the chance of an excessively large Fisher statistic. This conclusion still depends on independence and validity; discreteness does not repair invalid p-values.
Worked Example: Exact Binary Test and Continuous \(t\)-Test
Worked Example: Exact Binary Test and Continuous \(t\)-Test
Hypothetical trial. At stage 1, ten participants receive an intervention and the binary outcome is success or no success. Under the stage 1 null, each participant has success probability \(0.5\), and the planned one-sided alternative is a higher success probability. Six successes are observed. At stage 2, a separate group contributes a continuous symptom-change outcome. The planned one-sample \(t\)-test is for a positive mean change relative to zero; its sample size is \(n=16\), sample mean is \(\bar{x}=2.40\) points, and sample standard deviation is \(s=4.00\) points. Assume the stage 2 observations are independent and normally distributed under the model. The two stages use distinct participants, and the tests and combination rule were specified before examining outcomes.
Stage 1 p-value. Under the binary null, \(X\sim\operatorname{Binomial}(10,0.5)\). The exact one-sided p-value is the probability of observing at least six successes:
This exact calculation is valid for the stated binomial null. Because the binomial distribution is discrete, the p-value is not generally continuously uniform under the null, but it is valid in the sense that it does not exceed the nominal tail probability too often.
Stage 2 p-value. Under the one-sample normal model, the standard error is \(SE=s/\sqrt{n}=4.00/\sqrt{16}=1.00\) point. The test statistic and degrees of freedom are
The one-sided \(t\)-tail probability is \(P(T_{15}\geq 2.40)\approx 0.0150\), so \(P_2\approx0.0150\). The statistic is calibrated using the \(t\) distribution because the observations are assumed independent and normal, and the population standard deviation is estimated by \(s\).
Combination. Use the two stagewise p-values, not the success count and \(t\)-statistic themselves:
With two independent stages and valid p-values under the stated global null, the reference distribution is \(\chi^2_4\). Therefore,
At significance level \(\alpha=0.05\), this rejects the global null that both specified stagewise nulls hold. The evidence is contributed mainly by the continuous-outcome stage, whose p-value is smaller. The result is not evidence that both outcomes improved, nor does it supply an effect estimate in a common unit.
Assumptions and Design Checks
The reference calculation is justified only if the stagewise inputs have the properties required by the combination rule. Before using it, check the following conditions.
- Each p-value is valid for its stated stagewise null. The exact test must use the correct null distribution and sampling scheme. An asymptotic test must have an adequate approximation for its statistic and sample size. The null and tail must match the pre-specified scientific question.
- The combination rule is pre-specified. Stagewise tests, endpoints, p-value tails, and the combination method should be fixed before results are inspected. Choosing whichever test or endpoint gives the smallest p-value can invalidate the advertised calibration.
- The p-values are independent under the global null, or the reference distribution is otherwise justified. Separate, non-overlapping groups recruited independently can support independence when the data-generating processes and analyses are also independent under the null. Shared participants, shared controls, or correlated outcomes may violate it. Independence should not be inferred merely because the stages have different names.
- The global null is stated carefully. When stages test different endpoints, a natural global null is that every stagewise null holds. Rejection indicates evidence against that joint null; it does not establish a single common estimand or prove a benefit on each endpoint.
Useful diagnostics include checking that every stage used the planned analysis population and endpoint definition; verifying the exact or reference distribution used for each stagewise p-value; and confirming that independence is credible from the design, not just assumed after looking at results. Also inspect whether a p-value was rounded or reported only as “less than” a threshold. Since \(\log(P_i)\) is used, a recorded zero cannot be entered as if it were the true p-value; the underlying tail probability or a justified bound is needed.
Worked Example: Combining an Exact Test and a Large-Sample Test
Hypothetical trial. In a second two-stage design, stage 1 observes 8 successes among 12 participants. Under a binomial null success probability of \(0.5\), the planned one-sided exact test gives
At stage 2, an independent continuous-outcome comparison has a pre-specified large-sample normal statistic \(z=2.05\) for a positive mean difference. Suppose the large-sample approximation is justified for this test. Its one-sided p-value is \(P_2=1-\Phi(2.05)\approx0.02018\), where \(\Phi\) is the standard normal cumulative distribution function. The two tests use different statistics and different reference distributions, but each has been converted to a p-value for its planned alternative.
The product is \(P_1P_2=(0.193848)(0.02018)\approx0.003912\). Thus,
Provided the stages are independent and the p-values are valid under the joint null, the combined result is significant at the 0.05 level. The exact binary calculation and the normal approximation are not being averaged or pooled; their p-values are the inputs to a single calibrated test.
What the Combination Does—and Does Not Do
Fisher’s method is useful when stagewise analyses answer different but pre-specified questions and a single omnibus decision is wanted. It permits the binary stage to use a discrete exact test and the continuous stage to use a parametric or asymptotic test without requiring their raw statistics to share a scale. The method rewards a small p-value strongly, so a particularly informative stage can drive the combined result even if another stage is unpersuasive.
That feature also limits the interpretation. The combined p-value is not a weighted average of effect sizes, and it does not say how large a treatment effect is. Where endpoints differ, report each stage’s estimate and uncertainty alongside the global test. If the scientific claim requires success on every endpoint rather than evidence against the joint null, Fisher’s global test alone does not answer that question.
| Approach | What is combined | Key condition or limitation |
|---|---|---|
| Fisher combination | Stagewise p-values through \(-2\sum_i\log(P_i)\) | Reference calibration requires independent, valid p-values or another justified calibration |
| Pooling raw observations | Participant-level outcomes in a common analysis | Requires a meaningful common endpoint and a model that accounts for the design; different endpoint scales cannot simply be pooled |
| Inverse normal combination | Normal quantiles of stagewise p-values | Uses a different combination statistic and weighting choices; it still requires valid stagewise inputs and justified dependence assumptions |
These methods answer related but not identical design questions. If stages have different endpoints, pooling raw observations is generally not a direct alternative to an omnibus p-value combination. An inverse normal procedure also works with p-values, but its statistic and any weights must be chosen and calibrated appropriately. The choice of combination method should follow the intended global claim, not be selected after comparing which result is most favorable.
Common Mistake
Wrong reasoning: “The binary test produces a count and the continuous test produces a \(t\)-statistic, so I will add the count and the \(t\)-statistic, or average them after standardizing by their observed values.”
Correction: Raw statistics have different scales and reference distributions, so such an operation has no general null calibration. First obtain a valid p-value from each pre-specified stagewise test. Then apply the chosen combination function to those p-values and use its reference distribution only when its assumptions—especially independence here—are met.
Check Your Understanding
Answer each question before reading the solutions.
- Why is it generally inappropriate to add a binary-test statistic directly to a continuous-outcome \(t\)-statistic?
- For two independent continuous uniform p-values, what is the null reference distribution of \(-2\log(P_1P_2)\)?
- In a trial with a valid exact p-value of 0.20 and an independent valid continuous-test p-value of 0.01, which value is entered into the Fisher statistic for the first stage: the observed success count or 0.20?
- Does a significant Fisher test across a binary endpoint and a continuous endpoint show that treatment improved both endpoints?
- Name one design feature that could make the usual independent-p-value calibration questionable.
Solutions. (1) The statistics have different scales and null distributions; their sum has no general reference distribution. Convert each to a valid p-value and use a calibrated combination rule. (2) A chi-square distribution with 4 degrees of freedom. (3) Enter 0.20, the stagewise p-value, not the count. (4) No. Rejection is evidence against the global null that both stagewise nulls hold; it does not establish improvement on each endpoint. (5) Examples include shared participants, shared controls, or correlated outcomes that make the stagewise p-values dependent.