Worked Analysis: Selecting and Applying a Combination Test

From Stagewise Evidence to One Trial Decision

In a multi-stage clinical trial, each stage may produce a p-value for the same scientific question. The final analysis must combine those p-values in a way that accounts for the planned analysis and controls the probability of a false positive. The choice is not merely a matter of selecting a convenient formula: the hypothesis direction, stagewise validity, dependence structure, combination function, and weights all matter.

Suppose larger values of a treatment-effect parameter \(\theta\) represent greater benefit. A one-sided efficacy analysis might test \(H_0:\theta\leq 0\) against \(H_a:\theta>0\). The direction must be fixed before examining stage results. Each stage’s p-value should measure evidence in that same direction; a small p-value then consistently represents evidence for benefit.

Validity must hold under the null values relevant to the trial, not just at one convenient parameter value. In addition, the calibration of a familiar combination statistic generally relies on independence of the stagewise p-values. Independent stage data can support that condition, but the analysis must justify it from the trial design and the construction of the tests. If the p-values are dependent, the independent-p-value calibration cannot simply be used.

Selecting a Combination Function and Weights

Two common choices are the inverse-normal combination and Fisher’s combination. The inverse-normal method transforms each one-sided p-value to a standard-normal score and combines the scores. Fisher’s method adds transformed p-values. These procedures emphasize evidence differently: inverse-normal weights can reflect planned stage information, whereas Fisher’s statistic gives equal algebraic treatment to the p-values.

Weights are commonly set proportional to the square root of the planned stage information. If the information amounts are \(I_1,\ldots,I_K\), this gives \(w_k=\sqrt{I_k/\sum_j I_j}\). The weights are nonnegative, sum to one in squared value, and must be fixed without using the observed stagewise results. They encode how the analysis will combine stages; they are not estimates to be tuned after seeing which stage produced the smaller p-value.

The assumption behind each reference distribution should be stated rather than inferred from the formula. For inverse-normal combination, independent uniform p-values yield independent standard-normal \(Z_k\), so a fixed weighted sum with squared weights summing to one is standard normal. For Fisher combination, independent uniform p-values make each \(-2\log(p_k)\) chi-square with two degrees of freedom, and their sum has \(2K\) degrees of freedom.

Worked Analysis: Information-Weighted Inverse Normal

Information weighting is appropriate here because the stage information was planned, the same directional question is tested in both stages, and independent increments support the null calibration. It also gives the more informative second stage greater influence. Those reasons would not justify changing the weights after seeing \(p_1\) or \(p_2\).

Justifying Composite-Null Control

The distinction between an exact uniform p-value and a merely valid p-value matters for a composite null such as \(\theta\leq0\). At the boundary, stagewise p-values may be uniform and the inverse-normal statistic may have the exact standard-normal reference distribution. Away from the boundary, a stagewise test may be conservative. Nonnegative weights ensure that this conservativeness cannot turn into excess upper-tail rejection for the inverse-normal procedure.

Here is a short justification. For any stagewise valid p-value, the score \(Z_k=\Phi^{-1}(1-p_k)\) is stochastically no larger than a standard normal variable \(N_k\): for any real \(z\),

With independent stagewise p-values, the scores are independent. They can be coupled with independent standard normal variables so that \(Z_k\leq N_k\) for each stage. If every \(w_k\geq0\), this implies \(\sum_k w_kZ_k\leq\sum_k w_kN_k\) under that coupling. The normal sum on the right has mean zero and variance \(\sum_k w_k^2=1\). Consequently, its upper-tail probability at any boundary is at least that of the combined score, so the standard-normal upper-tail test controls the type I error rate.

The nonnegative-weight condition is essential to this argument. A negative weight reverses the coordinatewise inequality for that stage, so a conservative stagewise score can contribute in the direction of rejection. Squared weights summing to one alone are not enough to guarantee composite-null control.

Compare the Main Alternatives

Fisher’s method offers a different valid choice when its assumptions hold. In the scenario above, the product of the p-values is \(0.120(0.00800)=0.000960\), giving

For \(K=2\), the reference distribution is \(\chi^2_4\). Its upper-tail probability is \(e^{-T_F/2}(1+T_F/2)\), which here is approximately \(0.00763\). Thus Fisher’s method also rejects at \(\alpha=0.025\), though its combined p-value differs from the inverse-normal result. The methods need not rank every possible pair of stagewise p-values in the same way.

A reasonable selection is made in advance, based on the scientific alternative and the planned design. Information-weighted inverse normal is useful when stage information should determine relative influence and a directional normal-score combination is intended. Fisher’s method is a natural alternative when the chosen rule is to accumulate evidence through the sum of log p-values. Neither method removes the need for valid stagewise tests, an appropriate dependence assumption, and a calibrated boundary.

Discrete Tests, Diagnostics, and Limitations

Before applying a combination test, check that every stage addresses the same prespecified hypothesis direction, that its p-value is valid under the relevant null, and that the dependence assumptions match the calibration. For inverse normal, also verify that weights are nonnegative, fixed in advance, and have squared sum one. For Fisher, verify the number of stages and use the corresponding chi-square degrees of freedom.

Warning signs include stagewise tests that use opposite directions, weights chosen to favor observed results, or stage outcomes that share data despite an independence-based reference distribution. Discrete tests may yield conservative combination tests, reducing power; validity does not promise an exact size. A combination-test p-value also does not measure effect magnitude, prove clinical benefit, or fix a poorly chosen hypothesis. It is a calibrated summary of evidence for the stated test.

Common Mistake: Allowing Signed Weights

A second frequent mistake is selecting the combination rule or weights after inspecting the stage results. The nominal calibration refers to a prespecified procedure. Result-driven choices among rules or weights can increase the probability of rejection beyond the advertised level unless that selection is itself accounted for in the calibration.