Why the Combination Function Matters
A combination test reduces evidence from separate trial stages to a single statistic. But there is no single mathematical rule for doing this. One function may respond strongly when several stages each provide moderate evidence; another may be especially sensitive to one very small stagewise p-value. A design may also deliberately give one stage more influence than another. Choosing the function therefore expresses a statistical design goal, not just a computational preference.
The combination-test principle relies on valid stagewise evidence and an appropriate joint null distribution. The function itself describes how the p-values are mapped; it does not, by itself, establish a valid type I error rate. That requires a rejection rule calibrated to the joint null distribution and the design’s assumptions. Here the focus is on understanding what different mappings reward.
A stagewise p-value measures how incompatible the evidence from that stage is with its null hypothesis, under the test used for that stage. For a one-sided test, the p-values must correspond to the same prespecified direction of benefit if they are to be interpreted as evidence for that direction together. A small p-value is stronger evidence against the relevant null than a large one, but p-values are not effect estimates and do not measure the clinical size of an effect.
Assumptions Behind Comparing Functions
To compare function behavior clearly, suppose there are \(K\) stages, the stagewise p-values are calculated using prespecified valid tests, and the direction of testing is consistent across stages. Under the global null, each p-value is super-uniform: for every \(u\) between 0 and 1, \(P(p_k \leq u) \leq u\). For the simple reference comparisons below, also suppose the p-values are independent and continuously uniform under the null. This is an idealized reference case, not an automatic property of every adaptive trial.
In an adaptive design, the validity of a stagewise p-value and the joint behavior of the stagewise values depend on how stages are defined, how adaptations are made, and what information is used. For example, reusing data across stages can undermine an independence assumption. The earlier material on independent stage increments explains why independence matters. If the assumptions used to derive a reference distribution do not hold, the test may need a different valid calibration; choosing an attractive-looking function does not repair invalid inputs.
Two other design choices should be fixed before interpreting a combination statistic: the function and the rule that turns its value into a decision. For example, a statistic that increases with evidence needs an upper rejection threshold, while one that decreases with evidence needs a lower threshold. A numerical combination score is not automatically a p-value, and a threshold that would be appropriate for one function cannot simply be transferred to another.
Functions That Reward Different Evidence Patterns
Fisher’s combination function aggregates the logarithms of the stagewise p-values. Its score increases as any input p-value decreases, and every stage contributes to the total.
The product \(\prod_{k=1}^{K}p_k\) gives the same ordering as Fisher’s score, but in the opposite direction: a smaller product gives a larger \(F\). Fisher’s logarithms make the contributions additive and make it easier to see how each stage enters. They do not mean that the product itself is the combined p-value.
A minimum-p function focuses more directly on the strongest single stage. Define \(M=\min(p_1,\ldots,p_K)\); smaller values indicate stronger evidence by this statistic. Because selecting the smallest among several p-values creates a multiple-comparisons effect, \(M\) needs its own calibration. Treating \(M\) as if it were an ordinary single-stage p-value is generally invalid.
A weighted log score makes stage contributions unequal. If prespecified weights \(w_k\) are nonnegative and sum to 1, one possible score is \(S=-\sum_{k=1}^{K}w_k\ln(p_k)\). A larger weight gives that stage more influence on the score. This can be useful when the design has a defensible reason to emphasize one stage, but weights should not be chosen after looking at the observed p-values.
Worked Example: Comparing Consistent Evidence With One Strong Stage
Suppose three hypothetical stages produce p-values \(0.04\), \(0.03\), and \(0.05\). Compare their Fisher score with the score for a different hypothetical pattern, \(0.001\), \(0.80\), and \(0.80\). These calculations compare the evidence patterns; they do not decide whether either pattern crosses a calibrated rejection threshold.
For the first pattern, the product is \(0.04\times0.03\times0.05=0.0000600\). Therefore, \(F=-2\ln(0.0000600)=19.44\). All three stages contribute evidence in the same direction.
For the second pattern, the product is \(0.001\times0.80\times0.80=0.000640\). Therefore, \(F=-2\ln(0.000640)=14.71\). This pattern contains one very small p-value but two values that provide little evidence against the null.
The first pattern has the larger Fisher score, \(19.44\) compared with \(14.71\). Thus, for these inputs, Fisher’s function gives more combined score to the set of three consistently small p-values than to the set with one exceptionally small value and two large values. The minimum-p statistic ranks the patterns differently: their minimum values are \(0.03\) and \(0.001\), respectively, so it favors the second pattern. The rankings reflect different design goals, not a contradiction.
The reason the Fisher score and product have the same ordering follows directly from the logarithm’s properties. The natural logarithm is strictly increasing, so taking the negative of twice the log reverses the ordering of positive products.
Consequently, if one set of p-values has a smaller product than another, its Fisher score is larger. This identity explains the score’s direction and aggregation; it does not establish a rejection cutoff. The cutoff must still reflect the null distribution of the chosen function under the design’s assumptions.
Unequal Contributions and Stage Emphasis
A weighted score changes the influence of a stage while keeping the same p-values as inputs. The weights are part of the test specification. They can reflect a planned information allocation or a prespecified design priority, but they do not automatically correspond to the fraction of patients enrolled in each stage. Nor does a larger weight mean that the stage’s treatment effect is larger; it means only that the chosen combination rule gives its p-value more influence.
Worked Example: Reversing the Stage Order Under Unequal Weights
Consider two hypothetical stages and the weighted score \(S=-0.8\ln(p_1)-0.2\ln(p_2)\). The weights sum to 1, and Stage 1 has the larger prespecified contribution. Compare p-values \((0.01,0.20)\) with the same values reversed, \((0.20,0.01)\).
For \((0.01,0.20)\), \(S=-0.8\ln(0.01)-0.2\ln(0.20)\) \(=0.8(4.6052)+0.2(1.6094)=4.006\).
For \((0.20,0.01)\), \(S=-0.8\ln(0.20)-0.2\ln(0.01)\) \(=0.8(1.6094)+0.2(4.6052)=2.209\).
The two patterns contain the same pair of p-values, but their stage assignments differ. The first gives the smaller p-value to the stage with weight \(0.8\), so it has the larger score. This is the intended effect of unequal weights. It would be misleading to describe the scores as evidence that the first stage had a larger clinical effect: the calculation uses p-values, not effect estimates.
Weights can also create sensitivity to a pattern that is clinically or operationally important, but that sensitivity comes with a commitment: a stage given little weight has little opportunity to influence the final statistic. The choice should therefore be defended using design considerations established before observing the stagewise results. A data-dependent choice of weights can inflate error rates unless the selection is incorporated into the testing procedure.
How to Choose and Check a Design
The design question is not “Which function is universally best?” It is “Which evidence patterns should this test reward, and can that behavior be justified and calibrated for this trial?” The following sequence helps make the choice explicit.
Define the stages, the null and alternative hypotheses, and the stagewise tests. Ensure that p-values have a common interpretation and, for directional claims, a common direction.
Decide whether the design should respond to evidence spread across stages, to a very strong result in one stage, or to a planned difference in stage influence.
Write the formula, its evidence direction, and any weights in the protocol or analysis plan before examining the stagewise results.
Assess whether the joint null behavior assumed by the chosen test is appropriate, then determine a rejection rule that controls the target type I error rate.
Diagnostics should target both the inputs and the intended behavior. Check that each stagewise p-value was computed by the specified test, that its direction and analysis population are consistent with the plan, and that no unplanned choice among several p-values was made. Examine whether the stages share data or otherwise have dependence not covered by the reference assumptions. If simulations are used to assess operating characteristics, include patterns with evidence distributed across stages, evidence concentrated in one stage, and plausible null configurations. A function’s ranking of a few illustrative inputs is not a substitute for checking the operating behavior of the full design.
The main alternative is not simply another combination function. A pooled-data test analyzes the accumulated patient-level data under a model, whereas a combination test combines stagewise evidence. As discussed in the earlier material on combination tests versus pooled-data tests, their assumptions and information use can differ. A pooled analysis may be more directly tied to an effect estimate under its model; a combination test can retain a stagewise evidence structure. Neither approach is automatically preferable. The choice depends on the design, data availability, and the validity of the corresponding analysis.
Common Mistakes and Limitations
- Calling a score a p-value. A Fisher score or weighted score is a statistic. It needs a null calibration before it can support a rejection decision.
- Multiplying p-values and interpreting the product as a probability. The product is an input to a combination rule, not generally the probability that the null is true or a calibrated combined p-value.
- Using the smallest stagewise p-value without adjustment. Selecting a minimum changes its null distribution. The minimum-p statistic requires a threshold appropriate to the number of stages and their joint behavior.
- Changing weights after seeing results. That makes the test depend on the observed evidence and can invalidate a calibration that assumed fixed weights.
- Ignoring direction or stage dependence. Opposite-direction effects or dependent stagewise p-values may make a chosen function’s interpretation or reference distribution inappropriate.
All such functions compress information. A single score does not show the stagewise effect estimates, their uncertainty, or whether the directions of estimated effects agree. The function can also be sensitive to p-value conventions, such as discreteness or rounding. These limitations are reasons to report stagewise results and the prespecified combination rule alongside the final statistic, rather than presenting the combined number alone.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For two stages, which pattern has the larger Fisher score: p-values \((0.04,0.04)\) or \((0.001,0.80)\)? Show the scores to three significant figures.
- Why is the minimum of three valid stagewise p-values not automatically a valid combined p-value?
- Using \(S=-0.7\ln(p_1)-0.3\ln(p_2)\), calculate the score for \((p_1,p_2)=(0.05,0.20)\). Which stage has greater influence?
- State two checks required before using a combination function to make a trial-level decision.
Solutions. For question 1, the first product is \(0.04(0.04)=0.0016\), giving \(F=-2\ln(0.0016)=12.9\). The second product is \(0.001(0.80)=0.0008\), giving \(F=-2\ln(0.0008)=14.3\); thus the second pattern has the larger Fisher score, although the first has more consistent evidence. For question 2, selecting the minimum is a selection among three values, so its null distribution differs from that of one prespecified p-value; it needs calibration that accounts for the joint distribution. For question 3, \(S=0.7(2.9957)+0.3(1.6094)=2.580\), and Stage 1 has greater influence because its weight is \(0.7\). For question 4, suitable checks include verifying the stagewise tests and directions, assessing dependence and other joint-null assumptions, and calibrating the rejection rule to control the planned type I error rate.