Robustness to Unequal Stage Effects

What Robustness to Unequal Stage Effects Means

A combination test can give a valid test of a global clinical hypothesis even when the treatment effect is not the same in every stage. That is a statement about error control, not a claim that the stages share one effect size. A positive result can support evidence of benefit somewhere in the prespecified stagewise sense while leaving open how effects vary across stages.

For clarity, let \(\theta_k\) denote the treatment effect in stage \(k\), with positive values indicating benefit, and let \(p_k\) be a one-sided stagewise p-value for benefit. The global no-benefit hypothesis is composite: a harmful or null effect in any stage is allowed under it. This differs from the sharper hypothesis that every stage effect is exactly zero.

Definition: For \(K\) stages and a prespecified beneficial direction, the global no-benefit null is \(H_0:\theta_k\leq 0\) for every \(k=1,\ldots,K\). Its alternative is \(H_a:\theta_k>0\) for at least one stage. The sharper all-zero null is \(H_0:\theta_k=0\) for every stage; it is only a subset of the global no-benefit null.

A combination test addresses the global null by combining stagewise evidence. For the standard inverse normal and Fisher reference calibrations used below, assume that the stages are independent and that the stagewise one-sided p-values are valid throughout the composite null: for every stage and every \(u\) between 0 and 1, \(P(p_k\leq u)\leq u\) whenever \(\theta_k\leq0\). The examples use continuous normal tests, for which the p-values are exactly uniform at the zero-effect boundary and conservative under harmful effects. The combination weights are fixed by the design, not selected after seeing the stage results.

Why the Null Calibration Does Not Require a Common Effect

For the inverse normal method, choose prespecified nonnegative weights \(w_k\geq0\) with \(\sum_{k=1}^K w_k^2=1\). The combination statistic and its one-sided p-value are

Z_{\mathrm{comb}}=\sum_{k=1}^K w_k\Phi^{-1}(1-p_k), \qquad p_{\mathrm{comb}}=1-\Phi(Z_{\mathrm{comb}}).

Here \(\Phi\) is the standard normal cumulative distribution function. If the stagewise p-values are independent uniforms under the all-zero boundary, each \(\Phi^{-1}(1-p_k)\) is standard normal. The weighted sum is therefore standard normal because the squared weights sum to one. With valid but conservative stagewise p-values under the composite null, each transformed value is stochastically no larger than a standard normal variable. Independence then makes the combined upper tail no larger than the calibrated standard normal upper tail. Thus the test controls its type I error for the global no-benefit null under these assumptions; it does not require \(\theta_1=\cdots=\theta_K\).

Fisher’s combination uses

F=-2\sum_{k=1}^K\log(p_k).

Under independent uniform p-values, each \(-2\log(p_k)\) has a chi-square distribution with 2 degrees of freedom. Their sum has a chi-square distribution with \(2K\) degrees of freedom. This gives the standard Fisher calibration at the all-zero boundary. For independent p-values that are valid but conservative under the composite null, each term is stochastically no larger than its uniform-null counterpart, so the calibrated upper-tail test remains valid. This argument concerns validity of the global test, not estimation of a shared effect.

The distinction is especially clear in a normal stagewise model. Suppose independent estimates satisfy \(\hat{\theta}_k\sim N(\theta_k,1/I_k)\), where \(I_k\) is the stage information, and define \(Z_k=\hat{\theta}_k\sqrt{I_k}\). Then the one-sided p-value is \(p_k=1-\Phi(Z_k)\), and \(Z_k\sim N(\theta_k\sqrt{I_k},1)\). Under the inverse normal combination,

Z_{\mathrm{comb}}=\sum_{k=1}^K w_k Z_k \sim N\left(\sum_{k=1}^K w_k\theta_k\sqrt{I_k},\,1\right).

The variance is one because the stage statistics are independent and \(\sum w_k^2=1\). The mean depends on a weighted sum of the stage effects, not on whether those effects are equal. This explains both the method’s robustness for null control and its limitation: quite different effect patterns can produce the same combined mean.

Worked Calculations and Interpretation

Worked Example: Significant Inverse Normal Evidence With Unequal Estimates

Suppose two independent hypothetical stages each have a known standard error of \(0.20\) units. Their estimated beneficial effects are \(0.50\) and \(0.04\) units. Use equal inverse normal weights, \(w_1=w_2=1/\sqrt{2}\), and one-sided normal p-values.

The stage statistics are \(Z_1=0.50/0.20=2.50\) and \(Z_2=0.04/0.20=0.20\). The corresponding p-values are \(p_1=1-\Phi(2.50)\approx0.006210\) and \(p_2=1-\Phi(0.20)\approx0.4207\). Applying the prespecified combination rule gives

Z_{\mathrm{comb}}=\frac{2.50+0.20}{\sqrt{2}}=1.909, \qquad p_{\mathrm{comb}}=1-\Phi(1.909)\approx0.02812.

At a one-sided significance level of \(0.05\), this is evidence against the global no-benefit null under the stated assumptions. It is not evidence that both stages have the same effect: the observed estimates differ substantially, and the combination calculation tests neither equality nor a prespecified common effect size.

Worked Example: Fisher’s Test Uses the Same Evidence Differently

Use the two p-values from the preceding hypothetical stages in Fisher’s method. With \(K=2\), the reference distribution is chi-square with 4 degrees of freedom.

F=-2\{\log(0.006210)+\log(0.4207)\}\approx11.89.

For a chi-square variable with 4 degrees of freedom, the upper-tail probability at \(f\) is \(e^{-f/2}(1+f/2)\). Therefore,

p_{\mathrm{Fisher}} =e^{-11.89486/2}(1+11.89486/2) \approx0.01815.

This also rejects the global no-benefit null at level \(0.05\), using the independent-stage calibration. Fisher’s result differs from the inverse normal p-value because the methods weight the pattern of p-values differently. Neither p-value measures agreement between stage effects or establishes a common effect size.

Worked Example: A Shared Combined Mean Can Hide Different Effects

Consider two hypothetical independent stages with information \(I_1=I_2=25\), equal inverse normal weights, and stage effects \(\theta_1=0.50\) and \(\theta_2=-0.10\) units. The stagewise standard errors are both \(1/\sqrt{25}=0.20\). Under this alternative, the mean of \(Z_{\mathrm{comb}}\) is

\frac{1}{\sqrt{2}}\left(0.50\sqrt{25}+(-0.10)\sqrt{25}\right) =\frac{2.50-0.50}{\sqrt{2}}=1.414.

At one-sided \(\alpha=0.025\), the standard normal critical value is approximately \(1.96\). Since \(Z_{\mathrm{comb}}\sim N(1.414,1)\), the rejection probability is

P(Z_{\mathrm{comb}}>1.96) =1-\Phi(1.96-1.414) =1-\Phi(0.546) \approx0.2925.

If instead both effects were \(0.20\) units, the mean would also be \((1+1)/\sqrt{2}=1.414\), giving the same inverse normal power under this model. The same power calculation can therefore arise from a common modest benefit or from a larger benefit in one stage offset by harm in the other. The test’s global evidence does not distinguish those patterns.

Checking Heterogeneity Without Overclaiming

The combined test and the question of a common effect answer different questions. If stage-specific estimates matter clinically, display them with their standard errors or intervals and describe their directions and magnitudes. A combination p-value alone compresses that information. A separate assessment of effect variation may be appropriate, but a nonsignificant difference is not proof that the effects are equal.

Worked Example: A Difference Interval Is Not Proof of Equality

Return to the estimates \(0.50\) and \(0.04\) units, each with standard error \(0.20\). Under independent normal estimation, the estimated difference is \(0.50-0.04=0.46\) units, with standard error

SE(\hat{\theta}_1-\hat{\theta}_2) =\sqrt{0.20^2+0.20^2} =0.2828.

A 95% normal interval for the difference is \(0.46\pm1.96(0.2828)\), or approximately \((-0.094,1.014)\) units. Because it includes zero, these data do not establish a difference at the corresponding two-sided 5% level. But the interval is also wide and includes substantial differences in either direction. It does not show that the effects are equal; establishing practical similarity would require a prespecified equivalence margin and an analysis designed for that question.

Diagnostics should focus on whether the planned combination procedure is being applied to inputs that support its calibration and interpretation:

Check that every stagewise p-value tests benefit in the same prespecified direction. Reversing one stage’s direction can make a combined result meaningless.
Check that the planned weights and reference calibration match the design assumptions, including independence for the standard calibrations described here.
Compare stagewise estimates and uncertainty, not just p-values. A small p-value can reflect high information as well as a large estimated effect.
Distinguish a robust test of the global null from robustness to missing outcomes, dependence, or changes in the analysis. The test’s validity does not automatically cover those problems.

A common mistake is to reason, “The combined test is significant, so the treatment has the same beneficial effect in every stage.” The conclusion does not follow. The global alternative requires benefit in at least one stage; other stages may have null or harmful effects. The correction is to report the global test as evidence against the stated global null, then describe stage-specific estimates and uncertainty separately. Likewise, a nonsignificant heterogeneity check should not be translated into “the effects are the same.”

Key takeaway: With valid stagewise p-values and the assumptions required by its calibration, a combination test can control type I error for the composite global no-benefit null despite unequal stage effects. Its p-value is evidence about that null, not proof of a common effect size or a summary estimate of treatment benefit.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

With a beneficial direction defined as positive, write the global no-benefit null for three stages. How does it differ from the all-zero null?
For two independent stages with \(Z_1=1.0\), \(Z_2=1.0\), and equal inverse normal weights, calculate \(Z_{\mathrm{comb}}\) and its one-sided p-value.
Does a significant combination test establish that every stage has a positive effect? Explain.
Why does a nonsignificant test or interval comparison of stage effects not establish that those effects are equal?

Solutions. (1) The global null is \(\theta_1\leq0,\theta_2\leq0,\theta_3\leq0\); the sharper all-zero null sets each effect exactly to zero and excludes harmful values. (2) \(Z_{\mathrm{comb}}=(1+1)/\sqrt{2}=1.414\), so \(p=1-\Phi(1.414)\approx0.07865\). (3) No. The alternative is that at least one stage has a positive effect; other stage effects need not be positive. (4) Failure to detect a difference may reflect imprecision. Equality or practical similarity requires an appropriately designed analysis and, for equivalence, a prespecified margin.