Robust Combination Tests Under Model Misspecification

Why Combination Calibration Depends on the Stagewise Tests

A combination test is calibrated by combining evidence from multiple stages according to a reference distribution. That calibration relies on a key input: each stagewise p-value must have the stated null behavior. If a stage statistic is analyzed using a reference distribution that does not describe its actual null distribution, its reported p-value may be too small too often. Combining such p-values does not remove the error; it can carry the anti-conservatism into the final test.

This matters in sample size re-estimation because the second-stage design may be chosen after interim data are observed. A sound analysis must account both for the stagewise test’s null calibration and for the fact that the second-stage choice depends on earlier information. The robust principle is to use p-values that remain valid under the null, including conditionally on the information used to choose the second-stage design.

For an adaptive second stage, conditional validity is the relevant version of this requirement. Given the first-stage data and the rule that selects the second-stage sample size or test, the stage-two p-value must still be superuniform under the null. The stage-two data must also be collected and analyzed as specified by that conditional null calibration. This condition permits the second-stage design to depend on interim information; it does not permit an invalid stage-two reference distribution.

A Misspecified Reference Distribution Can Break the Test

Suppose a one-sided stagewise statistic \(T\) is treated as standard normal, so the reported p-value is \(1-\Phi(T)\). That p-value is valid only if the null distribution of \(T\) is standard normal, or if another argument establishes its validity. Correctly calculating a normal tail area is not enough: the tail area must correspond to the statistic’s actual null distribution.

The example also shows why a statistic having mean zero and variance one is not enough to justify standard normal p-values. The full null distribution, or a valid calibration of its tails, matters. In practice, a mismatch may arise from an inaccurate distributional assumption, small-sample approximations, or a test whose calibration does not account for the actual data-generating process.

Robust Combination Through Valid Null P-Values

A simple robust choice is to calibrate each stagewise test from its actual null distribution when that distribution is available. For a statistic where larger values are more extreme, the exact upper-tail p-value is the null probability of obtaining a statistic at least as large as the observed one. With discrete outcomes, this tail p-value is generally superuniform rather than exactly uniform, because the statistic cannot attain every possible tail probability.

Exact calibration is not always available. A different robust option is a conservative p-value whose null tail probability is bounded by the claimed level. For a composite null, validity must hold at every null parameter value, not just at one convenient value. One way to formulate a conservative p-value is to use the largest relevant null tail probability over the null parameter set. Such a choice can cost power, but it avoids claiming more evidence than the null calibration supports.

Here, “increasing evidence” means that replacing a p-value by a smaller one cannot turn rejection into non-rejection. The conditional requirement is important: the second-stage p-value cannot be treated as valid merely because it would be valid for a fixed design chosen in advance. It must be valid for the design actually selected, conditional on the information used to select it.

Justification for the Inverse-Normal Combination

For \(K\) stages, let \(p_i\) be the stagewise p-value and define \(Z_i=\Phi^{-1}(1-p_i)\). Choose fixed positive weights \(w_i\) such that \(\sum_{i=1}^K w_i^2=1\), and use \(Z_C=\sum_{i=1}^K w_iZ_i\). Under independent exactly uniform null p-values, the \(Z_i\) are independent standard normal variables, so \(Z_C\sim N(0,1)\). A one-sided level-\(\alpha\) test rejects when \(Z_C>z_{1-\alpha}\), where \(z_{1-\alpha}=\Phi^{-1}(1-\alpha)\).

The robustness argument follows from stochastic ordering. If \(p_i\) is superuniform, then \(Z_i=\Phi^{-1}(1-p_i)\) is stochastically no larger than a standard normal variable: for every real \(x\), $$ P(Z_i>x)=P(p_i<1-\Phi(x))\le 1-\Phi(x). $$ For two stages, conditional validity of \(p_2\) gives the same bound for \(Z_2\) conditional on the first-stage information. Since \(w_2>0\), conditional on \(Z_1=z\), $$ P(Z_C>c\mid Z_1=z) \le 1-\Phi\left(\frac{c-w_1z}{w_2}\right). $$ The right-hand side increases with \(z\). Because \(Z_1\) is itself stochastically no larger than a standard normal variable, averaging this increasing function over \(Z_1\) gives a probability no greater than the average under a standard normal \(Z_1\). Under that reference, \(w_1Z_1+w_2Z_2\sim N(0,1)\), so $$ P_0(Z_C>z_{1-\alpha})\le \alpha. $$ The same reasoning extends stage by stage when each new p-value is conditionally superuniform given the preceding data and adaptation decisions.

Fisher’s combination provides another example of the same principle. For independent exactly uniform p-values, \(Q=-2\sum_{i=1}^K\log(p_i)\) has a chi-square distribution with \(2K\) degrees of freedom. Smaller p-values make \(Q\) larger, so its rejection region is monotone in the evidence. If the p-values are independent and superuniform instead, the Fisher test calibrated using that chi-square reference is conservative or exact, not anti-conservative. With adaptation, the required conditional validity must still be established for the stagewise p-values.

Choosing and Checking a Robust Procedure

The practical decision is not simply whether to use Fisher or inverse normal. First establish how each stagewise p-value is calibrated under the null. Then choose a combination function whose reference distribution matches the required stagewise assumptions. Exact uniformity supports the usual reference calibration; superuniformity supports conservative control for monotone combination rules under the stated independence or conditional-validity conditions.

Useful diagnostics begin with the stagewise null calibration, not with the final combined statistic alone. Check whether the assumed reference distribution is justified for the outcome type, sample size, and test used. For an adaptive second stage, verify that the p-value remains valid for each design the rule can select and conditional on the information used to make that selection. If calibration is assessed by simulation, inspect null rejection frequencies over relevant null parameter values and adaptation paths, while accounting for Monte Carlo uncertainty.

A common failure is to see a familiar-looking standardized statistic and use a normal-reference p-value automatically. In the earlier discrete example, the statistic has a known null distribution that gives a valid tail p-value; applying an unrelated continuous reference may not. Another failure is to verify stage-two calibration only for a fixed sample size, then reuse that p-value after selecting the sample size from interim results without checking conditional validity.

The robust guarantee also has limits. Superuniformity does not promise a particular power, and conservative stagewise p-values can make a combination test less powerful than its exact-uniform benchmark. If stagewise p-values are dependent in ways not covered by independence or conditional validity, the standard combination reference may not control Type I error. Finally, no combination function repairs p-values that are anti-conservative under the null; the remedy is to improve their calibration or use a method whose error control is justified under the actual null behavior.