Why Calibration Needs a Dependence Condition
A combination test turns stagewise p-values into one final decision. Its critical value is often calibrated using independent uniform p-values under the null. But stagewise p-values in a trial need not be independent: later-stage data may share features with earlier data, and the design may induce dependence. The central question is therefore not whether the p-values are independent, but whether their joint behavior is sufficiently controlled for the chosen calibration to remain valid.
For a fixed combination rule that rejects for sufficiently strong evidence across stages, a useful sufficient condition is conditional stagewise validity. This condition permits dependence, but it is stronger than requiring each stagewise p-value to be valid on its own. The distinction matters because global-null control concerns the probability of the entire combination rule rejecting, not the validity of any one input considered separately.
The global null here means that all stagewise null hypotheses are true. If the null is composite, the condition must hold for every parameter value \(\theta\) in that global null. Conditional validity allows the distribution of \(P_j\) to depend on earlier information, provided that after conditioning it is still super-uniform: the probability of a p-value at most \(t\) is no greater than \(t\).
A combination rule is monotone in the evidence if making any input p-value smaller, while leaving the others unchanged, cannot turn a rejection into a non-rejection. Fixed-weight inverse normal and Fisher combination tests have this property. Their usual calibration chooses a boundary using independent \(U_1,\ldots,U_K\), where each \(U_j\) is uniform on \([0,1]\). The result below explains when that reference calibration remains conservative even if the actual stagewise p-values are dependent.
Global-Null Control: Statement and Justification
Here is the key argument. For any decreasing set \(A\), hold the first \(K-1\) coordinates fixed at \(y=(p_1,\ldots,p_{K-1})\). The values of the last coordinate that place the vector in \(A\) form an interval from zero up to some endpoint \(c(y)\), possibly empty. Conditional validity at stage \(K\) therefore gives $$ P_\theta\big((P_1,\ldots,P_K)\in A\mid\mathcal{F}_{K-1}\big) \le c(P_1,\ldots,P_{K-1}). $$
The endpoint \(c(y)\) is nonincreasing in each coordinate of \(y\): making an earlier p-value smaller cannot shrink the permitted interval for the last one. Apply the same argument to the first \(K-1\) coordinates, and continue backwards. This proves the theorem by induction. More explicitly, the induction compares expectations of bounded coordinatewise nonincreasing functions: write such a function \(g\), with values in \([0,1]\), as \(g(y)=\int_0^1 \mathbf{1}\{g(y)\ge s\}\,ds\). Each set \(\{y:g(y)\ge s\}\) is decreasing, so the induction hypothesis bounds its probability by the probability for independent uniforms. Integrating over \(s\) gives the expectation bound. The one-coordinate starting case is exactly conditional super-uniformity.
This proof uses no independence assumption about the actual \(P_j\). Instead, it uses conditional validity at each stage to bound the probability of a decreasing rejection event by the probability of the same event for independent uniforms. If the independent-uniform calibration gives exactly \(\alpha\), the actual rejection probability is at most \(\alpha\); it can be smaller, for example when stagewise p-values are discrete and conservative.
Worked Example: Inverse Normal Combination With Conditional Validity
Consider a hypothetical two-stage one-sided test with fixed weights \(w_1=0.6\) and \(w_2=0.8\), so \(w_1^2+w_2^2=1\). Suppose the stagewise p-values are \(P_1=0.20\) and \(P_2=0.0100\), and the design satisfies conditional validity under the global null. Use the inverse normal statistic $$ Z_{\mathrm{comb}}=w_1\Phi^{-1}(1-P_1)+w_2\Phi^{-1}(1-P_2), $$ where \(\Phi\) is the standard normal cumulative distribution function.
The stagewise normal scores are \(\Phi^{-1}(0.80)=0.8416\) and \(\Phi^{-1}(0.9900)=2.3263\). Thus $$ Z_{\mathrm{comb}}=(0.6)(0.8416)+(0.8)(2.3263) =0.5050+1.8610 =2.3660. $$
For a one-sided test at \(\alpha=0.05\), the independent-uniform reference boundary is \(z_{0.95}=1.6449\). Since \(2.3660>1.6449\), the combination test rejects the global null. Its corresponding normal-tail p-value is approximately \(1-\Phi(2.3660)=0.0090\). Conditional validity ensures that, over repeated trials under any global-null parameter value, this fixed test rejects no more often than the independent-uniform calibration permits. This conclusion does not require the observed stagewise p-values to be independent.
Worked Example: Fisher Combination Under the Same Control Principle
Suppose a hypothetical two-stage trial uses Fisher's statistic with \(P_1=0.20\) and \(P_2=0.0100\). Assume the p-values are conditionally valid under the global null. Fisher's statistic is $$ X=-2\{\log(P_1)+\log(P_2)\} =-2\log(P_1P_2). $$
Substitution gives $$ X=-2\log\big((0.20)(0.0100)\big) =-2\log(0.00200) =12.4292. $$
Under independent uniform inputs, the reference distribution is chi-square with \(4\) degrees of freedom. Its upper-tail probability at \(x\) is \(e^{-x/2}(1+x/2)\). Therefore the calibrated combination p-value is $$ e^{-12.4292/2}\left(1+\frac{12.4292}{2}\right) =(0.00200)(1+6.2146) =0.01443. $$
This is below \(0.05\), so the test rejects at the 5% level. The rejection region is decreasing in each input p-value: decreasing either p-value increases \(X\). The theorem therefore applies when conditional validity holds, even if the inputs are dependent. The calculation does not establish that the inputs are conditionally valid; that is a property of the stagewise procedures and trial design that must be justified separately.
Why Marginal Validity Is Not Enough
A p-value can be valid marginally while being strongly dependent on other p-values in a way that makes a combination test anti-conservative. Marginal validity only says \(P_\theta(P_j\le t)\le t\) for each stage separately. It does not constrain the conditional distribution given earlier information, which is what the proof requires.
Worked Example: Identical Uniform P-Values Break Fisher Calibration
Under a hypothetical global null, let \(U\sim\mathrm{Uniform}(0,1)\) and set \(P_1=U\) and \(P_2=U\). Each p-value is exactly uniform marginally, but the pair is perfectly dependent. Apply the ordinary two-stage Fisher test calibrated at \(\alpha=0.05\), whose chi-square critical value is approximately \(9.4877\).
The Fisher statistic is $$ X=-2\log(P_1P_2)=-4\log U. $$ It exceeds the critical value precisely when $$ -4\log U>9.4877 \quad\Longleftrightarrow\quad U<\exp(-9.4877/4) \approx 0.0932. $$
Consequently, the actual rejection probability is approximately \(0.0932\), not \(0.05\). Both p-values are valid marginally, but \(P_2\) is not conditionally super-uniform given \(P_1\): once \(P_1\) is known, \(P_2=P_1\) is fixed. This example shows why standard independent-uniform calibration cannot be justified by marginal validity alone.
Assumptions, Diagnostics, and Alternatives
Before applying the theorem, check that its conditions match the trial and the test. In particular, the number of stages, combination rule, and weights must be fixed for the calibration being used. The rejection event must be decreasing in the stagewise p-values. Most importantly, each stagewise p-value must be conditionally valid under the global null, for every relevant null parameter value. A p-value procedure that is valid only for a simple null, but not uniformly over a composite null, does not meet this requirement without further justification.
Conditional validity is a design-based or model-based property, not something that can usually be confirmed from the single set of p-values observed in a trial. Useful checks include:
- Confirm that the stagewise test's null calibration holds for the actual stagewise sample space, including discreteness or nuisance parameters.
- Check whether validity is established conditional on all earlier information, not just after averaging over that information.
- Verify that the chosen combination function is monotone in each p-value and that its critical value corresponds to the stated independent-uniform reference calibration.
- Use null simulations across relevant parameter configurations as a diagnostic for implementation errors or unexpected size inflation. Simulation supports a check but does not replace a proof of the assumptions.
A common alternative is a multiplicity procedure such as Bonferroni, which can control error using marginal p-value validity without requiring the same sequential conditional condition. It may be conservative and does not generally exploit the particular pattern of evidence targeted by a combination rule. Another possibility is to calibrate the combination statistic using its actual joint null distribution, if that distribution is known or can be justified. That approach can accommodate a specified dependence structure, but its guarantee depends on the accuracy of that joint model.
The theorem is a sufficient condition, not a claim that conditional validity is necessary for every possible calibrated test. A test may have global-null control under a different, correctly justified joint calibration. Conversely, using weights or changing the testing rule after observing interim results can invalidate the fixed-rule argument. Such changes require their own validity justification; they are not covered simply because the original stagewise p-values were marginally valid.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What is the conditional-validity requirement for stage \(j\), and what information is conditioned on?
- Why does a decreasing rejection region matter for the global-null control result?
- If every stagewise p-value is valid marginally, may the standard independent-uniform Fisher calibration be used under arbitrary dependence? Explain.
- A test's independent-uniform calibration has rejection probability \(0.04\) at a nominal level of \(0.05\). Under the theorem's assumptions, what upper bound applies to its global-null rejection probability?
Solutions. (1) For every global-null parameter value and every \(t\in[0,1]\), \(P_\theta(P_j\le t\mid\mathcal{F}_{j-1})\le t\) almost surely, where \(\mathcal{F}_{j-1}\) contains all information available before stage \(j\)'s data. (2) A decreasing region ensures that smaller p-values can only preserve or strengthen rejection; conditional super-uniformity can then bound its probability by the independent-uniform reference probability. (3) No. Marginal validity does not imply conditional validity, and dependent marginally uniform p-values can inflate the combination-test rejection probability, as in the identical-p-values example. (4) The global-null rejection probability is at most \(0.04\), since the theorem bounds it by the independent-uniform calibration probability.