Why the Timing of Weight Choice Matters
An inverse-normal combination test turns stagewise p-values into normal scores and combines those scores using weights. With weights fixed independently of the stagewise results, the usual calibration gives a standard normal statistic under the global null. The important qualification is fixed independently of the observed evidence. If the analyst looks at the stagewise results and then chooses weights that favor the more promising result, the resulting statistic need not have the standard normal distribution used for the ordinary cutoff.
This is not a claim that every data-dependent weight rule inflates type I error. The error is to use the fixed-weight reference distribution without showing that it remains valid for the selection rule. The examples below isolate the issue with two independent, one-sided stagewise tests under the global null.
Assumptions Behind the Ordinary Cutoff
For the basic two-stage result, assume that under the global null each stagewise p-value is valid and that the two stagewise p-values are independent. In the continuous exact case, \(p_1\) and \(p_2\) are independent Uniform(0,1) random variables. Consequently, \(Z_1\) and \(Z_2\) are independent standard normal random variables. The direction of evidence is one-sided and set in advance, so smaller p-values correspond to larger \(Z_i\).
If the weights are fixed before the stagewise results are observed, the statistic has mean zero and variance one under these assumptions:
A linear combination of independent normal variables is normal, so \(C\sim N(0,1)\). Therefore,
This argument uses both the distributional assumptions and the fact that the weights are fixed independently of \(Z_1,Z_2\). If weights are selected from those same scores, the statistic is no longer necessarily a fixed linear combination of independent standard normal variables. The variance normalization alone does not establish its null distribution.
Worked Example: A Prespecified Information-Based Weight
Suppose a hypothetical two-stage trial plans weights \(w_1=0.6\) and \(w_2=0.8\) before seeing either stagewise result. They are normalized because \(0.6^2+0.8^2=1\). The observed one-sided p-values are \(p_1=0.040\) and \(p_2=0.200\). Use a one-sided significance level \(\alpha=0.05\).
The normal scores are \(Z_1=\Phi^{-1}(0.960)=1.751\) and \(Z_2=\Phi^{-1}(0.800)=0.8416\). The combination statistic is
The one-sided critical value is \(z_{0.95}=1.645\). Since \(1.724>1.645\), the test rejects \(H_0\) at the 0.05 level. Its combined p-value is \(1-\Phi(1.724)\approx 0.0424\). Under the stated independence and null assumptions, and because the weights were prespecified, this comparison uses the standard fixed-weight calibration.
How Outcome-Driven Selection Inflates Error
A particularly transparent selection rule chooses whichever stage produced the larger normal score and assigns all the weight to that stage. Formally, choose \((w_1,w_2)=(1,0)\) when \(Z_1\ge Z_2\), and \((w_1,w_2)=(0,1)\) otherwise. Each possible weight vector is normalized. But after the choice, the reported statistic is
Using the ordinary one-sided cutoff \(z_{1-\alpha}\) now rejects if either stage score exceeds that cutoff. Under the independent standard normal null model, the probability of this event is not \(\alpha\). The calculation is direct:
The second equality uses independence; the final equality uses \(\Phi(z_{1-\alpha})=1-\alpha\). At \(\alpha=0.05\), the rejection probability is \(2(0.05)-(0.05)^2=0.0975\), almost twice the intended level. The nominal 5% test is therefore not a 5% test under this selection rule.
Worked Example: Choosing the Better-Looking Stage
In a hypothetical trial, suppose the design uses the selection rule above. Under the global null, \(Z_1\) and \(Z_2\) are independent standard normal scores. The analyst applies the usual 5% cutoff \(1.645\) after choosing the larger score.
The test rejects whenever at least one score exceeds 1.645. Each score separately exceeds it with probability \(0.05\), but the chance that at least one does is
Thus the actual type I error is 9.75%, not 5%. For a particular hypothetical realization \(Z_1=1.80\), \(Z_2=0.40\), the selection rule chooses \((1,0)\), giving \(C=1.80\). Comparing this with 1.645 would produce a rejection. The example's calibration calculation shows why that individual decision cannot be justified by the ordinary fixed-weight 5% cutoff.
Calibrating for a Prespecified Selection Rule
The selection rule can be incorporated into the calibration. For the rule that selects the larger of two independent stage scores, a Bonferroni threshold tests each score at \(\alpha/2\). The resulting rule rejects when
By the union bound, the probability of rejection is at most \(\alpha\). Under the independent continuous null model here, its exact size is \(1-(1-\alpha/2)^2\), which is slightly less than \(\alpha\). Equivalently, if the rule selects the smaller of two stagewise p-values, an adjusted p-value is \(\min\{1,2\min(p_1,p_2)\}\). This simple correction applies to this particular choice between two stages; it is not a general correction for arbitrary weight-selection rules.
Worked Example: Adjusting for Selecting the Smaller P-Value
Suppose a hypothetical analysis chooses the stage with the smaller observed p-value: \(p_1=0.018\) and \(p_2=0.110\). The selection rule picks stage 1. If its p-value were reported as though stage 1 had been chosen in advance, the selection would be ignored.
For this two-stage selection rule, the adjusted p-value is
Because \(0.036<0.05\), this selection-adjusted test rejects at the 5% level. The correction accounts for having chosen the more favorable of two p-values. It does not imply that multiplying by two is appropriate for every adaptive inverse-normal analysis; the full selection mechanism and its null distribution must determine the calibration.
Recognizing and Avoiding the Error
A useful diagnostic is to ask whether the weights could have been written down before anyone saw the stagewise p-values or statistics. Weights based on planned stage information, such as prespecified information fractions, can be fixed in advance. Weights chosen because one stage had a smaller p-value, a larger treatment estimate, or a more favorable interim trend use observed evidence. Reporting the conventional inverse-normal p-value after such a choice requires additional justification.
When the weights depend on results, the proper response is not to assume that all inference is impossible. Instead, specify the complete selection rule and calibrate the test for that rule, using a justified null distribution or a valid error-control procedure. In particular, an adjustment must cover all candidate choices the rule could make, not only the choice that happened to be selected. If dependence between stagewise p-values is present, that dependence must also be reflected in the calibration; independence cannot be presumed merely because the stages are distinct.
Specify whether weights are fixed, what information may determine them, and what choices are available.
If observed stagewise results affect the weights, do not automatically use the fixed-weight standard normal cutoff.
Use a justified selection-adjusted procedure or establish the null distribution of the complete rule, including any dependence and constraints.
The main alternatives differ in what they protect. A fixed inverse-normal test preserves the simple normal calibration when weights are specified independently of the results. A selection-adjusted test accounts for a stated outcome-driven rule, potentially with some loss of power. Other combination methods, such as Fisher's method, change how stagewise evidence is combined, but switching methods does not by itself remove selection bias: choosing the method or its inputs after seeing results also requires valid calibration.
| Approach | What is chosen using results? | Calibration requirement |
|---|---|---|
| Prespecified inverse-normal weights | Nothing about the weights is selected from stagewise evidence | Use the fixed-weight normal cutoff when its assumptions hold |
| Outcome-driven weights with ordinary cutoff | Weights favor observed stage results | Generally invalid without further justification; type I error can be inflated |
| Outcome-driven rule with adjusted calibration | A stated rule selects among weights or stages | Account for the entire selection rule under the null |
A common mistake is: “The selected weights still satisfy \(w_1^2+w_2^2=1\), so \(C\) is standard normal.” The correction is that normalization only controls the variance of a fixed weighted sum of independent standard normals. When the weights depend on the observed scores, selection changes the distribution of the sum; in the stage-selection example, it turns the statistic into \(\max(Z_1,Z_2)\), whose upper tail is \(1-\Phi(c)^2\), not \(1-\Phi(c)\).
The limitation of the simple examples is deliberate: they use two independent, continuous, one-sided null p-values and a fully specified selection rule. With more stages, dependent tests, discrete p-values, or other weight-selection algorithms, the numerical correction can differ. The general lesson remains that the inferential procedure includes the rule used to choose weights, not just the formula evaluated after those weights have been chosen.
Check Your Understanding
Answer each question using the assumptions stated in this tutorial. Brief solutions follow the questions.
- Under independent Uniform(0,1) stagewise p-values, what is the null distribution of each inverse-normal score?
- At a one-sided \(\alpha=0.05\), find the type I error when the analyst selects the larger of two independent standard normal stage scores and uses the ordinary 1.645 cutoff.
- For the rule that selects the smaller of two p-values, compute the adjusted p-value when the observed values are 0.030 and 0.070.
- Why does \(w_1^2+w_2^2=1\) fail to justify the standard normal cutoff if the weights were selected after viewing the stagewise results?
Solutions. (1) If \(p_i\sim\text{Uniform}(0,1)\), then \(Z_i=\Phi^{-1}(1-p_i)\sim N(0,1)\); independence of the p-values also gives independence of the scores. (2) The test rejects if either score exceeds 1.645, so its size is \(1-(0.95)^2=0.0975\), or 9.75%. (3) \(p_{\text{adj}}=\min\{1,2(0.030)\}=0.060\). (4) The sum-of-squares condition normalizes a fixed weighted sum, but result-dependent weights change the distribution through selection; the ordinary fixed-weight normal calibration therefore need not hold.