Why a Familiar Final P Value Can Mislead
A final analysis may use a familiar test statistic and a standard normal reference distribution, yet still have an invalid p-value. The problem is not necessarily the arithmetic in the final test. It can be the path by which the tested treatment or hypothesis was chosen: if interim data influence that choice, the final statistic may no longer have the reference distribution assumed by the conventional p-value.
This tutorial studies a stylized two-stage, one-sided test under a global null. It isolates one mechanism: select the candidate with the more favorable interim result, then combine that selected interim result with later information and assess the combination as though the candidate had been fixed in advance. The example is deliberately simple enough to derive the null rejection probability exactly up to a one-dimensional integral.
A Two-Stage Selection Model
Suppose two candidate treatments, A and B, are compared with a common control. At the interim, let \(X_A\) and \(X_B\) denote their standardized, one-sided treatment-versus-control statistics. Under the global null that neither candidate has an effect, assume \(X_A\) and \(X_B\) are independent standard normal random variables. The trial selects the candidate with the larger interim statistic. Write \(M=\max(X_A,X_B)\) for that selected interim statistic.
At the final analysis, suppose a common standardized information increment \(Z_2\) is independent of the interim statistics and is standard normal under the global null. This shared increment represents a simplified setting in which the later-stage component contributes the same noise term to either candidate's combined statistic. For each candidate, the equally weighted combined statistic is \(U=(X_A+Z_2)/\sqrt{2}\) or \(V=(X_B+Z_2)/\sqrt{2}\). Equal weighting corresponds to equal information in the two stages. Since the common increment is added to both, the selected final statistic is \(T=(M+Z_2)/\sqrt{2}=\max(U,V)\).
The following assumptions are essential to this calculation: the global null is true; the interim statistics are independent standard normals; \(Z_2\) is independent of both interim statistics and standard normal under the null; and the stage weights are as specified. The analysis uses a fixed, upper-tail test. This is a model for demonstrating the selection effect, not a universal joint distribution for all adaptive trials.
Worked Example: The Conventional P Value After Selection
Worked Example: The Conventional P Value After Selection
Suppose the hypothetical interim statistics are \(X_A=1.20\) and \(X_B=0.40\), and the later-stage common increment is \(Z_2=1.00\). Candidate A is selected because \(1.20>0.40\). The conventional equal-weight combination for A is
If this were a prespecified candidate, the one-sided standard normal p-value would be \(1-\Phi(1.556)\approx0.0599\), where \(\Phi\) is the standard normal cumulative distribution function. That calculation is arithmetically correct for a prespecified candidate under the stated model. It is not, by itself, a valid p-value for the adaptively selected candidate: its reference distribution has to account for selecting the larger interim statistic.
The p-value of approximately 0.0599 does not establish an effect, and it is not the probability that the null is true. Its calibration is the issue: in repeated trials under the null, does this procedure produce p-values at or below a given threshold no more often than that threshold? The next calculation shows that it does not.
Deriving the Null Rejection Probability
The conventional one-sided five percent standard normal cutoff is \(c=1.6449\), since \(P(Z>c)=0.05\) for \(Z\sim N(0,1)\). Using this cutoff on the selected statistic rejects when
The maximum \(M\) has cumulative distribution function \(P(M\leq m)=P(X_A\leq m,X_B\leq m)=\Phi(m)^2\), by independence. Differentiating gives its density \(f_M(m)=2\phi(m)\Phi(m)\), where \(\phi\) is the standard normal density. Conditional on \(M=m\), the independent variable \(Z_2\) exceeds \(\sqrt{2}c-m\) with probability \(1-\Phi(\sqrt{2}c-m)\). Averaging this conditional probability over the density of \(M\) gives the unconditional rejection probability:
For \(c=1.6449\), \(\sqrt{2}c\approx2.3262\), and numerical evaluation yields approximately \(0.0878\). Thus the nominal five percent test rejects in about 8.78% of repeated trials under this model's global null.
Worked Example: Checking the Error Rate Using a Bivariate Normal Representation
The same calculation can be checked without integrating over \(M\). Define \(U=(X_A+Z_2)/\sqrt{2}\) and \(V=(X_B+Z_2)/\sqrt{2}\). Each is standard normal. Their covariance is
Because both marginal variances equal 1, their correlation is 0.5. Selection means the test rejects if \(U>c\) or \(V>c\). At \(c=1.6449\), each event has probability 0.05. Their intersection has probability approximately 0.01219 under the bivariate standard normal distribution with correlation 0.5. By inclusion-exclusion,
This agrees with the integral calculation. The intersection is subtracted because trials in which both candidate statistics exceed the cutoff are counted in both marginal tail probabilities. The result is about 8.78%, not 5%: the final statistic is the larger of two correlated statistics, so its upper tail is larger than the upper tail of either single standard normal statistic.
What the Calculation Establishes
A conventional p-value \(1-\Phi(T)\) treats \(T\) as if it were a prespecified standard normal statistic. In this model, the actual statistic is \(T=\max(U,V)\). At the nominal threshold 0.05, the conventional p-value therefore satisfies \(P(1-\Phi(T)\leq0.05)\approx0.0878>0.05\). This one inequality is enough to show that the conventional p-value is not valid under the stated global null. In particular, its null distribution is not uniform, and it is anti-conservative at this threshold.
The result is not a claim that every data-dependent change invalidates every p-value. The conclusion depends on what data determined the change, how the final statistic is constructed, and whether its reference distribution accounts for those choices. For a candidate fixed in advance, \(U\) alone has a standard normal null distribution in this example. For the candidate selected from the interim evidence, using that same single-candidate reference distribution misses the selection step.
Worked Example: A Selection-Adjusted P Value at the Conventional Cutoff
Suppose the observed selected statistic is exactly \(t=1.6449\). Its conventional one-sided standard normal p-value is 0.05. Under the two-candidate global-null model, however, the exact selection-adjusted p-value for the statistic \(T=\max(U,V)\) is
The two marginal upper-tail probabilities are 0.05, and their joint upper-tail probability is approximately 0.01219. Therefore,
For this particular observed statistic, the selection-adjusted p-value is about 0.0878, rather than 0.05. A simpler conservative adjustment uses the union bound: the probability that either of two marginal tests exceeds its threshold is at most the sum of the two marginal probabilities. At this threshold that bound is \(0.05+0.05=0.10\). The exact adjustment is smaller here because the two statistics are positively correlated, but it still exceeds 0.05.
Alternatives, Diagnostics, and Limits
The appropriate analysis depends on the design and the inferential target. The comparison below distinguishes three broad strategies. They are not interchangeable: each relies on a different account of selection or information.
| Approach | What it does | Key requirement or cost |
|---|---|---|
| Naive pooled test after selection | Uses a single-candidate reference distribution for the selected pooled statistic. | Invalid here because selection changes the null distribution; it may inflate Type I error. |
| Selection-adjusted test | Uses the distribution of the maximum or otherwise accounts for the candidates searched. | Must represent the actual selection rule, dependence, and analysis statistic correctly; may be less powerful. |
| Independent final-stage test | Uses only a final-stage statistic whose null distribution remains valid after the interim choice. | Requires valid independence and a well-defined final-stage comparison; it may discard useful interim information. |
An important diagnostic is to reproduce the complete decision rule under the null, not merely to check the final test formula. In a simulation, generate interim and later data under a null configuration, apply the actual selection rule, calculate the reported p-value, and estimate \(P(P\leq\alpha)\). If the design is correctly calibrated at level \(\alpha\), this rate should not exceed \(\alpha\), apart from Monte Carlo error. In the stylized model here, simulation should approach 0.0878 at \(\alpha=0.05\) when the naive procedure is used.
Also inspect which information affects the selection, whether stage statistics are independent or correlated, whether the final statistic reuses interim data, and whether the test is one-sided or two-sided. Changes to any of these can change the null distribution. A calculation based on two independent candidates would not automatically apply to several candidates, unequal stage weights, a different selection rule, or a composite null with nuisance parameters.
Selection adjustment is only as reliable as its model and its implementation. If the selection rule or dependence structure is misspecified, the adjusted p-value can also be miscalibrated. An independent final-stage analysis can avoid reusing the selected interim evidence in the test statistic, but independence must genuinely hold under the null, and the inferential claim must match that analysis. Methods based on conditional error provide another design-based route when their assumptions and error-control conditions are met; they do not justify simply attaching an ordinary p-value to a selected pooled statistic.
Common Pitfall
- Wrong reasoning: “The final statistic is a Z statistic, so its p-value is \(1-\Phi(T)\).”
- Correction: A standard normal reference distribution is justified only if the statistic being tested has that null distribution under the full design. Here, the selected statistic is a maximum, and its distribution differs from that of one prespecified standard normal statistic.
- Wrong reasoning: “The selected candidate's final statistic is significant by the usual cutoff, so the selection cannot matter.”
- Correction: Selection is part of the repeated-trial procedure. Type I error counts rejection after all design choices have been made, including choices prompted by interim noise.
Check Your Understanding
Answer each question using the two-stage model above. Brief solutions follow the questions.
- Under the global null, what is the distribution of the interim maximum \(M=\max(X_A,X_B)\)?
- Why is the selected statistic \(T\) not distributed as a single standard normal variable?
- At the conventional five percent cutoff, what null rejection probability does the model give, and what does that imply about the naive p-value?
- If the candidates had been fixed before interim data were observed, what is the null distribution of \(U\) under the model?
- Why must a selection-adjusted analysis reproduce the actual selection rule and dependence structure?
Solutions. (1) Independence gives \(P(M\leq m)=\Phi(m)^2\), with density \(2\phi(m)\Phi(m)\). (2) \(T\) is the maximum of the two correlated combined statistics \(U\) and \(V\), so its upper tail includes rejection through either candidate. (3) The probability is approximately 0.0878, which exceeds 0.05; the naive p-value is anti-conservative at that threshold. (4) \(U\) is standard normal because it is the equally weighted sum of independent standard normal variables divided by \(\sqrt{2}\). (5) Both selection and dependence determine the null distribution of the reported statistic; using a different rule or dependence model can give the wrong calibration.