Why the Ordinary Z-Test P-Value Can Mislead After Stopping

Why Stopping Changes the Question

A one-sided ordinary Z-test p-value is calculated from a standard normal reference distribution: for an observed statistic \(z\), it is \(1-\Phi(z)\). At a single prespecified analysis, this is calibrated so that, under the null hypothesis, the probability of a p-value no greater than \(\alpha\) is \(\alpha\). In a group sequential design, however, the reported statistic may come from a look selected by the data. Its null distribution need not be the standard normal distribution used by the ordinary formula.

The issue is not that the Z-statistic at either planned look is incorrectly standardized. Under the null, each planned-look statistic can have a standard normal marginal distribution. The issue is that the trial does not always report the statistic from a predetermined look: it may stop early when interim evidence is sufficiently strong. The reported statistic is therefore selected by a stopping rule.

Definition: A p-value \(P\) is valid under \(H_0\) if it is superuniform: for every \(u\) between 0 and 1, \(P_{H_0}(P\leq u)\leq u\). A fixed-sample p-value formula is not automatically valid for a sequentially stopped outcome. Its validity must be assessed under the null distribution of the complete stopping-and-reporting procedure.

Two-Look Normal Model and Stopping Rule

Consider a one-sided test of \(H_0\) against an upper-direction alternative, with two planned analyses at information levels \(I_1<I_2\). Under the canonical normal model under \(H_0\), the standardized statistics \(Z_1\) and \(Z_2\) satisfy

(Z_1,Z_2)\sim N\left( \begin{pmatrix}0\\0\end{pmatrix}, \begin{pmatrix}1&\rho\\\rho&1\end{pmatrix} \right), \qquad \rho=\sqrt{\frac{I_1}{I_2}}.

This model assumes that the information levels and analysis times are fixed in advance, the null score increments over disjoint information intervals are independent, and the standardization gives each planned-look statistic null variance 1. The correlation follows because the score accumulated by the first look is included in the score at the second. The normal model is an approximation in many applications; the calculation below is exact within the stated model.

Use the following stopping rule: stop for efficacy at look 1 if \(Z_1\geq c\); otherwise continue to look 2 and stop there. There is no futility boundary in this example. Define the terminal statistic \(Z_{\mathrm{stop}}\) as the statistic at the look where the trial stops. The ordinary terminal one-sided p-value is

P_{\mathrm{ord}}=1-\Phi(Z_{\mathrm{stop}}).

This formula treats the reported statistic as though it came from a single prespecified standard normal analysis. To test its validity, we must calculate \(P_{H_0}(P_{\mathrm{ord}}\leq u)\) using the stopping rule as well as the null distribution.

Worked Example: A Nominal 5% Boundary at Two Looks

Suppose the trial uses the upper efficacy boundary \(c=\Phi^{-1}(0.95)\approx1.644854\), and the information levels are \(I_1=50\) and \(I_2=100\) units. Under the canonical null model, \(\rho=\sqrt{50/100}=1/\sqrt{2}\approx0.7071\). The same boundary is used at both looks: stop and reject \(H_0\) if \(Z_1\geq c\), or, after continuing, reject if \(Z_2\geq c\).

At a prespecified look, \(P(Z_k\geq c)=0.05\). With stopping, rejection occurs either at look 1 or at look 2 after not crossing at look 1. Therefore

\begin{aligned} P_{H_0}(\text{reject}) &=P(Z_1\geq c)+P(Z_1<c,\ Z_2\geq c)\\ &=0.05+P(Z_1<c,\ Z_2\geq c). \end{aligned}

The second probability is strictly positive: there are null outcomes for which the first statistic is below the boundary but the second crosses it. Given \(Z_1=x\), the canonical model gives \(Z_2\mid Z_1=x\sim N(\rho x,1-\rho^2)\). Hence

P(Z_1<c,\ Z_2\geq c) =\int_{-\infty}^{c} \phi(x)\left[ 1-\Phi\left(\frac{c-\rho x}{\sqrt{1-\rho^2}}\right) \right]\,dx.

Numerical integration with \(c=\Phi^{-1}(0.95)\) and \(\rho=1/\sqrt{2}\) gives \(P(Z_1<c,\ Z_2\geq c)\approx0.0301\). Thus the probability that the ordinary terminal p-value is at most 0.05 is approximately \(0.05+0.0301=0.0801\), not 0.05. The ordinary p-value is anti-conservative for this stopped testing procedure.

In this example the design boundary is exactly \(c=\Phi^{-1}(0.95)\), so \(P_{\mathrm{ord}}\leq0.05\) is equivalent to \(Z_{\mathrm{stop}}\geq c\). If a design instead rounds the boundary to 1.645, that equivalence is only approximate: the ordinary p-value threshold corresponds to 1.644854, not exactly 1.645.

Why the Ordinary Reference Distribution Fails

Before stopping, each \(Z_k\) has a standard normal null marginal. But the terminal statistic is not simply \(Z_1\) or \(Z_2\) selected independently of their values. It equals \(Z_1\) in the upper tail where \(Z_1\geq c\), and equals \(Z_2\) only on the event \(Z_1<c\). That restriction changes the distribution of the reported statistic. In particular, under the null it gives the procedure two opportunities to cross the same nominal threshold.

For a general one-sided level \(\alpha\), let \(c=\Phi^{-1}(1-\alpha)\). In this two-look design,

\begin{aligned} P_{H_0}(P_{\mathrm{ord}}\leq\alpha) &=P(Z_1\geq c)+P(Z_1<c,\ Z_2\geq c)\\ &=\alpha+P(Z_1<c,\ Z_2\geq c). \end{aligned}

When the second term is positive, the ordinary terminal p-value fails superuniformity at \(u=\alpha\). This is enough to show that it is not a valid p-value for the full stopped procedure. The ordinary p-value may still describe how extreme the terminal statistic looks against a single-look standard normal reference; that description does not establish sequential validity.

Worked Example: A Specific Early Stop

Suppose, hypothetically, that the first-look statistic is \(Z_1=1.70\). With the boundary \(c\approx1.644854\), the trial stops for efficacy at look 1. The reported ordinary one-sided p-value is

P_{\mathrm{ord}}=1-\Phi(1.70)\approx0.04457.

This value is the tail area for a standard normal statistic at one prespecified analysis. The observed path does meet the design's efficacy boundary, so the design's rejection decision is clear. But the value 0.04457 does not account for the possibility of having continued and then crossed the boundary at look 2. That additional route to rejection is why the ordinary p-value does not control the overall null rejection probability at 0.05.

For contrast, suppose a different hypothetical path has \(Z_1=1.20\), so the trial continues, and \(Z_2=1.70\). The terminal statistic is again 1.70, and the ordinary formula again gives approximately 0.04457. The matching p-values arise because the formula uses only the terminal Z-statistic; it does not encode whether that value came from an early or final analysis. The path is nevertheless part of the same two-look rejection event counted in the probability calculation above.

What the Calculation Assumes—and How It Can Go Wrong

The numerical inflation just calculated depends on the specified design and model. The two planned looks, information ratio, null joint distribution, stopping boundary, direction of testing, and reporting rule all matter. If there are additional looks, a futility stop, a different boundary at each look, or a different dependence structure, the probability must be recalculated for that complete procedure. The value 0.0801 is not a universal adjustment for sequential trials.

A useful diagnostic is to state the event corresponding to \(P_{\mathrm{ord}}\leq u\), then evaluate its probability under \(H_0\) while respecting the stopping rule. In the example, for \(u=0.05\), that event is the union of an early crossing and a later crossing after continuation. Calculating only \(P(Z_2\geq c)=0.05\) omits the early-stop route; treating the two looks as independent would also be wrong because their correlation is approximately 0.7071.

Check the analysis schedule. If the look was chosen based on accumulating results rather than fixed in advance, the assumed joint distribution may not describe the procedure.
Check the stopping rule. The probability calculation must reflect every efficacy or futility condition that can determine which statistic is reported.
Check the reference distribution. A marginal standard normal distribution at each planned look does not imply a standard normal distribution for the selected terminal statistic.
Check numerical calculations. For boundary probabilities, verify that the integration region matches the continuation event and that computed stage probabilities sum to the overall rejection probability.

What to Report Instead—and What Not to Conclude

The ordinary terminal Z-test p-value is appropriate when the analysis is genuinely a single prespecified look with no data-dependent selection of that look, under the model supporting the Z-test. It can also be reported descriptively after a sequential stop, but it should not be presented as a sequentially valid p-value without a calibration argument. To obtain a p-value for the stopped design, the null distribution used for calibration must include the stopping rule and the outcome being reported. The choice of how to order stopped outcomes by evidence is a separate design decision; the present calculation establishes why an unadjusted fixed-sample tail area alone is insufficient.

Likewise, a nominal 0.05 boundary at each of two looks does not imply a 0.05 overall type I error rate. The overall type I error is the probability, under \(H_0\), that the complete design rejects at any allowed look. Nor does an observed ordinary p-value of 0.04457 mean there is a 4.457% probability that \(H_0\) is true. It is a tail probability under a specified reference distribution, and the ordinary reference distribution does not account for the stopping process here.

Worked Example: When the Ordinary P-Value Is Calibrated

Suppose instead that a study has only one analysis at information \(I_2=100\), selected in advance, and under \(H_0\), \(Z_2\sim N(0,1)\). For a hypothetical observed \(Z_2=1.70\), the ordinary one-sided p-value is \(1-\Phi(1.70)\approx0.04457\). Under this fixed-look design, the event that the p-value is at most 0.05 is \(Z_2\geq\Phi^{-1}(0.95)\), whose null probability is 0.05. Thus the ordinary calculation is calibrated in this setting.

This comparison isolates the issue: the same statistic and formula can be valid for a prespecified single analysis but fail to be valid for the two-look stopped procedure. The difference is not the arithmetic of the normal tail area; it is the sampling and stopping plan that determines the reference distribution.

Common Mistake

Wrong reasoning: “At every planned look, the Z-statistic is standard normal under \(H_0\), so the ordinary p-value at whichever look occurs is valid.” The premise about each planned look's marginal distribution does not justify the conclusion about a data-selected look.

Correction: Define the reported terminal outcome and calculate its null distribution under the actual stopping rule. Then check the superuniformity condition \(P_{H_0}(P\leq u)\leq u\). In the two-look example, the extra probability of a look-2 crossing after a non-crossing at look 1 makes the ordinary terminal p-value fail this condition at 0.05.

Key takeaway: A standard normal marginal at each planned look does not guarantee that the ordinary Z-test p-value is valid after data-dependent stopping. Sequential validity is a property of the p-value together with the full stopping-and-reporting procedure.

Check Your Understanding

Use the two-look upper-boundary setup above unless a question states otherwise.

Why does \(Z_1\sim N(0,1)\) and \(Z_2\sim N(0,1)\) under \(H_0\) not establish that \(Z_{\mathrm{stop}}\sim N(0,1)\)?
For a nominal level \(\alpha\) and common boundary \(c=\Phi^{-1}(1-\alpha)\), write the two disjoint events that make the trial reject.
In the 5% example, what feature of the stopping procedure makes the ordinary terminal p-value anti-conservative?
Would the ordinary one-sided p-value be calibrated for a single analysis planned in advance, under the stated standard normal null model? Explain.

Solutions. (1) The reported look depends on the data: \(Z_1\) is reported when it crosses the boundary, and \(Z_2\) is reported only when \(Z_1<c\). This selection changes the terminal distribution. (2) The events are \(Z_1\geq c\) and \(Z_1<c,\ Z_2\geq c\); they are disjoint because the first requires \(Z_1\geq c\) and the second requires \(Z_1<c\). (3) There are two opportunities to cross the nominal boundary, and the probability of crossing at look 2 after continuing is positive, so the total rejection probability exceeds 0.05. (4) Yes. With one prespecified analysis, \(P(Z_2\geq\Phi^{-1}(0.95))=0.05\) under \(H_0\), so the ordinary one-sided p-value is calibrated under the stated model.