Nominal P-Values At Interim Looks

Why an Interim P-Value Is Not the Whole Design

In a group sequential design, data are analyzed at planned interim looks as well as at the final analysis. Each look produces a test statistic and may produce a p-value. That p-value describes the evidence at that particular look under its stated null model. It does not, by itself, describe the probability that the entire sequence of analyses will produce a false efficacy conclusion.

The distinction matters because the design’s efficacy rule is based on the planned sequence of looks. A result can be statistically significant by the usual single-analysis criterion, such as a one-sided nominal \(p<0.05\), but still fail to cross the more stringent boundary specified for that interim look. Conversely, a result that crosses the boundary is interpreted within the sequential design, whose overall false-positive rate is calibrated across the planned analyses.

The introductory material on classical boundaries establishes that these boundaries depend on the analysis plan, including the number and timing of looks. Here the focus is the meaning of the p-value reported at an individual look and how it relates to, but differs from, that design-wide rule.

Definitions and the Null Model

Definition: A nominal p-value at look \(i\) is the tail probability, calculated as if the statistic at that one look were the only analysis, of obtaining a result at least as extreme as the observed statistic under \(H_0\). For a prespecified upper-tailed test with standard normal statistic \(Z_i=z_i\), it is \(P(Z_i\ge z_i\mid H_0)=1-\Phi(z_i)\). A two-sided nominal p-value is \(2[1-\Phi(|z_i|)]\).

The nominal p-value is tied to the direction and null distribution of the test. In an upper-tailed efficacy test, positive values of \(Z_i\) support the prespecified alternative. If the test is two-sided, evidence in either direction contributes to the p-value. A one-sided p-value should not be substituted for a two-sided one after seeing which direction the data went.

For the standard canonical group sequential model under the global null, suppose the planned looks occur at increasing information levels \(I_1<\cdots<I_K\). The standardized statistics have mean zero and variance one, and their joint normal dependence is described by \(\operatorname{Corr}(Z_i,Z_j)=\sqrt{I_i/I_j}\) for \(i<j\). This model is appropriate when the planned information levels are fixed and the statistics have the usual independent-increment, approximately normal structure. The correlation reflects that later analyses reuse the earlier participants’ information; the statistics are not independent repeated tests.

(Z_1,\ldots,Z_K)\mid H_0 \text{ is jointly normal},\qquad E(Z_i\mid H_0)=0,\quad \operatorname{Var}(Z_i\mid H_0)=1,\quad \operatorname{Corr}(Z_i,Z_j\mid H_0)=\sqrt{\frac{I_i}{I_j}}\quad(i<j).

For an upper-boundary efficacy design, let \(c_i\) be the prespecified critical value at look \(i\). The design declares efficacy if the statistic crosses the efficacy boundary at any planned look: \(Z_i\ge c_i\) for at least one \(i\). The overall type I error is the probability, under the global null, of crossing at least one of those boundaries. A calibrated design chooses the boundary values so this probability is controlled at the design’s target \(\alpha\), under the assumed look schedule and joint distribution.

\alpha_{\text{overall}} = P_{H_0}\left(\bigcup_{i=1}^{K}\{Z_i\ge c_i\}\right).

This is not generally the same as the nominal level used to describe one look. The nominal p-value answers a single-look question; the overall type I error concerns the full boundary-crossing rule. The boundary need not correspond to the ordinary single-look cutoff for \(\alpha\).

Why Nominal Significance Can Differ From Boundary Crossing

For a one-sided test with nominal level \(0.05\), the usual single-look critical value is approximately \(1.645\), since \(P(Z\ge1.645)\approx0.05\) for \(Z\sim N(0,1)\). But a sequential design may require a larger value at a particular interim look. Thus an observed \(Z_i\) can exceed \(1.645\), giving a nominal \(p<0.05\), while remaining below \(c_i\). The correct sequential decision is determined by the prespecified boundary, not by applying the single-look cutoff independently at every look.

The reason is that checking repeatedly creates multiple opportunities to cross a single-look cutoff. Under the global null, the probability of at least one crossing is the probability of a union of events. For two looks, the exact relationship is

P_{H_0}(A_1\cup A_2) = P_{H_0}(A_1)+P_{H_0}(A_2)-P_{H_0}(A_1\cap A_2), \qquad A_i=\{Z_i\ge c_i\}.

The overlap term matters because the statistics are correlated, but repeated opportunities still mean the overall error is not simply the single-look error. Boundary calibration accounts for the look schedule and dependence. It does not turn each interim nominal p-value into an “adjusted p-value”; it defines a sequential decision rule whose probability of a false boundary crossing is controlled.

Worked Example: Nominally Significant but Below the Interim Boundary

Suppose a hypothetical trial has a prespecified upper-tailed efficacy test at an interim look. The statistic is \(Z_1=2.05\), and the design’s calibrated boundary at this look is \(c_1=2.35\). Assume the test statistic follows the standard normal null model at this look, the direction was prespecified, and the boundary belongs to the planned design.

The one-sided nominal p-value is

p_{\mathrm{nom},1}=1-\Phi(2.05)\approx0.0202.

Because \(0.0202<0.05\), the result is nominally significant at the conventional one-sided 0.05 level. However, the sequential comparison is \(2.05<2.35\), so the statistic does not cross the interim efficacy boundary. The design therefore does not declare efficacy at this look. The nominal result indicates evidence against \(H_0\) at this analysis; it is not permission to replace the prespecified sequential rule with a single-look rule.

Worked Comparisons in Context

Worked Example: Calculating a Nominal P-Value From an Estimated Effect

In a hypothetical study, the estimated treatment benefit on a continuous outcome is \(4.2\) points, with standard error \(2.0\) points at an interim look. The prespecified alternative is that the treatment benefit is positive. Assume the null distribution of the standardized statistic is approximately standard normal and the planned interim boundary is \(c_i=2.30\).

The statistic is the estimate divided by its standard error:

Z_i=\frac{4.2\text{ points}}{2.0\text{ points}}=2.10.

The units cancel, as required for a standardized statistic. The one-sided nominal p-value is \(1-\Phi(2.10)\approx0.0179\). This is below \(0.05\), so the result is nominally significant by the ordinary one-look criterion. But \(2.10<2.30\), so it does not cross the stated sequential boundary. Under the design rule, this look alone does not support an efficacy stop. The calculation assumes the reported standard error and normal approximation are suitable for this analysis; it also relies on using the prespecified positive-benefit direction.

Worked Example: A Later Look Crosses Its Boundary

Consider a separate hypothetical upper-tailed design. At a planned later look, the estimated benefit is \(5.1\) points with standard error \(2.5\) points, and the design’s boundary at that look is \(c_i=1.98\). Assume the statistic has the stated standard normal null distribution and the boundary was calibrated for this design’s planned analyses.

The observed statistic is

Z_i=\frac{5.1\text{ points}}{2.5\text{ points}}=2.04.

The nominal one-sided p-value is \(1-\Phi(2.04)\approx0.0207\). The statistic exceeds the boundary because \(2.04\ge1.98\), so the efficacy criterion is met at this look. The nominal p-value is still a look-specific quantity; it should not be reported as though it were the probability that the null hypothesis is true or as though it were the design’s overall type I error. The sequential conclusion follows from the boundary crossing, with the design-wide error controlled by the calibration of the whole boundary sequence.

Derivation: The Single-Look Error Is Not the Sequential Error

The distinction can be seen directly from the definitions. If the same single-look cutoff \(c\) were used at each of two looks, each marginal null probability would be \(P(Z_i\ge c\mid H_0)\). Yet the chance of at least one crossing is

P(Z_1\ge c\text{ or }Z_2\ge c\mid H_0) = P(Z_1\ge c\mid H_0)+P(Z_2\ge c\mid H_0) -P(Z_1\ge c,Z_2\ge c\mid H_0).

Unless the crossing events are identical, the union probability exceeds either one marginal probability. If the two looks were independent, the expression would be \(1-[1-P(Z\ge c)]^2\); in a group sequential design, the looks are usually dependent, so the joint probability must use their correlation. Either way, a per-look cutoff alone does not establish the design-wide type I error. A calibrated sequential boundary is chosen for the joint event of crossing at one or more looks.

This argument does not require that all interim analyses be independent. It only requires recognizing that the overall false-positive event is a union of look-specific crossing events. For actual calibration, the dependence structure and the exact planned boundaries matter.

Practical Checks, Alternatives, and Limitations

When reviewing an interim result, keep the analysis-level evidence and the design decision separate. First verify the test direction, statistic, and nominal p-value calculation. Then compare the statistic with the boundary at that specific look. Finally, interpret the decision under the complete prespecified design, not as a fresh single-analysis test.

Check the tail. A one-sided nominal p-value is appropriate only when the direction was specified in advance. For a two-sided test, calculate the two-sided p-value and use boundaries calibrated for that testing convention.
Check the look index. Compare \(Z_i\) with \(c_i\), not with a boundary from an earlier or later analysis.
Check the schedule. Boundaries are tied to planned information levels and looks. Changed timing or additional unplanned analyses can invalidate the original calibration.
Check the model. The normal approximation, information levels, and dependence assumptions should be credible. A misspecified information schedule or invalid statistic can make the nominal p-value and boundary calculation unreliable.

A commonly used alternative is to conduct a single final analysis, avoiding interim efficacy decisions. That approach does not provide the same opportunity for early stopping and is not the same design. Another approach is to use a sequential boundary calibrated for the planned looks. It permits interim decisions while controlling the probability of a false efficacy crossing under the specified assumptions. Neither approach makes the nominal p-value itself a measure of the design-wide error.

A common pitfall is: “The interim p-value is below 0.05, so the trial has met its efficacy criterion.” This reasoning applies a single-look cutoff repeatedly, even though the design uses a sequential rule. The correction is to say: “The interim result has a nominal one-sided p-value below 0.05, but efficacy is declared at this look only if the statistic crosses its prespecified boundary.” If it does not cross, the result may still be scientifically informative, but it has not met that design’s stopping criterion.

These interpretations depend on following the planned analysis and on the validity of its statistical model. A nominal p-value does not account for unplanned looks, and a calibrated boundary does not repair a changed analysis plan or a seriously inadequate model. The boundary-crossing rule addresses a specific error probability under specified assumptions, not every source of uncertainty in a trial.

Key takeaway: A nominal p-value describes one look as if it were the only analysis. Overall type I error describes the probability of at least one false boundary crossing across the planned sequence. Use the prespecified interim boundary for the sequential efficacy decision.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

At an upper-tailed interim look, \(Z_i=1.80\) and the boundary is \(2.20\). Is the result nominally significant at one-sided \(0.05\), and is the boundary crossed?
In one sentence, distinguish the nominal p-value at look \(i\) from overall type I error.
Why does using the ordinary single-look \(0.05\) cutoff at every interim analysis generally fail to define a design with overall type I error \(0.05\)?
A prespecified two-sided analysis gives \(Z_i=-2.00\). What is its approximate nominal p-value under the standard normal null model, and what extra information is needed to decide whether the sequential boundary is crossed?

Solutions. (1) The nominal one-sided p-value is \(1-\Phi(1.80)\approx0.0359\), so it is below \(0.05\); however, \(1.80<2.20\), so the boundary is not crossed. (2) The nominal p-value is the single-look null tail probability for the observed statistic, whereas overall type I error is the null probability of crossing at least one boundary in the planned sequence. (3) Repeated analyses create a union of opportunities to cross; the joint crossing probability depends on the number and dependence of looks and is not generally the one-look error rate. (4) The two-sided nominal p-value is \(2[1-\Phi(2.00)]\approx0.0455\); to decide boundary crossing, one needs the prespecified boundary for that look and the relevant boundary convention.