P-Values Under Actual Interim Look Timing

Why Actual Look Timing Matters

A sequential p-value is a probability under the stopping rule that was actually applied. If an interim analysis occurs at a different information time from the planned time, the standardized statistic at that look may still have a standard normal marginal distribution under the null, but its dependence on statistics at other looks changes. Since a stagewise p-value includes probabilities of crossing at earlier looks as well as at the observed look, this change in dependence can change the p-value.

The boundary convention matters too. Some designs assign a fixed boundary to each numbered look. Others specify a boundary function of information time. In the first case, use the boundary assigned to the look that actually occurred. In the second, evaluate the specified function at the actual information time. Do not silently substitute a boundary from the planned schedule or redesign the boundary after seeing the data.

This explicit ordering is important: a sequential p-value is not determined by the observed \(z\)-statistic alone. If the analysis also has futility stops, two-sided boundaries, or a different ordering, the relevant outcome regions must reflect those features. The calculations below focus on the stated upper-tail efficacy ordering.

Model and Calculation Under the Actual Schedule

Assume the canonical Gaussian model under \(H_0\): information time is measured on a common scale, the score process has independent increments, and standardization gives \(Z_i\sim N(0,1)\) at each look. For deterministic information times, its joint null distribution satisfies

Thus, for two looks, the correlation is \(\rho=\sqrt{t_1/t_2}\). This is generally not the correlation obtained by inserting the planned information times when the actual times differ. The model also assumes the information times are fixed in advance or are determined independently of the accumulating outcomes. If the timing itself responds to the emerging results, conditioning on the observed schedule alone may not give the correct null distribution.

For an efficacy stop at look \(k\) with observed statistic \(z\), let \(C_j\) denote the event of an efficacy stop at look \(j\). Under the ordering just defined, the p-value is

Because the observed outcome is an efficacy stop, \(z\ge b_k\), so the final two requirements are equivalent to \(Z_k\ge z\). This probability is evaluated using the joint distribution at the actual information times and the boundaries applicable under the design.

For two looks and an observed efficacy stop at look 2, the result reduces to

These two events are disjoint: the first is an efficacy stop at look 1; the second is no efficacy stop at look 1 followed by a look-2 statistic at least as large as the observed value. Under the canonical model, the second probability can be computed as

where \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The conditional distribution used here is \(Z_2\mid Z_1=x\sim N(\rho x,1-\rho^2)\). The formula follows by integrating the conditional probability \(P_0(Z_2\ge z\mid Z_1=x)\) over values \(x<b_1\), then adding the disjoint probability of an earlier efficacy stop.

Boundary Conventions and Practical Checks

Before calculating, establish what the protocol means by a boundary when timing shifts. If \(b_i\) is a fixed value attached to look number \(i\), use that value at the actual \(i\)th look. If the boundary is specified as \(b(t)\), evaluate \(b(t_i)\). These are different design conventions; neither should be assumed from the numerical boundary alone. A boundary table calculated only at planned times does not automatically define a boundary at every possible actual time.

A practical calculation should be checked against the data and design record, not just against software output. Confirm the actual information times, their ordering, the look number, the stopping status, the boundary convention, and the observed statistic. Then verify that the covariance matrix is positive definite and consistent with those times. For example, with two looks the calculated correlation must be between 0 and 1; a value outside that range signals an error in the time inputs or formula.

Numerical integration or multivariate normal software can evaluate the required probabilities, but the integration limits and event region must represent the stopping rule. Simulation can be a useful check: simulate joint statistics under the null at the actual times, apply the actual boundaries, and estimate the proportion in the stagewise tail region. Agreement within Monte Carlo error is a diagnostic, not a substitute for specifying the correct probability.

What Recalculation Does—and Does Not—Do

Recalculating the p-value under actual timing answers a specific question: under the assumed null model and the stopping rule as applied, how much null probability lies in outcomes at least as extreme as the observed stopped outcome under the stated ordering? It does not by itself restore a nominal type I error rate. If a design's boundaries were calibrated for one schedule and actual timing changes, the probability of crossing a boundary under \(H_0\) may differ from the planned error rate. A recalculated p-value describes the observed outcome using the actual rule; it does not retroactively recalibrate that rule.

The main alternative is to analyze using planned look times and their planned joint distribution. That is appropriate only when those times genuinely describe the analyses conducted or when a prespecified design-based convention justifies that analysis. Another possibility is to use a design explicitly defined by an information-time boundary function. In either case, the analysis must match the protocol and the actual stopping mechanism; choosing whichever schedule yields a smaller p-value is not valid.

The canonical Gaussian calculation also has limits. It may be inadequate if information is not correctly measured, the test statistic is not well approximated by the assumed process, increments are not independent, or actual analysis timing depends on emerging outcomes in a way not represented by the model. With outcome-dependent timing, the timing process may need to be included in the null probability calculation. Merely substituting the observed times into a fixed-time covariance formula does not automatically account for selection induced by that timing.

A common mistake is to argue that the same observed \(Z_k\) must imply the same sequential p-value because each \(Z_i\) has a standard normal null marginal distribution. The correction is that the stagewise p-value includes earlier stopping regions and their joint probability with the observed look. Marginal distributions alone do not determine that probability: the actual information times determine the dependence between looks and hence the probability of the ordered stopped outcomes.