From a Stopped-Data P-Value to a Test
A sequential trial can stop at an interim look or continue to its planned final analysis. The observed result is therefore not just a test statistic: it includes the stopping outcome, such as the look at which the trial ended. A stagewise p-value is designed to assess that stopped outcome using its place in an ordering of possible stopped outcomes. The key question here is what follows once that p-value has been calculated.
The answer depends on validity under the null hypothesis. If a stagewise p-value is valid for the full sequential procedure, then comparing it with the planned significance level defines a sequential test with controlled Type I error. Validity must account for the stopping rule and the complete null distribution of the stopped data; a p-value calculated as though the observed look had been fixed in advance does not automatically have this property.
The probability in this definition is calculated over repeated trials that follow the specified sequential design, including all planned looks, the stopping rule, and the resulting stopped-data ordering. It is not a probability that \(H_0\) is true. Rather, it describes how often the procedure would produce a p-value at or below a chosen threshold if a particular null parameter value generated the data.
Why the Threshold Rule Controls Type I Error
For a chosen significance level \(\alpha\), define the test to reject \(H_0\) exactly when the observed stagewise p-value is at most \(\alpha\). Its rejection event is therefore the event \(P_{\mathrm{seq}}\le\alpha\). Under any null parameter value, validity gives an immediate bound on the probability of that event.
The largest rejection probability over the null parameter space, called the test’s size, is consequently at most \(\alpha\). This is the defining level-\(\alpha\) guarantee. The result does not depend on whether the trial stopped early or at its final look: those possibilities are already included in the distribution used to establish p-value validity.
This implication is short, but its premise is substantive. It is not enough that the number is labelled “stagewise p-value.” The calculation must give a valid p-value for the actual design and ordering. Calibration Across Different Look Schedules is one reason this matters: the same numerical statistic, or even the same nominal p-value formula, need not have the same null distribution under different monitoring schedules.
Worked Example: A Two-Stage Stopping Rule
Worked Example: A Two-Stage Stopping Rule
Consider a hypothetical experiment with independent binary outcomes. Each observation is a success with probability \(q\) and a failure with probability \(1-q\). The trial stops after one observation if it is a success, denoted \(S\). If the first observation is a failure, the trial takes one more observation and stops; the possible stopped outcomes are \(FS\) and \(FF\). Test \(H_0:q\le 0.5\) against \(H_a:q>0.5\). The independence and constant-success-probability assumptions make the stopped-path probabilities well-defined. For ordering evidence, use the likelihood ratio of each stopped path under the fixed alternative value \(q_1=0.75\) against the null boundary \(q_0=0.5\).
Under \(q_0=0.5\), the stopped-path probabilities are $$ P_{0.5}(S)=0.5,\qquad P_{0.5}(FS)=(0.5)(0.5)=0.25,\qquad P_{0.5}(FF)=(0.5)(0.5)=0.25. $$ They sum to 1, as required. Under \(q_1=0.75\), the corresponding probabilities are \(0.75\), \(0.1875\), and \(0.0625\). Dividing each alternative probability by its null probability gives the likelihood ratios: $$ LR(S)=\frac{0.75}{0.5}=1.5,\qquad LR(FS)=\frac{0.1875}{0.25}=0.75,\qquad LR(FF)=\frac{0.0625}{0.25}=0.25. $$ Thus the evidence ordering is \(S\), then \(FS\), then \(FF\), from most to least supportive of \(H_a\).
The stagewise p-value is the null probability of outcomes at least as extreme as the observed outcome in this ordering. Hence $$ P_{\mathrm{seq}}(S)=0.5,\qquad P_{\mathrm{seq}}(FS)=0.5+0.25=0.75,\qquad P_{\mathrm{seq}}(FF)=1. $$ The tie between \(FS\) and \(FF\) under the null does not arise in this chosen ordering, but both outcomes are included when calculating the tail for \(FF\). The p-values are not smaller than the null probabilities of their corresponding upper-tail sets.
To check validity across the composite null \(q\le0.5\), consider any threshold \(u\). If \(0\le u<0.5\), no outcome has p-value at most \(u\), so the rejection probability is 0. If \(0.5\le u<0.75\), only \(S\) is included, and $$ P_q(P_{\mathrm{seq}}\le u)=P_q(S)=q\le0.5\le u. $$ If \(0.75\le u<1\), both \(S\) and \(FS\) are included, and $$ P_q(P_{\mathrm{seq}}\le u)=q+(1-q)q=2q-q^2\le 2(0.5)-(0.5)^2=0.75\le u. $$ The inequality follows because \(2q-q^2\) increases on \(0\le q\le0.5\). At \(u=1\), all outcomes are included and the probability is 1. Thus the p-value is valid for every \(q\in H_0\).
At \(\alpha=0.60\), the rule \(P_{\mathrm{seq}}\le\alpha\) rejects only for \(S\). Its rejection probability under \(q=0.5\) is \(0.5\), which is at most \(0.60\); for any \(q\le0.5\), it is \(q\le0.5\). At \(\alpha=0.75\), the rule rejects for \(S\) or \(FS\), with null-boundary rejection probability \(0.5+0.25=0.75\). The p-value threshold and the corresponding sequential rejection region agree exactly in both cases.
The likelihood ordering above is strict for any fixed alternative success probability \(q_1\) with \(0.5<q_1<1\): \(S\) ranks above \(FS\), and \(FS\) above \(FF\). At \(q_1=1\), however, \(P_1(FS)=P_1(FF)=0\); those two paths tie in likelihood rather than having a strict ordering. The example uses \(q_1=0.75\), so its stated strict ordering is justified.
The Equivalence and What It Does Not Say
For any fixed \(\alpha\), the decision is exactly equivalent to membership in the set of stopped outcomes whose stagewise p-values are at most \(\alpha\). In the example, that set is \(\{S\}\) when \(\alpha=0.60\) and \(\{S,FS\}\) when \(\alpha=0.75\). This is a direct equivalence between two descriptions of the same decision rule, not a claim that every sequential boundary procedure has automatically been represented by a particular stagewise p-value.
For the converse to hold—namely, for a prespecified sequential test to be reproduced by thresholding a p-value—its rejection regions must match the p-value’s threshold sets at the relevant \(\alpha\). A valid p-value guarantees error control when thresholded; validity alone does not guarantee that its ordering reproduces every alternative test’s rejection region. The ordering used to calculate the p-value matters.
A simple derivation explains why discrete p-values can be conservative. Suppose the distinct p-values are ordered as \(r_1<r_2<\cdots<r_m\). For a threshold \(u\) between two successive values, the event \(P_{\mathrm{seq}}\le u\) is unchanged until \(u\) reaches the next value. Its probability therefore stays fixed while the permitted upper bound \(u\) increases. The probability can be strictly less than \(u\), but validity requires only that it never exceed \(u\).
Worked Example: A Threshold Between Attainable P-Values
Use the two-stage example under the null boundary \(q=0.5\), where the possible p-values are 0.50, 0.75, and 1.00. For \(\alpha=0.70\), the rejection event remains \(P_{\mathrm{seq}}=0.50\), which occurs only for \(S\). Its null probability is \(0.50\), so $$ P_{0.5}(P_{\mathrm{seq}}\le0.70)=0.50\le0.70. $$ The test is level 0.70, although its size at the null boundary is only 0.50. The gap is due to discreteness: there is no attainable p-value between 0.50 and 0.75 that would add a further outcome to the rejection region.
Comparisons, Diagnostics, and Limitations
| Approach | What controls the rejection decision? | Key condition or risk |
|---|---|---|
| Threshold a valid stagewise p-value | Reject when the p-value for the stopped outcome is at most \(\alpha\). | Validity must account for the full stopping procedure and the stated null. |
| Use a fixed-look p-value after stopping | Compare a nominal p-value for the observed look with \(\alpha\). | That p-value may ignore selection of the stopping look and need not control sequential Type I error. |
| Use a prespecified sequential boundary | Reject when the observed statistic crosses a design boundary. | Its error control comes from calibration of the sequential design. A stagewise p-value reproduces its decision only if their rejection regions match. |
Several checks help detect a mismatch between the reported p-value and the test actually used:
- Check the null scope. Validity under a single null value does not by itself establish level control over a composite \(H_0\).
- Check the stopping mechanism. The null distribution must include the planned looks, the stopping rule, and the stopped outcomes that could have occurred.
- Check the ordering and ties. In a discrete sample space, specify how equally extreme outcomes are included. Omitting tied outcomes can make a tail probability too small.
- Check conditional versus unconditional calibration. A p-value valid conditional on a particular stopping look is not automatically valid unconditionally, and an unconditional p-value need not provide a conditional guarantee at each look.
- Check the decision rule actually applied. If the protocol rejects using a boundary crossing but the report compares a different p-value with \(\alpha\), confirm that the two rejection regions agree before describing them as equivalent.
A common mistake is to calculate a conventional one-sided p-value as if the observed look had been fixed in advance, then conclude that comparing it with \(\alpha\) gives a level-\(\alpha\) sequential test. That conclusion does not follow merely from the fact that the number is a p-value at that look. The correction is to use a p-value whose super-uniformity is established for the full stopped-data procedure, or to use a sequential test whose error control has been calibrated directly. This tutorial’s result is conditional on validity; it does not validate an uncalibrated calculation.
There are practical limitations even when validity is established. A valid p-value can be conservative, particularly with discrete stopped outcomes, so its threshold test may use less than the nominal level. Its value also depends on the ordering used to define “at least as extreme.” Different orderings can assign different p-values to the same observed path while preserving validity, and therefore can produce different rejection regions at a given \(\alpha\). Finally, the test controls the probability of rejecting under the null; it does not imply that a small p-value is the probability that the null is true.
Worked Example: Using a Calibrated Three-Look Result
Suppose a hypothetical three-look analysis has a prespecified stopped-outcome ordering. For the observed stopped result, an exact calculation under a simple null gives a stagewise tail probability with contributions \(0.0061\), \(0.0044\), and \(0.0022\) from the relevant disjoint sets of stopped outcomes. Assume the null calculation includes all three looks and the stopping rule, and that the resulting p-value is valid under the null.
The observed stagewise p-value is $$ P_{\mathrm{seq}}=0.0061+0.0044+0.0022=0.0127. $$ At \(\alpha=0.01\), \(0.0127>0.01\), so this test does not reject \(H_0\). At \(\alpha=0.02\), \(0.0127\le0.02\), so it rejects. Under the stated validity assumption, the latter threshold rule has null rejection probability at most 0.02 over repeated applications of this same sequential procedure. The comparison is justified by the calibrated stagewise p-value, not by treating the observed look as if it had been selected in advance.
Check Your Understanding
Answer each question using the definition of validity and the role of the stopping rule.
- If \(P_{\mathrm{seq}}\) is valid under every \(\theta\in H_0\), what inequality bounds the probability of rejecting when the test uses \(P_{\mathrm{seq}}\le\alpha\)?
- In the two-stage example, which stopped outcomes are rejected at \(\alpha=0.74\), and what is the rejection probability at \(q=0.5\)?
- Why can a discrete valid p-value produce a test whose size is less than \(\alpha\)?
- Does validity at the null boundary alone always establish level control for a composite null? Explain.
- Why is a fixed-look p-value not automatically valid after a sequential stopping decision?
Solutions. (1) \(P_{\theta}(P_{\mathrm{seq}}\le\alpha)\le\alpha\) for every \(\theta\in H_0\), so the test has size at most \(\alpha\). (2) Since \(0.50\le0.74<0.75\), only \(S\) is rejected; at \(q=0.5\), its probability is 0.50. (3) The rejection event may stay unchanged between attainable p-values, leaving its probability below the threshold. (4) No. Control at one boundary value does not establish control at every null value unless a separate argument shows the bound holds over the full null space. (5) Its null distribution may not account for the chance that the observed look was selected by the stopping rule, so super-uniformity for the stopped procedure is not guaranteed.