Why Ties Matter After Stopping
A stopped trial can have a discrete set of possible outcomes: the trial may stop at different looks, and the observed statistic at a look may itself take only a limited set of values. An ordering of stopped outcomes assigns each outcome a level of evidence against \(H_0\). When distinct outcomes have the same evidence under that ordering, they are tied. Their probability cannot be ignored when calculating a tail probability.
This tutorial assumes a prespecified ordering and a null distribution for the full stopped outcome, including the stopping look and any recorded statistic needed by the ordering. The null probabilities of all possible stopped outcomes sum to one. The main results below are stated for a simple null distribution, or for a reference distribution already known to give valid null probabilities. The ordering must be fixed independently of the observed outcome, and larger ordering scores will mean stronger evidence against \(H_0\).
The tie mass includes the null probabilities of all stopped outcomes assigned the observed score, even when they differ in look or in their original data values. It is the ordering that defines a tie, not a requirement that the raw outcomes be identical. Conversely, outcomes with different scores must not be combined merely because they occurred at the same look.
Calculating the Two p-Values
For a one-sided test in which larger scores are more extreme, the inclusive-tail p-value adds the probability of every outcome with score at least as large as the observed score. This includes the entire tie class. The randomized version includes the strict tail and a random fraction of the tie mass. Because \(U\) is uniform, that fraction is not always one half.
Worked Example: An Observed Score in a Discrete Null Distribution
Suppose a hypothetical stopped-outcome distribution under \(H_0\) has the following score probabilities. The observed score is \(R=3\).
| Score \(R\) | Null probability |
|---|---|
| 4 | 0.08 |
| 3 | 0.17 |
| 2 | 0.25 |
| 1 | 0.30 |
| 0 | 0.20 |
The probabilities sum to \(0.08+0.17+0.25+0.30+0.20=1.00\). Scores greater than 3 have probability \(0.08\), and the tie mass at score 3 is \(0.17\). Thus the conservative p-value is
If the independently generated random number is \(U=0.40\), the randomized p-value is
The randomized value is smaller here because this particular \(U\) assigns 40% of the tie mass. It is not a deterministic alternative calculation for the same observed data: another independent draw of \(U\) could produce a different randomized p-value.
Why the Inclusive Tail Is Superuniform
A p-value is superuniform under \(H_0\) if \(P_0(p\le a)\le a\) for every \(a\) between 0 and 1. This is the relevant validity guarantee: at any nominal level \(a\), rejecting when \(p\le a\) has null rejection probability no greater than \(a\). In a discrete distribution, the inclusive-tail p-value usually takes only a few values, so its null distribution can be conservative rather than exactly uniform.
Write them from most to least extreme, \(r_1>r_2>\cdots>r_m\), and let \(q_j=P_0(R=r_j)\). Each \(q_j\) is the total probability of that score's tie class.
Set \(C_0=0\) and \(C_j=\sum_{i=1}^{j}q_i\). At score \(r_j\), the inclusive-tail p-value is \(C_j\).
For any \(a\), the outcomes satisfying \(p_{\mathrm{cons}}\le a\) are an initial segment of these ordered score levels. If the last included level is \(k\), their total probability is \(C_k\le a\). If no level is included, the probability is zero. Hence \(P_0(p_{\mathrm{cons}}\le a)\le a\).
This argument depends on using the null probabilities for the complete stopped-outcome space and on including the entire observed tie class. It does not require each possible outcome to have a different score.
For the randomized p-value, conditional on observing score \(r_j\), the value is \(C_{j-1}+Uq_j\). It is uniformly distributed over the interval from \(C_{j-1}\) to \(C_j\). These intervals partition \([0,1]\), and score \(r_j\) occurs with probability \(q_j\), exactly the length of its interval. Therefore the randomized p-value is uniform on \([0,1]\) under the specified null distribution. This is exact calibration over repeated trials together with the independent randomization; it does not mean that the unknown parameter is random.
Ties Across Looks and Stopped Outcomes
In a group sequential setting, the null distribution must assign probability to outcomes according to the stopping rule. A tie class may contain outcomes from more than one look if the prespecified ordering gives them the same score. The look labels remain part of the outcome space for determining the probability mass, even when the ordering groups those outcomes together.
Worked Example: Combining a Tie Class Across Looks
Consider a hypothetical one-sided stopped-outcome distribution. Its scores and null probabilities are summarized below; two distinct outcomes at different looks receive score 3. The observed outcome is one of those score-3 outcomes.
| Stopped outcome | Score | Null probability |
|---|---|---|
| Efficacy stop at look 1, outcome A | 4 | 0.03 |
| Efficacy stop at look 2, outcome B | 4 | 0.02 |
| Efficacy stop at look 1, outcome C | 3 | 0.07 |
| Final-look outcome D | 3 | 0.05 |
| Other stopped outcomes with score 2 | 2 | 0.33 |
| Other stopped outcomes with score 1 | 1 | 0.48 |
| Other stopped outcomes with score 0 | 0 | 0.02 |
The probabilities total \(0.03+0.02+0.07+0.05+0.33+0.48+0.02=1.00\). The score-4 strict tail has mass \(0.03+0.02=0.05\). The score-3 tie mass is \(0.07+0.05=0.12\), not merely the probability of the observed outcome's particular look. Therefore
If an independent draw gives \(U=0.75\), then
The calculation uses the unconditional null distribution of stopped outcomes. Replacing it with probabilities conditional on stopping at the observed look would answer a different, conditional question and would not be this unconditional stagewise p-value.
Why a Mid-p Value Is Not the Same as Randomization
A common shortcut is to take half of the tie mass, giving \(P_0(R>r)+\tfrac12P_0(R=r)\). This is often called a mid-p value. It equals the randomized formula only when \(U\) happens to be 0.5. Fixing the fraction at one half removes the randomization that produced exact calibration, so a mid-p value is not guaranteed to be superuniform.
Worked Example: A Mid-p Value Can Be Anti-Conservative
Suppose a hypothetical null distribution has a high-evidence score with probability \(0.20\) and a low-evidence score with probability \(0.80\). The high score is observed. Its strict tail mass is zero and its tie mass is \(0.20\). The inclusive p-value is \(0.20\), while the mid-p value is
At nominal level \(a=0.15\), the mid-p value is at most \(0.15\) whenever the high score occurs. That event has null probability \(0.20\), so
Thus this mid-p calculation is anti-conservative at that level. By contrast, the inclusive p-value is never at most \(0.15\) in this example, and so its rejection probability is zero. The randomized value is \(0.20U\) at the high score and \(0.20+0.80U\) at the low score; over both outcomes and the independent draw of \(U\), it is uniform under \(H_0\).
Practical Checks and Limitations
Before reporting either value, verify that the score direction matches the alternative, that every outcome in the observed tie class has been included, and that the null probabilities cover all possible stopping outcomes. A useful numerical check is that the probabilities of all distinct score classes sum to one and that the cumulative tail masses are nondecreasing from the most extreme score to the least extreme score.
- Omitting the tie mass: using only \(P_0(R>r)\) makes the result smaller than the inclusive p-value and can invalidate the superuniform guarantee.
- Breaking ties after seeing the data: an outcome-specific ranking invented after observing the result is not the prespecified ordering for which the null calibration was established.
- Using the wrong stopping distribution: ignoring the stopping rule, or silently conditioning on the observed look, can give the wrong probability mass for the stopped outcome.
- Treating a randomized value as reproducible: the reported randomized p-value depends on the independent draw \(U\); document the randomization if it is used.
The inclusive-tail version is usually preferred when a deterministic, reproducible p-value with a direct type-I error guarantee is required, although discreteness can make it conservative. Randomization gives exact null calibration for a fully specified distribution, but introduces an auxiliary random choice that may be undesirable in practice. A mid-p value can reduce conservatism, but unlike either the inclusive-tail value or the properly randomized value, it has no general superuniformity guarantee. For a composite null, the null probabilities must also be handled so validity holds across the null parameter space; the simple-null derivation alone does not establish that extension.
Check Your Understanding
Answer each question, then compare your reasoning with the concise solutions below.
- An observed score has strict tail mass 0.06 and tie mass 0.14. What are the conservative and randomized p-values if \(U=0.30\)?
- Why must tied outcomes at different looks be combined when the prespecified score assigns them the same value?
- State the superuniformity condition for a p-value and identify which of the two main versions guarantees it under the stated null distribution.
- In the two-score example, what is the probability that the mid-p value is at most 0.15?
Solutions. (1) \(p_{\mathrm{cons}}=0.06+0.14=0.20\), and \(p_{\mathrm{rand}}=0.06+(0.30)(0.14)=0.102\). (2) The tie mass is the total null probability of all outcomes assigned the observed score, regardless of their look labels; omitting some of that mass understates the inclusive tail. (3) Superuniformity means \(P_0(p\le a)\le a\) for every \(a\in[0,1]\). The inclusive-tail p-value guarantees it, and the independently randomized version is exactly uniform under the specified null distribution. (4) The high score occurs with probability 0.20 and yields mid-p 0.10, so the probability is 0.20, which exceeds 0.15.