Anytime-Valid Randomization P-Values

Why Sequential Monitoring Needs a Different Guarantee

A randomization test calibrated for one prespecified analysis time can control its type I error at that time. But repeatedly calculating such p-values and stopping as soon as one is small changes the procedure: the chance of finding at least one apparently significant result can exceed the nominal level. An anytime-valid p-value is designed for this monitoring situation. It controls the chance of crossing a rejection threshold at any time, not just at one chosen look.

The construction here uses the sequential assignment probabilities and randomization history established in the earlier material on adaptive randomization. Its reference null is a sharp null: for every unit, the outcome would be the same under either assignment. Conditional on those fixed outcomes, the only randomness used for inference is the trial’s assignment mechanism.

Definition: Let $\mathcal{F}_t$ denote the information available through assignment $t$, including assignments and observed outcomes, and let $\mathcal{F}_0$ be the initial information. A nonnegative process $E_t$ is a test supermartingale under the null if $E_0\le 1$ and $$ E_t\ge 0,\qquad \mathbb{E}(E_t\mid\mathcal{F}_{t-1})\le E_{t-1} $$ at every step. If equality holds in the conditional expectation, it is a test martingale. The process accumulates evidence against the null while its expected value cannot increase under the null.

The conditional expectation is taken under the actual sequential randomization design and the sharp null. Write $A_t$ for the assignment at step $t$, and $\pi_t(a)=P(A_t=a\mid\mathcal{F}_{t-1})$ for its known conditional assignment probability. A convenient way to build a test martingale is to choose a nonnegative factor $R_t$, calculated from the history and the new assignment and outcome, such that $$ \sum_a \pi_t(a)R_t(a,Y_t,\mathcal{F}_{t-1})=1 $$ under the sharp null. Then $E_t=\prod_{j=1}^t R_j$ is a nonnegative martingale. Under the sharp null, $Y_t$ is the same whichever assignment unit $t$ receives; conditional on the past, it is fixed for this randomization calculation. The factor’s conditional expectation is therefore taken over the assignment, using its actual probability $\pi_t(a)$.

From a Test-Martingale to an Anytime P-Value

A large value of $E_t$ is evidence against the null. To protect against repeated monitoring, the evidence must be compared with a boundary using its running maximum, not just its value at the current look.

Definition: For a nonnegative test supermartingale $E_t$ with $E_0=1$, define $$ p_t=\min\left\{1,\frac{1}{\max_{0\le s\le t}E_s}\right\}. $$ This is an anytime-valid p-value process if, under the null, for every $0<\alpha<1$, $$ P\left(\text{there is some }t\ge 0\text{ such that }p_t\le\alpha\right)\le\alpha. $$

Because the running maximum cannot decrease, $p_t$ cannot increase. A process that has crossed a rejection boundary does not later “undo” that crossing if its current evidence falls. For any particular time $t$, the event $p_t\le\alpha$ is the event that the running maximum has reached at least $1/\alpha$. The same event over all times is a boundary-crossing event, so a single bound controls repeated looks and stopping at a data-dependent time.

Why the Boundary-Crossing Bound Holds

The key justification is Ville’s inequality: for a nonnegative test supermartingale starting at most at 1, the probability it ever reaches $c>0$ is at most $1/c$. A finite-horizon argument shows the essential logic. Let $\tau$ be the first time $E_t\ge c$, and stop at $\tau$ or at a fixed horizon $T$, whichever comes first. The stopped process has expectation at most 1. On the event that the process crosses $c$ by time $T$, its stopped value is at least $c$; otherwise its stopped value remains nonnegative. Thus $$ 1\ge \mathbb{E}(E_{\tau\wedge T}) \ge c\,P(\tau\le T). $$ It follows that $P(\tau\le T)\le 1/c$. Letting $T$ increase gives the same bound for crossing at any finite time. Setting $c=1/\alpha$ proves the anytime guarantee for $p_t$.

This argument also explains the role of nonnegativity. Without it, values below zero could offset values above the crossing boundary in the expectation, and the displayed probability bound would not follow. For an unbounded monitoring horizon, the guarantee applies to crossing at any finite time; it does not require a prespecified final look.

Worked Example: A Response-Adaptive Test Factor

Consider a hypothetical two-arm trial with binary outcomes $Y_t\in\{0,1\}$, where $A_t=1$ denotes treatment and $A_t=0$ control. The conditional probability of treatment may change with the observed history; call it $\pi_t=P(A_t=1\mid\mathcal{F}_{t-1})$. Under the sharp null, use $$ R_t=1+(A_t-\pi_t)Y_t. $$ For $Y_t=0$, the factor is 1 for either assignment. For $Y_t=1$, it is $1-\pi_t$ under control and $2-\pi_t$ under treatment, both nonnegative for $0\le\pi_t\le1$. Its conditional expectation under the null is $$ \pi_t\{1+(1-\pi_t)Y_t\}+(1-\pi_t)\{1-\pi_tY_t\}=1. $$ So the product of these factors is a test martingale, provided the $\pi_t$ values are the design’s true conditional assignment probabilities.

Suppose a hypothetical observed path has $(\pi_1,A_1,Y_1)=(0.5,1,1)$, $(\pi_2,A_2,Y_2)=(0.2,1,1)$, and $(\pi_3,A_3,Y_3)=(0.7,0,1)$. The factors are $$ R_1=1+(1-0.5)(1)=1.5,\quad R_2=1+(1-0.2)(1)=1.8,\quad R_3=1+(0-0.7)(1)=0.3. $$ Thus $E_1=1.5$, $E_2=1.5(1.8)=2.7$, and $E_3=2.7(0.3)=0.81$. The running maximum through time 3 is 2.7, so $$ p_3=\min\{1,1/2.7\}=0.3704. $$ The running p-values at the first three looks are approximately 0.6667, 0.3704, and 0.3704. At level $\alpha=0.4$, this path crosses the rejection boundary at look 2. The value 0.3704 is not the probability that the sharp null is true; it is the anytime-valid p-value calculated from this observed path.

Repeated Looks and the Difference from Fixed-Time Tests

At a single prespecified look, a nonnegative test martingale also gives a valid fixed-time p-value $\min\{1,1/E_t\}$: Markov’s inequality bounds the probability that $E_t\ge1/\alpha$ by $\alpha$. But applying that rule separately at every look and rejecting when any one of those values is small does not provide the same repeated-monitoring guarantee. The anytime construction instead uses $\max_{s\le t}E_s$; its boundary is calibrated for the possibility of crossing at any look.

The issue is not unique to randomization inference. For illustration, if ten independent null p-values were each uniform on $[0,1]$, the probability that at least one is at most 0.05 would be $$ 1-(1-0.05)^{10}=0.4013. $$ This is a simple illustration of repeated testing, not a model for every trial’s sequence of p-values. The anytime guarantee does not require independence between looks: it follows from the test-supermartingale condition.

A conventional randomization test can remain appropriate when the analysis time is fixed in advance and its randomization distribution is correctly constructed. The anytime p-value addresses a different requirement: it remains calibrated when the analyst may examine results repeatedly and choose a stopping time based on the observed history. It does not make an invalid randomization distribution valid, nor does it automatically convert a test of a sharp null into a test of a weak average-effect null.

Worked Example: Checking the Adaptive Assignment Probability

Suppose that, in a hypothetical trial, the design assigns treatment with conditional probability $\pi_t=0.8$ at the current look. An analyst incorrectly uses the fixed probability 0.5 in the factor $R_t^{\text{wrong}}=1+(A_t-0.5)Y_t$. Under the sharp null, if $Y_t=1$, its conditional expectation is $$ 0.8\{1+(1-0.5)\}+0.2\{1+(0-0.5)\} =0.8(1.5)+0.2(0.5)=1.3. $$ The factor has expected value 1.3, not at most 1, so it does not satisfy the test-martingale condition. A small value calculated from a product of such factors has no established anytime-valid interpretation.

Using the actual assignment probability gives $R_t=1+(A_t-0.8)Y_t$. With $Y_t=1$, the treatment factor is 1.2 and the control factor is 0.2, so $$ 0.8(1.2)+0.2(0.2)=1.00. $$ If the observed assignment is treatment, the one-step evidence value is 1.2 and its corresponding single-look p-value is $1/1.2=0.8333$. This calculation checks the factor’s calibration under the sharp null; it does not establish that the selected factor will be powerful against every alternative.

Combining Evidence Processes

Different nonnegative test martingales can represent different directions or patterns of departure from the null. A fixed weighted average of valid processes is also a valid process: conditional expectation is linear, and a weighted average of nonnegative values remains nonnegative. For weights $w_k\ge0$ that sum to 1, if each $E_t^{(k)}$ is a test supermartingale, then $\sum_k w_k E_t^{(k)}$ is one as well. The weights must be specified without using future trial outcomes; choosing weights after inspecting the results can invalidate the argument.

Worked Example: Averaging Two Directional Processes

For the same binary-outcome setting, define factors $R_t^+=1+(A_t-\pi_t)Y_t$ and $R_t^-=1-(A_t-\pi_t)Y_t$. Each is nonnegative, and each has conditional expectation 1 under the sharp null because $E(A_t-\pi_t\mid\mathcal{F}_{t-1})=0$ and $Y_t$ is fixed under that null. Let $E_t^+$ and $E_t^-$ be their respective products. Then $E_t=(E_t^++E_t^-)/2$ is a test martingale.

Suppose a hypothetical two-step path has $(\pi_1,A_1,Y_1)=(0.5,1,1)$ and $(\pi_2,A_2,Y_2)=(0.25,1,1)$. The plus factors are 1.5 and 1.75, giving $E_2^+=2.625$; the minus factors are 0.5 and 0.25, giving $E_2^-=0.125$. Their average is $E_2=1.375$. Since $E_0=1$ and $E_1=(1.5+0.5)/2=1$, the running maximum is 1.375 and the anytime p-value at the second look is $$ p_2=1/1.375=0.7273. $$ The mixture is valid, but this particular observed path gives little evidence against the null. A valid construction guarantees error control, not high power.

Diagnostics, Limitations, and Common Mistakes

Before interpreting an anytime p-value, check that the claimed process really meets its conditions under the randomization distribution. In particular, verify the assignment probabilities at each step, calculate the factor’s conditional expectation under the sharp null, and confirm that every factor is nonnegative. When assignments depend on previous outcomes, substituting a fixed allocation probability for the history-dependent probability can break the conditional expectation calculation, as the previous example showed.

Using only the current evidence. The value $1/E_t$ is a fixed-look calibration, not a repeated-monitoring rule. Use the running maximum of $E_s$ for the anytime p-value.
Ignoring the sharp null. The argument treats each unit’s outcome as fixed across assignments. A weak null about an average effect does not generally make those outcomes fixed, so the same factor needs a separate justification for that null.
Assuming any score is a test factor. A score must be converted into a nonnegative factor with conditional expectation at most 1 under the null. A large or intuitive score alone does not establish validity.
Using incorrect assignment probabilities. The conditional expectation is taken over the design’s actual next assignment. If the probabilities cannot be reconstructed correctly, this validity argument does not apply.

Anytime-validity is a calibration guarantee, not a guarantee of informative evidence. A factor can be valid yet have little power, and extreme or deterministic assignment probabilities can restrict what the data can reveal. The result also does not select the best test factor: different alternatives can favor different evidence processes.

Check Your Understanding

Answer each question, then compare with the solutions below.

Why does the anytime p-value use the running maximum of $E_s$ rather than $E_t$ alone?
A process starts at $E_0=1$ and has running maximum 4 by the current look. What is its anytime p-value, and does this path cross the 0.2 boundary?
Under the sharp null, a factor has conditional expectation 1.08 at a particular step. Does it satisfy the test-supermartingale condition at that step?
Why does a test martingale justified under a sharp null not automatically test a weak average-effect null?

Solutions. (1) The running maximum accounts for a crossing at any earlier look, which is what controls repeated monitoring. (2) The p-value is $\min\{1,1/4\}=0.25$; it does not cross 0.2 because 0.25 is greater than 0.2. (3) No. The conditional expectation must be at most the previous process value; for a multiplicative factor with previous value positive, a factor expectation of 1.08 increases that value and violates the condition. (4) A sharp null fixes every unit’s outcome across assignments, allowing the conditional randomization calculation. A weak average-effect null does not generally impose that unit-level equality, so the same calculation need not hold.