Adaptive Tests for Discrete Outcomes

Why Discrete Outcomes Need Careful Combination

For a continuous outcome, a valid null p-value is often approximately uniform, which makes familiar combination rules easy to interpret. With a discrete outcome, attainable p-values are restricted to a set of possible values. A valid p-value can therefore be conservative: its probability of being as small as a specified cutoff is no greater than that cutoff, and may be substantially smaller. In an adaptive design, this distinction matters because the second-stage sample size or test may be chosen after the first-stage data are observed.

The central safeguard is to use stagewise p-values that remain valid under the null, including conditionally after the first-stage information and the adaptation decision are known. A conditional-error bound then limits how often the adapted second stage can reject. The approach below makes this bound explicit and allows the second-stage discrete test or sample size to change.

Definition: A p-value \(P\) is super-uniform under the null if, for every \(u\in[0,1]\) and every parameter value satisfying \(H_0\), \(P(P\leq u)\leq u\). For a discrete test statistic \(T\), a nonrandomized upper-tail p-value may be defined as \(P_{\theta_0}(T\geq T_{\mathrm{obs}})\) under a simple null value \(\theta_0\). For a composite null, the p-value procedure must control this tail probability for every parameter value in \(H_0\), for example by using a valid worst-case tail probability.

For an adaptive two-stage test, assume that the first-stage p-value \(P_1\) is super-uniform under \(H_0\). The second-stage p-value \(P_2\) must also be super-uniform conditional on the first-stage information and the chosen second-stage design, under every null parameter value. This conditional validity generally follows when stagewise data are independent under the null, the second-stage sample size is selected using only first-stage information, and the selected second-stage test is valid at that sample size. These conditions must be justified for the actual design; a p-value that is valid marginally but not conditionally is not enough.

Fixed Inverse-Normal Combination With Discrete P-Values

A standard reference rule combines two one-sided stagewise p-values using fixed positive weights \(w_1,w_2\), chosen before the stagewise results are examined, with \(w_1^2+w_2^2=1\). Write \(\Phi\) for the standard normal cumulative distribution function, \(\Phi^{-1}\) for its quantile function, and \(z_{1-\alpha}=\Phi^{-1}(1-\alpha)\). The rule rejects for sufficiently large values of the inverse-normal combination statistic.

C=w_1\Phi^{-1}(1-P_1)+w_2\Phi^{-1}(1-P_2),\qquad \text{reject }H_0\text{ if }C\geq z_{1-\alpha}.

If both p-values are independent uniform random variables under the null, then each transformed value is standard normal and \(C\sim N(0,1)\). For discrete p-values, each transformed value is instead stochastically no larger than a standard normal variable. Because the stages are independent and the weights are positive, the weighted sum is stochastically no larger than the corresponding sum of independent standard normal variables. Thus the fixed-rule rejection probability is at most \(\alpha\). It may be strictly less than \(\alpha\), which is the conservatism introduced by discreteness.

Worked Example: Evaluating a Fixed Combination Statistic

Suppose a hypothetical one-sided two-stage test uses \(P_1=0.040\), \(P_2=0.060\), equal weights, and \(\alpha=0.025\). The values are assumed to come from stagewise tests whose p-values are valid under the null, with independent stages.

For equal weights, \(w_1=w_2=1/\sqrt{2}\). The observed combination statistic is

C=\frac{\Phi^{-1}(0.960)+\Phi^{-1}(0.940)}{\sqrt{2}} =\frac{1.7507+1.5548}{1.4142} =2.337.

The one-sided critical value is \(z_{0.975}=1.960\). Since \(2.337>1.960\), this rule rejects \(H_0\) at the 0.025 level. This conclusion concerns the prespecified combination rule; the p-values themselves do not imply that either stage separately passes a 0.025 threshold. If the stagewise p-values are discrete, the procedure's actual null rejection probability can be below 0.025 even though the critical value is set using the standard normal distribution.

Using Conditional Error When the Second Stage Changes

A fixed combination rule is a useful reference, but simply changing its weights after seeing the first-stage result does not automatically preserve its Type I error. The adaptation can select weights that favor unusually promising first-stage results. A direct way to protect the error rate is to calculate the original rule's conditional error and require the adapted second stage to spend no more than that amount.

For the fixed inverse-normal rule, after observing \(P_1=p_1\), define the conditional error \(A(p_1)\) as the probability that the original combination rule would reject if its second-stage p-value were uniform under the null. Put \(z_1=\Phi^{-1}(1-p_1)\). The original boundary requires the second-stage transformed value to be at least \((z_{1-\alpha}-w_1z_1)/w_2\), so

A(p_1) =1-\Phi\!\left(\frac{z_{1-\alpha}-w_1\Phi^{-1}(1-p_1)}{w_2}\right) =\Phi\!\left(\frac{w_1\Phi^{-1}(1-p_1)-z_{1-\alpha}}{w_2}\right).

An adapted second-stage test can reject only when its conditionally valid p-value satisfies \(P_2\leq A(p_1)\). This rule may be implemented with a different second-stage sample size or a different valid discrete test. The key requirement is that the resulting \(P_2\) remains super-uniform conditional on the first-stage result and the selected design.

Theorem: Suppose \(P_1\) is super-uniform under \(H_0\), and, conditional on the first-stage information and adaptation decision, \(P_2\) is super-uniform under \(H_0\). If \(A(p_1)\) is the conditional-error function for a fixed inverse-normal rule with positive weights satisfying \(w_1^2+w_2^2=1\), then the adapted rule \(P_2\leq A(P_1)\) has Type I error at most \(\alpha\), for every null parameter value.

The argument has two parts. Conditional validity of \(P_2\) limits the chance of rejection after the first stage. Then super-uniformity of \(P_1\) ensures that averaging this conditional allowance cannot exceed the original level.

Bound the adapted second-stage rejection chance.
For any realized first-stage information and selected design, \(P(P_2\leq A(P_1)\mid\text{first stage and design})\leq A(P_1)\).

Average the conditional bound.
The total rejection probability is at most \(E[A(P_1)]\).

Compare with the continuous reference.
\(A(p_1)\) is decreasing in \(p_1\), since \(\Phi^{-1}(1-p_1)\) decreases as \(p_1\) increases. A super-uniform \(P_1\) is stochastically no smaller than a uniform random variable, so \(E[A(P_1)]\leq E[A(U_1)]=\alpha\), where \(U_1\) is uniform on \([0,1]\). The final equality follows because the fixed inverse-normal rule has null rejection probability \(\alpha\) for two independent uniform p-values.

The inequality can be strict at either stage: a discrete \(P_1\) can reduce the average conditional error, and a discrete \(P_2\) can reject less often than its conditional allowance. Thus the error guarantee is a bound, not a promise that the test uses exactly \(\alpha\).

Worked Example: Calculating an Error Allowance Before Re-Estimation

Consider a hypothetical test with \(\alpha=0.025\), \(w_1=0.6\), \(w_2=0.8\), and an observed first-stage p-value \(p_1=0.100\). The weights satisfy \(0.6^2+0.8^2=1\). Suppose the design now selects a larger second-stage sample size using only the first-stage information, and the selected second-stage test has a conditionally valid p-value.

First calculate the first-stage normal score and the critical value:

z_1=\Phi^{-1}(0.900)=1.2816,\qquad z_{0.975}=1.960.

The conditional-error allowance is

A(0.100) =1-\Phi\!\left(\frac{1.960-(0.6)(1.2816)}{0.8}\right) =1-\Phi(1.4888) \approx 0.0683.

The adapted design may reject if its second-stage p-value is at most 0.0683. For example, if the selected second-stage test reports \(P_2=0.050\), then \(0.050<0.0683\), so the adapted rule rejects. This is not an assertion that the conditional null rejection probability equals 0.0683: for a discrete second-stage test it is no greater than 0.0683 and can be smaller. The error guarantee applies if the conditional validity assumptions hold for the selected sample size and test.

What Discreteness Changes in Practice

With a nonrandomized discrete test, the p-value can jump over a nominal cutoff. For example, a rule requiring \(P_2\leq A(p_1)\) may have a conditional rejection probability far below \(A(p_1)\) if no attainable p-value lies near the boundary. This is not a Type I error failure; it is conservatism. It can, however, reduce power and make the realized operating characteristics sensitive to the support of the stagewise statistic.

Worked Example: A Discrete Second-Stage Boundary

Suppose a hypothetical second-stage test uses \(X\sim\mathrm{Binomial}(10,0.5)\) under its simple null, with large values providing evidence against \(H_0\). Its nonrandomized upper-tail p-value is \(P(X\geq x)\). For \(x=8\),

P(X\geq 8)=\frac{\binom{10}{8}+\binom{10}{9}+\binom{10}{10}}{2^{10}} =\frac{45+10+1}{1024} =0.05469.

For \(x=9\),

P(X\geq 9)=\frac{10+1}{1024}=0.01074.

Use this test for the second stage in the preceding example, where \(A(0.100)\approx0.0683\). The condition \(P_2\leq0.0683\) is met for \(x\geq8\), so the conditional null rejection probability is \(P(X\geq8)=0.05469\), below the allowance. The next attainable upper-tail p-value after 0.05469 is 0.01074; there is no attainable p-value between these values. That gap illustrates why a valid discrete test can use substantially less than its allowed conditional error.

This simple binomial calculation also verifies validity at the conventional 0.05 cutoff: rejection requires \(x\geq9\), giving a null rejection probability of 0.01074, while \(x\geq8\) would have probability 0.05469 and exceed 0.05. The nonrandomized rule therefore does not reject at \(x=8\) for a 0.05 test. In the adaptive example, by contrast, its allowance is about 0.0683, so \(x=8\) is included.

Assumptions, Diagnostics, and Alternatives

The error bound is only as reliable as the stagewise p-values and the adaptation mechanism. Before using it, verify the following:

Null validity: each stagewise p-value controls its tail probability for every parameter value in the null hypothesis, including relevant nuisance-parameter values.
Conditional second-stage validity: after conditioning on first-stage information and the selected sample size or test, the second-stage p-value remains super-uniform under the null.
Valid information flow: the second-stage design is selected using permitted first-stage information, not unaccounted-for second-stage outcomes.
Correct direction and scale: the p-values use the intended one-sided alternative, and the inverse-normal transformation and critical value match the prespecified test.

A useful diagnostic is to list, for each possible first-stage result that can change the design, the selected second-stage test and its null p-value behavior. Check that its conditional rejection probability at the calculated allowance is no greater than that allowance. For a discrete test this can be checked from its exact null distribution. If the second-stage test is only approximately calibrated, the error bound is only as accurate as that approximation.

The main alternative is to retain a fixed combination rule with prespecified weights. It is straightforward when both stages have valid independent p-values, but it does not justify choosing new weights after inspecting the first-stage result. Conditional-error control is more flexible: it allows a different stage-two sample size or test, provided the conditional error is respected. A normal approximation to a discrete test may make a cutoff easier to compute, but can misstate the attainable rejection probability, particularly in a tail or with small samples. Exact p-values avoid that approximation but can be conservative.

The method does not repair an invalid stagewise test, dependence that violates the conditional validity argument, or an adaptation rule that uses information outside the stated conditioning. It also does not ensure that the overall test has rejection probability exactly \(\alpha\). If exact use of the nominal level is essential, further design choices may be needed; they must be justified without weakening the validity conditions.

Common Mistake

Wrong reasoning: “The first-stage p-value is valid, so I can choose whichever second-stage test or weights look best and then use the original fixed critical value.”

Correction: Validity of each stagewise p-value does not by itself validate a data-dependent combination rule. The adaptation must be accounted for. One defensible construction is to calculate the original rule's conditional error from the first-stage result and require the selected second-stage test to reject with conditional probability no greater than that allowance. The second-stage p-value must remain valid after conditioning on the adaptation.

Key takeaway: Discrete stagewise p-values are often super-uniform rather than exactly uniform. This makes a correctly constructed combination test conservative, while conditional-error control permits adaptation without exceeding the nominal Type I error rate when the required conditional validity assumptions hold.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A null p-value satisfies \(P(P\leq0.04)=0.02\). Is this consistent with super-uniformity? What does it say about rejection at the 0.04 cutoff?
For \(\alpha=0.025\), \(w_1=0.6\), \(w_2=0.8\), and \(p_1=0.100\), what is the conditional-error allowance, to three significant figures?
Why is conditional validity of the second-stage p-value required when its sample size is selected using first-stage data?
In the binomial example, what is the null rejection probability if the second-stage allowance is 0.05 and the test rejects when its p-value is at most the allowance?

Solutions. (1) Yes. The probability 0.02 is no greater than 0.04, as required. Rejection at that cutoff occurs with probability 0.02 for this null distribution, so the test is conservative relative to 0.04. (2) Using \(z_1=\Phi^{-1}(0.9)=1.2816\), \(A=\Phi(((0.6)(1.2816)-1.960)/0.8)=\Phi(-1.4888)\approx0.0683\). (3) The conditional-error argument bounds the rejection chance after the first-stage result and design choice are known. A p-value that is valid only before conditioning may have a conditional rejection probability greater than the allowance, invalidating that bound. (4) At cutoff 0.05, \(x=8\) has p-value 0.05469 and does not qualify, while \(x=9\) has p-value 0.01074. Rejection therefore requires \(x\geq9\), and its null probability is \(11/1024=0.01074\), below 0.05.