Constructing Tests From Exact Conditional Distributions

From a Conditional Distribution to a Test

Conditioning can remove nuisance parameters from an inference problem, leaving a distribution under the null hypothesis that is fully specified once the conditioning statistic is observed. The next step is to use that distribution to decide which outcomes count as evidence against the null. This requires an ordering: a rule that identifies which outcomes are more extreme for the alternative of interest.

The ordering matters. A one-sided alternative naturally gives a direction, while a two-sided alternative can be represented by several different measures of extremeness. Once an ordering is chosen, exact tail probabilities give a test that controls its finite-sample type I error. Because conditional distributions are often discrete, the attainable rejection probabilities may not equal the desired level.

Definition: Let \(Y\) be the observed data and \(C\) a conditioning statistic. For an observed value \(C=c\) with \(P(C=c)>0\), a conditional test is a function \(\phi(Y,c)\) taking values in \([0,1]\). Its conditional rejection probability under the null is \(E_0[\phi(Y,c)\mid C=c]\). The test has conditional size at most \(\alpha\) if this probability is at most \(\alpha\) for every relevant \(c\). Values strictly between 0 and 1 represent randomization.

In the usual nuisance-elimination setting, the conditional distribution under the null does not depend on the nuisance parameter. If it still does, a valid conditional test must control its rejection probability for every allowed nuisance value. The constructions below assume that the conditional null distribution is specified. Write its probability mass function as \(q(y\mid c)\), on the conditional sample space \(\Omega_c\).

Ordering Outcomes and Computing Tail Probabilities

For an upper-tail test, choose a statistic \(T(Y)\) for which larger values count as more extreme in the direction of the alternative. Its inclusive-tail p-value at the observed data \(y\) is

p(y\mid c)=P_0\{T(Y)\geq T(y)\mid C=c\} =\sum_{z\in\Omega_c:\,T(z)\geq T(y)}q(z\mid c).

The inequality includes outcomes tied with the observed statistic. This convention makes the p-value conservative rather than anti-conservative when there are ties. For a lower-tail test, reverse the inequality. For a two-sided test, first define a measure of extremeness, such as distance from a null value, and then use the inclusive tail of that measure. There is no single two-sided ordering that is appropriate for every problem.

Result: If \(T\) has a specified conditional null distribution, the test that rejects when \(p(Y\mid c)\leq\alpha\) has conditional size at most \(\alpha\). This is a finite-sample statement; it does not depend on a large-sample approximation.

To see why discreteness can make the test conservative, list the distinct possible values of \(T\) as \(t_1<t_2<\cdots<t_k\), with conditional probabilities \(r_j=P_0(T=t_j\mid C=c)\). The inclusive upper-tail probabilities are

s_j=P_0(T\geq t_j\mid C=c)=\sum_{\ell=j}^k r_\ell.

A nonrandomized test can reject only for entire groups of outcomes having the same statistic value. The cumulative probability of the groups rejected may therefore jump past some possible target sizes. The following proof shows that this jump cannot make the test exceed its target.

Keep the conditioning value fixed.
All probabilities in the argument are computed under the conditional null distribution given \(C=c\).

Identify rejected statistic values.
For an upper-tail test, \(T=t_j\) is rejected exactly when \(s_j\leq\alpha\). Since \(s_j\) decreases as \(j\) increases, the rejected values form a terminal segment of the ordered support, or the empty set.

Sum their probabilities.
If the first rejected value is \(t_m\), the rejection probability is \(\sum_{j=m}^k r_j=s_m\leq\alpha\). If no value is rejected, it is zero.

Thus the test has conditional size at most \(\alpha\). If it has this bound for every relevant value of \(C\), its unconditional rejection probability is also at most \(\alpha\), because

P_0(\text{reject}) =E_0\!\left[P_0(\text{reject}\mid C)\right] \leq E_0[\alpha] =\alpha.

The last identity uses the law of total expectation. The same argument applies when the rejection probability is calculated by a randomized test.

Worked Examples

Worked Example: A One-Sided Binomial Conditional Test

Suppose that, at a fixed conditioning value \(C=c\), the null distribution of a count is \(T\mid C=c\sim\operatorname{Binomial}(4,1/2)\). Consider the upper-tail alternative and target size \(\alpha=0.10\). Under the null, the probabilities for \(T=0,1,2,3,4\) are \(1/16,4/16,6/16,4/16,1/16\).

The inclusive upper tails are \(P(T\geq4\mid c)=1/16=0.0625\), \(P(T\geq3\mid c)=5/16=0.3125\), \(P(T\geq2\mid c)=11/16=0.6875\), \(P(T\geq1\mid c)=15/16=0.9375\), and \(P(T\geq0\mid c)=1\). Only \(T=4\) has a p-value at most 0.10. The nonrandomized test therefore rejects only when \(T=4\), and its conditional size is \(1/16=0.0625\), which is below the target.

To attain size exactly 0.10, always reject at \(T=4\), and reject with probability \(\gamma\) at \(T=3\). The required boundary probability is determined by

\frac{1}{16}+\gamma\frac{4}{16}=0.10 \quad\Longrightarrow\quad \gamma=\frac{0.10-0.0625}{0.25}=0.15.

The randomized test has size \(1/16+0.15(4/16)=0.0625+0.0375=0.10\). Without randomization, the next available rejection region would include \(T=3\) as well as \(T=4\), and its size would be \(5/16=0.3125\), far above the target.

Worked Example: A Two-Sided Ordering by Distance

Use the same conditional null distribution, \(T\mid C=c\sim\operatorname{Binomial}(4,1/2)\), but now test against departures in either direction from the null center 2. Define the statistic \(D=|T-2|\); larger values are more extreme. This definition of two-sided extremeness is part of the test and should be chosen before looking at the observed outcome.

The outcomes \(T=0\) and \(T=4\) have \(D=2\) and total null probability \(2/16=0.125\). The outcomes \(T=1\) and \(T=3\) have \(D=1\) and total probability \(8/16=0.5\). The outcome \(T=2\) has \(D=0\) and probability \(6/16=0.375\). Thus the inclusive-tail p-values are \(0.125\) for either endpoint, \(0.125+0.5=0.625\) for \(T=1\) or \(T=3\), and 1 for \(T=2\).

At target size \(\alpha=0.10\), the nonrandomized test rejects no outcomes: even the most extreme outcomes have p-value \(0.125\). To obtain size exactly 0.10, randomize at the extreme boundary. Reject at \(T=0\) or \(T=4\) with probability \(\gamma=0.10/0.125=0.8\). The resulting size is \(0.8(0.125)=0.10\). This example shows why a valid exact test need not reject any outcome at a particular nominal level.

Worked Example: Ordering by a Likelihood Ratio

Suppose the conditional sample space is \(\{a,b,c,d\}\). Under a simple null, the conditional probabilities are \(0.10,0.20,0.30,0.40\), respectively. Under a specified simple alternative, they are \(0.40,0.30,0.20,0.10\). The likelihood ratios, alternative probability divided by null probability, are \(4,1.5,2/3,0.25\). For this alternative, order outcomes from largest to smallest likelihood ratio: \(a,b,c,d\).

The null tail probabilities in that order are \(P_0(Y=a\mid c)=0.10\), \(P_0(Y\in\{a,b\}\mid c)=0.30\), \(P_0(Y\in\{a,b,c\}\mid c)=0.60\), and 1. For target size \(\alpha=0.25\), the nonrandomized test rejects only at \(a\), giving size 0.10. To reach size 0.25, reject at \(a\) and reject at \(b\) with probability \(\gamma\). Then

0.10+0.20\gamma=0.25 \quad\Longrightarrow\quad \gamma=0.75.

The conditional size is \(0.10+0.75(0.20)=0.25\). The likelihood ratio supplies an ordering suited to this specified alternative; the null probabilities, not the alternative probabilities, are what determine the size and the boundary randomization.

Randomization and Exact p-Values

The boundary procedure has a general form. Let \(A\) be the null probability of outcomes strictly more extreme than the boundary, and let \(B\) be the null probability of outcomes tied at the boundary. If \(A\leq\alpha\leq A+B\), reject all outcomes more extreme than the boundary and reject at the boundary with probability

\gamma=\frac{\alpha-A}{B},

provided \(B>0\). Since \(0\leq\alpha-A\leq B\), this gives \(0\leq\gamma\leq1\), and the resulting size is \(A+\gamma B=\alpha\). If \(\alpha\) equals a cumulative tail probability, no partial rejection is needed. This is a way to attain a precise size, not a requirement that every practical test use randomization.

There is also a randomized p-value that expresses the same boundary principle. For an upper-tail statistic, let \(U\) be independent of the data and uniform on \([0,1]\), and define

p_{\mathrm{rand}}(Y\mid c) =P_0(T>T(Y)\mid C=c)+U\,P_0(T=T(Y)\mid C=c).

Under the conditional null, this p-value is uniform on \([0,1]\). To verify the claim, use the distinct statistic values \(t_1<\cdots<t_k\) with probabilities \(r_j\). When \(T=t_j\), the randomized p-value is uniform on the interval from \(s_{j+1}\) to \(s_j\), where \(s_j=\sum_{\ell=j}^k r_\ell\) and \(s_{k+1}=0\). That interval has length \(r_j\), exactly the probability that \(T=t_j\). The intervals for \(j=1,\ldots,k\) fill \([0,1]\) without overlap except at endpoints. On each interval, the conditional density contributed by \(T=t_j\) is \(r_j/r_j=1\). Thus the total density is 1 throughout \([0,1]\), proving uniformity. Consequently, rejecting when \(p_{\mathrm{rand}}\leq\alpha\) has conditional size exactly \(\alpha\). \(\square\)

The ordinary inclusive-tail p-value is usually preferred for reporting because it is computed without an auxiliary random draw. Its null distribution is generally not uniform: it takes a limited set of values and satisfies \(P_0(p\leq\alpha\mid C=c)\leq\alpha\). That inequality, rather than uniformity, is the relevant validity guarantee for a nonrandomized exact test.

Common Mistakes and Practical Choices

Choosing an ordering after seeing the outcome. Trying several statistics and reporting the smallest resulting tail probability does not preserve the stated size automatically. Specify the statistic or extremeness rule as part of the test.
Dropping tied outcomes from the tail. The inclusive tail includes every outcome at least as extreme as the observed statistic, including ties. Omitting ties can make a reported p-value too small.
Assuming “exact” means “equal to the target.” Exact refers to finite-sample control under the specified null distribution. A nonrandomized test may be conservative because the available probabilities are discrete.
Treating every two-sided test as the same test. Distance from a center, a likelihood-ratio ordering, and other defensible orderings can give different rejection regions. State which one defines extremeness.
Using only a conditional calculation without checking its scope. A conditional size bound implies an unconditional bound when it holds for every relevant conditioning value. If the conditional null law still depends on nuisance parameters, control must also account for those values.

Key takeaway: An exact conditional test is built from a specified conditional null distribution and a prespecified ordering of outcomes. Inclusive tail probabilities give a nonrandomized test whose conditional size cannot exceed its target; randomizing at a boundary can attain the target exactly.

Check Your Understanding

Work through these questions using the definitions and calculations above.

For a discrete upper-tail statistic, why does the inclusive-tail p-value include outcomes tied with the observation?
A conditional null distribution assigns probability 0.07 to outcomes more extreme than a boundary and 0.18 to outcomes tied there. What boundary randomization probability gives conditional size 0.16?
Why can a nonrandomized exact test have size strictly below its target even when the null distribution is known exactly?
In a two-sided test, what must be specified before calculating the tail probability, and why can different choices produce different p-values?
What additional condition is needed to infer an unconditional size bound from conditional size control?