Mid-P Values and Exact-Test Conservatism

Why Discrete Exact Tests Can Be Conservative

In an exact conditional test, inference is based on the null distribution of a statistic given the quantities on which the analysis conditions. That distribution often has only a finite set of possible values. Consequently, a nonrandomized test cannot always attain every desired significance level: its rejection probability changes in jumps as outcomes are added to the rejection region. The result is that a test calibrated to have size at most \(\alpha\) may have size substantially below \(\alpha\).

A mid-P adjustment addresses this discreteness by counting half, rather than all, of the null probability at the observed outcome. This usually reduces a conventional exact p-value, but it also changes what can be guaranteed: a mid-P test need not have type I error probability at most its stated level. To compare the methods precisely, first fix how “at least as extreme” is defined.

The statistic \(S\) also specifies the ordering for a two-sided test. For example, an analysis might define a statistic that increases as outcomes become more discordant with the null, then use the upper tail of that statistic. Alternatively, it might define a two-sided p-value by a particular ordering of outcomes or by combining tail probabilities. Those conventions need not give identical p-values. Once the ordering is fixed, the same comparison applies: the conventional p-value includes the full probability of outcomes tied with the observation, while the mid-P value includes half.

Calculating the Two Values

For a test that rejects when its p-value is at most \(\alpha\), the test size under a fixed null distribution is the probability of rejection under that null. Conventional p-values have a useful finite-sample guarantee, even when the statistic is discrete.

Proof. List the distinct support values in increasing order, \(s_1<s_2<\cdots<s_m\), and write \(q_j=P_0(S=s_j)\). The tail probabilities \(p(s_j)=\sum_{k=j}^m q_k\) are nonincreasing in \(j\). Therefore the set of indices \(j\) for which \(p(s_j)\leq\alpha\) is either empty or consists of \(j_0,j_0+1,\ldots,m\) for some \(j_0\). In the empty case the rejection probability is zero. Otherwise,

This proves the claimed bound. The inequality can be strict because the tail probabilities jump over some values of \(\alpha\). \(\square\)

What the Mid-P Adjustment Gives Up

The super-uniformity proof depends on including the whole probability mass at each statistic value in the tail. A mid-P value does not do that, so the proof does not apply. A mid-P test can be less conservative, but there is no general guarantee that its size is at most \(\alpha\) for every \(\alpha\).

The example shows why “less conservative” is not equivalent to “exactly calibrated.” A mid-P procedure can be closer to the nominal level in some settings, but its rejection probability must be assessed as a property of the particular null distribution and decision rule. It may also be below the nominal level; the direction and amount of the discrepancy are not fixed by the definition alone.

Connection to Randomized Exact Tests

There is a precise way to split the tied probability while retaining exact calibration: randomize within the observed tie. Let \(U\sim\operatorname{Uniform}(0,1)\) be independent of \(S\). Define the randomized upper-tail p-value by

Conditional on the observed value \(S=s\), its expectation over the auxiliary randomization is the mid-P value:

This identity explains the “half” in mid-P. But averaging the randomized value does not preserve its rejection probability: comparing the average to a threshold is not the same as randomly comparing the original value to that threshold.

Proof. Write the support in decreasing order, \(s_1>s_2>\cdots>s_m\), and let \(q_j=P_0(S=s_j)\). Set \(a_j=\sum_{k<j}q_k\), the probability of values more extreme than \(s_j\). When \(S=s_j\), the randomized p-value is \(a_j+Uq_j\), which is uniform on the interval \((a_j,a_j+q_j)\). These intervals partition \((0,1)\), apart from their endpoints, and the probability of selecting interval \(j\) is \(q_j\), its length. For any interval \(B\subseteq(a_j,a_j+q_j)\), the probability that the randomized p-value lies in \(B\) is

Adding this equality over the partition shows that the randomized p-value assigns each interval its length, which is the uniform distribution on \((0,1)\). Hence \(P_0\{p_{\mathrm{rand}}\leq\alpha\}=\alpha\). \(\square\)

A Conditional Table Calculation

Interpreting and Reporting the Trade-Off

Conventional exact p-values and mid-P values answer closely related but distinct calibration questions. A conventional p-value includes the observed outcome’s full probability and is super-uniform under the specified null distribution. This guarantees a test size no greater than its nominal level, though discreteness may make the test conservative. A mid-P value halves the observed tie. It can reduce that conservatism, but it is not generally super-uniform and does not guarantee type I error control at every nominal level.

In conditional inference, the guarantee is tied to the conditional null distribution actually used. If a test is applied separately at a fixed conditioning outcome, the conventional bound applies under that conditional law. The mid-P size calculation likewise depends on that law. If a procedure averages over different conditioning outcomes, its overall error rate depends on how those outcomes are distributed and how the test is applied within them.

A useful report identifies the convention rather than presenting a p-value as though there were only one possible discrete calculation. State the test statistic or ordering, the null distribution and conditioning event, and whether the reported value is conventional or mid-P. For consequential decisions, compare the mid-P conclusion with the conventional exact result when the distinction could change the decision.

• Use the conventional value when guaranteed finite-sample size control at the stated level is required.
• Consider the mid-P value when reducing discreteness-related conservatism is important, while acknowledging that its type I error can exceed the nominal level.
• Use randomization only deliberately when exact equality to a target size is desired and randomized decisions are acceptable.

A common mistake is to argue, “The mid-P value is smaller, so the test is more powerful and still exact.” The correction is that a smaller p-value can produce more rejections, but the added rejections may raise the null rejection probability above \(\alpha\). “Exact” should therefore be used carefully: the conventional tail calculation gives a finite-sample size bound, whereas a mid-P adjustment is a different convention with no universal size bound.