Why Mid-P Values Need Special Care
In a discrete test, possible test statistics have probability masses rather than a continuous range of outcomes. An exact tail probability includes the entire probability at the observed boundary. A mid-P value includes only half of that boundary mass. This can reduce conservatism, but it also means a test that rejects when its mid-P value is at most \(\alpha\) need not have rejection probability at most \(\alpha\).
That distinction matters in adaptive designs. Conditional error control asks whether a procedure’s probability of rejection, under a null hypothesis, stays within its specified bound for each relevant interim history—not merely on average over histories. A mid-P value that is treated as if it supplied an exact conditional bound can exceed that bound. If the procedure uses such tests within a closed testing procedure, the resulting local-test error can also compromise strong familywise error control.
Binomial Exact and Mid-P Values
For a one-sided binomial test of \(H_0:p\leq p_0\) against \(H_a:p>p_0\), the boundary value \(p_0\) is least favorable: upper-tail probabilities are nondecreasing in \(p\). Thus, checking the rejection probability at \(p=p_0\) checks the null’s largest rejection probability. The calculations below use this simple boundary null.
The model assumes a fixed, prespecified sample size \(n\), independent Bernoulli observations with a common success probability under the null, and a prespecified upper-tail rejection rule. In a conditional calculation, these assumptions must hold for the distribution given the interim history being considered. For example, if \(Y\) is the count of successes in a later, independent stage, its conditional null distribution can be \(\operatorname{Bin}(n,p_0)\) even after conditioning on earlier data. If the stage data are not independent, or the conditional null distribution differs, the binomial probabilities must be replaced by the appropriate conditional distribution.
Worked Example: A Mid-P Test Exceeds 0.05
Suppose \(X\sim\operatorname{Bin}(4,0.5)\) under the null and use an upper-tail test with \(\alpha=0.05\). The four Bernoulli observations are assumed independent, and \(n=4\) was fixed in advance. The probability of \(X=4\) is \(0.5^4=0.0625\), while the probability of any value greater than 4 is zero.
At \(x=4\), the exact p-value is \(P(X\geq4)=0.0625\), so the exact test does not reject. The mid-P value is
The mid-P test rejects at \(x=4\), because \(0.03125\leq0.05\). It rejects only at that value, so its null rejection probability is \(P(X=4)=0.0625\), which exceeds 0.05. The mid-P value is therefore not an upper bound on this test’s rejection probability.
Why the Boundary Mass Creates the Problem
For an upper-tail test with a monotone rejection rule, let \(k\) be the smallest value of \(X\) in the rejection region. The probability of rejection under the null is the full upper tail at \(k\). By contrast, the mid-P value at \(k\) subtracts half the probability mass at \(k\). If the mid-P value crosses the threshold while the exact tail does not, the test includes the whole boundary mass even though the mid-P calculation counted only half.
If \(\operatorname{midP}(k)\leq\alpha\) but \(P_{p_0}(X\geq k)>\alpha\), the nominal mid-P rule is anti-conservative at that boundary. The excess is half the boundary mass relative to the mid-P value. This identity explains the mechanism; it does not imply that every mid-P test exceeds its level. The result depends on the sample size, null probability, significance level, and resulting rejection region.
Worked Example: Conditional Error at a Selection History
Imagine a two-stage design in which an interim rule selects a confirmatory path with probability 0.4 under the null. Write \(A\) for that selection event. Assume that, conditional on \(A\), the later-stage count \(Y\) is independent of the interim data and has distribution \(\operatorname{Bin}(4,0.5)\) under the null. The proposed confirmatory rule rejects only on path \(A\), using the mid-P test at level 0.05 from the previous calculation.
At the selected history, the conditional rejection probability is
Thus the conditional error on path \(A\) exceeds 0.05. If the procedure never rejects on the other path, its unconditional rejection probability is \(0.4(0.0625)=0.025\). That average is below 0.05, but it does not repair the conditional excess: the selected path itself has a rejection probability of 0.0625. Conditional error control concerns the conditional probability at the history in question, not only an average across histories.
The calculation relies on the stated conditional model. If selection changes the distribution of \(Y\), or if the later observations are dependent on the interim data, one cannot use the unconditional binomial probabilities without adjustment. The relevant probabilities must be calculated under the null conditional on the selection event or history.
Exact and Boundary-Randomized Alternatives
An exact nonrandomized test uses the full boundary mass. It may be conservative because discrete data often do not provide a rejection region whose probability is exactly \(\alpha\). A boundary-randomized test can attain the level exactly by rejecting with a specified probability at the boundary. Mid-P testing instead uses a half-mass adjustment in the p-value, but its usual deterministic rule does not actually randomize the decision.
| Procedure | Boundary treatment | Level implication |
|---|---|---|
| Exact nonrandomized | Includes the full boundary mass in the tail probability | Null rejection probability is at most \(\alpha\); it can be smaller |
| Boundary-randomized | Rejects at the boundary with a calibrated probability | Can attain \(\alpha\) exactly under the specified null model |
| Mid-P nonrandomized | Counts half the observed boundary mass in the p-value | Can be less conservative, but need not have rejection probability at most \(\alpha\) |
For an upper-tail boundary \(k\), let \(r=P_{p_0}(X>k)\) and \(m=P_{p_0}(X=k)\). If \(r\leq\alpha\leq r+m\), a randomized test rejects whenever \(X>k\), and rejects with probability \(\gamma\) when \(X=k\), where
This gives rejection probability \(r+\gamma m=\alpha\). The randomization must be independent of the data and implemented according to the specified probability. A study may prefer not to randomize a clinical conclusion; in that case, the exact nonrandomized test preserves the bound, usually at the cost of conservatism.
Worked Example: Calibrating a Boundary-Randomized Test
Return to \(X\sim\operatorname{Bin}(4,0.5)\) and \(\alpha=0.05\). Choose the upper boundary \(k=4\). Then \(r=P(X>4)=0\) and \(m=P(X=4)=0.0625\). The boundary rejection probability is
The test rejects with probability 1 when \(X>4\) (an event of probability zero) and with probability 0.8 when \(X=4\). Its null rejection probability is \(0+(0.8)(0.0625)=0.05\). This is different from the deterministic mid-P rule, which rejects every time \(X=4\), giving probability 0.0625. The randomized result depends on carrying out the randomization; simply reporting a mid-P value does not implement it.
Implications for Conditional Error and Closed Testing
A conditional error is the null probability of rejection after conditioning on a specified interim history. If a design’s adaptation rule selects a test or changes a critical value, its error argument must apply to the resulting conditional distribution at each history for which the bound is required. A marginal level calculation does not establish that condition. The calculation must also use the relevant null hypothesis: in a multiple-testing procedure, this includes intersection nulls, not just individual hypotheses.
Closed testing controls the familywise error rate by testing every intersection hypothesis at level \(\alpha\) and rejecting an elementary hypothesis only when all intersection hypotheses containing it are rejected. Its strong-control guarantee relies on the local tests having their claimed level under their respective intersection nulls. If a local mid-P test is treated as an exact level-\(\alpha\) test but actually has size above \(\alpha\), the usual premise for that guarantee is missing.
Worked Example: A Closed Procedure Can Inherit the Excess
Consider two hypotheses \(H_1\) and \(H_2\). For illustration, suppose their local tests and the test of \(H_1\cap H_2\) all use the same upper-tail mid-P rule based on \(X\sim\operatorname{Bin}(4,0.5)\) under the global null. The observations are independent Bernoulli trials under that null, and each test rejects at 0.05 when its mid-P value is at most 0.05. As calculated above, each local rule rejects exactly when \(X=4\).
In closed testing, \(H_1\) can be rejected only if both its own test and the intersection test reject; likewise for \(H_2\). Since all three rules reject on the same event, the procedure rejects both elementary hypotheses when \(X=4\). Under the global null, the familywise error probability is therefore
This example shows the risk, not a universal outcome for every closed procedure: the global-null error already exceeds the target, so strong control fails. Using a valid exact or appropriately boundary-randomized local test would address the level problem in this example, but each intersection test still needs a valid calibration under its own null and any conditioning required by the design.
Diagnostics and a Common Mistake
A practical audit should calculate rejection probabilities, rather than infer them from the numerical size of a p-value. For each relevant interim history and each null hypothesis or intersection null, identify the conditional distribution, enumerate the rejection region, and sum the probabilities of all outcomes in that region. For a binomial endpoint, this can be done directly from the probability mass function. Repeat the calculation at the least-favorable null parameter when the null is composite.
- Check whether the exact tail at the rejection boundary exceeds \(\alpha\), even when the mid-P value is below it.
- Check whether conditioning on selection changes the distribution used to calculate the p-value.
- Check every conditional branch, not only the design’s overall average error.
- For closed testing, verify level control for each intersection test under its corresponding null.
A common mistake is to reason, “The mid-P value is 0.03125, which is below 0.05, so the conditional rejection probability must be at most 0.05.” That conclusion confuses a statistic used to make a decision with the probability of making that decision under the null. In the four-trial example, the rule rejects on an event with probability 0.0625. The correction is to enumerate the rejection region and calculate its conditional null probability directly, or to use a procedure whose calibration guarantees the required bound.
Mid-P values can be useful as less conservative summaries for discrete data, but their reduced conservatism is precisely why they cannot automatically be substituted for exact conditional error bounds. The choice between exact nonrandomized, boundary-randomized, and mid-P procedures involves a trade-off among conservatism, randomization, and guaranteed error control. Under adaptation, the error guarantee must match the conditional and multiplicity structure actually used by the design.
Check Your Understanding
Answer each question, then compare with the solutions.
- For \(X\sim\operatorname{Bin}(4,0.5)\), what is the exact upper-tail p-value at \(X=4\), and what is the mid-P value?
- At level 0.05, what is the rejection probability of the deterministic mid-P test in that example?
- For a boundary with \(r=0.02\), \(m=0.06\), and \(\alpha=0.05\), calculate the boundary randomization probability and verify the test size.
- Why does an unconditional error rate below 0.05 not establish a conditional error bound of 0.05 on every adaptation path?
- What must be true of the local tests for the usual closed-testing strong-control argument to apply?
Solutions. (1) The exact value is \(P(X\geq4)=0.0625\); the mid-P value is \(0+\tfrac12(0.0625)=0.03125\). (2) The test rejects only at \(X=4\), so its rejection probability is \(0.0625\). (3) \(\gamma=(0.05-0.02)/0.06=0.5\); the size is \(r+\gamma m=0.02+(0.5)(0.06)=0.05\). (4) An average can be below 0.05 while one conditional branch exceeds 0.05; the conditional requirement must be checked at the relevant history. (5) Each intersection hypothesis must be tested by a valid level-\(\alpha\) test under its null, with the appropriate conditional distribution and any required conditioning respected.