Multiplicity With Discrete P-Values

Why Discrete P-Values Matter

Many tests used for binary or otherwise discrete outcomes can produce only a limited set of p-values. If a test statistic has a finite set of possible outcomes, its exact p-value changes in jumps rather than moving continuously between zero and one. A nominal cutoff such as 0.05 may therefore fall between attainable p-values. This can make a test more conservative than its nominal level: it rejects only when the data reach the next attainable value below the cutoff.

In a multi-arm trial, the issue arises at each comparison, and multiplicity procedures add further cutoffs. Bonferroni and Holm remain valid with discrete p-values when the individual p-values are valid. However, their nominal thresholds may not be attainable, and some of their steps may therefore never result in a rejection. Understanding this distinction helps separate error control, which is a probability guarantee, from the practical resolution of a particular test.

Definition: A p-value is discrete when it can take values only in a finite or countable set of attainable values under the test procedure. It is valid for its null hypothesis if, whenever that null is true, \(P(p\leq u)\leq u\) for every \(u\) between 0 and 1. This property is also called super-uniformity. A valid discrete p-value may be conservative because its rejection probability at a chosen cutoff can be strictly smaller than that cutoff.

For a common exact one-sided test, the p-value at an observed statistic is the null probability of obtaining a result at least as extreme as the observed result. The observed statistic can take only particular values, so its tail probabilities—and thus its p-values—also lie on a restricted set. Including the observed outcome in the tail is important for validity: it gives a conservative test in many discrete settings.

A Small Exact-Test Example

Suppose, hypothetically, that a one-sided paired sign test has six discordant pairs. Under the null hypothesis of no directional advantage, assume each discordant pair is equally likely to favour either condition, independently across pairs. If \(X\) is the number favouring the treatment, then \(X\sim \operatorname{Bin}(6,0.5)\). The exact upper-tail p-value for observing \(x\) treatment-favouring pairs is \(P(X\geq x)\).

P(X\geq x)=\sum_{j=x}^{6} {6\choose j}(0.5)^j(0.5)^{6-j}

Worked Example: Finding the Attainable P-Values

For the hypothetical sign test above, suppose six pairs favour the treatment. The observed count is \(x=6\), so only the outcome in which all six pairs favour treatment is at least as extreme.

Formula and substitution:

p=P(X\geq 6)={6\choose 6}(0.5)^6=\frac{1}{64}=0.015625

If five pairs favour treatment, the exact p-value is \(P(X\geq5)=\{{6\choose5}+{6\choose6}\}/64=7/64=0.1094\). If four favour treatment, it is \(P(X\geq4)=\{{6\choose4}+{6\choose5}+{6\choose6}\}/64=22/64=0.3438\). Thus, the p-value cannot take an arbitrary value between 0 and 1: for example, 0.05 is not attainable for this test. The next attainable p-value at or below 0.05 is 1/64.

The assumption about the distribution of \(X\) is part of the example, not a universal feature of paired data. In an application, the sign-test null model and the independence of pair signs must be appropriate. More generally, the exact null distribution must be correctly specified for the p-value to be valid. Discreteness does not rescue a misspecified null model.

What Discreteness Does to Bonferroni

For a family of \(m\) hypotheses, Bonferroni rejects a hypothesis when its valid p-value is at most \(\alpha/m\), where \(\alpha\) is the desired familywise error rate. This rule controls the probability of at least one false rejection at no more than \(\alpha\), regardless of dependence among tests. Discrete p-values do not change the rule or its guarantee. They can, however, make the cutoff unattainable or make the effective rejection boundary substantially lower.

Result: If each true-null p-value is valid, rejecting any hypothesis with \(p_i\leq\alpha/m\) gives familywise error rate at most \(\alpha\). In fact, if there are \(m_0\) true null hypotheses, then \(P(\text{at least one true null is rejected})\leq \sum_{i\text{ true}}P(p_i\leq\alpha/m)\leq m_0\alpha/m\leq\alpha\). The first inequality is the union bound and does not require independence.

For a particular test, let \(\mathcal{P}\) be its set of attainable p-values. Its effective Bonferroni boundary is the largest attainable value in \(\mathcal{P}\) that is no greater than \(\alpha/m\), if one exists. If none exists, that test cannot be rejected by this Bonferroni rule, whatever outcome is observed. This is a statement about the test’s resolution, not a failure of the multiplicity procedure.

Worked Example: Bonferroni Cutoffs on a Discrete Scale

Consider three hypothetical treatment-arm comparisons, each analysed by the six-pair exact test above, with \(\alpha=0.05\). Suppose their observed numbers of treatment-favouring pairs are 6, 5, and 4. The resulting p-values are \(1/64=0.015625\), \(7/64=0.1094\), and \(22/64=0.3438\).

Bonferroni cutoff:

\frac{\alpha}{m}=\frac{0.05}{3}=0.01667

Comparing each value with 0.01667, only 0.015625 is small enough to reject. The second attainable p-value, 0.1094, is already greater than the cutoff. The exact test’s next attainable value above 0.015625 is 0.1094, so the effective rejection boundary is 0.015625, not 0.01667.

Now keep three tests and the same exact-test p-value grid, but set \(\alpha=0.04\). The Bonferroni cutoff is \(0.04/3=0.01333\), below the smallest attainable p-value, 0.015625. No result from these tests can meet the cutoff. The procedure still controls familywise error; it simply has no attainable rejection at this familywise level with this test resolution.

Holm’s Steps and Attainability

Holm’s step-down procedure sorts the \(m\) p-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\). It compares \(p_{(1)}\) with \(\alpha/m\), then \(p_{(2)}\) with \(\alpha/(m-1)\), continuing with \(p_{(i)}\) against \(\alpha/(m-i+1)\). It rejects in order until the first comparison that fails; that hypothesis and all later ones are not rejected. If the p-values are discrete, each step’s nominal boundary may not itself be attainable. A step can be passed only if the observed p-value is an attainable value at or below its boundary.

Holm’s familywise guarantee requires valid marginal p-values, but not independence. The key argument also illustrates why the location of a true null in the ordered list must be handled carefully. Suppose there are \(m_0\) true null hypotheses, and let \(k\) be the rank of the first true null in the ordered list. At most \(m-m_0\) false nulls can precede it, so \(k\leq m-m_0+1\). Therefore \(m-k+1\geq m_0\), and the Holm boundary at rank \(k\) satisfies \(\alpha/(m-k+1)\leq\alpha/m_0\). If any true null is rejected, this first true null must pass its comparison. Thus at least one of the \(m_0\) true-null p-values is at most \(\alpha/m_0\). By validity and the union bound, the probability of that event is at most \(m_0(\alpha/m_0)=\alpha\).

Worked Example: Applying Holm to Three Arm Comparisons

Use the three hypothetical p-values \(0.015625\), \(0.1094\), and \(0.3438\) from the Bonferroni example, with \(m=3\) and \(\alpha=0.05\). They are already ordered from smallest to largest.

At rank 1, compare \(0.015625\) with \(0.05/3=0.01667\). The comparison passes, so reject the first hypothesis. At rank 2, compare \(0.1094\) with \(0.05/2=0.025\). This comparison fails, so stop: do not reject the second or third hypotheses. Holm therefore rejects only the first hypothesis.

For these tests the second attainable p-value is 0.1094, which exceeds the rank-2 boundary of 0.025. Consequently, rank 2 cannot be passed in this example’s p-value grid. This does not mean every discrete test makes Holm’s second step impossible: the attainable values can differ among tests, and a sufficiently small attainable p-value can pass the relevant boundary.

Why the Guarantee and the Actual Rejection Rate Differ

A cutoff is an upper bound on a rejection probability, not a promise that the rejection probability equals the cutoff. In the six-pair example, a single test at level 0.05 rejects only when \(X=6\). Under its null model, its actual rejection probability is \(P(X=6)=1/64=0.015625\), below 0.05. At a Bonferroni cutoff of 0.01667, it has the same rejection event and the same null rejection probability. The difference between nominal level and actual size is a consequence of the attainable tail probabilities.

This conservatism can reduce power: outcomes that would have produced a p-value just above the cutoff cannot be partially counted as evidence. But raising the cutoff simply to the next attainable p-value can break the intended error guarantee. More tailored exact procedures can sometimes use the null distribution and the family structure to improve efficiency, but their validity must be established for the procedure and assumptions actually used.

Bonferroni is a single-step rule: every test faces the same cutoff, so its discrete boundary can be identified separately for each test. Holm is step-down: its boundaries become less stringent at later ranks, but a failure stops the procedure. Unlike an unadjusted collection of nominal-level tests, both procedures account for multiplicity. Their validity applies whether tests are independent or dependent, provided the individual p-values are valid. Dependence may affect how often several small p-values occur together, but it is not needed for the stated guarantees.

Diagnostics and Common Pitfalls

Check the attainable scale. For an exact test, enumerate or calculate its possible tail probabilities. Compare the smallest attainable p-value and the next values with the Bonferroni or Holm boundaries.
Verify the null model. Exact p-values are only as reliable as the assumed randomization distribution or sampling model. Check, for example, that the paired sign-test assumptions are appropriate before using its binomial tail probabilities.
Keep validity distinct from equality. Validity requires \(P(p\leq u)\leq u\), not equality. A conservative p-value is not automatically a defective one.
Do not replace an exact p-value with a mid-p value without justification. A mid-p value reduces the contribution from outcomes tied with the observed statistic. This can improve average power, but it need not satisfy the super-uniformity condition and can therefore lose a guaranteed error bound.
Do not loosen a multiplicity cutoff because it is unattainable. A cutoff below the smallest attainable p-value means there is no rejection for that test at that boundary. It does not license substituting a larger cutoff.

Key takeaway: Discreteness changes which p-values a test can produce, not the Bonferroni or Holm logic. With valid p-values, both procedures retain familywise error control; their actual rejection opportunities may be fewer than their nominal cutoffs suggest.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A valid exact p-value can equal 0.05 only if 0.05 is attainable. True or false? Explain.
For three tests at \(\alpha=0.05\), what is the Bonferroni cutoff? If a test’s smallest attainable p-value is 0.02, can it be rejected by that rule?
Three ordered p-values are 0.01, 0.03, and 0.20. Apply Holm’s procedure at \(\alpha=0.05\).
Why does the Bonferroni union-bound argument not require independent tests?

Solutions. (1) False. A valid discrete p-value may or may not take 0.05; validity concerns the probability of being at or below a threshold, not whether that threshold is attainable. (2) The cutoff is \(0.05/3=0.01667\). Since 0.02 exceeds it and is the smallest attainable value, this test cannot be rejected by that Bonferroni rule. (3) At rank 1, \(0.01\leq0.05/3=0.01667\), so reject. At rank 2, \(0.03\leq0.05/2=0.025\) is false, so stop; do not reject the second or third hypotheses. (4) The probability of a union of events is at most the sum of their probabilities for any dependence structure. Applying that inequality to the true-null rejection events gives the Bonferroni bound.