Why Discrete Tests Have Coarse P-Values

Why P-Values Can Jump

A p-value is a probability calculated under a null model, but the possible outcomes of a study depend on the kind of data collected. If the data are a count, the count can take only whole-number values. Its null distribution therefore assigns probability to separate outcomes rather than spreading it continuously across a scale. Summing those probabilities can make exact p-values jump from one attainable value to another, leaving many numbers between them unavailable.

This matters when reading a result such as \(p=0.05\). An exact test based on a small count might not be able to produce that value at all. Its nearest possible p-values could be well above and well below 0.05. A continuous approximation can provide a smoother numerical estimate, but it is not the exact probability calculation for the discrete null model.

A Binomial Model and an Exact P-Value

Suppose \(X\) counts the number of successes in \(n\) independent trials, each with the same success probability \(p_0\) under the null hypothesis. Then the null model is \(X\sim\operatorname{Binomial}(n,p_0)\), with \[ P_0(X=k)={n\choose k}p_0^k(1-p_0)^{n-k}, \quad k=0,1,\ldots,n. \] The notation \(P_0\) emphasizes that these probabilities are calculated assuming the null model is true. To use this model, the trials must be independent, have two possible outcomes for the count, and share the same null success probability.

For a two-sided test, one must specify what counts as “at least as extreme” as the observed result. In the examples below, extremeness is measured by distance from the null mean \(np_0\). This gives the exact two-sided p-value \[ p_{\text{exact}}(x_{\text{obs}}) = P_0\left(\left|X-np_0\right|\geq\left|x_{\text{obs}}-np_0\right|\right). \] The probability is found by adding the null probabilities of all count outcomes at least as far from the null mean as the observed count. This is one standard two-sided convention; other conventions can order discrete outcomes differently, so the convention should be stated when it matters.

For a one-sided alternative that predicts unusually large counts, the exact p-value instead sums the upper tail: \[ p_{\text{exact}}=P_0(X\geq x_{\text{obs}}). \] In either case, the possible exact p-values come from sums of probabilities assigned to count outcomes. They are not freely selectable numbers.

Why the Set of Exact Values Is Limited

For fixed \(n\), a binomial count has only \(n+1\) possible outcomes: \(0,1,\ldots,n\). A test rule assigns a p-value to each possible outcome, so there can be no more than \(n+1\) different exact p-values for that fixed test. Often there are fewer, because different outcomes can produce the same p-value. For example, in a symmetric two-sided test with \(p_0=0.5\), counts equally far below and above the null mean have the same p-value.

The sizes of the jumps depend on the probabilities of the count outcomes. If one or more outcomes included in a tail have substantial probability, adding them can change the p-value noticeably. In the four-trial example, each extreme outcome has probability \(1/16\), so the smallest two-sided p-value under the chosen rule is 2/16. No observed count falls between the central outcomes and those extremes, so no exact p-value falls between 0.125 and 0.625.

Unequal Probabilities and Unequal Jumps

When \(p_0\) is not 0.5, binomial probabilities are generally not symmetric around the center. The probability attached to each count can differ substantially, and therefore so can the sizes of the steps between attainable tail probabilities. For an upper-tailed test, the exact p-value at count \(x\) is the sum of the probabilities for \(x,x+1,\ldots,n\). Moving the observed count up by one removes the probability at the old boundary from that sum.

A Short Justification: Why the Exact P-Value Is Calibrated

The discreteness of an exact p-value does not make it mathematically invalid. Under the null model, outcomes with p-values at most a chosen cutoff \(\alpha\) have total probability no greater than \(\alpha\). For the one-sided test, list the possible counts from most extreme to least extreme in the direction of the alternative. Their upper-tail probabilities decrease as the count increases. The counts whose p-values are at most \(\alpha\) therefore form an upper tail, or an empty set. If \(c\) is the smallest count in that rejection tail, then \[ P_0(\text{p-value}\leq\alpha)=P_0(X\geq c)=p_{\text{exact}}(c)\leq\alpha. \] If no count qualifies, that probability is zero.

The key inequality arises because the p-value for the boundary count \(c\) is the probability of the entire tail beginning at \(c\). That tail has probability at most \(\alpha\) by the rule for including \(c\). This argument also explains why discrete tests can be conservative at a chosen cutoff: the actual null probability of rejection may be less than \(\alpha\), because the next available tail probability might jump past the cutoff.

Common Mistake: Treating a Discrete P-Value as Any Number

A common mistake is to assume that an exact test must be able to return a p-value of exactly 0.05, or that a reported p-value near 0.05 can be adjusted to equal it. That reasoning ignores the possible outcomes and their null probabilities. For the four-trial example, the exact values were 0.125, 0.625, and 1; none is 0.05, and rounding cannot create an attainable exact value that was not calculated.

The correction is to distinguish the exact calculation from an approximation and report what was actually used. A small count may provide only a coarse set of exact p-values. A larger sample often creates more possible values and smaller probability jumps, but that does not guarantee that any particular target value is attainable. A continuous approximation can be useful when its conditions are reasonable, but it should not be described as the exact binomial tail probability.