Exact Inference for Sign Tests

From Sign Counts to Exact Inference

A sign test reduces observations to a count: how many lie above a proposed median, or how many paired differences are positive. Under the null hypothesis, that count has a binomial distribution. Exact inference uses this distribution directly rather than approximating it with a normal curve.

The binomial model also provides a way to build a confidence interval. Instead of testing just one proposed median, we can test a range of candidate values and retain those not rejected. Because a sign count can take only whole-number values, exact p-values and confidence levels are discrete; a nominal 95% interval may therefore have actual coverage greater than 95%.

Definition: For a proposed population median $m_0$, let $B$ be the number of non-tied observations greater than $m_0$, and let $n$ be the number of non-tied observations. Under the sign-test null $H_0$, with independent observations and no probability mass at $m_0$, each observation is equally likely to fall above or below $m_0$. Thus $B\sim\operatorname{Binomial}(n,0.5)$. For paired data, apply the same definition to the nonzero paired differences and a proposed median difference.

The null is commonly stated as “the population median equals $m_0$.” For the binomial sign model to represent that median null, the distribution must have equal probabilities of falling above and below $m_0$. A continuous distribution at $m_0$ ensures there is no probability mass exactly at the proposed median. The observations (or paired differences) must also be independent across sampled units.

If an observed value equals the proposed median, its sign is zero and it does not contribute to $B$ or to the effective sample size. Under the continuous model, exact ties have probability zero; in practice, rounded measurements can produce ties. The calculation must use the number of nonzero signs, not the original number of observations.

Calculating an Exact Two-Sided P-Value

For a two-sided sign test, an outcome is at least as extreme as the observed sign count when it is equally far into either tail of the binomial distribution. The conventional doubled-tail p-value is twice the smaller inclusive tail probability, capped at 1. The word inclusive matters: the observed count itself is included in its tail.

Exact two-sided sign-test p-value: If $B\sim\operatorname{Binomial}(n,0.5)$ under $H_0$, and the observed count of positive signs is $b$, then $$ p=\min\left(1,\;2\min\{P(B\leq b),P(B\geq b)\}\right), $$ where $$ P(B=j)=\binom{n}{j}(0.5)^n=\frac{\binom{n}{j}}{2^n}. $$

The calculation is exact because it sums the probabilities of binomial outcomes under the null model. It does not mean that the null hypothesis has probability $p$ of being true. Rather, it is the probability, assuming the null model, of a sign count at least as far into a tail as the observed count, using the stated two-sided convention.

Worked Example: An Exact One-Sample P-Value

Suppose eight independent, hypothetical measurements in millimetres are $12, 14, 15, 16, 17, 18, 19, 22$. Test $H_0$: the population median is $20$ mm, against a two-sided alternative.

Seven measurements are below 20 and one is above it, so there are no zero signs, $n=8$, and the observed count of positive signs is $b=1$. Under the null, $B\sim\operatorname{Binomial}(8,0.5)$. The smaller tail is the lower tail:

P(B\leq 1)=\frac{\binom{8}{0}+\binom{8}{1}}{2^8} =\frac{1+8}{256} =\frac{9}{256} =0.03515625.

Therefore,

p=2\left(\frac{9}{256}\right) =\frac{18}{256} =0.0703125\approx 0.07031.

Equivalently, the upper tail is $P(B\geq 7)=9/256=0.03515625$, and doubling that unrounded probability gives the same p-value. At significance level $\alpha=0.05$, the p-value is larger than $\alpha$, so these data do not provide sufficient evidence against a population median of 20 mm. This is not proof that the median equals 20 mm.

Exact Inference for Paired Differences

For paired observations, define each difference using one consistent order, such as “after minus before.” The sign test then concerns the median of the population of paired differences. Under a null median difference of zero, and with independent pairs and no mass at zero, each difference is equally likely to be positive or negative. The number of positive differences follows a binomial distribution with success probability 0.5.

Worked Example: Testing a Paired Median Difference

In a hypothetical study, nine independent participants have paired measurements, and each difference is calculated as after minus before in arbitrary score points. The nine differences are $2, 1, 4, 3, -1, 2, 3, 1, 5$. Test whether the population median difference is zero, using a two-sided sign test.

There are eight positive differences and one negative difference, with no zeros. Thus $n=9$, $b=8$, and under $H_0$, $B\sim\operatorname{Binomial}(9,0.5)$. By symmetry, the smaller tail is $P(B\geq 8)=P(B\leq 1)$:

P(B\leq 1) =\frac{\binom{9}{0}+\binom{9}{1}}{2^9} =\frac{1+9}{512} =\frac{10}{512} =0.01953125.

The two-sided p-value is

p=2\left(\frac{10}{512}\right) =\frac{20}{512} =0.0390625.

At $\alpha=0.05$, reject $H_0$. The data provide evidence that the population median paired difference is not zero. Given the difference convention, the signs lean positive; the test does not establish that the measurements changed because of an intervention. That conclusion would require an appropriate study design, and the sign test itself does not estimate the size of the median difference.

Inverting Sign Tests to Form a Confidence Interval

A confidence interval can be constructed by testing candidate median values. For each candidate $m_0$, count observations above and below it and apply the exact two-sided sign test. The confidence set contains the candidate values whose tests are not rejected at level $\alpha$. With continuous data and no ties, the resulting set can be written using order statistics.

Order-statistic interval: Sort $n$ independent observations as $X_{(1)}\leq\cdots\leq X_{(n)}$. Choose an integer $r$ so that $$ 2P(B\leq r-1)\leq\alpha,\qquad B\sim\operatorname{Binomial}(n,0.5). $$ Then $[X_{(r)},X_{(n-r+1)}]$ is a sign-test confidence interval for the population median, with coverage $$ 1-2P(B\leq r-1). $$ For a paired median difference, use the sorted paired differences in place of the observations.

If more than one value of $r$ meets the inequality, the largest such $r$ gives the narrowest interval in this symmetric order-statistic construction. The coverage is at least $1-\alpha$, but is often greater because the binomial distribution is discrete. This interval relies on independent observations from a common population and a continuous distribution around the median, so that ties at the median do not complicate the sign-count argument.

Why the Order-Statistic Interval Has Its Coverage

Let $\theta$ be the population median and let $B$ count how many of the $n$ observations fall below $\theta$. Under the continuous median model, each observation has probability $0.5$ of falling below $\theta$, independently, so $B\sim\operatorname{Binomial}(n,0.5)$. The interval $[X_{(r)},X_{(n-r+1)}]$ contains $\theta$ when at least $r$ observations are below it and at least $r$ are above it. In terms of $B$, this is the event $r\leq B\leq n-r$. The two ways to miss are symmetric tails, giving:

P\{X_{(r)}\leq\theta\leq X_{(n-r+1)}\} =P(r\leq B\leq n-r) =1-2P(B\leq r-1).

This derivation also explains why the interval is conservative when the tail probability cannot equal $\alpha/2$ exactly: the integer cutoff must be chosen so that the total probability of the two excluded tails does not exceed $\alpha$.

Worked Example: A Confidence Interval for a Median

Suppose 12 hypothetical independent measurements, in millimetres, sorted from smallest to largest are $4, 5, 7, 8, 9, 10, 12, 13, 15, 16, 18, 21$. Construct a two-sided confidence interval with at least 95% coverage for the population median.

Here $n=12$, so the binomial denominator is $2^{12}=4096$. For $r=3$, the lower-tail probability is

P(B\leq 2) =\frac{\binom{12}{0}+\binom{12}{1}+\binom{12}{2}}{4096} =\frac{1+12+66}{4096} =\frac{79}{4096} \approx 0.01929.

The total excluded-tail probability is $2(79/4096)=158/4096\approx0.03857$, which is below 0.05. The corresponding coverage is

1-\frac{158}{4096} =\frac{3938}{4096} \approx 0.9614.

The interval uses the third and tenth order statistics:

[X_{(3)},X_{(10)}]=[7,16]\text{ mm}.

For the next narrower symmetric interval, $r=4$, the excluded-tail probability is $2P(B\leq3)$, where $P(B\leq3)=(1+12+66+220)/4096=299/4096$. Thus the coverage would be $1-598/4096\approx0.8540$, below 95%. The selected interval is therefore the narrowest interval from this construction that reaches at least 95% coverage. In repeated sampling under the model, this procedure covers the population median about 96.14% of the time. It does not mean there is a 96.14% probability that the fixed median lies in this particular observed interval.

What Exactness Does and Does Not Mean

“Exact” refers to calculating probabilities from the specified binomial model without using a large-sample approximation. It does not remove the need for assumptions. If observations are dependent, if the pairs are not independent, or if signs are not equally likely under the null, the binomial probabilities need not describe the data.

Exact sign inference is useful when the sample is small or the measurement distribution is not well described by a normal model. Its simplicity comes with a trade-off: it uses only whether each observation is above or below the candidate median, not how far away it is. As a result, it may have less power than procedures that use more information when their additional assumptions are appropriate.

For a one-sample test, use observations relative to the hypothesized population median.
For paired data, calculate one difference per pair, then use signs of those differences.
For a confidence interval, choose an order-statistic cutoff using the binomial tail probability, rather than assuming the interval has exactly its nominal coverage.

Common Mistakes

Rounding before doubling. If a tail is $9/256=0.03515625$, doubling the exact fraction gives $18/256=0.0703125$. Doubling the rounded value $0.03516$ instead gives $0.07032$. Keep full precision until the final rounding.
Using a one-sided tail for a two-sided question. A two-sided p-value uses both tails under the stated doubled-smaller-tail convention.
Including zero signs in the binomial sample size. A zero difference contributes neither a positive nor a negative sign; exclude it and use the nonzero count as $n$.
Calling a confidence level the probability that the parameter is in the observed interval. The confidence level describes the long-run coverage of the interval procedure under its assumptions.
Interpreting a small p-value as the probability that the null is true. It is calculated assuming the null model, and measures how unusual the observed sign count, or a more extreme one, would be under that model.

Key takeaway: Under the sign-test null, the count of positive signs has a $\operatorname{Binomial}(n,0.5)$ distribution. Inclusive binomial tail probabilities give exact p-values, and appropriate binomial cutoffs give order-statistic confidence intervals for a population median or paired median difference.

Check Your Understanding

Use the binomial model and the order-statistic reasoning to answer these questions.

In a sign test with $n=7$, the observed count of positive signs is $b=6$. Write the exact two-sided p-value in terms of a binomial tail probability.
Why are observations equal to the proposed median omitted from the sign count?
For paired measurements, what quantity should be sorted to form a confidence interval for the paired median difference?
Explain why a sign-test interval with nominal 95% confidence can have actual coverage greater than 95%.
What does a two-sided exact p-value of 0.04 mean, and what does it not mean?

What You'll Learn