Worst-Case P-Values for Matched-Pair Tests

Why Worst-Case Sign-Test P-Values Matter

A matched-pair sign test asks whether the treated unit tends to have a higher outcome than its matched control. Under randomized treatment assignment, the sharp null of no treatment effect makes the two treatment labels within each pair exchangeable. Hidden bias weakens that symmetry: even within a matched pair, one unit may have been more likely to receive treatment. A worst-case p-value asks how large the relevant tail probability could be under any assignment probabilities allowed by the sensitivity model.

This is a sensitivity analysis, not a correction that removes hidden bias. It quantifies how the evidence against a sharp null changes when treatment assignment may depend on an unmeasured factor. The calculation below uses only the direction of each pair’s outcome difference; its magnitude is ignored.

Pairwise Assignment Bounds and the Sign Statistic

Suppose there are $m$ matched pairs, with exactly one treated unit and one control unit in each pair. Conditional on the matched pairs, let the odds of treatment for either member of a pair differ from those for the other member by at most a factor $\Gamma$, where $\Gamma\geq 1$. This is the pairwise version of Rosenbaum’s hidden-bias model. At $\Gamma=1$, the two members have equal treatment odds; larger values allow greater within-pair assignment imbalance.

Definition: For pair $i$, let $q_i$ be the probability, under the assignment mechanism and conditional on that pair, that treatment is assigned to the member whose fixed outcome is higher. The sensitivity model implies $$ \frac{1}{1+\Gamma}\leq q_i\leq \frac{\Gamma}{1+\Gamma}. $$ Let $S$ be the number of pairs in which the observed treated outcome exceeds the observed control outcome. Pairs with tied outcomes are omitted, so $m$ is the number of non-tied pairs.

The bound follows from the odds constraint. If $q_i$ is the probability of treating the higher-outcome member, the odds in its favor are $q_i/(1-q_i)$. Requiring those odds to be no more than $\Gamma$ gives $q_i\leq\Gamma/(1+\Gamma)$. Applying the same requirement to the other member gives $1-q_i\leq\Gamma/(1+\Gamma)$, or $q_i\geq1/(1+\Gamma)$.

Under the sharp null, each unit’s outcome is fixed regardless of treatment. If the treatment label is switched within a pair, the sign of its treated-minus-control difference switches too. Thus, for a pair whose observed sign is positive, the probability of a positive sign under reassignment is $q_i$; for a pair whose observed sign is negative, it is $1-q_i$. Both probabilities lie within the same bounds. Under the usual matched-pair conditional randomization model, assignments are independent across pairs. Consequently, $S$ has a sum-of-independent-Bernoulli distribution, with pair-specific success probabilities in the stated interval.

Worst-Case One-Sided P-Values

For the alternative that treatment tends to increase outcomes, the observed evidence is a large value of $S$. The one-sided worst-case p-value is the largest probability of obtaining at least the observed number of positive signs, over all assignment probabilities allowed by the sensitivity model. A larger success probability makes this upper-tail event more likely. Therefore, the maximum is attained by setting every pair’s probability to its upper bound.

Result: If $s$ of the $m$ non-tied pairs have positive signs, the worst-case upper-tail p-value is $$ P\!\left\{B\geq s\right\},\qquad B\sim\operatorname{Binomial}\!\left(m,\frac{\Gamma}{1+\Gamma}\right). $$ For a lower-tail alternative, maximize $P(S\leq s)$ by setting every success probability to its lower bound. The resulting worst-case p-value is $$ P\!\left\{B\leq s\right\},\qquad B\sim\operatorname{Binomial}\!\left(m,\frac{1}{1+\Gamma}\right). $$

These are exact finite-sample probabilities under the specified assignment model, not normal approximations. The one-sided direction must be chosen from the scientific question before inspecting which direction gives the smaller p-value. Reporting whichever direction looks more favorable without accounting for that choice does not preserve the intended one-sided test.

Worked Example: An Upper-Tail Test with Six Pairs

Suppose a hypothetical matched study has six non-tied pairs, five with a positive treated-minus-control difference. The prespecified alternative is that treatment increases the outcome, and the sensitivity analysis uses $\Gamma=2$. The model permits the probability of treating the higher-outcome member to be as large as $$ \frac{\Gamma}{1+\Gamma}=\frac{2}{3}. $$

The worst-case upper-tail p-value is the probability of at least five positive signs when $B\sim\operatorname{Binomial}(6,2/3)$: $$ P(B\geq5)=\binom{6}{5}\left(\frac{2}{3}\right)^5\left(\frac{1}{3}\right)+\binom{6}{6}\left(\frac{2}{3}\right)^6. $$ Substituting and simplifying gives $$ 6\left(\frac{32}{243}\right)\left(\frac{1}{3}\right)+\frac{64}{729} =\frac{192}{729}+\frac{64}{729} =\frac{256}{729}\approx0.3512. $$

At a significance level of $0.05$, this worst-case p-value is not small enough to reject the sharp null. This does not establish that the null is true; it says that, under the allowed hidden bias, an upper-tail result at least this extreme has probability as high as about $0.351$.

Worked Example: A Lower-Tail Test with Five Pairs

In a second hypothetical study, one of five non-tied pairs has a positive sign. Suppose the prespecified alternative is that treatment decreases the outcome and $\Gamma=3$. For this lower-tail test, the least favorable assignment probabilities for producing few positive signs are all at their lower bound, $1/(1+\Gamma)=1/4$.

Let $B\sim\operatorname{Binomial}(5,1/4)$. The worst-case p-value is $$ P(B\leq1)=\left(\frac{3}{4}\right)^5 +5\left(\frac{1}{4}\right)\left(\frac{3}{4}\right)^4 =\frac{243}{1024}+\frac{405}{1024} =\frac{648}{1024} =0.6328. $$

The one-sided evidence is weak at conventional significance levels: even the least favorable allowed assignment mechanism gives a probability of about $0.633$ of one or fewer positive signs under the sharp null. The direction of the alternative is essential here. Using the upper-tail formula instead would test a different claim.

Exact Worst-Case Two-Sided P-Values

A two-sided sign test treats a count far above or far below $m/2$ as evidence against the sharp null. If the observed count is $s$, define the two-sided event by $$ |2S-m|\geq |2s-m|. $$ Equivalently, when $s>m/2$, the event is $S\geq s$ or $S\leq m-s$; the corresponding cutoffs reverse when $s<m/2$. When $s=m/2$, every possible count is at least as close to the center as the observed count, so this definition gives a p-value of 1.

For $\Gamma>1$, the assignment probabilities need not all be equal. Maximizing the upper tail alone and maximizing the lower tail alone can use different assignment mechanisms; adding those two separate maxima can therefore overstate the exact maximum probability of the two-sided event. To get the exact worst-case two-sided p-value, maximize the probability of the union of the two tails under one allowed assignment mechanism.

Result: The two-sided worst-case p-value is the supremum of $P\{|2S-m|\geq|2s-m|\}$ over all $q_1,\ldots,q_m\in[1/(1+\Gamma),\Gamma/(1+\Gamma)]$. It is sufficient to check assignments in which each $q_i$ is at one of the two endpoints. If $k$ pairs use the upper endpoint and $m-k$ use the lower endpoint, then $$ S\ \sim\ \operatorname{Binomial}\!\left(k,\frac{\Gamma}{1+\Gamma}\right) +\operatorname{Binomial}\!\left(m-k,\frac{1}{1+\Gamma}\right), $$ where the two binomial variables are independent. Calculate the two-sided event probability for $k=0,1,\ldots,m$; the largest is the exact worst-case p-value.

The endpoint check is justified because the probability of any event for a sum of independent Bernoulli variables is affine in any one $q_i$ when all the other probabilities are fixed. An affine function on a closed interval attains its maximum at an endpoint. Starting with any allowed vector, replace its first coordinate by a maximizing endpoint, then its second, and continue. Each replacement cannot decrease the event probability, and the final vector has every coordinate at an endpoint. Since the event depends only on the sum, permuting which pairs receive the upper endpoint leaves the distribution unchanged; only the number $k$ matters. This gives a finite calculation with $m+1$ candidate distributions.

Worked Example: Maximizing a Two-Sided P-Value

Suppose all four non-tied pairs in a hypothetical study have positive signs, so $m=4$ and $s=4$. The two-sided event is $S=4$ or $S=0$. Let $\Gamma=2$, giving endpoint probabilities $1/3$ and $2/3$. If $k$ pairs have success probability $2/3$ and the other $4-k$ have probability $1/3$, the event probability is $$ \left(\frac{2}{3}\right)^k\left(\frac{1}{3}\right)^{4-k} +\left(\frac{1}{3}\right)^k\left(\frac{2}{3}\right)^{4-k}. $$

For $k=0$, this is $1/81+16/81=17/81$. For $k=1$, it is $2/81+8/81=10/81$. For $k=2$, it is $4/81+4/81=8/81$. The cases $k=3$ and $k=4$ repeat the values for $k=1$ and $k=0$, respectively. The maximum is therefore $$ \frac{17}{81}\approx0.2099. $$

Thus the exact worst-case two-sided p-value is approximately $0.210$. For comparison, summing the separately maximized upper and lower tails would give $16/81+16/81=32/81$, which is not the exact maximum: those two tail maxima require different assignment probabilities. The calculation illustrates why the two-sided event must be optimized as a whole.

Assumptions, Diagnostics, and Limitations

The calculations rely on several conditions. There must be one treated and one control unit in each pair, the sharp null must hold for the randomization reference distribution, and the sensitivity model must appropriately bound within-pair treatment odds by $\Gamma$. The binomial one-sided results and the two-sided endpoint calculation also rely on independence of treatment assignments across pairs, conditional on the matched design. If assignments across pairs are dependent, the marginal bounds alone do not justify these product-form distributions; the joint assignment mechanism must be specified and optimized instead.

Ties contribute no directional information to a sign test and are excluded from $m$. State how many ties were omitted, since removing many ties can leave little information. A useful diagnostic is to recompute the worst-case p-value across a range of $\Gamma$ values, including $\Gamma=1$. As $\Gamma$ increases, the permissible assignment mechanisms expand, so a worst-case p-value cannot decrease. A decreasing value as $\Gamma$ rises signals a calculation or implementation error. Also check that the one-sided direction matches the alternative, that $m$ excludes ties, and that the two-sided calculation evaluates both tails under each same candidate mechanism.

The sign test is robust to the sizes of outcome differences because it uses only their directions, but that is also a limitation: a tiny positive difference and a very large positive difference count equally. At $\Gamma=1$, it reduces to the ordinary paired sign test under equal assignment probabilities. Other paired analyses use more outcome information and require their own assumptions and sensitivity calculations; their results are not interchangeable with this sign-based analysis. In particular, do not substitute a sensitivity procedure designed for continuous ranked outcomes and assume it produces the same p-value.

Common Pitfalls

Using the fair-assignment probability when $\Gamma>1$. The value $1/2$ applies only at $\Gamma=1$. For an upper-tail worst case, use $\Gamma/(1+\Gamma)$.
Adding two individually maximized tails as if they came from one mechanism. This can be conservative but is not generally the exact two-sided worst-case p-value. Optimize the union event over a single probability vector.
Calling a large p-value evidence that the sharp null is true. A p-value describes the probability of a result at least as extreme under the null and the specified assignment model; it is not the probability that the null is true.
Ignoring the assignment assumptions. Pairwise probability bounds do not by themselves imply independence across pairs. The binomial formulas require the conditional independence stated above.

Key takeaway: For a one-sided matched-pair sign test, the worst-case tail is binomial with success probability at the appropriate endpoint of the Rosenbaum bound. For an exact two-sided p-value, maximize the probability of the full two-tail event under one assignment mechanism, checking all endpoint configurations.

Check Your Understanding

Answer these questions, then compare your reasoning with the solutions.

With $\Gamma=1$, what are the lower and upper bounds on the probability of treating the higher-outcome member of a pair?
For an upper-tail test with $m=3$, $s=3$, and $\Gamma=1$, calculate the p-value.
For a lower-tail test, why is the least favorable success probability the lower endpoint?
For an exact two-sided p-value, why is it enough to examine endpoint probabilities?
Name one assumption needed for the binomial formulas and one limitation of the sign test.

Solutions. (1) Both bounds are $1/2$, since $1/(1+1)=1/2$ and $1/(1+1)=1/2$. (2) The result is $P(B\geq3)$ for $B\sim\operatorname{Binomial}(3,1/2)$, so $p=(1/2)^3=0.125$. (3) A smaller success probability makes a small-count event $S\leq s$ more likely, maximizing its probability at $1/(1+\Gamma)$. (4) Holding all other probabilities fixed, the event probability is affine in each individual success probability, so a maximum over its allowed interval occurs at an endpoint; applying this successively gives an endpoint vector. (5) The formulas require conditional independence of assignments across pairs. The sign test discards the magnitudes of the outcome differences.