Worked Matched-Pair Sensitivity Analysis

Why Analyze Hidden Bias After Matching?

Matching can make treated and control units similar on recorded covariates, but it cannot guarantee similarity on covariates that were not measured. A matched-pair sensitivity analysis asks a focused question: how much could an unmeasured factor alter treatment assignment within pairs before the evidence against a no-effect hypothesis becomes inconclusive? Rosenbaum bounds quantify this question by restricting how different treatment assignment odds may be within a matched pair.

The analysis below uses a Wilcoxon signed-rank statistic for paired outcome differences. It is a test of a sharp null hypothesis under a stated sensitivity model, not an estimate of how large the treatment effect is. The one-sided direction must be chosen before inspecting the results. Here, a positive difference means a better outcome under treatment, and the question is whether treatment tends to improve the outcome.

Definition: In a matched-pair sensitivity model, $\Gamma \geq 1$ bounds the odds ratio for treatment assignment between the two units in any pair, conditional on the matched covariates. If one unit has assignment probability $\pi$ and the other has probability $1-\pi$, the bound is $$ \frac{1}{\Gamma} \leq \frac{\pi}{1-\pi} \leq \Gamma. $$ Equivalently, the probability of the unit favored for treatment is at most $q=\Gamma/(1+\Gamma)$. At $\Gamma=1$, assignment probabilities are equal within every pair. Larger $\Gamma$ permits greater hidden imbalance.

This model presumes one treated and one control unit in each pair, that the pairs are independent of one another for the test calculation, and that the assignment-odds restriction is a defensible sensitivity model. The sharp null says that treatment has no effect on any unit’s outcome. Under that null, outcomes are fixed while the treatment labels within pairs can be considered to vary. A small sensitivity p-value is evidence against the sharp null under the specified bound; it does not establish that the model captures every possible form of hidden bias.

From Paired Differences to the Signed-Rank Statistic

For pair $i$, let $D_i$ be the treated unit’s outcome minus the control unit’s outcome. Exclude pairs with $D_i=0$, then rank the nonzero values by $|D_i|$, using average ranks for tied absolute differences. The Wilcoxon signed-rank statistic $W^+$ is the sum of the ranks attached to positive differences. Larger $W^+$ supports the prespecified alternative that treatment tends to increase the outcome.

Under $\Gamma=1$, the treatment label is equally likely to fall on either unit in each pair. Conditional on the observed absolute differences, each signed rank is therefore positive with probability $1/2$. For $\Gamma>1$, the sensitivity bound allows the probability of a positive sign to be as large as $q=\Gamma/(1+\Gamma)$ in every pair. That is the assignment pattern most favorable to an unusually large $W^+$, and it gives a conservative upper-tail p-value bound.

Worked Example: Ranking Eight Paired Differences

Suppose a hypothetical matched-pair study records outcome differences, in outcome points, of $-1,-2,3,4,5,6,7,8$. Positive values favor treatment. The absolute differences are $1,2,3,4,5,6,7,8$, so their ranks are $1,2,3,4,5,6,7,8$; there are no ties or zero differences.

The positive differences have ranks 3 through 8. Thus the observed statistic is $$ W^+=3+4+5+6+7+8=33. $$ The total of all ranks is $1+2+\cdots+8=36$. The negative ranks sum to 3, which will make the upper-tail calculation especially transparent.

This calculation presumes that pairs are independent and that the outcome direction and one-sided alternative were specified in advance. It uses the paired structure: replacing the differences with two unrelated outcome samples would not be the same analysis.

Why the Sensitivity Bound Is Conservative

For a positive-tail test, write $B_i=1$ when pair $i$’s rank has a positive sign, and $B_i=0$ otherwise. Conditional on the absolute ranks $r_i$, the statistic is $W^+=\sum_i r_iB_i$. Under the sensitivity model, each sign probability is no greater than $q$ in the direction that maximizes the upper tail.

The bound follows by coupling the signs to independent uniform random variables $U_i$ on $[0,1]$. Set $B_i=1$ when $U_i\leq p_i$, where $p_i\leq q$, and define $B_i^*=1$ when $U_i\leq q$. Then $B_i\leq B_i^*$ for every pair, so $\sum_i r_iB_i\leq\sum_i r_iB_i^*$. Consequently, for any threshold $w$,

P(W^+\geq w)\leq P\left(\sum_i r_iB_i^*\geq w\right), \qquad B_i^*\sim \operatorname{Bernoulli}(q).

Thus the right-hand probability is a worst-case upper-tail bound. It is calculated as if every pair had the largest permitted probability $q$ of a positive sign. This justification relies on independent pair assignments in the sensitivity model; dependence between pairs requires a different analysis.

Worked Example: The Bound at No Hidden Bias

For the eight ranks above, an upper-tail result of at least 33 occurs when the negative ranks sum to at most 3, since the total rank sum is 36. The possible negative-rank subsets are the empty set, any one of ranks 1, 2, or 3, and the pair of ranks 1 and 2. There are five such subsets among $2^8=256$ equally likely sign patterns when $\Gamma=1$.

Therefore the one-sided p-value at $\Gamma=1$ is $$ P(W^+\geq 33)=\frac{5}{256}=0.01953. $$ At the conventional $\alpha=0.05$ level, this rejects the sharp null under equal treatment odds within pairs. This is not yet a conclusion robust to hidden bias; increasing $\Gamma$ increases the worst-case probability of positive signs.

Calculating the Gamma Tipping Point

For these ranks, let $q=\Gamma/(1+\Gamma)$. To have $W^+\geq33$, the negative signs must have total rank at most 3. There can be no negative signs, one negative sign on rank 1, 2, or 3, or negative signs on both ranks 1 and 2. Under the worst case, the probability of a positive sign is $q$ and of a negative sign is $1-q$, independently across pairs. The bound is therefore

p_{\mathrm{bound}}(q) =q^8+3q^7(1-q)+q^6(1-q)^2 =q^6(1+q-q^2).

The Gamma tipping point is the smallest $\Gamma$ at which this upper-tail bound reaches the chosen significance level. At $\alpha=0.05$, solve $q^6(1+q-q^2)=0.05$ numerically. The solution is approximately $q=0.585374$. Converting that probability to the odds-ratio scale gives

\Gamma=\frac{q}{1-q} \approx\frac{0.585374}{1-0.585374} =\frac{0.585374}{0.414626} \approx 1.4118.

The displayed values and quotient are consistent at the shown precision. The tipping point means that allowing the treatment odds within a pair to differ by about a factor of 1.412 is sufficient for this worst-case one-sided bound to reach 0.05. It does not mean that such an unmeasured confounder exists, nor does it express the strength of a confounder’s association with the outcome.

Worked Example: Checking the Conclusion at Two Gamma Values

At $\Gamma=1.25$, the maximum positive-sign probability is $q=1.25/2.25=5/9$. Substitution in the bound gives $$ p_{\mathrm{bound}} =\left(\frac{5}{9}\right)^6\left(1+\frac{5}{9}-\frac{25}{81}\right) =\left(\frac{5}{9}\right)^6\frac{101}{81} \approx 0.03666. $$ This remains below 0.05, so the result rejects the sharp null at that sensitivity level.

At $\Gamma=1.5$, $q=1.5/2.5=0.6$, and $$ p_{\mathrm{bound}}=0.6^6(1+0.6-0.6^2) =0.046656(1.24) =0.05785. $$ This is above 0.05, so the analysis does not reject the sharp null at this sensitivity level. The two calculations locate the change in conclusion between 1.25 and 1.5, consistent with the numerical tipping point of about 1.412.

Reading the Result and Diagnosing Problems

A useful report gives the observed statistic, the testing direction, the sensitivity parameter, the significance level, and the bound at substantively chosen values of $\Gamma$. The tipping point is a compact summary, but it should not replace the sensitivity curve or a few representative bound values. Its practical meaning depends on whether an odds imbalance of that size is plausible in the subject matter.

This analysis answers a different question from an E-value. A Rosenbaum bound describes how much hidden bias in treatment assignment, within matched pairs, can weaken a test result. An E-value summarizes the minimum association strength with treatment and outcome that would be needed to explain away an estimate under its assumptions. Neither quantity measures hidden bias directly, and their numerical values are not interchangeable. A paired t-test, by contrast, uses a model for the mean of the differences and does not by itself provide a bound for hidden treatment-assignment bias.

Check the pair structure. Each pair must contain one treated and one control unit, and the paired differences must correspond to the intended matched units.
Check zeros and ties. Zero differences are omitted from the signed-rank calculation; tied absolute differences receive average ranks. The rank set and tail probability must reflect those choices.
Pre-specify the direction. A positive-tail calculation cannot be presented as a two-sided test after seeing which direction the differences favor. A two-sided sensitivity analysis needs a two-sided test and its own bound calculation.
Check the independence assumption. If units or pairs share clusters or treatment-assignment mechanisms, independent pairwise sign calculations may not apply.
Do not overstate the tipping point. Losing significance at a particular $\Gamma$ does not prove there is confounding of that magnitude or that the treatment effect is zero.

The method is intentionally limited: it tests a sharp null rather than estimating an effect interval, and its interpretation is conditional on the matching design, the sensitivity model, and the chosen test. A small bound p-value indicates robustness of this test to the modeled degree of hidden assignment bias, not robustness to every possible misspecification.

Key takeaway: A matched-pair Rosenbaum analysis raises the largest permitted positive-sign probability to $q=\Gamma/(1+\Gamma)$, computes the resulting worst-case Wilcoxon tail probability, and identifies where that bound crosses the chosen significance level. The crossing quantifies sensitivity under the model; it is not a measurement of actual confounding.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For paired differences $-1,2,3,4$, with no ties or zeros and a positive one-sided alternative, find $W^+$ and the exact p-value at $\Gamma=1$.
Interpret $\Gamma=2$ in words. What is the largest probability $q$ for the unit favored for treatment?
Why are zero differences not counted as positive ranks in a signed-rank analysis?
In the eight-pair example, what is the decision at $\Gamma=1.5$ and $\alpha=0.05$? What does that decision not establish?

Solutions. (1) The ranks are 1, 2, 3, and 4. Positive ranks sum to $2+3+4=9$. The total is 10, so $W^+\geq9$ occurs when the negative ranks sum to at most 1: either no negative rank or rank 1 alone. Thus the one-sided exact p-value is $2/16=0.125$. (2) The treatment odds may be up to twice as large for one member of a pair as for the other, conditional on matched covariates; $q=\Gamma/(1+\Gamma)=2/3$. (3) A zero difference has no positive or negative sign and contributes no signed rank; it is omitted, and the ranks of the nonzero differences are assigned accordingly. (4) The worst-case bound is approximately 0.05785, above 0.05, so the sharp null is not rejected at that sensitivity level. This does not prove the null, show that hidden bias of that magnitude exists, or rule out a treatment effect.