Rosenbaum Bounds for Matched Observational Studies

Why Sensitivity Bounds Fit Matched Studies

Matching makes treated and control units more comparable on measured characteristics, but it cannot guarantee that treatment assignment is free of hidden bias. Rosenbaum bounds ask a focused question: how much could treatment odds differ between otherwise matched units because of an unmeasured factor, before a randomization-based conclusion changes? The method expresses that allowance with a sensitivity parameter $\Gamma$, then evaluates inference under the most unfavorable assignment probabilities permitted by the model.

The analysis is conditional on the matched sets and their treatment counts. For clarity, the definitions and calculations below use matched pairs with one treated unit and one control unit in each pair. The central logic extends to larger matched sets, where the sensitivity analysis optimizes over the allowed conditional assignment probabilities within each set.

Definition: In the paired Rosenbaum sensitivity model, $\Gamma\geq 1$ bounds the ratio of the treatment-assignment odds for the two units in a matched pair. Equivalently, if $p_i$ is the conditional probability that a particular member of pair $i$ receives treatment, then $$ \frac{1}{1+\Gamma}\leq p_i\leq\frac{\Gamma}{1+\Gamma}. $$ At $\Gamma=1$, both members have treatment probability $1/2$, as in a randomized assignment within each pair. Larger $\Gamma$ permits greater hidden assignment imbalance.

The model is a restriction on the assignment mechanism, not a model for how the outcome is generated. In observational data, the probabilities are not known to be randomized; the analysis instead asks what would follow if their departures from equal odds were no greater than the specified $\Gamma$. The bound does not identify the size of hidden bias from the data. Choosing $\Gamma$ is a sensitivity judgment.

Testing a Sharp Null Under the Sensitivity Model

A randomization test first specifies a sharp null hypothesis: for example, that treatment has no effect on any unit. Under that null, each unit's outcome under treatment equals its outcome under control, so the observed outcomes can be used to calculate the test statistic under every assignment allowed by the matched design. In a pair, swapping treatment labels reverses the sign of the pair's treated-minus-control difference.

For a simple sign test, let $K$ be the number of pairs whose treated-minus-control difference is positive. If the null holds and assignment is equally likely within every pair, each sign is positive with probability $1/2$. For a two-sided test, an observed count $k$ is at least as extreme as the observed result when $|K-n/2|\geq |k-n/2|$, where $n$ is the number of non-tied pairs. Under a sensitivity model, the p-value is the largest probability of that event over all allowed assignment probabilities. This is a worst-case, or upper-bound, p-value.

Definition: The Rosenbaum upper-bound p-value at sensitivity level $\Gamma$ is the supremum, over assignment mechanisms satisfying the $\Gamma$ model, of the probability under the sharp null of obtaining a test statistic at least as extreme as the observed statistic. A large upper-bound p-value means the data do not reject the null even under the most unfavorable assignment mechanism allowed by the model.

For a one-sided sign test of an excess of positive differences, the event $K\geq k$ becomes more likely as each pair's probability of a positive sign increases. Its worst case therefore uses $p_i=\Gamma/(1+\Gamma)$ in each pair. For two-sided tests, the upper tail and lower tail must be considered together; the worst-case calculation is an optimization over the permitted probabilities. In the special case where all $n$ observed signs point in one direction, the two-sided extreme event is $K=0$ or $K=n$. Its largest probability is $q^n+(1-q)^n$, where $q=\Gamma/(1+\Gamma)$. One way to see this is that the probability of the event is a function linear in each $p_i$ separately, so a maximum occurs at a corner of the allowed probability range; evaluating those corners shows that setting all probabilities to the same endpoint maximizes the sum.

Worked Example: A Worst-Case p-Value With Four Matched Pairs

Suppose four hypothetical matched pairs have positive treated-minus-control differences, so $K=4$ of $n=4$ signs are positive. Test the sharp null against a two-sided alternative at $\Gamma=2$. The observed sign pattern is maximally extreme, so the two-sided event is $K=4$ or $K=0$. Here $q=2/(1+2)=2/3$, and the lower probability bound is $1/3$.

The worst-case probability is $$ p_{\Gamma=2}=\left(\frac{2}{3}\right)^4+\left(\frac{1}{3}\right)^4 =\frac{16}{81}+\frac{1}{81} =\frac{17}{81} \approx 0.2099. $$

At $\Gamma=1$, the corresponding exact two-sided p-value is $2(1/2)^4=0.1250$. Allowing the assignment odds to differ by as much as a factor of two raises the worst-case p-value to about 0.210. At a significance level of 0.05, neither value rejects the sharp null. This example does not show that the treatment has no effect; it shows that this test does not reject the sharp null under the stated analysis.

How the Sensitivity Parameter Changes Inference

Increasing $\Gamma$ widens the permitted range of assignment probabilities. If $\Gamma_2\geq\Gamma_1$, every assignment mechanism allowed at $\Gamma_1$ is also allowed at $\Gamma_2$. Taking a supremum over a larger set cannot produce a smaller value. Thus worst-case p-values are nondecreasing as $\Gamma$ increases.

The same logic applies to confidence sets formed by inverting tests. For each candidate treatment effect, test the corresponding sharp null and retain the candidate when the upper-bound p-value exceeds the chosen significance level $\alpha$. A larger $\Gamma$ can only increase or preserve each candidate's upper-bound p-value, so a candidate retained at a smaller $\Gamma$ remains retained at a larger one. The resulting confidence set can widen, but cannot narrow.

Worked Example: A Sensitivity Threshold for Six Positive Signs

In a hypothetical study, all six matched-pair differences are positive. At $\Gamma=1$, the two-sided exact sign-test p-value is $$ 2\left(\frac{1}{2}\right)^6=\frac{2}{64}=0.03125. $$ At $\alpha=0.05$, this rejects the sharp null under equal assignment odds.

At $\Gamma=1.5$, $q=1.5/2.5=0.6$. The worst-case two-sided p-value becomes $$ (0.6)^6+(0.4)^6 =0.046656+0.004096 =0.050752. $$ This is just above 0.05, so the sharp null is no longer rejected using the upper-bound p-value. Solving $q^6+(1-q)^6=0.05$, then converting by $\Gamma=q/(1-q)$, gives a tipping value of approximately $\Gamma=1.488$. In this example, an allowance of roughly 1.49 for the within-pair assignment-odds ratio is enough to remove rejection at the 0.05 level. This threshold describes the test's sensitivity; it does not estimate the actual hidden bias.

Confidence Bounds by Inverting Tests

A confidence set for a constant additive treatment effect can be obtained by testing candidate values. Let $d_i$ denote the observed treated-minus-control difference in pair $i$. For a candidate effect $\tau$, subtract $\tau$ from each observed difference and test the sharp null that every unit's treatment effect equals $\tau$. The adjusted difference is $d_i-\tau$; its sign supplies the sign-test statistic. Tied adjusted differences contribute no sign and are omitted from that test.

The sensitivity confidence set at level $1-\alpha$ contains each candidate $\tau$ whose worst-case p-value is greater than $\alpha$. Inverting the tests gives coverage protection under the stated matched design, the constant additive-effect formulation, and the assignment model at the chosen $\Gamma$. It does not provide protection against assignment mechanisms outside that model or against a mismatch between the constant-effect assumption and the scientific setting.

Worked Example: Inverting a Sign Test at Γ = 1

Suppose six hypothetical matched pairs have observed differences, in outcome units, of $2,3,4,5,6,$ and $7$. Use the two-sided sign test at $\alpha=0.05$, with $\Gamma=1$, to form a confidence set for a constant additive effect $\tau$.

If $\tau<2$, all six adjusted differences are positive. The p-value is $2/2^6=0.03125$, so these candidates are rejected. If $\tau>7$, all six are negative and the same p-value results, so these candidates are rejected as well.

At $\tau=2$, one adjusted difference is zero and is omitted. The remaining five signs are all positive, giving $$ p=\frac{2}{2^5}=0.0625. $$ This candidate is retained because $0.0625>0.05$. At $\tau=7$, the five nonzero signs all point in the other direction, so its p-value is also 0.0625 and it is retained.

For a candidate strictly between 2 and 7 that is not equal to an observed difference, all six adjusted differences are nonzero. The least-extreme sign patterns in this range are 5-to-1 splits, for which $$ p=\frac{2\left\{\binom{6}{5}+\binom{6}{6}\right\}}{2^6} =\frac{2(6+1)}{64} =\frac{14}{64} =0.21875. $$ More balanced patterns have still larger two-sided p-values. At interior candidates equal to one of the observed differences, the zero is omitted; the remaining sign patterns also give p-values above 0.05. The retained set is therefore $[2,7]$ outcome units. It is a 95% confidence set from test inversion under the equal-odds model; it is not a statement that there is a 95% probability that the fixed effect lies in this particular interval.

At $\Gamma=2$, candidates below 2 still give six signs in one direction, but their worst-case p-value is $65/729\approx0.08916$, which exceeds 0.05. Candidates above 7 have the same bound. Thus the sensitivity confidence set extends beyond both observed endpoints at $\Gamma=2$; the equal-odds interval is not robust to that degree of hidden assignment bias.

Calculation Workflow and Interpretation

A Rosenbaum analysis is most transparent when the matched design, test statistic, sensitivity model, and interpretation are kept distinct. A practical workflow is:

Specify the matched assignment structure.
State the matched sets and the number treated in each. The paired formulas above assume one treated unit and one control unit per pair.

Choose the null and statistic.
For example, test a sharp null using the number of positive pair differences, or another statistic appropriate to the outcome and study question.

Set Γ and calculate the worst case.
Optimize the randomization tail probability over assignment probabilities satisfying the model. Report the sensitivity level and the upper-bound p-value.

For an effect interval, invert candidate tests.
Test each candidate effect at the same Γ and retain the candidates whose upper-bound p-values exceed $\alpha$.

The choice of statistic matters: Rosenbaum bounds are not one universal p-value attached to a matched data set. A test using signs, ranks, or a magnitude-sensitive statistic can have different sensitivity. For matched sets larger than pairs, the conditional assignment probabilities must respect the set's fixed treatment count, and the worst-case calculation must use that structure rather than treating all units as independent Bernoulli assignments.

Common Pitfall

Wrong: “The matched pairs control confounding, so $\Gamma=1$ is justified.” Matching controls measured characteristics used in the design; it does not establish equal treatment odds within pairs in the presence of unmeasured factors. Correction: Treat $\Gamma=1$ as the no-hidden-assignment-bias benchmark and report how conclusions change over larger values.
Wrong: “The upper-bound p-value is the probability that the null hypothesis is true.” Correction: It is the largest probability, over allowed assignment mechanisms under the null, of a result at least as extreme as the observed one.
Wrong: “A large $\Gamma$ proves that confounding of that size exists.” Correction: $\Gamma$ specifies a bound considered in the analysis; the observed matched outcomes do not reveal the actual assignment-odds ratio due to hidden factors.
Wrong: “A confidence interval that includes zero proves there is no effect.” Correction: It indicates that zero is not rejected under the specified test, effect model, confidence level, and sensitivity bound. It does not establish a zero causal effect.

Key takeaway: Rosenbaum bounds quantify how inference from matched observational sets changes when hidden factors are allowed to alter treatment odds by up to $\Gamma$. Larger $\Gamma$ expands the set of possible assignment mechanisms, so worst-case p-values cannot decrease and test-inverted confidence sets cannot become narrower.

Check Your Understanding

Answer each question using the matched-pair setup and the definitions above.

What range of treatment-assignment probabilities is allowed in a pair when $\Gamma=3$?
Why is the Rosenbaum upper-bound p-value nondecreasing as $\Gamma$ increases?
For six non-tied pairs with a 5-to-1 sign split under $\Gamma=1$, calculate the two-sided exact sign-test p-value.
When inverting tests for a constant additive effect, what is done with a pair whose adjusted difference $d_i-\tau$ is zero?
Does choosing $\Gamma=2$ establish that hidden factors actually double treatment odds within a pair? Explain.