Rosenbaum Gamma Sensitivity for Hidden Bias

Why Sensitivity to Hidden Bias Matters

Matching observational units on measured covariates can make treated and control units more comparable, but it cannot guarantee that they are comparable on unmeasured characteristics. If an unobserved factor affects both treatment assignment and the outcome, the matched comparison may still be biased. Rosenbaum’s Gamma sensitivity analysis asks a focused question: how strong would hidden treatment-assignment bias have to be to change the conclusion of a matched-study analysis?

The method does not estimate the size of an unmeasured confounder or remove its influence. Instead, it limits how different treatment-assignment odds may be within matched sets, then calculates how unfavorable the resulting test or confidence bound could be. The value of $\Gamma$ is a sensitivity parameter: the data do not generally identify it.

The Gamma Model for Matched Pairs

Consider $n$ matched pairs. Each pair contains one treated unit and one control unit, matched on observed covariates. Let $p_i$ be the probability that the first member of pair $i$ receives treatment, conditional on exactly one member of that pair being treated and on the matched covariates. Rosenbaum’s model bounds the odds ratio comparing the two members’ treatment chances.

Definition: The Rosenbaum sensitivity model with parameter $\Gamma \ge 1$ requires that, within every matched pair, the odds of treatment for one member relative to the other are at most $\Gamma$. Equivalently, conditional on one treated unit per pair, $$ \frac{1}{\Gamma} \le \frac{p_i}{1-p_i} \le \Gamma, \qquad\text{or}\qquad \frac{1}{1+\Gamma} \le p_i \le \frac{\Gamma}{1+\Gamma}. $$ When $\Gamma=1$, both members have equal treatment odds, so $p_i=1/2$. Larger values allow greater hidden imbalance.

This restriction is about the assignment mechanism, not the outcome model. It says that, even after matching on observed covariates, an unobserved factor may make one member more likely to receive treatment, but by no more than the specified odds ratio. A common interpretation uses an unobserved covariate $u$ that changes treatment odds by a factor such as $\exp(\gamma u)$, where the possible range of $u$ implies a maximum odds ratio $\Gamma=\exp(\gamma)$.

For the calculations below, assume treatment assignments are independent across pairs conditional on the matched sets and the sensitivity model. Outcomes are treated as fixed for randomization inference. The resulting bounds are worst-case over all allowed pair-specific assignment probabilities; they are not a claim that every pair actually has the same hidden bias.

Worked Example: Interpreting a Gamma Value

Suppose a matched-pair analysis considers $\Gamma=2.5$. What range of probabilities for treatment of the first member is allowed, conditional on one treated unit in the pair?

The model gives $$ \frac{1}{1+2.5} \le p_i \le \frac{2.5}{1+2.5}. $$ Thus $0.2857 \le p_i \le 0.7143$. Within a pair, the member with the higher treatment chance may have treatment odds up to 2.5 times the other member’s odds. This does not mean the probability itself differs by a factor of 2.5: the restriction applies to odds.

Worst-Case Bounds for a Matched-Pair Sign Test

A simple matched-pair statistic counts how many non-tied pair differences favor treatment. Write $D_i$ for the treated-minus-control outcome difference in pair $i$, and define $S=\sum_{i=1}^n I(D_i>0)$, excluding pairs tied under the null being tested. For a one-sided alternative that treatment increases the outcome, large values of $S$ count against the null.

Under the sharp null of no treatment effect for any unit, each pair’s two outcomes are fixed regardless of assignment. If the assignment within a pair is reversed, the sign of its treated-minus-control difference reverses. Thus, for each pair, the indicator of a positive difference is a Bernoulli variable under the assignment distribution. Its success probability is at most $q_\Gamma=\Gamma/(1+\Gamma)$. Across pairs these indicators are independent under the stated assignment assumption.

Theorem: Under the sharp null and the pairwise Gamma model, the largest possible upper-tail probability for the sign statistic is $$ P_\Gamma(S \ge s_{\mathrm{obs}}) \le P\!\left\{\operatorname{Binomial}\!\left(n,\frac{\Gamma}{1+\Gamma}\right) \ge s_{\mathrm{obs}}\right\}. $$ This is a worst-case one-sided p-value bound. For a lower-tail test, use success probability $1/(1+\Gamma)$ and calculate the corresponding lower tail.

The bound follows from monotonicity. Let $U_i$ be independent uniform random variables on $[0,1]$, and represent a Bernoulli indicator with success probability $q_i$ as $I(U_i\le q_i)$. If $q_i\le q_\Gamma$, then $I(U_i\le q_i)\le I(U_i\le q_\Gamma)$ for every $i$. Therefore the sum at the allowed probabilities cannot have a larger upper tail than the sum when every probability equals $q_\Gamma$. This also explains why the worst case is attained at the boundary of the Gamma restriction.

For a two-sided sign test, one conservative convention is to calculate the worst-case upper-tail probability and the worst-case lower-tail probability separately, double the smaller, and cap the result at 1. That is a valid conservative two-sided bound, though it need not equal a uniquely defined exact two-sided randomization p-value for every statistic. The tail convention should be stated when reporting results.

Worked Example: A Worst-Case Sign-Test P-Value

In six hypothetical matched pairs, suppose the observed differences in outcome points are $5, 3, 2, 1, -1,$ and $4$. Five of the six differences favor treatment, so $s_{\mathrm{obs}}=5$. Under $\Gamma=2$, $q_\Gamma=2/3$, and the worst-case one-sided p-value is $$ P\{\operatorname{Binomial}(6,2/3)\ge 5\} = {6\choose5}(2/3)^5(1/3)+(2/3)^6 = \frac{256}{729} \approx 0.3512. $$

At $\Gamma=1$, the same calculation uses success probability $1/2$, giving $$ P\{\operatorname{Binomial}(6,1/2)\ge5\} = \frac{6+1}{64} = 0.1094. $$ The evidence is already insufficient to reject at the 5% level when $\Gamma=1$, and it is weaker under the $\Gamma=2$ worst case. The result does not show that hidden bias exists; it shows the test’s behavior under the specified assignment restrictions.

Other Test Statistics and Their Sensitivity Bounds

The same principle extends beyond the sign test. For a paired statistic that is a weighted sum of signed pair differences, such as a Wilcoxon signed-rank statistic, the outcomes and their weights are fixed under a sharp null. A one-sided statistic that increases when more favorable signs occur has a worst-case distribution obtained by choosing allowed assignment probabilities that make favorable signs as likely as possible. Its tail probability can be computed from the resulting weighted Bernoulli distribution.

For a general statistic, the sensitivity calculation may require enumerating or optimizing over assignment probabilities within the Gamma limits rather than using a binomial formula. The simple binomial bound applies directly to the unweighted sign count because all pairs contribute one unit. It should not be substituted for a correctly calculated bound for a differently weighted statistic.

The p-value at a chosen $\Gamma$ is a worst-case probability, under the null and the allowed assignment mechanisms, of a statistic at least as unfavorable to the null as the observed one. It is not the probability that the null is true, nor the probability that hidden bias of size $\Gamma$ exists. Reporting p-values across several $\Gamma$ values can show where the inferential conclusion becomes sensitive to hidden bias.

Treatment-Effect Inference by Test Inversion

A treatment-effect confidence bound can be obtained by testing candidate effects and inverting the tests. A particularly transparent version assumes a constant additive treatment effect: for every unit, treatment changes the outcome by the same amount $\tau$. To test a candidate value $\tau_0$, subtract $\tau_0$ from each treated outcome. Under the sharp null that the effect equals $\tau_0$ for every unit, the adjusted outcomes have no treatment effect, so the Gamma sign-test calculation applies to the adjusted pair differences $D_i-\tau_0$.

Rejecting candidate values below the data-supported effect at level $\alpha$ yields a one-sided $1-\alpha$ lower confidence bound by inversion. The bound protects against all assignment probabilities allowed by the specified $\Gamma$, subject to the matched-pair and constant-effect assumptions. It is not a bound on an average treatment effect under unrestricted effect heterogeneity: a constant-effect sharp null is stronger than a null about the average alone.

Worked Example: A Sensitivity Lower Bound for a Constant Effect

Suppose ten hypothetical matched pairs have treated-minus-control differences of $1,2,3,4,5,6,7,8,9,$ and $10$ outcome points. Their sample mean difference is $$ \bar{D}=\frac{1+2+3+4+5+6+7+8+9+10}{10} =\frac{55}{10}=5.5\text{ points}. $$ This is the observed matched mean difference; the following sign-test inference instead uses the constant-effect model. Consider a one-sided 5% test with $\Gamma=1.5$, so $q_\Gamma=1.5/2.5=0.6$.

At candidate effect $\tau_0=1.5$ points, nine adjusted differences are positive and one is negative. The worst-case one-sided p-value is $$ P\{\operatorname{Binomial}(10,0.6)\ge9\} = {10\choose9}(0.6)^9(0.4)+(0.6)^{10} \approx 0.04636. $$ The candidate is rejected at the 5% level.

At $\tau_0=2.5$ points, eight adjusted differences are positive and two are negative. The bound is $$ P\{\operatorname{Binomial}(10,0.6)\ge8\} = {10\choose8}(0.6)^8(0.4)^2 +{10\choose9}(0.6)^9(0.4)+(0.6)^{10} \approx 0.1673. $$ This candidate is not rejected. Under the convention that adjusted ties are omitted from the sign test, $\tau_0=2$ has one tie; its nine non-tied differences include eight positive signs and one negative sign, giving an upper-tail bound of $9(0.6)^8(0.4)+(0.6)^9\approx0.07054$. Thus the one-sided 95% lower confidence bound is 2 outcome points for this test-inversion procedure: candidate effects below 2 are rejected, while 2 is not. This is a sensitivity confidence bound under a constant effect, not a point estimate or a general bound for heterogeneous effects.

How to Report and Interpret a Sensitivity Analysis

A useful report states the matched design, outcome and units, test statistic, direction of the alternative, treatment-effect assumption if effects are tested, and the values of $\Gamma$ considered. It should distinguish the ordinary matched analysis at $\Gamma=1$ from the worst-case results at larger values. For example, a result that remains below the chosen significance level at a specified $\Gamma$ is robust to assignment odds imbalances up to that level under the model; it does not rule out larger hidden bias or violations of the model.

Gamma is a scale for hidden assignment imbalance, not a directly measured confounder strength. It is often informative to describe the implied probability range alongside the odds-ratio interpretation. Sensitivity findings are conditional on the matching and assignment assumptions and do not by themselves establish that the treatment caused the observed outcome differences.

Common Mistake

Wrong: “$\Gamma=2$ means treatment is twice as likely for one member of every pair, so the p-value is the ordinary randomization p-value multiplied by two.”
Correction: $\Gamma$ bounds the ratio of treatment odds within each pair. The sensitivity p-value comes from the worst-case distribution over allowed assignment probabilities; it is not obtained by multiplying an ordinary p-value by $\Gamma$.
Wrong: “A sensitivity confidence bound from the sign test is a bound for the average treatment effect, regardless of effect heterogeneity.”
Correction: The example’s test inversion relies on a sharp constant additive effect. Inference for an average effect under unrestricted heterogeneity requires additional methods and assumptions.

Key takeaway: Rosenbaum’s $\Gamma$ specifies how far treatment odds may differ within matched pairs because of hidden bias. Use the worst-case assignment distribution to bound test evidence, and interpret effect bounds only under the assumptions used to construct them.

Check Your Understanding

Answer these questions using the model and calculations above.

For $\Gamma=3$, what is the allowed range of $p_i$ within a matched pair, conditional on one treated unit?
Why does the worst-case upper-tail sign-test probability use success probability $\Gamma/(1+\Gamma)$?
In a one-sided test, what does a sensitivity p-value bound mean, and what does it not mean?
Why is the sign-test binomial bound not automatically the correct calculation for a weighted signed-rank statistic?
What additional effect assumption is used when inverting the sign test to obtain the treatment-effect lower bound in the example?