Why Matched Results Need a Hidden-Bias Check
Matching can make treated and comparison individuals similar on measured covariates, but it cannot guarantee similarity on factors that were not recorded or were measured poorly. Rosenbaum sensitivity analysis asks a focused question: how much could an unmeasured factor change the treatment-assignment odds within a matched pair before the evidence against no treatment effect is no longer compelling?
The method does not estimate the size of hidden confounding or remove it. Instead, it calculates worst-case bounds on a test result under progressively stronger assumptions about how much hidden bias could influence treatment assignment. The analysis is therefore about the robustness of an inferential conclusion, not a substitute for the matched effect estimate.
For binary outcomes, a useful matched-pair test focuses on discordant pairs: pairs in which one member has the favorable outcome and the other does not. Tied pairs provide no direction for this particular test. Let \(m\) be the number of discordant pairs and \(X\) the number in which the treated member has the favorable outcome. Under no treatment effect and equal assignment odds within each pair, either member of a discordant pair is equally likely to be treated. Thus \(X\) has a binomial distribution with \(m\) trials and success probability \(1/2\).
The Sensitivity Model and Its Bound
Within pair \(i\), let \(\pi_i\) denote the probability that the first member receives treatment, conditional on exactly one member in the pair being treated. Rosenbaum’s sensitivity model requires that the odds of treatment for the two members cannot differ by more than a factor \(\Gamma\). Equivalently, the conditional probability that the member with the favorable outcome is treated is no greater than \(q_\Gamma=\Gamma/(1+\Gamma)\). This upper limit is \(1/2\) at \(\Gamma=1\), and increases as the allowed hidden bias increases.
The inference also relies on the null hypothesis of no treatment effect for any individual in the matched pairs, so that the outcomes would not change if treatment labels were exchanged. This is a sharp null for the test, not merely a statement that an average effect is zero. The pairs must be correctly formed, treatment assignment within pairs must be represented by the sensitivity model, and pairs are treated as independent for the binomial bound. For the binary-outcome sign test, the number of discordant pairs is held fixed under this sharp null.
The justification is a stochastic-domination argument. Under the sharp null, each discordant pair contributes an indicator \(I_i\) for whether the favorable-outcome member received treatment. Its probability \(p_i\) is at most \(q_\Gamma\). Represent these independent indicators using independent uniform random variables \(U_i\) on \([0,1]\): \(I_i=1\) when \(U_i\leq p_i\). Define \(J_i=1\) when \(U_i\leq q_\Gamma\). Since \(p_i\leq q_\Gamma\), \(I_i\leq J_i\) for every pair. Therefore \(\sum I_i\leq\sum J_i\), and \(\sum J_i\) has a binomial distribution with parameters \(m\) and \(q_\Gamma\). Consequently, the binomial upper tail bounds the probability of observing at least \(x\) favorable discordances under any assignment pattern allowed by the model.
For a two-sided assessment, a commonly used conservative bound is twice the one-sided bound in the observed direction, capped at 1. This doubling is a simple conservative convention, not the only possible construction of a two-sided sensitivity test. The convention should be stated when reporting results.
Worked Example: Nine Discordant Pairs and a Hidden-Bias Grid
Suppose a hypothetical matched study has \(m=9\) discordant pairs, of which \(x=8\) favor treatment. Calculate the one-sided and conservative two-sided bounds at several \(\Gamma\) values.
At \(\Gamma=1\), \(q_\Gamma=1/(1+1)=0.5\). The one-sided bound is \(P\{\operatorname{Binomial}(9,0.5)\geq8\}\). There are \(\binom{9}{8}=9\) outcomes with exactly eight successes and one with nine, out of \(2^9=512\) equally likely outcomes. Thus the one-sided bound is \((9+1)/512=0.01953125\), and the conservative two-sided bound is \(2(0.01953125)=0.0390625\).
At \(\Gamma=1.5\), \(q_\Gamma=1.5/2.5=0.6\). The upper tail is \(9(0.6)^8(0.4)+(0.6)^9\). Substitution gives \(9(0.01679616)(0.4)+0.010077696=0.060466176+0.010077696=0.070543872\). The conservative two-sided bound is \(2(0.070543872)=0.141087744\), approximately \(0.1411\).
The same calculation can be repeated at other sensitivity levels. For \(m=9\) and \(x=8\), the tail simplifies to \(q_\Gamma^8(9-8q_\Gamma)\).
| \(\Gamma\) | \(q_\Gamma\) | One-sided upper bound | Conservative two-sided bound | Conclusion at \(\alpha=0.05\) |
|---|---|---|---|---|
| 1.00 | 0.500 | 0.01953 | 0.03906 | Significant |
| 1.25 | 0.5556 | 0.04134 | 0.08268 | Not significant |
| 1.50 | 0.600 | 0.07054 | 0.1411 | Not significant |
| 2.00 | 0.6667 | 0.1431 | 0.2861 | Not significant |
For example, at \(\Gamma=2\), the one-sided bound is \((2/3)^8(9-8(2/3))=(256/6561)(11/3)=2816/19683\approx0.1431\). Under the stated conservative two-sided convention, the corresponding value is approximately \(0.2861\). The conclusion changes between \(\Gamma=1\) and \(\Gamma=1.25\): even a modest permitted departure from equal treatment odds is enough for this two-sided result to exceed 0.05.
Interpreting the Sensitivity Analysis
The sensitivity parameter is a hypothetical bound on relative treatment odds, not a measured property of the study. Reporting results over a range of \(\Gamma\) values makes the dependence of the conclusion visible. A result that remains below the chosen significance level for larger values is more robust to the particular hidden-bias model than one that crosses the threshold near 1. The analysis does not establish that hidden bias of any particular magnitude exists or does not exist.
Worked Example: A Result That Is Not Significant at \(\Gamma=1\)
In a second hypothetical study, \(m=10\) pairs are discordant and \(x=6\) favor treatment. First use \(\Gamma=1\), so each discordant pair has probability \(0.5\) of favoring treatment under the sharp null. The one-sided exact probability is
The conservative two-sided value is \(2(0.376953125)=0.75390625\), approximately \(0.7539\). This is not statistically significant at \(\alpha=0.05\) even under equal assignment odds. Therefore, increasing \(\Gamma\) cannot turn this particular upper-tail test into evidence against the null: its upper bound only becomes more conservative. For illustration, at \(\Gamma=1.5\), \(q_\Gamma=0.6\) and the one-sided bound is
The two-sided conservative value is capped at 1: \(\min\{1,2(0.633103258)\}=1\). Thus this example gives no evidence of a treatment effect under either assignment-odds assumption.
Worked Example: Reporting a Tipping Range
Return to the study with eight favorable outcomes among nine discordant pairs. At \(\Gamma=1\), the conservative two-sided bound is \(0.03906\), below \(0.05\); at \(\Gamma=1.25\), it is approximately \(0.08268\), above \(0.05\). The sensitivity threshold at which the bound crosses \(0.05\) must therefore lie between 1 and 1.25. Without solving for a more precise root, this interval is already an informative tipping range.
A careful report would say: “Under the specified conservative two-sided sign-test bound, the result is below 0.05 at \(\Gamma=1\) but above 0.05 by \(\Gamma=1.25\). Thus the statistical conclusion is sensitive to allowing a treatment-odds ratio difference somewhere in that range.” It would not claim that a hidden confounder of a particular size has been found. Nor does this calculation change the observed matched effect estimate; it describes how the test’s worst-case null probability changes.
Diagnostics, Alternatives, and Limitations
Before interpreting the bounds, check the matched analysis itself. Confirm that pairs contain one treated and one comparison individual, that the outcome’s favorable direction was specified consistently, and that \(m\) and \(x\) were counted from the same outcome definition. Examine measured-covariate balance and overlap: severe imbalance or poor overlap is a warning about the observed design, not something the sensitivity parameter repairs. A small number of discordant pairs also means the test may have limited resolution and power.
The standard paired test corresponds to \(\Gamma=1\); it is the natural reference point, not a guarantee that hidden bias is absent. Regression adjustment or propensity-score weighting can use measured covariates differently, but those methods also depend on assumptions about measured and unmeasured confounding. Rosenbaum bounds instead report how a matched-pair test behaves over a stated range of hidden treatment-odds differences. They do not by themselves provide a confidence interval for an effect adjusted for hidden bias, and the binary discordance calculation here should not be applied unchanged to continuous outcomes.
- Do not read \(\Gamma\) as a probability. \(\Gamma=1.5\) is a bound on a ratio of treatment odds, not a 50% probability of confounding.
- Do not treat a large \(\Gamma\) as proof of robustness. The meaningful range depends on subject-matter judgment about plausible unmeasured factors.
- Do not report only one arbitrary sensitivity value. A grid and, where useful, a tipping range reveal how the conclusion changes.
- Do not say that a nonsignificant bound proves no effect. It means the data do not reject the sharp null at the stated level under that bound and testing convention.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a matched-pair analysis, what does \(\Gamma=1\) mean in the sensitivity model?
- For \(m=9\), \(x=8\), and \(\Gamma=1.5\), calculate the conservative two-sided bound from the one-sided tail.
- For 10 discordant pairs with 6 favoring treatment, what is the one-sided exact p-value bound at \(\Gamma=1\)?
- Why does the binomial upper-tail calculation use \(q_\Gamma=\Gamma/(1+\Gamma)\) rather than \(1/2\) when \(\Gamma>1\)?
- Does a sensitivity analysis estimate the amount of unmeasured confounding or adjust the matched effect estimate?
Solutions. (1) The two members of each pair have equal treatment odds under the model. (2) The one-sided value is \(9(0.6)^8(0.4)+(0.6)^9=0.070543872\); doubling gives \(0.141087744\), approximately \(0.1411\). (3) The tail is \((210+120+45+10+1)/1024=386/1024=0.376953125\), approximately \(0.3770\). (4) The model permits the favorable-outcome member to have treatment probability as high as \(\Gamma/(1+\Gamma)\), so this gives the worst-case upper-tail probability. (5) No. It bounds inference under specified hidden-bias assumptions; it neither measures hidden confounding nor changes the observed matched effect estimate.