What a Rosenbaum Sensitivity Report Must Establish
A matched observational study can balance measured covariates without ensuring that treatment assignment is effectively random within matched sets. Rosenbaum sensitivity analysis asks how strongly an unmeasured factor would have to alter treatment assignment, within the specified model, before a test or interval changes its conclusion. A useful report therefore does more than give a single sensitivity number: it identifies the design and outcome statistic, states the values of \(\Gamma\) examined, and shows the worst-case inference at those values.
The parameter \(\Gamma\) describes a limit on how much treatment-assignment odds may differ between units in the same matched set, after conditioning on the observed matching variables. It is not an estimate of hidden bias, nor does it measure the association between an unmeasured factor and the outcome by itself. The sensitivity analysis asks what the inference would look like if hidden bias were as large as the stated bound.
A report should specify the target comparison and the null hypothesis, the matching structure (such as pair matching or matched sets of a stated size), how the sets were formed, and which units or sets were excluded. It should also identify the outcome, follow-up or measurement time, and the test statistic. For example, with pair-matched binary outcomes, a sign statistic can count the discordant pairs in which the treated member has the more favorable outcome. State whether the test is one-sided or two-sided and the direction in which a large statistic counts as evidence.
The interpretation relies on assumptions. The matched sets and observed covariates must be defined as described; the treatment-assignment odds within a set must satisfy the chosen \(\Gamma\) bound; and the test procedure must be valid for the matched design and its stated null. For the simple paired calculation below, pairs are treated as independent assignment units under the sensitivity model, and the sharp null says treatment changes no unit’s outcome. A sensitivity result does not repair poor matching, outcome mismeasurement, interference between units, or a wrongly specified analysis.
Worst-Case P-Values and a Short Derivation
For \(n\) independent matched pairs with binary outcomes, discard concordant pairs for a sign test because they do not distinguish treatment from control. Let \(T\) be the number of discordant pairs in which the treated member has the more favorable outcome. Under the sharp null and \(\Gamma=1\), each discordant pair favors treatment with probability one half. Under a bound \(\Gamma>1\), the probability can be as high as \(q=\Gamma/(1+\Gamma)\). To assess evidence in the favorable direction, the worst-case upper-tail p-value is therefore
This bound follows by considering the assignment probability in each discordant pair. Let \(p_i\) be the probability that the treated member is the one with the more favorable outcome, conditional on the observed pair and the sensitivity model. The odds bound implies \(p_i\leq q\). For an upper-tail event, increasing any \(p_i\) cannot decrease the probability of obtaining at least \(T_{\mathrm{obs}}\) favorable signs. Thus the largest tail probability occurs when every \(p_i=q\), giving the binomial expression when pair assignments are independent. For other statistics or matched-set structures, software may calculate a different worst-case bound; report the statistic and method actually used rather than assuming this formula applies.
A worst-case p-value is an upper bound on the p-value under the permitted assignment probabilities. If that bound is below the chosen significance level, the result remains statistically significant for every assignment mechanism allowed by that \(\Gamma\) model. If the bound is above the level, the sensitivity analysis does not establish that significance survives; it does not prove that hidden bias exists or that the null is true. For a two-sided analysis, state the two-sided test and the method used to obtain its worst-case p-value. Do not report a one-sided binomial tail as if it were two-sided.
Worked Example: Reporting the Tipping Point on a Gamma Grid
Suppose a hypothetical pair-matched study has 12 discordant pairs, with the treated member showing the favorable outcome in 10. A prespecified one-sided sign test assesses evidence of a favorable treatment effect. At significance level \(\alpha=0.05\), report the worst-case p-values at \(\Gamma=1\), \(1.25\), and \(1.50\).
At \(\Gamma=1\), \(q=1/(1+1)=0.5\). The upper-tail probability is
At \(\Gamma=1.25\), \(q=1.25/2.25=5/9\), and \(1-q=4/9\). Substitution gives
At \(\Gamma=1.50\), \(q=1.50/2.50=0.6\), so
The result is significant at \(\Gamma=1\) and \(\Gamma=1.25\), but not at \(\Gamma=1.50\), using the 0.05 threshold. A precise conclusion is: on the evaluated grid, the one-sided finding remains significant through \(\Gamma=1.25\), but not at \(\Gamma=1.50\). The calculations locate a change in conclusion between those grid points; they do not establish an exact continuous tipping value.
Choosing and Explaining the Gamma Grid
The grid should be chosen before interpreting the result and should include \(\Gamma=1\) as a reference, plus substantively meaningful larger values. A fine grid near a change in conclusion can help describe where the result becomes sensitive, but a long list of arbitrary increments can create a false impression of precision. Report the full grid, not only the most favorable value.
Calibration gives readers a way to judge whether the chosen values are small or large relative to observed features of the study. One approach is to compare the treatment-assignment odds associated with a strong measured prognostic covariate, conditional on the matching variables, with the odds differences permitted by candidate \(\Gamma\) values. The outcome prognostic strength of that covariate is also relevant when explaining why it is a useful benchmark. State how the comparison was made, including the covariate and model or summary used. A measured covariate is a reference point, not proof that all unmeasured factors are weaker.
Worked Example: Making Calibration Concrete
In a hypothetical matched study, investigators find that a strong measured prognostic factor is associated with a conditional treatment-assignment odds ratio of about 1.7 within the matched population. They evaluate \(\Gamma=1, 1.25, 1.50, 1.75,\) and \(2.00\), explaining that the values around 1.7 allow hidden assignment-odds differences roughly comparable to, and somewhat larger than, this measured benchmark.
For example, at \(\Gamma=1.70\), the pairwise probability bound is
Thus the sensitivity model allows the more treatment-prone member of a pair to have treatment probability as high as about 0.630, rather than 0.5 under \(\Gamma=1\). This is an explanation of the model’s assignment-odds bound, not a claim that the hidden factor has been measured or that its effect on the outcome matches the observed factor. A responsible report notes that calibration is a benchmark and that different hidden factors could have different relationships with treatment and outcome.
A compact table makes the sensitivity pattern auditable. Include the statistic and its observed value in the text or table notes, the test direction, and the significance threshold. Use enough digits to distinguish values near the threshold, and label every entry as a worst-case bound rather than an ordinary p-value when that is what the software returns.
| Gamma | Worst-case one-sided p-value | Conclusion at alpha = 0.05 |
|---|---|---|
| 1.00 | 0.01929 | Significant |
| 1.25 | 0.04571 | Significant |
| 1.50 | 0.08344 | Not significant |
Reporting Sensitivity Intervals
When the goal is estimation, a sensitivity analysis may report confidence intervals for a treatment-effect parameter over a grid of \(\Gamma\) values. A common construction in matched-pair analyses inverts tests of a constant additive effect: for each candidate effect \(\delta\), test the corresponding null, and retain values not rejected by the sensitivity-adjusted test. The resulting interval is conditional on the effect model, matched design, confidence level, and chosen \(\Gamma\). State the parameter and units, the interval construction, and whether the interval is a worst-case sensitivity interval.
Worked Example: Describing How an Interval Changes
Suppose a hypothetical matched analysis estimates a mean reduction in systolic blood pressure of 4 mmHg. Its 95% sensitivity intervals, obtained by inverting the stated matched-pair tests, are \([1,7]\) mmHg at \(\Gamma=1\), \([-1,10]\) mmHg at \(\Gamma=1.50\), and \([-4,13]\) mmHg at \(\Gamma=2.00\). The intervals are assumed here to be outputs of the specified analysis; they are not calculated from the paired sign-test example above.
At \(\Gamma=1\), the interval excludes 0 mmHg and is compatible with a reduction between 1 and 7 mmHg under this analysis. At \(\Gamma=1.50\), it includes 0 mmHg, so effects of no change are not ruled out at the 95% level under that sensitivity bound. The interval also includes increases as well as reductions. At \(\Gamma=2.00\), it is wider still. The correct report gives these values and their units; it does not say there is a 95% probability that the fixed effect lies in any one interval.
Intervals and p-values answer related but distinct questions. A p-value reports evidence against a specified null under a test; an interval shows which effect values remain compatible with the inversion procedure. If both are reported, their test, confidence level, and \(\Gamma\) assumptions should agree. An interval containing the null generally corresponds to failure to reject that null in the associated two-sided procedure, subject to the method’s conventions.
Diagnostics, Alternatives, and Common Reporting Errors
Before interpreting a sensitivity curve or table, check that the reported design matches the analysis: number and size of matched sets, treatment allocation within sets, outcome definition, and any exclusions. Confirm that the statistic and tail direction were selected for the scientific question, and that the reported \(\Gamma=1\) result agrees with the analysis’s no-hidden-bias reference. If software output is used, record the procedure and relevant settings, including whether the p-value is one- or two-sided and whether the interval is based on test inversion.
Rosenbaum bounds and E-values are alternative sensitivity summaries, not interchangeable scales. Rosenbaum analysis bounds within-set treatment-assignment odds and reports how worst-case inference changes under those bounds. E-values summarize the confounder strength needed, on a different association scale, to explain away an estimate under their assumptions. Neither establishes that unmeasured confounding is present or absent. A conventional matched analysis without a sensitivity bound addresses inference under its specified assignment assumptions; it does not by itself show robustness to departures from them.
A frequent mistake is to say, “The result is robust to unmeasured confounding because it is significant at \(\Gamma=1.25\).” That is too broad. The corrected statement identifies the exact test, outcome direction, and evaluated range: the worst-case one-sided p-value remains below 0.05 through \(\Gamma=1.25\), but exceeds 0.05 at \(\Gamma=1.50\). It does not claim robustness to every possible confounder or to values outside the examined grid.
- Do not treat \(\Gamma\) as a measured confounder effect. It bounds treatment-assignment odds within matched sets.
- Do not omit the matching design. A bound is hard to interpret without the set structure and outcome statistic.
- Do not hide the grid. Reporting only the largest \(\Gamma\) with a favorable conclusion conceals what happens nearby.
- Do not call non-significance evidence of no effect. It means the stated procedure does not reject the null at that bound and threshold.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a matched-pair sensitivity model, what does \(\Gamma=1\) mean, and what does a larger value permit?
- A prespecified one-sided worst-case p-value is 0.04 at \(\Gamma=1.2\) and 0.07 at \(\Gamma=1.5\). What conclusion is justified at \(\alpha=0.05\)?
- Why is a measured prognostic covariate useful for calibration, and why does it not validate the sensitivity model?
- A 95% sensitivity interval for a mean treatment effect in minutes is \([-2,5]\) at \(\Gamma=1.4\). What should be said about a zero effect?
Solutions. (1) At \(\Gamma=1\), treatment-assignment odds are equal within each matched set under the model; larger values permit greater within-set odds differences. (2) The finding remains significant at \(\Gamma=1.2\), but not at \(\Gamma=1.5\); on this grid the change occurs between those values, without identifying an exact tipping point. (3) Its conditional treatment-odds association provides a concrete observed benchmark, especially when its outcome prognostic strength is also described. It cannot show that all hidden factors are weaker or that the model captures every source of bias. (4) The interval includes 0 minutes, so the analysis does not rule out no effect at the 95% level under the stated \(\Gamma=1.4\) assumptions; it does not prove that the true effect is zero.