One Study, Two Hidden-Confounding Questions
Matching can improve comparability on measured covariates, but it cannot guarantee that treated and untreated people are comparable on factors that were not measured. Sensitivity analyses make this limitation more explicit by asking how strong residual bias would need to be to alter a conclusion. Two summaries often reported together are the E-value and the Rosenbaum sensitivity parameter \(\Gamma\). They are not two ways of measuring the same quantity: they use different models and address different questions.
The E-value asks how strongly an unmeasured confounder would need to be associated with treatment and outcome, conditional on measured covariates, to explain away a risk-ratio estimate. A matched-pair Rosenbaum analysis asks how much hidden bias in treatment assignment within matched pairs would be needed to make a test result no longer statistically significant under a specified null. The worked analysis below calculates both summaries for a clearly hypothetical study.
Worked Integrated Analysis
Consider a hypothetical matched observational study of a treatment and a binary health outcome during follow-up. Suppose there are 36 matched pairs. The table records whether the outcome occurred for each member of a pair.
| Outcome pattern within pair | Number of pairs |
|---|---|
| Outcome for both treated and untreated members | 6 |
| Outcome for treated member only | 18 |
| Outcome for untreated member only | 6 |
| Outcome for neither member | 6 |
Worked Example: E-Value and Rosenbaum Tipping Point
Step 1: Calculate the matched-cohort risk ratio. There are \(6+18=24\) outcome events among treated members and \(6+6=12\) among untreated members. Each group has 36 people, so the observed risks are \(24/36=0.6667\) and \(12/36=0.3333\), respectively. Thus
The treated group’s observed risk is twice the untreated group’s risk in this hypothetical matched cohort. That is an adjusted observational association; the ratio alone does not establish a causal effect.
Step 2: Obtain a confidence interval. For illustration, use a large-sample log-risk-ratio interval with a variance that accounts for the paired outcomes. Let \(Y_T\) and \(Y_C\) indicate an outcome for the treated and control members of a pair. Their sample proportions are \(p_T=24/36=2/3\) and \(p_C=12/36=1/3\). The estimated within-pair covariance is \(6/36-p_Tp_C=1/6-2/9=-1/18\). The estimated variance of the log risk ratio is
Therefore \(SE(\log RR)=\sqrt{0.08333}=0.2887\). The 95% Wald interval is
This approximate interval is above 1, though its width reflects substantial uncertainty. Its usual repeated-sampling interpretation is that the interval procedure has 95% coverage under its assumptions; it is not a 95% probability statement about this particular interval containing the true risk ratio.
Step 3: Calculate the E-values. For the point estimate,
For the confidence interval, use its limit nearest the null, 1.14. Using that displayed, rounded limit,
In the E-value framework, an unmeasured confounder would need associations of at least about 3.41-fold with both treatment and outcome, conditional on measured covariates, to explain away the point estimate under the framework’s assumptions. Associations of about 1.54-fold would suffice, under the same interpretation, to move the confidence limit nearest the null to 1. The confidence-limit E-value is smaller because the interval already includes values closer to the null than the point estimate.
Step 4: Calculate a matched-pair Rosenbaum bound. The outcome is discordant in 24 pairs: 18 have an outcome for the treated member only, and 6 have an outcome for the untreated member only. Under the sharp null of no treatment effect for any member, and with equal treatment odds within each pair (\(\Gamma=1\)), the number of discordant pairs favoring the treated member has a \(\mathrm{Binomial}(24,0.5)\) reference distribution. The one-sided tail probability for 18 or more is
For \(\Gamma>1\), a conservative one-sided bound uses the largest allowed probability of treatment assignment favoring the observed direction, \(q_\Gamma=\Gamma/(1+\Gamma)\). Under the independent-pair sensitivity model, the bound is \(P\{\mathrm{Binomial}(24,q_\Gamma)\geq18\}\). At \(\Gamma=1.30\), \(q_\Gamma=1.30/2.30\approx0.5652\), giving a tail probability of approximately \(0.04991\). The p-value reaches the conventional 0.05 threshold at a \(\Gamma\) of about 1.30. At \(\Gamma=1.50\), \(q_\Gamma=1.50/2.50=0.60\), and the corresponding tail probability is approximately \(0.0960\).
The Rosenbaum tipping point is therefore about 1.30 for this test: a hidden factor permitting treatment odds to differ by roughly 30% within a matched pair is enough for the conservative upper-bound p-value to reach 0.05. It is a tipping point for this test under its assumptions, not an estimate of how much confounding actually exists.
Why the Two Results Are Not Interchangeable
The E-value and \(\Gamma\) have different scales, assumptions, and targets. The E-value is tied to an effect estimate on the risk-ratio scale. It describes the minimum joint association strengths with treatment and outcome needed, under its bias framework, to move an estimate or confidence limit to the null. The Rosenbaum analysis bounds the odds of treatment assignment within a matched set and assesses how that bound changes inference under a sharp null. It produces a sensitivity analysis for a test statistic, not a confounder-to-outcome association strength.
| Summary | What is varied? | Question answered | What it does not establish |
|---|---|---|---|
| E-value | Strength of unmeasured confounder associations with treatment and outcome, on risk-ratio scales | How strong would confounding need to be to explain away the estimate or move a confidence limit to the null? | Whether such a confounder exists, or whether the estimate is causal |
| Rosenbaum \(\Gamma\) bound | Permitted treatment-odds imbalance within matched pairs from hidden factors | How much hidden assignment bias would make the test’s worst-case p-value cross a chosen threshold? | The size of the causal effect or the actual amount of hidden bias |
The two numbers should not be compared numerically. For example, \(\Gamma=1.30\) is not “weaker” than an E-value of 3.414 in a direct, common-unit sense. They refer to different associations and calculations. Their joint message in this example is more useful: the estimated risk ratio has a point-estimate E-value of 3.414, while the matched-pair test’s statistical significance is sensitive to a modest amount of hidden treatment-odds imbalance. One concerns moving an effect estimate to the null; the other concerns loss of a test’s significance under a worst-case assignment model.
Assumptions, Diagnostics, and Failure Modes
Neither sensitivity calculation repairs a poorly specified primary analysis. The matched-cohort risk ratio must be a suitable estimate for the target population and outcome period. Interpretation as a causal effect also depends on standard causal conditions, including consistency, positivity, and no residual confounding after accounting for measured covariates. The E-value does not verify these conditions, nor does it account automatically for selection bias, measurement error, or model misspecification.
The paired Rosenbaum calculation above is specific to matched pairs with a binary outcome, a sharp-null test, and independent pair-level assignment mechanisms under the sensitivity model. The binomial tail calculation uses only the discordant pairs. More complex matched sets, different test statistics, or other outcomes require corresponding bounds rather than this exact sign-test calculation. The choice of a one-sided alternative must also be justified by the scientific question; changing the direction after seeing the results invalidates the stated interpretation.
Before presenting either result, check that matching achieved reasonable balance on measured covariates and that the matched sample has adequate overlap. Examine the pair structure, missingness, and outcome coding. For the paired analysis, confirm the discordant counts and which direction is defined as favoring treatment. For the E-value, state the effect measure, point estimate, confidence interval, and which confidence limit was used. A small p-value at \(\Gamma=1\) does not imply robustness to substantial hidden bias; a large E-value does not show that a confounder with the required associations is implausible. Calibration against measured covariates can help judge scale, but does not turn either sensitivity summary into proof.
Worked Example: Reading an E-Value for a Smaller Risk Ratio
A separate hypothetical matched-cohort analysis reports a risk ratio of 1.50 with a 95% confidence interval of 1.05 to 2.10. Its point-estimate E-value is
The confidence limit nearest 1 is 1.05, so its E-value is
The point estimate and interval answer distinct questions: the point estimate would require stronger confounding to explain away than the confidence limit nearest the null would require to move that limit to 1. Reporting only 2.366 would conceal how little confounding strength is needed to make the interval compatible with the null.
Worked Example: A Rosenbaum Bound Beyond the Tipping Point
Return to the 24 discordant pairs in the integrated example. At \(\Gamma=1.50\), the maximum assignment probability in the observed direction is
Using the conservative binomial tail for 18 or more treatment-favoring pairs gives
Because \(0.0960>0.05\), the worst-case bound no longer rejects the sharp null at the 5% level at this \(\Gamma\). This does not mean that the treatment has no effect; it means that the test result is not significant under this level of hidden-bias allowance and the specified model.
Common Mistake: Treating a Tipping Point as a Confounder Estimate
A frequent error is to write, “The Rosenbaum analysis found that the hidden confounder changed treatment odds by 30%.” That reverses the meaning of the calculation. The analysis did not observe or estimate a hidden variable. It found that a sensitivity model allowing treatment odds to differ by about 30% within pairs is enough to bring the conservative p-value to the chosen threshold. Likewise, an E-value does not assert that a confounder has associations of 3.41 with both treatment and outcome. It describes a strength threshold within a specified framework.
The appropriate conclusion is conditional and limited: describe the sensitivity model, the value at which the result changes, and the effect measure or test being assessed. Then discuss whether that degree and pattern of bias seem plausible using substantive knowledge and measured-confounder benchmarks. Do not declare an analysis “robust” or “not robust” without saying what robustness means for the particular result.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A risk ratio is 2.00. Calculate its point-estimate E-value and state what it describes.
- For 24 discordant matched pairs, 18 favor treatment. At \(\Gamma=1.30\), what is \(q_\Gamma\), and what does a one-sided bound of 0.0499 imply at \(\alpha=0.05\)?
- Why is an E-value of 3.414 not directly comparable with a Rosenbaum tipping point of \(\Gamma=1.30\)?
- A 95% confidence interval for a risk ratio is 0.90 to 2.10. Which limit is used for the confidence-interval E-value, and can the interval be described as excluding the null?
Solutions. (1) \(2+\sqrt{2(2-1)}=3.414\). Under the E-value framework, that is the minimum equal strength of association with treatment and outcome, conditional on measured covariates, needed to explain away the point estimate. (2) \(q_{1.30}=1.30/2.30\approx0.5652\). Since \(0.0499<0.05\), the upper-bound test is just below the chosen threshold at that value; the tipping point is approximately 1.30. (3) The E-value concerns treatment and outcome associations on risk-ratio scales, whereas \(\Gamma\) bounds treatment-assignment odds within matched pairs and is used for a test’s worst-case inference. Their units and targets differ. (4) Use the limit nearest 1, which is 0.90. The interval includes 1, so it does not exclude the null; the usual confidence-limit E-value interpretation for an interval wholly above 1 does not apply as stated.