Reporting Sensitivity Analyses for Unmeasured Confounding

Why Report Sensitivity to Unmeasured Confounding?

Adjustment, matching, or weighting can address confounding measured in the available covariates, but cannot directly remove bias from a confounder that was not measured. A sensitivity analysis makes this limitation more concrete: it specifies how strongly an unmeasured factor would need to relate to treatment and outcome, or how much bias a set of assumed parameters would produce, to change the result. Its purpose is not to certify that the result is unbiased. It is to show readers how conclusions depend on explicit, interpretable assumptions.

A useful report starts from the effect estimate already selected for the study, with its scale, target population, and uncertainty stated. It then names the sensitivity method, describes its assumptions, gives the parameters and values examined, and reports the corresponding estimate, interval, or test bound. The report should also say what changes across the analysis: for example, whether the estimated effect approaches the null, whether an interval crosses the null, or whether a test no longer rejects a sharp null hypothesis.

What a Transparent Report Needs to State

A number labeled “sensitivity analysis” is not enough to reproduce or interpret an analysis. State the method by its standard name, the estimator or estimand to which it applies, and the assumptions that connect its sensitivity parameters to bias. Report the parameter values examined and why they are considered plausible, rather than presenting only the most favorable value. When the analysis produces a bound, identify it as a bound and say whether it is a bound on an estimate, a p-value, or another quantity.

For each parameter setting, report the resulting effect estimate and uncertainty measure when available. If the procedure instead gives a sensitivity bound on a test, report the bound and the null hypothesis being tested. State the tipping point—the parameter value at which a specified conclusion changes—only with a clear definition of “changes.” For example, “the confidence interval first includes the null” and “the point estimate reaches the null” are different tipping points.

The introductory material on communicating uncertainty and statistical inference distinguishes an estimate from its uncertainty. The same distinction matters here: sensitivity analysis changes the assumptions used to assess bias, while sampling uncertainty remains a separate issue. A sensitivity result does not automatically incorporate every source of uncertainty in the data, nor does a stable result prove that unmeasured confounding is absent.

Quantitative Bias Analysis: Make the Assumptions Visible

One direct approach is quantitative bias analysis (QBA): specify a model for an unmeasured confounder and calculate the bias it would induce under chosen parameter values. The following simple risk-difference version assumes a binary unmeasured covariate \(U\), an additive outcome-risk model with no treatment-by-\(U\) interaction, and no residual confounding other than \(U\). Let \(p_1=P(U=1\mid A=1)\) and \(p_0=P(U=1\mid A=0)\), where \(A\) indicates treatment. Let \(\delta\) be the increase in outcome risk associated with \(U=1\) rather than \(U=0\), at either treatment level. Under these assumptions, the confounding bias in the unadjusted risk difference is \(\delta(p_1-p_0)\). The formula is not a general correction for every outcome model or estimand; it is tied to these assumptions.

The justification is direct. If the risk model contains an additive treatment effect \(\tau\), a term \(\delta U\), and an intercept, the average risk difference between treatment groups is \(\tau+\delta E(U\mid A=1)-\delta E(U\mid A=0)\). Since \(U\) is binary, those conditional means are \(p_1\) and \(p_0\), giving \(\tau+\delta(p_1-p_0)\). Thus the correction subtracts the modeled confounding contribution. A report should explain how the values for \(p_1,p_0,\delta\) were chosen and check that the assumed risk model is sensible for the outcome.

A single parameter setting can conceal how results vary. In this example, hold the assumed prevalence difference at \(p_1-p_0=0.50\) and vary \(\delta\). The table displays the correction to the 10-percentage-point observed risk difference. Each row is a hypothetical scenario; the values are not estimated from the observed treatment and outcome data.

Under these selected assumptions, the adjusted risk difference remains positive but shrinks as the assumed confounder-outcome association grows. That statement is limited to the point estimates shown: without an uncertainty analysis, it says nothing about whether an interval includes zero. A fuller QBA can examine a grid of both prevalence and outcome-risk parameters, and may propagate uncertainty in those parameters. Reports should state whether parameter uncertainty was included or whether each row is simply a fixed scenario.

E-Values for Risk-Ratio Results

An E-value is a summary for a risk-ratio estimate. For a risk ratio \(RR\) greater than 1, its value is \(RR+\sqrt{RR(RR-1)}\). It describes the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need to have with both treatment and outcome—conditional on the measured covariates—to explain away the observed association, under the E-value framework. More precisely, the stated strength is a lower-bound summary when the two confounder associations are allowed to differ; equal-strength associations at least as strong as the E-value are sufficient in the corresponding worst-case formulation. It does not say that a confounder with that strength exists, and it is not a probability that the result is correct.

For an estimate whose confidence interval lies entirely above the null value 1, an E-value can also be calculated for the confidence limit closest to 1. This helps describe how strong unmeasured confounding would need to be to move that confidence limit to the null. For a risk ratio below 1, first express the effect in the opposite direction using its reciprocal. Interpretation requires attention to the effect measure and framework; this formula should not be transferred automatically to a risk difference or odds ratio without the relevant method.

Matched-Pair Sensitivity Analysis and \(\Gamma\)

For matched observational data, Rosenbaum sensitivity analysis is one way to assess how hidden bias in treatment assignment within matched sets could affect a test. In a matched pair, let the two individuals have the same measured covariates, but allow an unmeasured covariate to make one individual more likely to receive treatment. The sensitivity parameter \(\Gamma\geq1\) bounds the ratio of their treatment-assignment odds. When \(\Gamma=1\), the model assumes equal assignment odds within each pair, conditional on the measured covariates. Larger values allow greater hidden bias. This is an assignment mechanism assumption, not a direct estimate of a confounder’s prevalence or outcome effect.

Under a sharp null hypothesis of no treatment effect for any individual, a sensitivity analysis can bound the p-value over assignment mechanisms that satisfy the \(\Gamma\) restriction. Consider the simple special case of \(m\) discordant pairs, all favoring treatment, with the upper-tail sign test. The worst-case probability of a treatment-favoring discordance for each pair is \(q=\Gamma/(1+\Gamma)\). If the pairs are independent under the assignment model, the upper-tail probability of all \(m\) discordances is \(q^m\). This gives a transparent sensitivity calculation, but more general matched-set analyses can have different bounds and test statistics; the method and test must therefore be named.

Diagnostics, Comparisons, and Limitations

The methods answer different questions and should not be treated as interchangeable. QBA returns adjusted estimates under specified confounder parameters and a chosen outcome model. An E-value gives a compact strength-of-association benchmark for a risk-ratio result; it does not produce a corrected estimate. Rosenbaum analysis bounds inference over a specified range of hidden treatment-assignment bias in matched sets. A report should select the method that fits its estimator and design, rather than quote whichever sensitivity statistic is most convenient.

Before interpreting a result, check that the reported parameter values align with the method’s scale and assumptions. For QBA, check whether the assumed confounder prevalence and outcome association are credible, whether the model permits interactions, and whether implied risks are within 0 and 1. For an E-value, check that the estimate and confidence limit are risk ratios and that the confidence limit used is the one nearest the null. For Rosenbaum analysis, report the matched-set structure, \(\Gamma\) values, test direction, null, and the bound being reported. A sensitivity value without these details is difficult to evaluate.

Common failures include reporting only the sensitivity value that preserves a preferred conclusion; presenting a corrected estimate without its assumptions; treating an E-value as a probability or a guarantee; and describing a Rosenbaum \(\Gamma\) as the measured strength of a particular confounder. The correction is to show a range of substantively defensible parameter values, label estimates and bounds accurately, and explain what changes across that range. If the result changes under modest, plausible assumptions, say so. If it does not change under the examined assumptions, describe that limited robustness without claiming that unmeasured confounding has been ruled out.