Common Interpretation Errors in Confounding Sensitivity Analyses

Why Interpretation Matters

A sensitivity analysis asks how conclusions would change under specified departures from the assumptions used to identify or estimate a causal effect. In analyses of unmeasured confounding, those departures are often summarized by quantities such as a confounder’s prevalence imbalance, its outcome association, a treatment-assignment odds ratio, or a robustness value. These quantities can make an analysis easier to communicate, but only if their meanings and assumptions remain attached to them.

The central interpretive discipline is to distinguish what the data estimate from what the sensitivity analysis assumes or varies. A sensitivity analysis does not directly reveal the strength of an unmeasured confounder merely by producing a number. Nor does a large robustness value establish that no such confounder exists. It describes how demanding a specified model-based explanation would have to be, according to a particular metric and benchmark.

Definition: A sensitivity parameter is a quantity that indexes an assumption or hypothetical departure from the identifying assumptions of an analysis, such as the imbalance in an unmeasured confounder between treatment groups. A sensitivity analysis evaluates how an estimate or conclusion changes across specified values of such parameters. Unless the parameter is measured or otherwise identified from data, its value is not an empirical finding about the study.

Assumed Parameters Are Not Observed Facts

Consider a simple additive model with a binary unmeasured confounder \(U\), a binary treatment \(A\), and a continuous outcome \(Y\). Suppose the difference in outcome associated with \(U=1\) rather than \(U=0\), conditional on treatment and measured covariates, is a constant \(\gamma\). Let \(p_1=P(U=1\mid A=1)\) and \(p_0=P(U=1\mid A=0)\), within the analysis population and after any stated adjustment. Under this model, the confounding component of the unadjusted treated-minus-untreated outcome contrast is \((p_1-p_0)\gamma\). This calculation relies on the additive, constant-association model; it is not a general formula for every outcome scale or confounding structure.

\text{confounding component}=(p_1-p_0)\gamma

Both \(p_1-p_0\) and \(\gamma\) must be supplied, estimated from suitable data, or examined as hypothetical values. If \(U\) was not measured, choosing values for these quantities does not turn them into observed characteristics of the study. The result is conditional: if the prevalence difference and outcome association were as specified, and the model’s other assumptions held, then the implied bias would be as calculated.

Worked Example: Interpreting a Hypothetical Binary-Confounder Scenario

Suppose a hypothetical observational analysis compares a treatment and control group on a continuous outcome measured in days. Its adjusted contrast is 2.0 days. To examine an unmeasured binary confounder, an analyst posits that its prevalence is 0.50 in the treatment group and 0.38 in the control group, and that its conditional outcome association is 5 days under the additive model above. These are scenario inputs, not measurements in this example.

The assumed prevalence difference is \(p_1-p_0=0.50-0.38=0.12\). The implied confounding component is therefore

(0.12)(5\ \text{days})=0.60\ \text{days}.

If the assumed confounding acts in the same direction as the observed contrast, subtracting this component gives \(2.0-0.60=1.4\) days as the contrast corrected under this scenario. If the assumed direction is opposite, the adjustment would go the other way. The arithmetic does not establish that the confounder has a 12-percentage-point prevalence difference or a 5-day outcome association. It shows the consequence of those inputs under the stated model, including its additive and constant-association assumptions.

A common reporting error is to write, “The unmeasured confounder is associated with a 5-day outcome difference,” when 5 days was only selected for a sensitivity scenario. A more accurate statement is, “Under a hypothetical 5-day conditional outcome association and a 0.12 prevalence difference, the model implies 0.60 days of bias.” The second sentence preserves the distinction between data and assumptions.

Robustness Values: A Threshold, Not a Probability

A robustness value summarizes the strength of an unmeasured confounder that would be required, under a specified model and metric, to change a target conclusion. One common version is the robustness value for reducing a linear-regression treatment coefficient to zero. In the partial-\(R^2\) formulation for an omitted variable \(Z\), the relevant associations have different conditioning sets: treatment association is measured conditional on the included covariates \(X\), while outcome association is measured conditional on both treatment \(D\) and the included covariates \(X\).

Definition: In the linear-regression partial-\(R^2\) sensitivity framework, the robustness value for reducing an estimated treatment coefficient to zero is the minimum equal value \(r\) of the two partial \(R^2\) quantities, over the equal-strength benchmark, needed for an omitted confounder to explain away the estimate. The treatment partial \(R^2\) is \(R^2_{D\sim Z\mid X}\); the outcome partial \(R^2\) is \(R^2_{Y\sim Z\mid D,X}\). Thus the outcome association is conditional on treatment as well as the included covariates. The robustness value is a threshold in this metric, not a probability that confounding exists.

Here \(D\) denotes treatment, \(Y\) the outcome, \(X\) the included covariates, and \(Z\) a hypothetical omitted confounder. A partial \(R^2\) describes the proportion of residual variation accounted for by a variable after the stated conditioning. Under the conventional linear-regression sensitivity calculation, the absolute omitted-variable bias, expressed in units of the coefficient’s standard error, is

\frac{|\text{bias}|}{SE(\hat{\tau})} = \sqrt{ df\, \frac{ R^2_{Y\sim Z\mid D,X}\,R^2_{D\sim Z\mid X} }{ 1-R^2_{D\sim Z\mid X} } }.

This expression makes the conditioning sets consequential, not cosmetic. Replacing \(R^2_{Y\sim Z\mid D,X}\) with an outcome association conditional only on \(X\) would describe a different quantity and would not justify the same robustness-value interpretation. The model, outcome scale, included covariates, and target conclusion must therefore accompany a reported value.

Worked Example: Reading an Equal-Strength Robustness Value

Suppose a hypothetical linear regression estimates a treatment coefficient of 0.3983 outcome units per treatment unit, with standard error 0.1300 and \(df=100\). The test statistic is \(t=0.3983/0.1300\approx3.064\). For this illustration, use the equal-strength benchmark: set each of the two partial \(R^2\) values to \(q\), where \(q\) is an \(R^2\), not a correlation.

To reduce the coefficient to zero, the confounding bias must be as large as the observed coefficient. In standard-error units, this requires the bias expression to equal \(|t|\). Substituting the equal-strength values gives

|t|=\sqrt{df\,\frac{q^2}{1-q}}.

Squaring and rearranging yields \(dfq^2+t^2q-t^2=0\). With \(df=100\) and \(t=3.064\), the positive root is approximately \(q=0.263\). Accordingly, under this model and equal-strength benchmark, an omitted confounder would need partial \(R^2\) associations of about 26.3% with treatment conditional on the included covariates, and with outcome conditional on treatment and the included covariates, to reduce the estimated coefficient to zero.

This is not a statement that there is a 26.3% probability of confounding, or that the actual confounder has those associations. It also does not mean the confounder explains 26.3% of the outcome’s total variation: the outcome quantity is a partial \(R^2\) conditional on treatment and \(X\). The result is a model-specific threshold for the specified target, scale, covariates, and equal-strength comparison.

Different Robustness Targets Answer Different Questions

“Robustness” is incomplete unless the target conclusion is named. A threshold for reducing an estimate to zero is not automatically a threshold for making a confidence interval include zero, changing the sign of an estimate, or crossing a clinically important decision threshold. These are distinct targets. For example, a confidence interval can include zero even while the point estimate remains positive; the confounding strength needed to alter statistical significance can therefore differ from the strength needed to explain away the point estimate.

The benchmark also matters. An equal-strength robustness value asks what happens when treatment and outcome partial \(R^2\) values are set equal. A sensitivity analysis that allows unequal strengths can reach the same target with a stronger association on one side and a weaker association on the other. Likewise, a benchmark calibrated to measured covariates is a comparison with those covariates under a chosen metric; it is not evidence that the unmeasured confounder must resemble them.

Worked Example: Interpreting a Rosenbaum Sensitivity Parameter

Suppose a hypothetical matched observational analysis reports a sensitivity parameter \(\Gamma=1.5\). In a Rosenbaum-style matched-pair sensitivity model, \(\Gamma\) bounds how much the treatment-assignment odds may differ between two individuals in a matched pair because of an unobserved covariate, beyond the matching information. At \(\Gamma=1\), the model allows no such within-pair departure; at \(\Gamma=1.5\), the assignment odds may differ by as much as a factor of 1.5 under the bound.

The parameter does not mean there is a 50% chance of hidden bias, that treatment odds are observed to differ by exactly 50%, or that a confounder has a 1.5-fold association with the outcome. It defines a sensitivity model. A result that remains statistically significant at \(\Gamma=1.5\) is evidence that the test result withstands that specified assignment-odds departure, for the matched design and test being used. It is not proof that no unmeasured confounding exists or that every plausible departure has been covered.

Robustness Is Not Proof of No Confounding

A sensitivity analysis assesses how conclusions behave within a chosen set of models or parameter values. It cannot certify that the chosen set contains every plausible form of hidden bias. An apparently robust result may still be vulnerable to a confounder outside the investigated range, to multiple confounders acting jointly, to a different functional form, or to violations of assumptions unrelated to unmeasured confounding. Conversely, sensitivity to modest departures does not by itself prove that the causal effect is absent; it shows that the available conclusion depends materially on assumptions that deserve scrutiny.

Interpret numerical results with particular care when changing outcome scales. A bias amount expressed in days cannot be read as a risk difference, and a parameter defined on a ratio scale does not retain the same meaning after transformation. Sensitivity parameters and thresholds belong to the model and scale in which they were defined. When comparing analyses, check whether the outcome, treatment contrast, adjustment set, and target estimand are actually comparable.

Key takeaway: Report a sensitivity result conditionally and specifically: identify the target conclusion, the model and scale, the sensitivity metric, its conditioning set or benchmark, and the range assessed. A robustness threshold describes the strength required to alter that target under those choices; it is neither a probability nor proof that confounding is absent.