Why Benchmark Hidden Confounding?
A sensitivity analysis can describe how strong an unmeasured confounder would need to be to change an estimate. That strength is easier to judge when it has a reference point: for example, a measured covariate that is already in the analysis and has a known relationship with treatment and outcome. Benchmarking makes the comparison more concrete, but it does not show that an unmeasured confounder exists or that it resembles the measured variable.
This tutorial uses partial \(R^2\) values to make that comparison in a linear regression setting. One partial \(R^2\) describes how much a variable explains residual treatment variation; another describes how much it explains residual outcome variation after accounting for treatment and the measured covariates. The pair matters: a variable that strongly predicts only one of treatment or outcome may still have limited capacity to create omitted-variable bias.
A partial \(R^2\) lies between 0 and 1. It is conditional on the variables listed after the vertical bar: it is not the variable’s ordinary, unadjusted correlation with treatment or outcome. To benchmark a measured covariate \(X_j\), compare models that differ by whether \(X_j\) is added to the relevant adjustment variables. For treatment strength, compare \(D\sim X_{-j}\) with \(D\sim X_{-j},X_j\); for outcome strength, compare \(Y\sim D,X_{-j}\) with \(Y\sim D,X_{-j},X_j\). Here \(X_{-j}\) denotes the other measured covariates.
From Partial R-Squared Values to a Bias Benchmark
In the linear regression sensitivity framework, the two partial \(R^2\) values jointly determine a bound on omitted-variable bias. The expression below gives the largest bias magnitude, in standard-error units, compatible with the specified strengths under the model. The sign of actual bias depends on the directions of the relevant associations; the partial \(R^2\) values alone do not supply that sign.
This result is used under a linear, additive regression specification, with the same analysis sample and a single omitted variable added to the stated adjustment set. It describes sensitivity of the adjusted regression coefficient; it does not by itself establish that the coefficient is a causal effect. A causal interpretation still requires the relevant causal assumptions, including adequate control of confounding. In practice, use partial \(R^2\) values calculated for the same covariate set and sample, and use a consistent residual degrees of freedom convention.
The main ingredients of the expression can be understood through residualization. First remove the linear association with measured covariates from treatment, leaving its residual variation. The treatment partial \(R^2\) measures how strongly \(U\) accounts for that variation. The outcome partial \(R^2\) measures how much outcome variation remains associated with \(U\) after controlling for treatment and measured covariates. The two strengths combine multiplicatively: if either one is zero, this omitted variable cannot create bias through this model. The denominator \(1-r_D\) accounts for how much residual treatment variation remains after \(U\) is included.
Worked Examples
Worked Example: Benchmarking Against a Measured Covariate
Hypothetical example. Suppose a fitted linear regression estimates a treatment coefficient of 2.4 outcome units, with \(SE=0.8\) outcome units and \(df=400\). Thus the estimated coefficient is 3 standard errors above zero. A measured covariate, when considered as a benchmark, has treatment partial \(R^2\) of 0.080 and outcome partial \(R^2\) of 0.180. Assume these values were calculated from the same analysis sample and the relevant nested regressions.
The maximum bias associated with this benchmark, in standard-error units, is
In outcome units, this is \(2.502(0.8)=2.002\) units of possible bias. If the bias acts in the direction that reduces the positive estimate, subtracting this maximum gives \(2.4-2.002=0.398\) outcome units, or about \(0.498\) standard errors. This benchmark alone would not quite explain away the point estimate. It does not tell us that the actual bias has this size or direction; it says what the specified strength pair can accommodate at most under the framework.
Worked Example: The Benchmark Choice Changes the Comparison
Hypothetical example. Keep the same estimated coefficient, standard error, and \(df\) as above, but compare two possible measured-covariate benchmarks. A weaker covariate has \(r_D=0.040\) and \(r_Y=0.090\). A stronger one has \(r_D=0.080\) and \(r_Y=0.180\). In each case, assume the partial \(R^2\) values come from appropriate nested regressions for the same sample and adjustment set.
For the weaker benchmark, the calculation is
standard errors. The corresponding maximum bias in outcome units is \(1.225(0.8)=0.980\), leaving \(2.4-0.980=1.420\) outcome units if the bias reduces the estimate.
For the stronger benchmark, the bias bound is
That is \(2.002\) outcome units of possible bias in the reducing direction, leaving \(0.398\) outcome units. The stronger benchmark therefore makes the estimate look less robust to a confounder with matching strengths. The difference is not a change in the fitted estimate; it is a change in the reference strength used to describe a hypothetical confounder.
Worked Example: Relating an Explain-Away Robustness Value to a Benchmark
Hypothetical example. For the estimate above, \(|t|=|2.4/0.8|=3.0\), with \(df=400\). An explain-away robustness value under equal-strength treatment and outcome associations asks what common partial \(R^2\), \(r\), would make the maximum bias equal to the estimate’s 3-standard-error distance from zero. Set \(r_D=r_Y=r\) in the bias formula and solve:
Rearranging gives \(r^2+0.0225r-0.0225=0\). The nonnegative solution is
Thus, under this equal-strength convention, an omitted variable with partial \(R^2\) of about 13.9% with treatment and 13.9% with outcome could explain away the point estimate in the most bias-inducing direction. The earlier measured-covariate benchmark was asymmetric, at 8.0% and 18.0%; its bias bound was about 2.502 standard errors, less than the 3.0 required to explain away the estimate. These are complementary comparisons, not interchangeable descriptions: an equal-strength robustness value is not a claim that both associations must match a particular covariate’s values.
How to Choose and Report a Benchmark
A useful benchmark is substantively interpretable and measured comparably to the hypothetical confounder. A single covariate may be a reasonable reference if its role and strength are familiar to the intended audience. Alternatively, several covariates can be reported to show a range of reference strengths. For example, a prognostic baseline measure may be a more informative outcome benchmark than a demographic variable, while a strong predictor of treatment may help anchor the treatment dimension. The choice should be explained rather than selected only because it makes the result appear robust or fragile.
A benchmark is not automatically a conservative upper bound on hidden confounding. An unmeasured variable could be weaker than every measured covariate, stronger than all of them, or strong in one dimension and weak in the other. Also, two measured covariates considered separately may have different partial \(R^2\) values from a joint group of variables. State clearly whether the reference is one covariate, a group, or a chosen multiple of a covariate’s strengths.
When reporting a robustness value alongside benchmarks, keep the questions distinct. The robustness value summarizes the equal-strength association needed to reach a specified target, such as explaining away the point estimate. A benchmark translates an observed covariate’s particular pair of strengths into a bias bound. Neither number is a probability that hidden confounding exists, and neither establishes the truth of a causal claim.
Specify the treatment, outcome, adjustment set, analysis sample, and target for the sensitivity statement.
For each selected measured covariate, obtain its treatment and outcome partial \(R^2\) values from the appropriate nested regressions.
Use the bias formula with the stated residual degrees of freedom, and express the result in standard-error units and, where useful, outcome units.
Explain what the benchmark says about a hypothetical confounder of that strength, and state that the actual hidden confounder may differ.
Common Pitfall
A common mistake is to say, “The measured covariate explains 18% of the outcome, so an unmeasured confounder explaining 18% would create the same bias.” This reasoning uses only the outcome dimension and may confuse an ordinary \(R^2\) with a partial \(R^2\). The bias calculation depends on both the treatment and outcome partial \(R^2\) values, with each conditioned on the appropriate variables. The correction is to report the pair, identify the adjustment sets used to calculate it, and avoid treating a measured covariate as proof of the strength of hidden confounding.
Check Your Understanding
Answer each question using the definitions and calculations in this tutorial.
- What does \(R^2_{D\sim U\mid X}\) measure, and why is it not generally the same as an unadjusted association between \(D\) and \(U\)?
- A hypothetical confounder has zero outcome partial \(R^2\) conditional on treatment and measured covariates. What does the bias formula imply, regardless of its treatment partial \(R^2\)?
- Why should a measured covariate benchmark be calculated using nested regressions that correspond to the stated adjustment set?
- In the first worked example, why does a bias bound of 2.502 standard errors not quite explain away an estimate 3.0 standard errors from zero?
- What additional information does an asymmetric measured-covariate benchmark provide beyond an equal-strength explain-away robustness value?