Worked Quantitative Bias Analysis for an Observational Study

From an Adjusted Estimate to a Robustness Question

An adjusted estimate from an observational study is not automatically a causal effect: adjustment controls only the confounders that were measured and handled adequately. Quantitative bias analysis makes this limitation concrete. It asks how the estimate would change if an unmeasured confounder had specified relationships with treatment and outcome. The result is conditional on those assumptions; it does not discover the strength of hidden confounding from the observed data alone.

This tutorial works through an illustrative study of a program intended to reduce symptom severity. The outcome is standardized, so a difference of one unit corresponds to one standard deviation of the outcome. Negative treatment effects indicate lower symptom severity in the program group. The adjusted estimate is \(-0.060\) standard deviations. The central task is to determine what unmeasured-confounding assumptions could move that estimate, and how the conclusion changes under plausible values.

Definition: Let \(\hat{\theta}_{adj}\) be an estimate adjusted for measured covariates, with negative values indicating benefit in this example. Define \(B\) as the bias correction added to that estimate under a specified sensitivity scenario. The bias-adjusted estimate is \(\hat{\theta}_{adj}(B)=\hat{\theta}_{adj}+B\). Thus, positive \(B\) moves a negative estimated effect toward zero. This convention defines the direction of \(B\); it is not the same as defining bias as estimate minus truth.

A Simple Model for an Omitted Binary Confounder

To turn confounder assumptions into a numerical correction, consider a binary unmeasured confounder \(U\), such as a baseline characteristic that is more common among people who do not enter the program and is associated with symptom severity. Let \(C\) denote the measured covariates. Suppose the conditional mean outcome follows an additive model with no treatment-by-\(U\) interaction:

E(Y\mid A=a,C=c,U=u)=m(c)+\tau a+\gamma u.

Here \(A\) is the binary treatment, \(\tau\) is the constant conditional treatment effect, and \(\gamma\) is the difference in mean outcome associated with \(U=1\) rather than \(U=0\), holding treatment and \(C\) fixed. For the derivation below, assume consistency, conditional exchangeability of treatment and potential outcomes given \(C,U\), and positivity for treatment within relevant covariate strata. Also assume the stated additive outcome model is adequate for this bias calculation. These are substantive assumptions, not consequences of the numerical analysis.

Use the same target distribution of measured covariates for the treated and untreated conditional means. Define \(\Delta\) as the weighted average, over that target distribution, of the conditional prevalence difference in \(U\):

\Delta = E_w\!\left[ P(U=1\mid A=1,C)-P(U=1\mid A=0,C) \right].

A negative \(\Delta\) means that \(U=1\) is less prevalent among treated people, on average, than among untreated people. In this simple model, the treatment contrast omitting \(U\) is shifted by \(\gamma\Delta\). The correction to add back is therefore \(B=-\gamma\Delta\).

Why the Bias Formula Has This Sign

The calculation follows by averaging the conditional mean model over the distribution of \(U\) within each treatment group and covariate stratum. Write \(p_a(c)=P(U=1\mid A=a,C=c)\). Since \(U\) is binary, its conditional mean is \(p_a(c)\), so:

E(Y\mid A=a,C=c)=m(c)+\tau a+\gamma p_a(c).

Subtract the untreated conditional mean from the treated conditional mean. The terms involving \(m(c)\) cancel, leaving the treatment effect plus the contribution from the difference in \(U\) prevalence:

E(Y\mid A=1,C=c)-E(Y\mid A=0,C=c) = \tau+\gamma\{p_1(c)-p_0(c)\}.

Averaging these contrasts with the same weights over \(C\) gives an adjusted contrast of \(\tau+\gamma\Delta\). If the estimate targets that contrast, its confounding shift is \(\gamma\Delta\); subtracting the shift is equivalent to adding \(B=-\gamma\Delta\). This result is exact for the stated model and common standardization weights. A generic regression coefficient need not obey this simple formula when its model, weights, or effect structure differ.

Worked Example: Establishing the Unadjusted Sensitivity Baseline

Suppose a hypothetical observational study estimates a standardized mean difference of \(\hat{\theta}_{adj}=-0.060\) standard deviations, with \(SE=0.025\). Treat the adjusted estimate and its standard error as the starting point for sensitivity analysis. A normal-based 95% interval is:

-0.060\ \pm\ 1.96(0.025) = -0.060\ \pm\ 0.049 = [-0.109,\,-0.011]\text{ standard deviations}.

The test statistic for a zero effect is \(z=-0.060/0.025=-2.40\), corresponding to a two-sided \(p\)-value of approximately \(0.0164\) under the normal approximation. Under the model used for that interval and test, the adjusted estimate is statistically distinguishable from zero at the 5% level. This is not yet a claim that the causal effect is distinguishable from zero: the calculation has not accounted for unmeasured confounding.

The interval describes the repeated-sampling uncertainty represented by the standard error, conditional on the fitted analysis. It does not assign a 95% probability to the fixed parameter being inside this particular interval. For the sensitivity calculations below, initially hold the standard error fixed; that convention isolates the effect of changing the assumed bias correction.

Specifying a Confounder Scenario

A quantitative bias analysis requires explicit parameter choices. For a binary \(U\) under the model above, those choices can be expressed as (1) the outcome association \(\gamma\), in outcome standard deviations, and (2) the prevalence contrast \(\Delta\), on the probability scale. Their product \(\gamma\Delta\) is the shift in the adjusted contrast, and \(B=-\gamma\Delta\) is the correction added to the estimate.

Worked Example: Translating Confounder Assumptions into a Correction

Suppose investigators judge that a hypothetical unmeasured characteristic is associated with a \(0.30\)-standard-deviation higher outcome, so \(\gamma=0.30\). They also judge that it is 20 percentage points less prevalent among treated people than untreated people after measured-covariate standardization: \(P(U=1\mid A=1,C)=0.20\) and \(P(U=1\mid A=0,C)=0.40\) in the simplified scenario. Thus, \(\Delta=0.20-0.40=-0.20\).

The omitted-confounder shift is:

\gamma\Delta=(0.30)(-0.20)=-0.060\text{ standard deviations}.

The corresponding bias correction is \(B=-\gamma\Delta=0.060\) standard deviations. Adding it to the adjusted estimate gives:

\hat{\theta}_{adj}(B)=-0.060+0.060=0.000\text{ standard deviations}.

Under this particular scenario, the point estimate reaches the null. The signs are important: the confounder is associated with a higher outcome but is less common among treated people, so omitting it shifts the observed treatment contrast downward. The correction is positive and offsets that downward shift. The scenario is an assumption-based calculation, not evidence that \(U\) actually has these parameter values.

Point Tipping and Statistical Tipping

A point-estimate tipping point is the correction that moves the point estimate to a chosen threshold, here zero. Solving \(\hat{\theta}_{adj}+B=0\) gives \(B=-\hat{\theta}_{adj}\). For the estimate \(-0.060\), that correction is \(0.060\) standard deviations. This is distinct from the correction that makes the confidence interval include zero. Since the upper endpoint is initially \(-0.011\), the interval first reaches zero when \(B=0.011\).

If a fixed correction is added while the standard error remains \(0.025\), the sensitivity-adjusted interval is the original interval shifted by that correction. This procedure treats the bias parameters as fixed. It does not incorporate uncertainty about \(\gamma\), \(\Delta\), their product, or the standard error.

Worked Example: Finding Both Tipping Points

For the point estimate, set the corrected estimate to zero and solve for \(B\):

-0.060+B=0 \quad\Longrightarrow\quad B=0.060\text{ standard deviations}.

For the interval, add \(B\) to both endpoints. Its upper endpoint is \(-0.011+B\); it reaches zero when:

-0.011+B=0 \quad\Longrightarrow\quad B=0.011\text{ standard deviations}.

For example, at \(B=0.030\), the corrected estimate is \(-0.060+0.030=-0.030\) standard deviations and the shifted interval is \([-0.079,0.019]\). The interval includes zero, although the point estimate remains negative. At \(B=0.060\), the point estimate is zero and the interval is \([-0.049,0.049]\). Thus, loss of conventional statistical significance occurs at a smaller correction than the point-estimate tipping point.

Calibrating and Reporting a Robustness Conclusion

The numerical tipping point becomes useful when compared with defensible benchmarks. One approach is to compare a hypothesized hidden confounder with a measured prognostic characteristic: how strongly is that measured characteristic associated with the outcome, and how imbalanced is it across treatment groups after adjustment? Such a comparison can help readers judge scale, but it cannot establish that an unmeasured confounder is weaker or stronger. The benchmark may not capture the hidden variable’s joint association with treatment and outcome.

Worked Example: Comparing a Benchmark Scenario with the Tipping Point

Suppose a measured baseline characteristic is associated with a \(0.25\)-standard-deviation higher outcome, and a plausible benchmark prevalence difference is 12 percentage points, with lower prevalence among treated people. Using \(\gamma=0.25\) and \(\Delta=-0.12\), the correction is:

B=-\gamma\Delta=-(0.25)(-0.12)=0.030\text{ standard deviations}.

The bias-adjusted point estimate is \(-0.060+0.030=-0.030\) standard deviations. Shifting the original interval by the same amount gives \([-0.109+0.030,\,-0.011+0.030]=[-0.079,0.019]\). This benchmark does not reverse the point estimate, but it is large enough to make the interval include zero under the fixed-standard-error calculation.

A calibrated conclusion should state both facts rather than call the result simply “robust” or “not robust.” For instance: “The adjusted estimate is \(-0.060\) standard deviations. A correction of \(0.011\) standard deviations would make its normal-based 95% interval include zero, while a correction of \(0.060\) would move the point estimate to zero. A benchmark scenario giving a \(0.030\)-standard-deviation correction retains a negative point estimate but yields an interval that includes zero.” This describes the sensitivity of the evidence without asserting that the benchmark is the true confounding strength.

A Practical Analysis Sequence

A transparent report makes it possible to see where the conclusion comes from. The following sequence keeps the estimate, assumptions, and interpretation distinct.

State the target and scale.
Name the adjusted effect, its units, the target population or standardization distribution, and which sign represents benefit.

Write the bias model.
Define the correction direction and identify the assumptions linking the unmeasured confounder to the treatment contrast and outcome.

Justify parameter values.
Explain the range or benchmark for each confounder parameter; do not present chosen values as if the data had estimated them.

Calculate corrected estimates and uncertainty.
Show the arithmetic for each scenario and say whether intervals hold the original standard error fixed.

Report the tipping points and calibration.
Distinguish the correction that moves the estimate to the null from the correction that changes the interval’s relation to the null.

Common Mistake

Wrong reasoning: “The estimate is \(-0.060\), so a bias correction of \(-0.060\) moves it to zero.” This applies the wrong sign for the correction convention used here: \(-0.060+(-0.060)=-0.120\), which moves the estimate farther from zero.

Correction: Solve the equation using the stated definition. Because the corrected estimate is \(\hat{\theta}_{adj}+B\), the point tipping value satisfies \(-0.060+B=0\), hence \(B=0.060\). Always write the correction formula before interpreting the sign. If a different convention defines bias as estimate minus truth, the equation and sign must be changed consistently.

A second caution is that a fixed-bias correction does not automatically account for sampling uncertainty in the sensitivity parameters. If \(\gamma\) and \(\Delta\) are themselves uncertain, the shifted interval is only a conditional summary for the chosen values. A fuller analysis would need to propagate that uncertainty rather than imply that the fixed-parameter interval covers every source of uncertainty.

Key takeaway: Quantitative bias analysis connects explicit confounder assumptions to a numerical correction. Its strength is transparency: readers can see how large a correction is required and compare it with stated benchmarks. Its limitation is equally important: the conclusion remains conditional on the bias model, parameter values, and uncertainty calculation.

Check Your Understanding

Use the correction convention \(\hat{\theta}_{adj}(B)=\hat{\theta}_{adj}+B\) for these questions.

If an adjusted estimate is \(-0.080\) standard deviations, what correction moves its point estimate to zero?
For \(\gamma=0.40\) and \(\Delta=-0.10\), calculate the omitted-confounder shift and the bias correction. State their signs.
Why can an interval include zero before the point estimate reaches zero?
What additional uncertainty is not represented when a fixed bias correction is simply added to both confidence-interval endpoints?
Why is a measured covariate benchmark informative but not proof of the strength of an unmeasured confounder?