Test Sensitivity to Alternative Adjustment Sets

Why Compare Adjustment Sets?

In an observational study, an adjusted estimate depends partly on which covariates enter the analysis. Some choices are guided by subject-matter knowledge and a causal diagram; others may be plausible alternatives when the role of a measured variable is uncertain. Comparing estimates across defensible choices can show how dependent the result is on those choices. It is a sensitivity analysis, not a procedure that selects the “correct” model by whichever estimate looks most convincing.

The comparison is useful only when its meaning is clear. A change might reflect ordinary sampling or model variation, a change in the target population or causal contrast, or an adjustment choice that fails to control confounding—or that controls for a variable inappropriate to the question. A stable estimate is not proof that the causal assumptions hold, and an unstable estimate does not by itself reveal which model is right.

Define the Contrast Before Comparing Estimates

Let \(A\) denote a binary exposure, with \(A=1\) for exposure and \(A=0\) for no exposure. Let \(Y(1)\) and \(Y(0)\) be the outcomes an individual would have under those two exposure conditions. A common target is the average treatment effect (ATE) in a specified population: the average difference between the outcomes that population would have under exposure and no exposure. The target population must be stated; it might be all eligible individuals or a defined study population.

The earlier material on causal diagrams, backdoor adjustment sets, and identification assumptions gives the basis for assessing whether a set is appropriate. For the ATE, the essential assumptions include consistency (the observed outcome equals the potential outcome under the exposure actually received), conditional exchangeability given the adjustment set (within covariate profiles, exposure assignment is independent of the potential outcomes), and positivity (each relevant profile has a positive probability of each exposure level). Estimation also requires an adequate outcome model if regression predictions are used. These are substantive and statistical assumptions; comparing estimates does not verify them.

Why Valid Sets Should Often Agree

Suppose \(C\) is a measured adjustment set that satisfies the assumptions above. Under conditional exchangeability, the mean potential outcome under exposure level \(a\) can be expressed as an average of observed conditional means over the target population’s covariate distribution. This is the standardization formula:

Taking the difference between the two exposure conditions gives the ATE. The following justification shows why a valid set can identify that contrast.

The first equality is the law of total expectation. The second follows from conditional exchangeability: at each value of \(C\), the mean potential outcome under \(a\) is the same among those with \(A=a\) as in the full covariate profile. The third follows from consistency for individuals with \(A=a\). Positivity ensures that the conditional mean for each exposure level can be learned for the profiles being averaged. Subtracting the expression for \(a=0\) from the expression for \(a=1\) yields the ATE.

If two defensible sets both identify the same ATE in the same target population, their correctly estimated standardized contrasts should target the same quantity. In finite samples, their estimates can still differ because of sampling variation, model specification, or estimation choices. If the sets are not valid for the same contrast, or the estimates average over different populations, agreement is not expected.

A Structured Comparison

A fair sensitivity analysis begins by specifying the causal contrast, the target population, the outcome scale, and a reason for considering each adjustment set. Then compare estimates that are genuinely comparable—for instance, standardized mean differences in the same target population—not a conditional regression coefficient from one model and a population-average contrast from another.

When the Target Changes

Not every difference between estimates is evidence of sensitivity to adjustment choices for one fixed estimand. If effect modification is present, the average effect depends on the population over which subgroup effects are averaged. Changing the standardization population changes the target even if the subgroup contrasts themselves stay the same. Likewise, a conditional regression coefficient is not automatically the same quantity as a population-average standardized contrast. Before interpreting a change, check whether both estimates answer the same question.

When Adjustment Changes the Question

A set of variables should not be treated as a reasonable alternative merely because it can be added to a regression model. In particular, for a total-effect question, adjusting for a mediator can remove part of the pathway through which exposure affects the outcome. The resulting regression coefficient is not simply a more carefully adjusted estimate of the same total effect. Its causal interpretation requires additional assumptions and a separately defined target.

Interpreting Stability and Instability

When valid sets produce similar estimates, the result is less sensitive to those particular measured-covariate choices, given the models and data used. It does not establish that there is no residual confounding, that the covariates were measured accurately, or that the causal assumptions hold. When estimates differ, possible explanations include chance variation, model dependence, different target populations, or an adjustment set that does not identify the stated contrast. The difference itself cannot distinguish among these explanations.

For a useful comparison, report each set’s variables and rationale, the estimate and uncertainty interval, the analysis sample, and the target population. State whether estimates are standardized to a common population and use the same outcome scale. If adjustment sets are nested, identify the added covariates and why they are plausible alternatives. Avoid using significance as the deciding criterion: whether a confidence interval excludes zero is not a test of which adjustment set is causally appropriate.

Common Pitfall

• Wrong reasoning: “The estimate barely changed after adjustment, so confounding cannot be important.”
• Correction: Similar estimates across the sets examined show stability only to those measured-covariate choices and model specifications. They do not rule out unmeasured confounding or other violations of identification assumptions.
• Wrong reasoning: “The model with the most covariates must be best.”
• Correction: Covariates should be selected for a causal reason. Adding a mediator, collider, or otherwise inappropriate variable can change the target or introduce bias.
• Wrong reasoning: “Any two adjusted coefficients can be compared directly.”
• Correction: First verify that they refer to the same contrast, outcome scale, and target population. Conditional and population-average quantities need not match.