Why Check Sensitivity to Covariate Choices?
An adjusted regression coefficient describes a relationship conditional on the covariates included in the model. Since more than one adjustment set may be plausible, it is useful to ask whether the estimated exposure association changes when those sets change. A structured sensitivity check makes that comparison visible instead of relying on a single, potentially arbitrary model.
The aim is not to try many combinations and choose the one with the most appealing result. Before comparing estimates, specify a small set of defensible adjustment choices using subject-matter reasoning. Then compare the exposure coefficient and its uncertainty across them, keeping other modeling choices as consistent as possible. This can reveal dependence on a particular adjustment decision, but it cannot establish that the chosen estimate is unbiased or that the covariates are causally appropriate.
Set Up a Fair Comparison
Let \(Y_i\) be the outcome for individual \(i\), and let \(X_i\) be the exposure or predictor of interest. For an adjustment set \(C\), a multiple linear regression model can be written as
Here, \(C_{ij}\) denotes the value of covariate \(j\) for individual \(i\), \(\beta_X\) is the population coefficient for \(X\) in this specified model, and \(\epsilon_i\) is the model error. The fitted coefficient \(\hat{\beta}_X\) estimates the expected change in \(Y\), in outcome units, associated with a one-unit increase in \(X\), holding the included covariates fixed under the model. It is an adjusted association; a causal interpretation requires additional justification.
For an ordinary least squares comparison, the model should use an appropriate linear conditional mean for the outcome, and the errors should have conditional mean zero given the included predictors. Observations should be independent for the usual model-based uncertainty calculations. Conventional standard errors also rely on assumptions such as constant conditional error variance; small-sample inference may additionally rely on a suitable error distribution. There must not be perfect linear dependence among the model predictors. These assumptions do not guarantee that a covariate set removes confounding. They concern the regression model and its uncertainty calculations.
A meaningful comparison also requires consistency in the analysis itself. Keep the outcome definition, exposure coding and units, functional form, and study population fixed. Ideally, fit every model to the same observations. Otherwise, an estimate may change because the included people changed, not because the covariates changed. If the same sample cannot be used, report that limitation and avoid attributing the whole difference to adjustment.
Choose a reference model in advance, such as the main adjustment set justified by subject-matter knowledge. For every alternative set, record \(\hat{\beta}_X\), its standard error, the sample size, and the covariates included. The standard error (SE) is the model-based estimate of the sampling variability of a coefficient estimate. The SEs describe uncertainty under their respective models; comparing them is not a formal test that two coefficients differ.
Summarize Changes Transparently
For an alternative model \(m\) and reference model \(r\), the absolute change in the exposure estimate is
If the reference coefficient is not close to zero, a relative change can also help describe the size of the shift:
This is a descriptive summary, not a universal threshold for deciding whether a result is robust. When the reference estimate is near zero, the percentage can become extremely large or unstable, so report the absolute change instead. Always include the coefficients themselves, with units, so readers can see their direction and magnitude.
A useful comparison table might look like this:
| Model | Adjustment set | \(\hat{\beta}_X\) | SE | \(n\) |
|---|---|---|---|---|
| Reference | Prespecified main set | Estimate in outcome units | Reported SE | Analysis sample size |
| Alternative | Plausible alternative set | Estimate in outcome units | Reported SE | Same sample, if possible |
Worked Example: Comparing Two Plausible Adjustment Sets
Suppose a hypothetical analysis estimates the association between a one-hour increase in weekly exercise (\(X\)) and systolic blood pressure (\(Y\), in mmHg). Both models use the same 240 adults and the same linear model form. Subject-matter reasoning supports either adjusting for age and smoking, or for age, smoking, and a baseline measure of overall health. The latter is included as a plausible prognostic covariate, not because it was selected to produce a preferred estimate.
The reference model, adjusted for age and smoking, gives \(\hat{\beta}_{X,r}=-0.80\) mmHg per additional weekly exercise hour, with \(SE=0.30\) mmHg. The alternative model gives \(\hat{\beta}_{X,m}=-0.68\) mmHg per hour, with \(SE=0.34\) mmHg.
Absolute change: applying the formula gives \(\Delta_m=-0.68-(-0.80)=0.12\) mmHg per hour. The estimate is 0.12 mmHg per hour closer to zero in the alternative model.
Relative change: \((0.12/|-0.80|)\times100\%=15.0\%\). The alternative estimate is 15.0% smaller in absolute magnitude than the reference estimate.
The two estimates have the same negative direction and are reasonably similar in magnitude, while the reported SE increases from 0.30 to 0.34 mmHg per hour. This is evidence that the point estimate is not highly sensitive to these two particular covariate choices in this analysis. It does not show that either estimate is unbiased, nor does the SE comparison establish whether the estimates differ beyond sampling variation.
Interpret Changes in Light of the Adjustment Rationale
A coefficient shift is a signal to examine the models and the reasoning behind them, not an automatic instruction to add or remove a variable. A covariate may be a confounder, mediator, collider, or prognostic variable, and those roles have different implications. The appropriate set depends on the question and the causal structure being considered. Sensitivity checks complement subject-matter knowledge; they do not replace it.
For each plausible model, ask: Was the covariate measured before the exposure, or could it be affected by the exposure? Is there a substantive reason to regard it as a common cause or otherwise necessary for the target comparison? Does including it change the analysis sample? Is the fitted coefficient still describing the same exposure contrast? A model that adjusts for a post-exposure variable may answer a different question from one that adjusts only for pre-exposure covariates. A stable number across such models does not make the interpretations interchangeable.
Worked Example: A Change After Adding a Post-Exposure Variable
In a hypothetical analysis of a treatment and a later health score, the treatment coefficient is \(-3.2\) score points after adjustment for age and baseline severity. A team also fits a model adding a measure of treatment-related activity recorded after treatment begins; the coefficient becomes \(-1.1\) points. Assume both models use the same 180 people and that higher scores mean worse health.
Absolute change: \(-1.1-(-3.2)=2.1\) score points. The adjusted association is 2.1 points closer to zero after adding the post-treatment measure.
This comparison reveals a substantial change across the two specifications. But because the added variable is measured after treatment starts, its inclusion may change the question and could introduce overadjustment. The change alone does not show that the first estimate is correct, that the second is biased, or that the treatment effect is 2.1 points smaller. The team should explain the timing and causal role of the added variable and state which adjustment set addresses its intended question.
Uncertainty and Comparability
Coefficient estimates and their SEs can move in different directions across models. Adding a covariate can account for outcome variation and sometimes reduce an SE, but it can also make predictors more redundant or reduce the number of usable observations, sometimes increasing an SE. The direction is not guaranteed. Report the actual SEs rather than assuming that a larger adjustment set must make the estimate more precise or more uncertain.
If every model uses the same observations and coefficient scale, compare the SEs as a description of model-based uncertainty. For instance, an SE of 0.40 outcome units is larger than an SE of 0.30 outcome units for the same coefficient scale. This does not quantify the uncertainty in the difference between the two fitted coefficients: the estimates come from overlapping data and are generally related. A formal comparison would require methods specifically designed for that question, rather than treating the two SEs as independent.
Worked Example: Separating Coefficient Stability From SE Stability
A hypothetical study models a recovery-time outcome in days using the same 150 patients in every model. The exposure is a one-unit increase in a symptom score. The reference model adjusts for age and baseline severity and reports \(\hat{\beta}_{X,r}=1.50\) days per score unit with \(SE=0.25\) days. A plausible alternative adds a pre-exposure prognostic variable and reports \(\hat{\beta}_{X,m}=1.56\) days per score unit with \(SE=0.40\) days.
Coefficient change: \(1.56-1.50=0.06\) days per score unit. Relative to the reference magnitude, this is \((0.06/1.50)\times100\%=4.0\%\).
SE change: \(0.40-0.25=0.15\) days per score unit. The alternative model's SE is \((0.15/0.25)\times100\%=60.0\%\) larger than the reference SE.
The coefficient is stable in magnitude across these models, while the estimated uncertainty is not. This may happen because the added variable changes how precisely the exposure coefficient is estimated, even when its point estimate changes little. The table should report both quantities: saying only that “the result was stable” would hide the difference in uncertainty. These arithmetic comparisons do not provide a test of coefficient equality or a confidence interval for the difference.
A Structured Sensitivity-Check Workflow
A compact workflow helps keep the analysis focused on plausible alternatives rather than model searching. It also makes clear which changes can be attributed to covariate choices and which require a separate explanation.
Name the outcome, exposure contrast, units, and intended interpretation. Clarify whether a causal interpretation is being considered and what assumptions it would require.
Use subject-matter knowledge to explain why each covariate is included or omitted. Do not select a set because its coefficient, SE, or significance result looks preferable.
Use the same outcome and exposure coding, model form, and, where possible, the same observations. Record any unavoidable changes in sample size or model specification.
Report each exposure coefficient, its SE, the adjustment set, and \(n\). Calculate absolute changes from the prespecified reference; use relative changes cautiously when that reference is near zero.
Describe whether direction and magnitude are stable across the choices examined, note changes in uncertainty, and discuss what the comparison cannot establish.
Robustness Is Not Proof of Unbiasedness
When an estimate remains similar across several plausible adjustment sets, it is reasonable to say that it is robust to the covariate choices examined. That statement is deliberately limited. All compared models may share an omitted confounder, use a poorly measured covariate, or rely on an unsuitable functional form. The same biased estimate can therefore appear stable across every model in the comparison.
Conversely, a shift does not by itself identify the “right” estimate. It could reflect meaningful control of confounding, adjustment for a mediator or collider, changes in the sample, or other modeling differences. The right response is to investigate the reason for the shift using the timing and substantive role of each variable, then report the defensible analysis and its limitations. A sensitivity check is evidence about dependence on specified choices—not a certificate of validity.
Common Mistake
- Wrong reasoning: “The exposure coefficient barely changed when I added covariates, so omitted-variable bias is impossible and the estimate is unbiased.”
- Correction: Similar estimates show only that the result is not very sensitive to the covariate sets actually compared. Shared omissions, measurement error, selection, model misspecification, or inappropriate adjustment can still affect all of them.
- Another trap: Comparing models fitted to different samples and attributing every coefficient change to adjustment. Check and report \(n\); whenever feasible, refit on a common sample so the comparison isolates the covariate choice.
Check Your Understanding
Use the definitions and comparison principles above to answer these questions.
- A reference coefficient is \(-2.0\) outcome units and an alternative is \(-1.6\). What is the absolute change, and what is the relative change in magnitude?
- Why is it useful to fit plausible adjustment-set models to the same observations?
- Two models have similar exposure coefficients, but both omit a plausible common cause of exposure and outcome. What can and cannot be concluded from their similarity?
- An SE rises from 0.20 to 0.32 outcome units while the coefficient changes very little. How should this pattern be summarized?
- Why is adding a variable measured after exposure begins not automatically a better adjustment choice?