A Worked Coefficient Sensitivity Analysis

Why Compare Closely Related Specifications?

A regression coefficient describes a relationship conditional on the other predictors in its model. When two predictors carry overlapping information, including both can change the estimated relationship for either one. A coefficient may become smaller, change sign, or have a wider confidence interval. A sensitivity analysis makes those changes visible: it asks whether a substantive conclusion persists across plausible, closely related ways of specifying the regression.

The comparison is meaningful only if it is organized carefully. For each model, the response, sample, units, and coefficient being compared should be the same. The predictor sets can differ, but the models should address closely related questions. A changed estimate is evidence that the reported association depends on the specification; it is not, by itself, evidence that one model is correct or that either association is causal.

Models and Conditions for the Comparison

Consider hypothetical data on 120 adults. The response \(Y\) is systolic blood pressure at a 12-week follow-up, measured in mmHg. The candidate predictors are age, measured in years; an activity score, measured in points; and weekly exercise time, measured in minutes. Suppose activity score and exercise time have a sample correlation of 0.90, so they contain substantially overlapping information.

Compare two ordinary least-squares models fitted to the same 120 adults. Model A includes age and activity score. Model B includes age, activity score, and weekly exercise time. In each model, an activity-score coefficient describes the estimated difference in follow-up blood pressure for a one-point difference in activity score, holding the model’s other predictors fixed. Because the models condition on different predictors, those coefficients answer related but not identical questions.

The models use the usual linear regression conditions for coefficient estimates and inference: the conditional mean is adequately represented by the specified linear form; observations and errors are independent across adults; the error variance is constant for the stated standard errors; and there is no exact linear dependence among the predictors. For the usual small-sample \(t\)-based confidence intervals to be exact, the errors are also normally distributed. Approximate normality of the errors supports approximate intervals in larger samples.

For this hypothetical analysis, assume residual-versus-fitted plots show no pronounced curvature or changing spread, and a normal probability plot shows no severe departure from normality. The adults are treated as independent observations, and neither model has exact predictor redundancy. These checks support using the reported ordinary least-squares standard errors and \(t\)-intervals; they do not guarantee that the models capture every relevant feature of the data.

Worked Comparison of Estimates and Intervals

For a coefficient estimate \(\hat{\beta}_j\), its standard error \(SE(\hat{\beta}_j)\) measures its estimated sampling variability under the fitted model. A two-sided confidence interval has the form \(\hat{\beta}_j \pm t^*SE(\hat{\beta}_j)\), where \(t^*\) is the appropriate \(t\) critical value and the residual degrees of freedom depend on the model.

Why Can the Coefficient Change?

A short derivation clarifies one source of coefficient sensitivity. Consider centered variables and a two-predictor population model \(Y=\beta_1X_1+\beta_2X_2+\varepsilon\), where the error is uncorrelated with both predictors. If a reduced regression uses \(X_1\) alone, its population slope is \(\operatorname{Cov}(X_1,Y)/\operatorname{Var}(X_1)\). Substituting the full model into that expression gives

The reduced-model slope equals the full-model coefficient \(\beta_1\) plus a term involving the association between the predictors and the effect of \(X_2\) on \(Y\). Thus, if \(X_1\) and \(X_2\) are correlated and \(X_2\) is related to \(Y\), omitting \(X_2\) can change the slope attributed to \(X_1\). This calculation is a simplified population result, not a general claim that every omitted predictor creates bias. It helps explain why closely related specifications can produce different coefficients.

Including an overlapping predictor can also increase a coefficient’s standard error: the model has less information for separating the predictors’ individual associations. A confidence interval reflects both the estimate and its standard error, so it is important to compare both rather than treating a change in statistical significance as the whole story.

Which Conclusions Persist?

A sensitivity analysis should describe the pattern of evidence with care. It can report that a coefficient keeps its sign, that its magnitude changes substantially, that its interval changes width, or that an interval changes whether it contains zero. These are distinct observations. For example, in the activity comparison, the estimate remains negative in both models, but its magnitude is smaller and its uncertainty is greater in Model B. The sign persists; the strength of evidence against a zero coefficient does not.

A coefficient’s interval is conditional on its model. Overlap or non-overlap between two separate confidence intervals is not, by itself, a formal test that the coefficients differ. The estimates are obtained from the same observations and are generally dependent. The goal here is a transparent comparison across specifications, not an unreported test of the difference.

Summarize the Comparison Without Overclaiming

A useful report states the shared setup, names the difference between the specifications, and summarizes the estimate and uncertainty in context. It also distinguishes a persistent feature from a specification-sensitive one. In this hypothetical analysis, a concise summary would be: “Across models fitted to the same 120 adults, the estimated age coefficient remained positive and its 95% interval excluded zero. The activity-score coefficient remained negative, but became smaller and less precise after weekly exercise time was added; its interval then included zero.”

That summary does not identify a preferred model. Whether a predictor belongs in a regression depends on the question and substantive justification for the specification, not simply on which model produces a smaller p-value. Nor does an association establish that changing a predictor would change blood pressure. Those limits matter especially when two predictors represent related aspects of the same underlying behavior.

Common Mistake

• Wrong reasoning: “The activity coefficient is significant in Model A but not Model B, so adding exercise proves that activity has no relationship with blood pressure.”
• Correction: The estimates differ in size and precision, and the Model B interval includes zero. That shows the evidence for a nonzero conditional activity coefficient is sensitive to this specification. It does not prove the coefficient is zero, settle which specification answers the scientific question, or establish causation.
• Another trap: Comparing coefficients from models fitted to different samples, outcomes, or measurement scales can confuse specification changes with changes in the data or units. Keep those features fixed when interpreting a coefficient comparison.