Model Terms and Coefficient Uncertainty

Why Model Terms Matter for Uncertainty

A regression coefficient is not a number with one fixed standard error regardless of how the model is written. The estimate and its standard error depend on which predictors are included. Adding a predictor changes what the other coefficients describe; it can also change how much information the data provide for estimating them. Removing a predictor can likewise change both the estimated coefficient and its uncertainty.

Consider estimating the association between weekly exercise and resting heart rate. A regression of heart rate on exercise alone describes the fitted change in heart rate across exercise levels without adjusting for other measured characteristics. A regression that also includes age describes the exercise coefficient while holding age fixed. Those are different model terms, and the coefficients need not be equal. This comparison is about statistical associations, not evidence that exercise causes a change in heart rate.

Specifications and Coefficients

Definition: A regression specification is the set of terms used to describe the response’s mean. For example, a simple linear regression specification includes an intercept and one predictor; a multiple regression specification includes an intercept and two or more predictors. A coefficient describes the fitted change in the response for a one-unit increase in its predictor, with the other included predictors held fixed. In a simple regression there are no other predictors to hold fixed.

For clarity, consider the simple model and a model that adds a second predictor:

Y_i=\beta_0+\beta_1 x_i+\varepsilon_i \qquad\text{and}\qquad Y_i=\beta_0+\beta_1 x_i+\beta_2 z_i+\varepsilon_i.

Here \(Y_i\) is the response for observation \(i\), \(x_i\) and \(z_i\) are predictors, and the \(\beta\) terms are population regression coefficients for the stated specification. The error \(\varepsilon_i\) is the difference between the response and its model mean. The first model’s coefficient for \(x\) describes its fitted linear association with \(Y\) without adjusting for \(z\). In the second model, the coefficient for \(x\) describes the fitted association with \(Y\) among observations held at the same value of \(z\).

The usual conventional standard error formulas below are based on a linear mean model with independent errors of mean zero and common variance \(\sigma^2\), conditional on the predictor values. The standard error estimates the sampling variability of a coefficient estimate under that specified model. The formulas do not guarantee that the chosen specification is substantively appropriate.

How the Standard Error Depends on the Other Predictors

For a coefficient of \(x\), first imagine removing from \(x\) the part that can be linearly predicted by the other predictors and the intercept. Call the remaining, residualized values \(x_i^*\). Their sum of squares, \(\sum_i (x_i^*)^2\), measures how much variation in \(x\) remains to distinguish its contribution from the other included terms. In a simple regression, there are no other predictors to remove, so \(x_i^*=x_i-\bar{x}\).

Result: In conventional least squares regression, the estimated standard error of the coefficient for \(x\) is

SE(\hat{\beta}_1)=\frac{s}{\sqrt{\sum_i (x_i^*)^2}}, \qquad s=\sqrt{\frac{SSE}{n-p}},

where \(SSE\) is the sum of squared residuals and \(p\) is the number of estimated coefficients, including the intercept. In the simple regression case, this reduces to \(SE(\hat{\beta}_1)=s/\sqrt{S_{xx}}\), where \(S_{xx}=\sum_i(x_i-\bar{x})^2\).

The formula has two competing parts. A larger residual standard deviation \(s\) increases the standard error. More remaining variation in \(x\), after accounting for the other predictors, increases the denominator and decreases the standard error. Thus, adding a predictor can make \(x\) harder to distinguish from the other terms, but it can also explain response variation and reduce \(s\). The net change is not determined by the number of terms alone.

With one other predictor \(z\), let \(r_{xz}\) be the sample correlation between \(x\) and \(z\). The sum of squares remaining in \(x\) after accounting for \(z\) is \(S_{xx}(1-r_{xz}^2)\). Therefore:

SE(\hat{\beta}_1\text{ in the model with }z) = \frac{s_{\text{with }z}}{\sqrt{S_{xx}(1-r_{xz}^2)}}.

This expression makes the effect of overlap visible: when \(x\) and \(z\) are strongly correlated, \(1-r_{xz}^2\) is small, so little distinct variation in \(x\) remains. However, \(s_{\text{with }z}\) may be smaller than the residual standard deviation in the model without \(z\). Both changes matter.

Why the Formula Has This Form

A short derivation explains the role of predictor overlap. After accounting for the intercept and the other predictors, the information about the coefficient of \(x\) comes from the residualized predictor values \(x_i^*\). The least squares coefficient’s estimation error can be written as a weighted sum of the regression errors:

\hat{\beta}_1-\beta_1 = \frac{\sum_i x_i^*\varepsilon_i}{\sum_i(x_i^*)^2}.

Conditional on the predictors, the \(x_i^*\) values are fixed. Independent errors with variance \(\sigma^2\) make the variance of the numerator \(\sigma^2\sum_i(x_i^*)^2\). Dividing by the squared denominator gives:

\operatorname{Var}(\hat{\beta}_1\mid x,z) = \frac{\sigma^2}{\sum_i(x_i^*)^2}.

Replacing the unknown \(\sigma\) with the residual estimate \(s\) gives the stated standard error. This derivation also shows why the coefficient’s precision depends on variation in \(x\) that is not shared with the other terms, rather than on the total spread of \(x\) alone.

Worked Example: A Correlated Predictor Increases the Standard Error

Suppose a hypothetical sample of 30 adults is used to relate resting heart rate, in beats per minute, to weekly exercise hours \(x\). The simple regression has \(S_{xx}=100\) hours squared and residual standard deviation \(s=4\) beats per minute. A second specification adds weekly walking hours \(z\). For this comparison, suppose \(r_{xz}=0.80\) and the added model’s residual standard deviation is also \(4\) beats per minute. Calculate the conventional standard error for the exercise coefficient in each model.

In the simple regression:

SE(\hat{\beta}_1)=\frac{4}{\sqrt{100}} =\frac{4}{10} =0.400\ \text{beats per minute per exercise hour}.

In the model with walking hours, the remaining sum of squares for exercise is:

S_{xx}(1-r_{xz}^2)=100(1-0.80^2)=100(0.36)=36.

So the adjusted coefficient’s standard error is:

SE(\hat{\beta}_1)=\frac{4}{\sqrt{36}} =\frac{4}{6} =0.667\ \text{beats per minute per exercise hour}.

With residual variation held equal, adding the correlated predictor raises the standard error from 0.400 to 0.667. Exercise and walking hours overlap substantially in this hypothetical sample, leaving less distinct variation with which to estimate the exercise coefficient while holding walking hours fixed. The result concerns precision, not whether either specification is the right one.

Worked Example: A Lower Residual Standard Deviation Can Offset Overlap

In a separate hypothetical analysis, a researcher relates a weekly symptom score to hours of sleep \(x\). The simple model has \(S_{xx}=80\) hours squared, residual standard deviation \(s=6\) score points, and estimated slope \(-1.25\) score points per additional sleep hour. A model adding work hours \(z\) has \(r_{xz}=0.50\), residual standard deviation \(s=4\) points, and estimated sleep coefficient \(-1.10\) score points per hour. Compare the sleep coefficient standard errors.

For the simple model:

SE(\hat{\beta}_1)=\frac{6}{\sqrt{80}} =\frac{6}{8.944} =0.671\ \text{score points per hour}.

For the model with work hours, the remaining sum of squares for sleep is:

$$ 80(1-0.50^2)=80(0.75)=60. $$

Thus:

SE(\hat{\beta}_1)=\frac{4}{\sqrt{60}} =\frac{4}{7.746} =0.516\ \text{score points per hour}.

Although sleep and work hours share some variation, the added model’s residual standard deviation is sufficiently smaller that the estimated standard error falls from 0.671 to 0.516. The point estimate also changes, from \(-1.25\) to \(-1.10\). These coefficients answer different questions: the first is unadjusted for work hours, while the second is adjusted for them. Neither the smaller standard error nor the change in estimate, by itself, establishes which specification is more useful.

Removing a Term Can Change the Coefficient Too

Changing a specification can change the estimated coefficient as well as its standard error. The following identity illustrates what happens when a linear predictor \(z\) is removed from a model with \(x\) and \(z\). For the same observations and an intercept in both regressions, the slope from the reduced regression of \(Y\) on \(x\) equals the full model’s \(x\) coefficient plus the full model’s \(z\) coefficient multiplied by the slope from regressing \(z\) on \(x\):

\hat{\beta}_{x,\text{reduced}} = \hat{\beta}_{x,\text{full}} + \hat{\beta}_{z,\text{full}}\hat{\gamma}_x, \qquad \hat{\gamma}_x=\frac{S_{xz}}{S_{xx}}.

Here \(S_{xz}=\sum_i(x_i-\bar{x})(z_i-\bar{z})\). The identity follows by substituting the full model’s fitted linear part into the reduced slope formula: the slope of the fitted response on \(x\) is the full \(x\) slope plus the full \(z\) slope times the slope of \(z\) on \(x\). The omitted predictor’s relationship with \(x\) therefore contributes to the reduced coefficient. This is an algebraic comparison of fitted coefficients; deciding which coefficient is substantively relevant requires deciding which conditional relationship the question asks about.

Worked Example: Removing a Predictor Changes the Estimate and Its Uncertainty

Suppose a hypothetical sample of 40 patients is used to model a symptom score \(Y\) from weekly exercise hours \(x\) and age \(z\), with age measured in decades. In the full model, the estimated exercise coefficient is \(-2.0\) score points per hour and the estimated age coefficient is \(1.5\) score points per decade. In the sample, the slope from regressing age on exercise is \(-0.60\) decades per exercise hour. The full model has \(s=3\), \(S_{xx}=120\), and \(r_{xz}=0.60\). The reduced model, containing exercise alone, has \(s=4.5\). Compare the exercise estimates and standard errors.

First, the reduced-model exercise slope is:

\hat{\beta}_{x,\text{reduced}} = -2.0+(1.5)(-0.60) =-2.0-0.90 =-2.90\ \text{score points per hour}.

The full-model standard error for exercise uses the variation in exercise remaining after accounting for age:

SE(\hat{\beta}_{x,\text{full}}) = \frac{3}{\sqrt{120(1-0.60^2)}} = \frac{3}{\sqrt{76.8}} = 0.342\ \text{score points per hour}.

The reduced model’s standard error is:

SE(\hat{\beta}_{x,\text{reduced}}) = \frac{4.5}{\sqrt{120}} = \frac{4.5}{10.954} = 0.411\ \text{score points per hour}.

In this hypothetical comparison, removing age changes the exercise estimate from \(-2.0\) to \(-2.90\) and the standard error from 0.342 to 0.411. The reduced coefficient is unadjusted for age; the full coefficient compares patients at the same age. A change in either value is not automatically evidence of an error: the specifications describe different relationships.

How to Compare Specifications Carefully

A coefficient standard error is meaningful only in relation to its model term and specification. When comparing outputs, first check that the response, observations, measurement units, and predictor coding match. Then identify what each coefficient means. If one model adjusts for a predictor and another does not, the coefficient estimates may not target the same conditional association.

It is also useful to separate three sources of change:

Residual variation: a model that explains more of the response variation may have a smaller \(s\), which tends to reduce standard errors.
Predictor overlap: an added term can leave less independent variation in a predictor, increasing its standard error when other quantities are held fixed.
Residual degrees of freedom: adding a coefficient changes \(p\), and therefore changes the estimate \(s=\sqrt{SSE/(n-p)}\) as well as potentially changing \(SSE\).

A smaller standard error means less estimated sampling variability for that coefficient under that specification and its assumptions. It does not mean the model is automatically better, the coefficient is unbiased for every scientific question, or the relationship is causal. Model terms should be guided by the question and relevant subject knowledge, not selected solely to obtain a smaller standard error.

Common Mistake

Wrong reasoning: “Adding a predictor must make every coefficient less precise, because the predictors become more correlated.” This holds only for the comparison that fixes the residual standard deviation and compares the remaining predictor variation. In an actual refit, the added term may reduce residual variation enough to offset the loss of distinct predictor variation; the degrees of freedom also change.

Correction: Recalculate the standard error for each specification using its own residual standard deviation and its own remaining predictor variation. Also state whether the coefficient’s interpretation changed. A numerical comparison without that context can be misleading.

Key takeaway: Including or removing terms can change both a coefficient estimate and its standard error. Predictor overlap tends to reduce the information available for separating coefficients, while a lower residual standard deviation can improve precision. Interpret each estimate within the specification that produced it.

Check Your Understanding

Answer these questions using the formulas and distinctions in this tutorial.

In a multiple regression, what does the coefficient for \(x\) describe that the simple-regression slope does not?
If the residual standard deviation stays fixed and \(x\) is strongly correlated with an added predictor, what happens to the remaining variation in \(x\), and why can its coefficient standard error increase?
Can adding a predictor reduce a coefficient’s standard error? Identify the two competing changes that determine the result.
A predictor is removed, and the coefficient for \(x\) changes. Why does that change not, by itself, show that one estimate is incorrect?
Why is choosing a regression specification solely because it gives the smallest standard error a poor strategy?