How Residual Variability Affects Coefficient Precision

Why Residual Variability Matters

Regression estimates a relationship between a response and one or more predictors. Even when the predictor values are held fixed, the response values may lie close to the fitted relationship in one setting and scatter farther from it in another. That scatter affects how precisely the regression coefficients can be estimated: with more residual variability, a coefficient estimate typically has greater sampling uncertainty.

The key comparison in this tutorial keeps the predictor design and sample size unchanged. This isolates the role of residual variability. It does not mean that residual scatter changes the coefficient estimate by a fixed amount; rather, it changes the estimated standard error, and therefore the width of a confidence interval.

Definition: In a regression model, the residual standard deviation \(s\) estimates the standard deviation \(\sigma\) of the model errors, in response-variable units. A coefficient standard error estimates the sampling standard deviation of that coefficient estimate, in coefficient units. The two quantities are related, but they are not interchangeable.

The Fixed-Design Relationship

For a multiple linear regression with \(n\) observations and \(p\) predictors, let \(X\) be the design matrix, including a column for the intercept. Assume the model is linear in its coefficients, the errors have mean zero and a common variance \(\sigma^2\), observations have independent errors, and \(X\) has full column rank. For conventional coefficient confidence intervals, the errors are also assumed to be normally distributed. The residual variance is estimated using \(s^2\), with \(df=n-p-1\).

If \(\hat{\beta}_j\) is the estimate of coefficient \(\beta_j\), its estimated standard error is

SE(\hat{\beta}_j)=s\sqrt{\left[(X^\mathsf{T}X)^{-1}\right]_{jj}}.

The diagonal entry depends on the predictor design, not on the response values. Consequently, when the same observations, predictors, and model terms are used, that entry stays fixed. The standard error then changes in direct proportion to \(s\): multiplying the residual standard deviation by a factor multiplies every coefficient standard error by that same factor.

A conventional two-sided confidence interval for \(\beta_j\) is

\hat{\beta}_j \ \pm\ t^* SE(\hat{\beta}_j),

where \(t^*\) is the appropriate \(t\) critical value for the chosen confidence level and residual degrees of freedom. If the sample size and model remain the same, \(df\) and \(t^*\) remain the same too. The full interval width is \(2t^*SE(\hat{\beta}_j)\), so it also increases in direct proportion to \(s\).

Key result: With the same full-rank predictor design, sample size, and model terms, increasing the residual standard deviation increases each coefficient standard error proportionally. It also widens each conventional coefficient confidence interval proportionally, provided the confidence level and degrees of freedom are unchanged.

Worked Paired Comparisons

The examples below compare hypothetical regression fits. In each pair, the predictor design is held fixed; only the residual variability differs. The conventional interval calculations use the model assumptions stated above. In practice, those assumptions should be assessed for the analysis at hand.

Worked Example: Doubling Residual Variability

Suppose two hypothetical regressions use the same 10 clinic dose values to estimate the association between dose, in dose units, and a response, in outcome points. Both fits give a dose coefficient estimate of \(0.800\) outcome points per dose unit. The dose coefficient’s design factor, \(\sqrt{[(X^\mathsf{T}X)^{-1}]_{jj}}\), is \(0.1741\) per dose unit in both fits. The residual standard deviations are \(1.00\) outcome point for Fit A and \(2.00\) outcome points for Fit B. There are no other predictors, so \(df=10-1-1=8\). Assume the stated regression conditions are appropriate.

For Fit A, the estimated standard error is

SE_A=1.00(0.1741)=0.1741\text{ outcome points per dose unit}.

For Fit B, it is

SE_B=2.00(0.1741)=0.3482\text{ outcome points per dose unit}.

Thus, doubling the residual standard deviation doubles the coefficient standard error. For a 95% interval with \(df=8\), use \(t^*=2.306\). For Fit A, the margin of error is \(2.306(0.1741)=0.4014746\), giving

0.800\pm 2.306(0.1741) =[0.3985,\ 1.2015]\text{ outcome points per dose unit}.

For Fit B, the margin is \(2.306(0.3482)=0.8029492\), so

0.800\pm 2.306(0.3482) =[-0.0029,\ 1.6029]\text{ outcome points per dose unit}.

The estimates are the same, but Fit B’s interval is twice as wide because its standard error is twice as large. Under the usual repeated-sampling interpretation of a 95% confidence procedure, the method is designed to cover the fixed coefficient in about 95% of repeated samples when its assumptions hold. The particular interval does not assign a 95% probability to the coefficient.

Worked Example: Residual Variability in a Multiple Regression

Consider two hypothetical models using the same patients, predictor values, and terms, including dose and other adjustment variables. The estimated dose coefficient in both models is \(10.0\) outcome points per dose unit. The dose entry on the diagonal of \((X^\mathsf{T}X)^{-1}\) is \(0.25\) per dose unit squared, so its square root is \(0.500\) per dose unit. The models have \(df=18\); their residual standard deviations are \(3.00\) and \(6.00\) outcome points. Assume the independent, equal-variance, normal-error model is appropriate.

For the first fit,

SE_1=3.00\sqrt{0.25}=3.00(0.500)=1.50 \text{ outcome points per dose unit}.

For the second fit,

SE_2=6.00\sqrt{0.25}=6.00(0.500)=3.00 \text{ outcome points per dose unit}.

The diagonal entry has units of inverse dose squared; its square root has units of inverse dose. Multiplying by the residual standard deviation in outcome points therefore gives a standard error in outcome points per dose unit, as required.

For a 95% interval with \(df=18\), take \(t^*=2.101\). The first interval has margin \(2.101(1.50)=3.1515\), giving \([6.8485,\ 13.1515]\) outcome points per dose unit. The second has margin \(2.101(3.00)=6.303\), giving \([3.697,\ 16.303]\) outcome points per dose unit. With the design and degrees of freedom fixed, doubling \(s\) doubled the standard error and the interval width.

Worked Example: A 50% Increase in Residual Standard Deviation

Two hypothetical regressions use the same design and model, with \(df=18\). For a particular coefficient, suppose the square root of the relevant diagonal entry of \((X^\mathsf{T}X)^{-1}\) is \(0.200\) per predictor unit, and both coefficient estimates are \(-0.500\) response units per predictor unit. The residual standard deviations are \(2.40\) and \(3.60\) response units. Assume the model conditions for a conventional interval are reasonable.

The standard errors are

SE_1=2.40(0.200)=0.480,\qquad SE_2=3.60(0.200)=0.720 \text{ response units per predictor unit}.

The residual standard deviation increases by a factor of \(3.60/2.40=1.50\), and the standard error also increases by \(0.720/0.480=1.50\). Using \(t^*=2.101\), the interval margins are \(2.101(0.480)=1.00848\) and \(2.101(0.720)=1.51272\). The intervals are therefore approximately \([-1.5085,\ 0.5085]\) and \([-2.0127,\ 1.0127]\), respectively, in response units per predictor unit. Their widths are \(2(1.00848)=2.01696\) and \(2(1.51272)=3.02544\). The second width is 1.50 times the first, matching the increase in residual standard deviation.

Why the Coefficient Estimate Need Not Change

Residual variability describes how far the observed responses depart from the fitted values. It is possible for two response sets to have the same fitted coefficient estimate but residuals of different sizes. For example, multiplying every residual from a fit by a positive constant while keeping the fitted values fixed preserves the least-squares orthogonality conditions: the residuals still sum to zero and have zero cross-product with each predictor column. The fitted coefficients therefore remain the same, while the residual standard deviation is multiplied by that constant.

This gives a direct justification for the paired comparisons. In the standard-error formula, the coefficient estimate does not appear; with the design fixed, the design factor does not change either. The change in \(s\) alone accounts for the change in standard error. The interval remains centered on the estimate, but its endpoints move farther from that center as the margin of error grows.

This matters when reading regression output. A coefficient’s estimated size and its precision answer different questions. The coefficient estimate describes the fitted change in the response associated with a one-unit change in the predictor, holding the other included predictors fixed where applicable. Its standard error describes the estimate’s sampling uncertainty under the model. More residual scatter can make an estimate less precise even when its numerical value is unchanged.

Common Mistakes and a Practical Check

Confusing residual standard deviation with coefficient standard error. They generally have different units. Use \(SE(\hat{\beta}_j)=s\sqrt{[(X^\mathsf{T}X)^{-1}]_{jj}}\), rather than reporting \(s\) as the coefficient’s uncertainty.
Assuming a larger \(s\) must change the coefficient estimate. The estimate may stay fixed while its standard error increases; the paired examples demonstrate this possibility.
Comparing intervals while other features change. If the sample size, predictor values, or model terms differ, the design factor or \(t^*\) may differ too. The proportional comparison applies when those features and the confidence level are held fixed.
Using incorrect units for the design factor. For a dose coefficient, the square root of the corresponding diagonal entry has units of inverse dose. Multiplication by residual standard deviation in outcome units gives coefficient units of outcome per dose.
Reading a wider interval as proof of a smaller coefficient. Width concerns precision, not the direction or size of the coefficient estimate. Compare the interval’s center and spread separately.

A useful check is to state what changed before comparing two outputs. If only residual standard deviation changed by a factor of \(c\), with the design, model, degrees of freedom, and confidence level fixed, then each standard error and confidence interval width should also change by a factor of \(c\). If the reported numbers do not follow that relationship, check whether one of the supposedly fixed features actually differs or whether the calculation was rounded inconsistently.

Check Your Understanding

Use the fixed-design relationship and distinguish the coefficient estimate from its uncertainty.

With the same design, \(s\) rises from \(2.0\) to \(5.0\). By what factor does a coefficient standard error change?
Why does a coefficient confidence interval become wider when its standard error increases but its estimate and \(t^*\) stay fixed?
A dose coefficient’s design-factor square root is in inverse dose units, and \(s\) is measured in outcome points. What are the units of the coefficient standard error?
Can two fits have the same coefficient estimate but different standard errors? Explain briefly.
Name two features, other than residual standard deviation, that can affect interval width if they are not held fixed.