Standard Errors in Multiple Regression

Why a Multiple-Regression Coefficient Has a Standard Error

A multiple-regression coefficient describes the association between one predictor and the response while the other included predictors are held fixed. Its estimate varies from sample to sample, even when the underlying relationship is unchanged. The standard error estimates the standard deviation of that sampling variation.

In simple regression, the precision of a slope depends in part on how much the predictor varies. In multiple regression, the relevant variation is more specific: it is the part of the predictor that cannot be explained by the other predictors in the model. If two predictors largely move together, there may be little information in the data to distinguish their separate associations with the response.

Model and Definition

Consider a multiple linear regression with \(p\) predictors and \(n\) observations:

Here, \(Y_i\) is the response for observation \(i\), \(x_{ij}\) is its value of predictor \(j\), \(\beta_0\) is the intercept, and \(\beta_j\) is the population coefficient for predictor \(j\). The fitted coefficient is \(b_j\). For the conventional standard error, assume the conditional mean is correctly represented by this linear model, the errors have mean zero and common variance \(\sigma^2\), and errors from different observations are uncorrelated. The predictor values are treated as fixed for the calculation. Normal errors are not needed for the variance formula itself.

To state the formula, first regress predictor \(j\) on the other predictors and an intercept. For each observation, subtract the value predicted by that auxiliary regression from the observed \(x_{ij}\). Call this difference \(q_{ij}\). These differences are the residualized predictor values; their sum of squares, \(\sum_{i=1}^n q_{ij}^2\), measures the predictor’s remaining variation. It is sometimes called its partial sum of squares in this context.

Let \(SSE\) be the sum of squared residuals from the response regression. With \(p\) predictors, the residual degrees of freedom are \(n-p-1\). If the design matrix has full column rank and \(n>p+1\), the error variance estimate is \(s^2=SSE/(n-p-1)\), and the conventional coefficient standard error is:

Full column rank means that no included predictor is an exact linear combination of the other predictors and the intercept. In particular, \(\sum q_{ij}^2\) must be positive for the coefficient to be separately estimable. The condition \(n>p+1\) provides positive residual degrees of freedom so that \(s^2\) can be estimated. If \(n=p+1\), this conventional estimate and standard error are unavailable.

Why the Formula Has These Two Parts

The formula separates two sources of uncertainty. The numerator \(s\) measures the typical unexplained response variation around the fitted regression. The denominator measures the predictor’s usable information after the other included predictors are taken into account. More unexplained variation tends to make coefficient estimates less stable; more unique predictor variation tends to make them more stable.

A short derivation makes the role of residualized predictor variation precise. Let \(Z\) contain the intercept column and all predictors except predictor \(j\), and let \(x_j\) be the column of observed values for predictor \(j\). Define \(q_j\) as the residual vector from regressing \(x_j\) on \(Z\). In matrix notation, when \(Z\) has full column rank,

The bracketed matrix removes the part explained by \(Z\), so \(q_j\) is orthogonal to every column of \(Z\). This means the multiple-regression estimate for predictor \(j\) is the slope obtained by relating the part of the response left after accounting for \(Z\) to \(q_j\). Equivalently, substituting the model into that slope gives:

The equality follows because \(q_j\) is orthogonal to the intercept and the other predictors, while \(q_j^\mathsf{T}x_j=q_j^\mathsf{T}q_j\). With uncorrelated errors of common variance \(\sigma^2\), the variance of \(q_j^\mathsf{T}\varepsilon\), conditional on the predictor values, is \(\sigma^2q_j^\mathsf{T}q_j\). Therefore:

Replacing the unknown \(\sigma^2\) with \(s^2\) gives the conventional estimated standard error. This derivation explains why raw predictor spread alone is not enough: only the spread that remains after adjustment for the other included predictors appears in the denominator.

What the Standard Error Tells You—and What It Does Not

A coefficient’s standard error is expressed in the same units as that coefficient. If the response is in points and the predictor is in hours, the standard error is in points per hour. It describes estimated sampling variability, not the coefficient’s size and not the amount of unexplained variation in the response.

The two components in the formula are useful for understanding practical differences between coefficients. A predictor may have a large raw range but little variation that is distinct from the other predictors. Its coefficient can then have a relatively large standard error. Conversely, substantial unique variation can improve precision, provided the residual variability and other conditions are comparable. No single predictor’s standard error should be interpreted without considering which other predictors are included: changing the adjustment set can change the residualized predictor values and the fitted residuals.

The formula is a model-based calculation. The assumptions about the error mean, common variance, and lack of correlation are not proved by the formula, and summary quantities such as \(SSE\) and residualized predictor values cannot verify them by themselves. In an actual analysis, the model specification and residual behavior need to be considered. If the conventional assumptions are seriously inappropriate, the conventional standard error may not describe the estimator’s sampling variability well.

Common Mistake