Standard Errors for Regression Coefficients

Why Coefficient Standard Errors Matter

A fitted regression coefficient summarizes a relationship estimated from a particular sample. If a different sample were collected under the same conditions, the estimate would generally change. A coefficient’s standard error measures the estimated sampling variability of that coefficient: smaller values indicate estimates that would tend to vary less from sample to sample, given the model and predictor information.

A standard error is not the same as the residual standard deviation. The residual standard deviation describes typical variation in observations around the regression mean. A coefficient standard error describes variation in an estimated coefficient across repeated samples. The two are connected, but the predictor values and their arrangement also affect coefficient standard errors.

This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. It focuses on calculating and interpreting standard errors, not on using them to conduct tests or construct confidence intervals.

Model and Standard Error Formula

Consider a linear regression with $n$ observations and $p$ fitted coefficients, including the intercept when the model has one. Write the model in matrix form as $Y=X\beta+\varepsilon$, where $Y$ is the $n$-vector of responses, $X$ is the $n$-by-$p$ design matrix, and $\beta$ is the vector of unknown coefficients. Assume $X$ has full column rank, so the coefficients are estimable.

For the usual ordinary least squares standard error formula, assume that, conditional on $X$, the errors have mean zero, common variance $\sigma^2$, and zero covariance between distinct observations. In matrix notation, these conditions are $E(\varepsilon\mid X)=0$ and $\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I$. Normal errors are not needed for the variance formula itself. They are needed, together with these conditions, for the exact normal sampling distribution of the coefficient estimates.

Definition: The estimated error variance is $s^2=\mathrm{SSE}/(n-p)$, where $\mathrm{SSE}$ is the sum of squared residuals. The estimated covariance matrix of the least squares coefficient estimates is $s^2(X^\mathsf{T}X)^{-1}$. The standard error of coefficient $\hat{\beta}_j$ is the square root of the $j$th diagonal entry of this matrix: $$ \operatorname{SE}(\hat{\beta}_j) = \sqrt{s^2\left[(X^\mathsf{T}X)^{-1}\right]_{jj}}. $$

The degrees of freedom are $n-p$, not $n-1$ unless $p=1$, which is not the usual regression setup with an intercept. Each estimated coefficient uses information from the data; accounting for all $p$ fitted coefficients gives the residual degrees of freedom.

Why the Matrix Formula Holds

The formula follows from how the least squares estimator depends on the observed responses. With full column rank, the estimator is $\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y$. Substitute the model $Y=X\beta+\varepsilon$:

\hat{\beta} =(X^\mathsf{T}X)^{-1}X^\mathsf{T}(X\beta+\varepsilon) =\beta+(X^\mathsf{T}X)^{-1}X^\mathsf{T}\varepsilon.

Conditional on $X$, the first term is fixed and the second is a linear transformation of the errors. For any fixed matrix $A$, the covariance of $A\varepsilon$ is $A\operatorname{Var}(\varepsilon\mid X)A^\mathsf{T}$. Taking $A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ and using $\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I$ gives:

\operatorname{Var}(\hat{\beta}\mid X) =\sigma^2 (X^\mathsf{T}X)^{-1}X^\mathsf{T}X(X^\mathsf{T}X)^{-1} =\sigma^2(X^\mathsf{T}X)^{-1}.

Because $\sigma^2$ is unknown, replace it with $s^2$ to estimate the covariance matrix. A coefficient’s estimated variance is the corresponding diagonal entry; taking its square root produces its standard error. This also shows why the formula depends on both the estimated error variance and the predictor information in $X$.

Simple Linear Regression Formulas

For simple linear regression with an intercept, let $x_i$ be the predictor value for observation $i$, and define $\bar{x}$ as the sample mean of the predictor and $S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2$. The matrix formula reduces to familiar expressions:

\operatorname{SE}(\hat{\beta}_1)=\frac{s}{\sqrt{S_{xx}}}, \qquad \operatorname{SE}(\hat{\beta}_0) =s\sqrt{\frac{1}{n}+\frac{\bar{x}^2}{S_{xx}}}, \qquad s=\sqrt{\frac{\mathrm{SSE}}{n-2}}.

The slope standard error decreases when the predictor values are more spread out, because a wider range of $x$ provides more information about the slope. The intercept standard error also depends on where zero lies relative to the observed predictor values. If the predictor values are far from zero, estimating the response mean at $x=0$ can be less precise.

Worked Example: Standard Errors in Simple Regression

A hypothetical insurer models an account’s annual service cost using account tenure $x$, measured in years. A simple linear regression with an intercept is fitted to $n=6$ accounts. The fitted model has $\bar{x}=2.5$, $S_{xx}=17.5$, and $\mathrm{SSE}=20$. Calculate the estimated error variance and the standard errors of the slope and intercept.

Method. There are two fitted coefficients, the intercept and slope, so the residual degrees of freedom are $n-p=6-2=4$. Thus $s^2=20/4=5$, and $s=\sqrt{5}$.

For the slope, use $S_{xx}=17.5$: $$ \operatorname{SE}(\hat{\beta}_1) =\sqrt{\frac{s^2}{S_{xx}}} =\sqrt{\frac{5}{17.5}} =\sqrt{\frac{2}{7}} \approx 0.5345. $$

For the intercept: $$ \operatorname{SE}(\hat{\beta}_0) =\sqrt{5\left(\frac{1}{6}+\frac{2.5^2}{17.5}\right)} =\sqrt{5\left(\frac{1}{6}+\frac{6.25}{17.5}\right)} =\sqrt{5\left(\frac{11}{21}\right)} =\sqrt{\frac{55}{21}} \approx 1.6183. $$

Check and trap avoided. The estimated error variance is $5$, so the residual standard deviation is $\sqrt{5}$, not $5$. The coefficient standard errors use $s^2$ together with predictor information; neither coefficient standard error is simply the residual standard deviation. Rounded to four decimals, the slope and intercept standard errors are $0.5345$ and $1.6183$, respectively.

Multiple Regression and Predictor Information

In multiple regression, the diagonal of $(X^\mathsf{T}X)^{-1}$ accounts for the way each predictor relates to the other predictors, as well as its own spread. A predictor with little variation after accounting for the other predictors provides limited information for separating its coefficient from the others. This can make its coefficient standard error large.

When the intercept and predictor columns are centered and mutually orthogonal, the calculation is especially direct: the diagonal entries of $X^\mathsf{T}X$ are $n$ for the intercept and the sums of squares for the predictors. When predictor columns are correlated, the inverse generally has larger diagonal entries than it would with the same predictor sums of squares but no correlation. This is one way that collinearity makes individual coefficient estimates less precise.

Worked Example: Orthogonal Predictors in Multiple Regression

A hypothetical model predicts monthly claim-processing time using two centered predictors: the number of documents submitted and the number of prior contacts. There are $n=8$ observations and three coefficients: an intercept and two slopes. The predictor columns are orthogonal, with sums of squares 16 and 8. The fitted model has $\mathrm{SSE}=18$. Find the standard error of each coefficient.

Method. The residual degrees of freedom are $8-3=5$, so $s^2=18/5=3.6$. Since the predictor columns are centered, each has sum zero and is orthogonal to the intercept column. With the predictors also orthogonal to each other, $X^\mathsf{T}X$ is diagonal, with entries $8$, $16$, and $8$. Its inverse has diagonal entries $1/8$, $1/16$, and $1/8$.

The intercept standard error is: $$ \operatorname{SE}(\hat{\beta}_0) =\sqrt{3.6\left(\frac{1}{8}\right)} =\sqrt{0.45} \approx 0.6708. $$

The first slope standard error is: $$ \operatorname{SE}(\hat{\beta}_1) =\sqrt{3.6\left(\frac{1}{16}\right)} =\sqrt{0.225} \approx 0.4743. $$

The second slope standard error is: $$ \operatorname{SE}(\hat{\beta}_2) =\sqrt{3.6\left(\frac{1}{8}\right)} =\sqrt{0.45} \approx 0.6708. $$

Check and trap avoided. The second predictor has half the sum of squares of the first, so its coefficient standard error is larger, despite both predictors being measured in this same model. The residual standard deviation is $\sqrt{3.6}\approx1.8974$, which is not any of the three coefficient standard errors. Rounded to four decimals, the standard errors are $0.6708$, $0.4743$, and $0.6708$.

Worked Example: Correlated Predictors and Larger Standard Errors

A hypothetical risk model uses two centered predictors to estimate a policy’s annual administrative expense. There are $n=8$ observations and an intercept, so $p=3$. The intercept column is orthogonal to both centered predictors, and the predictor cross-product matrix is: $$ \begin{pmatrix} 10 & 6\\ 6 & 10 \end{pmatrix}. $$ The model’s $\mathrm{SSE}$ is 12. Calculate the two slope standard errors and compare them with the values that would result if the predictors were orthogonal with the same sums of squares.

Method. The residual degrees of freedom are $8-3=5$, giving $s^2=12/5=2.4$. For the two predictor columns, the determinant of the cross-product matrix is $10(10)-6(6)=64$. Its inverse is: $$ \begin{pmatrix} 10 & 6\\ 6 & 10 \end{pmatrix}^{-1} = \frac{1}{64} \begin{pmatrix} 10 & -6\\ -6 & 10 \end{pmatrix}. $$

Both slope variances use the diagonal entry $10/64$, so: $$ \operatorname{SE}(\hat{\beta}_1) =\operatorname{SE}(\hat{\beta}_2) =\sqrt{2.4\left(\frac{10}{64}\right)} =\sqrt{0.375} \approx 0.6124. $$

If the predictors were orthogonal but still had sums of squares 10 and 10, the corresponding inverse diagonal entries would each be $1/10$. Each slope standard error would then be: $$ \sqrt{2.4\left(\frac{1}{10}\right)} =\sqrt{0.24} \approx 0.4899. $$

Check and trap avoided. The actual standard errors, $0.6124$, are larger than $0.4899$ because the predictors are correlated. Using $1/10$ as an inverse diagonal entry would incorrectly treat the predictors as orthogonal. The off-diagonal entry of the inverse is $-6/64$, reflecting their joint relationship, but it is the diagonal entries that give the individual coefficient variances. Rounded to four decimals, the actual slope standard errors are both $0.6124$.

Interpreting and Checking Standard Errors

A coefficient standard error is conditional on the model and the observed predictor information. It does not describe uncertainty caused by choosing the wrong model, omitted predictors, measurement error, or violations of the error assumptions. If the errors have unequal variances or are correlated, the usual covariance formula $s^2(X^\mathsf{T}X)^{-1}$ need not be appropriate.

A useful calculation sequence is:

Count fitted coefficients.
Include the intercept when present, then calculate residual degrees of freedom as $n-p$.

Estimate error variance.
Calculate $s^2=\mathrm{SSE}/(n-p)$; use $s^2$, not $s$, in the covariance formula.

Use predictor information.
Find the relevant diagonal entry of $(X^\mathsf{T}X)^{-1}$, or use the simple-regression formulas when applicable.

Take the square root and interpret.
The result is in the units of the coefficient and estimates its sampling variability under the stated model.

A common exam trap is to report $s=\sqrt{\mathrm{SSE}/(n-p)}$ as the standard error of every coefficient. That quantity estimates the error standard deviation, not a coefficient standard error. Another trap is to ignore predictor geometry: two coefficients from one model can have different standard errors even though they share the same estimated error variance.

Key takeaway: Under the ordinary least squares assumptions, coefficient uncertainty is estimated by $s^2(X^\mathsf{T}X)^{-1}$. The estimated error variance sets the overall scale, while the design matrix determines how much information the predictors provide about each coefficient.

Check Your Understanding

For each problem, calculate the requested quantity before checking the answer.

A simple regression with an intercept uses $n=10$ observations, has $S_{xx}=45$, and has $\mathrm{SSE}=24$. Calculate the residual degrees of freedom, $s^2$, and the slope standard error. (Answer: $8$, $3$, and $0.2582$, respectively.)
A multiple regression has an intercept and three mutually orthogonal, centered predictors, with $n=12$ and $\mathrm{SSE}=32$. The predictor sums of squares are 16, 25, and 4. Calculate the intercept standard error and the three slope standard errors, in that order. (Answer: $0.5774,\ 0.5000,\ 0.4000,\ 1.0000$.)
A regression with an intercept and two centered predictors has $n=9$, $\mathrm{SSE}=30$, and predictor cross-product matrix $\begin{pmatrix}20&10\\10&8\end{pmatrix}$. Find both slope standard errors. The determinant of the matrix is $20(8)-10(10)=60$, so its inverse diagonal entries are $8/60$ and $20/60$. (Answer: $0.8165$ for the first slope and $1.2910$ for the second.)
A regression has $n=15$, $p=3$, $\mathrm{SSE}=45$, and the relevant diagonal entry of $(X^\mathsf{T}X)^{-1}$ is $0.08$. Calculate the standard error of that coefficient. (Answer: $0.5477$, since $s^2=45/12=3.75$ and $\sqrt{3.75(0.08)}=\sqrt{0.3}$.)
In ordinary least squares, what does a coefficient standard error describe, and why is it not generally equal to the residual standard deviation? (Answer: It estimates the coefficient’s sampling variability under the model; it depends on both the estimated error variance and the predictor information in $(X^\mathsf{T}X)^{-1}$, whereas the residual standard deviation estimates the error scale.)

What You'll Learn