Deriving the Simple Regression Slope Variance

Why Derive the Slope Variance?

A least-squares slope is calculated from a sample, so it can vary from one sample of responses to another, even when the predictor values stay the same. The model’s error variability is one source of this uncertainty. The arrangement of the predictor values is another: a slope is more or less sensitive to response changes depending on how far the predictor values lie from their mean.

The aim here is to derive the variance of the least-squares slope, rather than to estimate that variance from a particular data set. The derivation separates the roles of the model’s error variance and the centered predictor values. It also makes clear which assumptions are needed for the result.

Model and Assumptions

Suppose there are \(n\) observations with fixed predictor values \(x_1,\ldots,x_n\). The simple linear regression model for the response \(Y_i\) is

Y_i=\beta_0+\beta_1x_i+\varepsilon_i,\qquad i=1,\ldots,n.

Here \(\beta_0\) is the population intercept, \(\beta_1\) is the population slope, and \(\varepsilon_i\) is the error for observation \(i\). The derivation uses these assumptions:

The conditional mean follows the stated linear model: \(E(Y_i\mid x_i)=\beta_0+\beta_1x_i\), or equivalently \(E(\varepsilon_i\mid x_1,\ldots,x_n)=0\).
Given the fixed predictor values, the errors are independent.
Each error has the same finite variance, \(\operatorname{Var}(\varepsilon_i\mid x_1,\ldots,x_n)=\sigma^2\).
The predictor values are not all identical, so there is information to estimate a slope.

The common error variance \(\sigma^2\) describes response variation around the model’s mean at a fixed predictor value. The derivation does not require normally distributed errors. Normality is needed for some exact distributional results used in regression inference, but not for this variance calculation.

Definition: Write \(\bar{x}=\sum_{i=1}^n x_i/n\) for the mean of the fixed predictor values, and define their centered sum of squares by \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\). When \(S_{xx}>0\), the conditional variance of the least-squares slope under the assumptions above is \(\operatorname{Var}(\hat{\beta}_1\mid x_1,\ldots,x_n)=\sigma^2/S_{xx}\).

Deriving the Formula

For simple linear regression with an intercept, the least-squares slope can be written as the centered cross-product of predictor and response values divided by the centered sum of squares of the predictor values:

\hat{\beta}_1= \frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{S_{xx}}, \qquad \bar{Y}=\frac{1}{n}\sum_{i=1}^nY_i.

This formula follows from the least-squares fit with an intercept: the fitted line passes through \((\bar{x},\bar{Y})\), and minimizing the sum of squared residuals gives the stated ratio when \(S_{xx}>0\). To find the slope’s variance, substitute the model for each \(Y_i\). The intercept and the average response disappear from the slope’s error expression because the centered predictor values sum to zero.

Substitute the model.
Since \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), centering the response gives \(Y_i-\bar{Y}=\beta_1(x_i-\bar{x})+(\varepsilon_i-\bar{\varepsilon})\), where \(\bar{\varepsilon}=\sum_i\varepsilon_i/n\).

Simplify the slope.
In the cross-product, the term involving \(\bar{\varepsilon}\) vanishes because \(\sum_i(x_i-\bar{x})=0\). Thus \(\hat{\beta}_1=\beta_1+\sum_i(x_i-\bar{x})\varepsilon_i/S_{xx}\).

Take the conditional variance.
The fixed values \(x_i\) and the denominator \(S_{xx}\) are constants. Independent errors with common variance \(\sigma^2\) therefore give \(\operatorname{Var}(\hat{\beta}_1\mid x_1,\ldots,x_n)=\sigma^2\sum_i(x_i-\bar{x})^2/S_{xx}^2=\sigma^2/S_{xx}\).

The last step uses the variance rule for a weighted sum of independent errors. In detail, if \(a_i=(x_i-\bar{x})/S_{xx}\), then the random part of the slope is \(\sum_i a_i\varepsilon_i\), and its variance is \(\sum_i a_i^2\operatorname{Var}(\varepsilon_i)\). Independence makes the covariance terms zero, and the common variance makes this \(\sigma^2\sum_i a_i^2\). Since \(\sum_i(x_i-\bar{x})^2=S_{xx}\), the result is \(\sigma^2/S_{xx}\).

Derivation result: With fixed predictor values, independent errors of mean zero and common variance \(\sigma^2\), and \(S_{xx}>0\), the slope is unbiased conditional on those predictor values and has variance \(\sigma^2/S_{xx}\). Its standard deviation across repeated responses with the same predictor values is \(\sigma/\sqrt{S_{xx}}\). The latter is the standard error when \(\sigma\) is known.

Worked Examples

Worked Example: Calculating the Variance From Fixed Predictor Values

In a hypothetical setting, a response is measured in milligrams per litre and a predictor in grams. The fixed predictor values are \(0,1,3,4\) grams. Assume the stated linear mean model, independent errors with mean zero, and a common error standard deviation of \(2\) milligrams per litre. Find the variance of the least-squares slope.

First calculate \(\bar{x}=(0+1+3+4)/4=2\) grams. Then

S_{xx}=(0-2)^2+(1-2)^2+(3-2)^2+(4-2)^2 =4+1+1+4=10\ \text{grams}^2.

The common error variance is \(\sigma^2=(2\ \text{mg/L})^2=4\ (\text{mg/L})^2\). Applying the derived formula,

\operatorname{Var}(\hat{\beta}_1\mid x_1,\ldots,x_4) =\frac{4\ (\text{mg/L})^2}{10\ \text{grams}^2} =0.400\ \frac{(\text{mg/L})^2}{\text{gram}^2}.

The corresponding standard deviation is \(\sqrt{0.400}=0.6325\) milligrams per litre per gram. This describes the slope’s sampling variability under the specified model and fixed predictor values; it is not the standard deviation of individual responses around the regression line.

Worked Example: Seeing the Error Weights in the Slope

Consider a hypothetical model in which the predictor values are fixed at \(2,5,8\) hours and the response is measured in points. Assume the linear mean model and independent, mean-zero errors, each with variance \(9\) points squared. Show the error weights in the slope and use them to calculate its variance.

The predictor mean is \(\bar{x}=(2+5+8)/3=5\) hours, so \(S_{xx}=(-3)^2+0^2+3^2=18\) hours squared. From the slope expression, the coefficient multiplying each error is \((x_i-\bar{x})/S_{xx}\). The three weights are therefore \(-3/18=-1/6\), \(0/18=0\), and \(3/18=1/6\) per hour.

The response errors enter the slope as \(-\varepsilon_1/6+\varepsilon_3/6\); the error for the observation at the mean predictor value has weight zero. Independence means the variance of this weighted sum is the sum of the weighted error variances:

\operatorname{Var}(\hat{\beta}_1\mid x_1,x_2,x_3) =\left(\frac{1}{6}\right)^2(9)+0^2(9)+\left(\frac{1}{6}\right)^2(9) =\frac{18}{36}=0.500\ \frac{\text{points}^2}{\text{hour}^2}.

Equivalently, the formula gives \(9/18=0.500\) points squared per hour squared. The example shows why the centered predictor values matter: they determine each response error’s weight in the slope.

Worked Example: Comparing Two Predictor Arrangements

Two hypothetical designs each use four observations and the same response scale. Assume in both that the conditional mean is linear, errors are independent with mean zero, and the error variance is \(\sigma^2=4\) response-units squared. Design A has fixed predictor values \(0,1,2,3\); Design B has values \(0,2,4,6\). Calculate the slope variance for each.

For Design A, \(\bar{x}=1.5\), so

S_{xx,A}=(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2=5.

Thus its slope variance is \(4/5=0.800\) response-units squared per predictor-unit squared.

For Design B, \(\bar{x}=3\), and

S_{xx,B}=(-3)^2+(-1)^2+(1)^2+(3)^2=20.

Its slope variance is \(4/20=0.200\) response-units squared per predictor-unit squared. Both calculations use the same error variance and sample size, but the centered predictor values differ. The formula gives a smaller slope variance for Design B because its \(S_{xx}\) is larger. This is a statement about the variance under the assumed models and designs, not evidence that either fitted slope is closer to a causal effect.

What the Formula Does—and Does Not—Say

The numerator, \(\sigma^2\), represents the common error variance: greater response variation around the linear mean produces greater slope variance. The denominator, \(S_{xx}\), summarizes the squared distances of the predictor values from their mean. It is not the uncentered sum \(\sum_i x_i^2\). Centering matters because a common shift in all predictor values changes the intercept’s reference point, but does not change the information in their relative positions for estimating the slope.

The formula is conditional on the fixed predictor values. If predictor values are treated as random in a study, the same calculation can be understood conditional on the observed values, provided the model assumptions hold conditionally. The formula is not justified merely by having a straight-line fitted to the data: the error assumptions support the variance calculation.

The assumptions also explain how the result could change. If error variances differ across observations, the step that factors out one common \(\sigma^2\) no longer applies. If errors are correlated, covariance terms need not be zero. In either case, \(\sigma^2/S_{xx}\) is not generally the correct variance. The linear conditional mean and nonidentical predictor requirement matter too: if all predictor values are equal, \(S_{xx}=0\), so a slope cannot be estimated from these data.

Common Mistake

Pitfall: Using \(\sum_i x_i^2\) in the denominator because it looks like a measure of predictor size. The slope formula with an intercept depends on \(S_{xx}=\sum_i(x_i-\bar{x})^2\), not \(\sum_i x_i^2\). The correction is to center the predictor values first. For example, adding the same constant to every \(x_i\) changes \(\sum_i x_i^2\), but leaves every \(x_i-\bar{x}\), and therefore \(S_{xx}\), unchanged.

Another important distinction is between a variance and a standard deviation. The slope variance is \(\sigma^2/S_{xx}\); its square root is the standard deviation of the slope across repeated responses under the model. Their units differ: the variance has squared slope units, while the standard deviation has slope units.

Key takeaway: Under a linear conditional mean with fixed, nonidentical predictor values and independent, equal-variance errors, the least-squares slope has conditional variance \(\sigma^2/S_{xx}\). Error variability sets the numerator; squared centered predictor values set the denominator.

Check Your Understanding

Answer each question using the model and assumptions stated in the tutorial.

What does \(S_{xx}\) measure, and why must it be positive to estimate a slope?
For fixed predictor values, how does the slope variance change if the common error variance is multiplied by four?
Why do the error covariances disappear from the variance derivation under the independent-errors assumption?
In the three-point predictor example, why does the error at the mean predictor value have zero weight in the slope?
Would the formula \(\sigma^2/S_{xx}\) necessarily hold if errors were correlated or had different variances? Explain.