Sampling Variability of the Estimated Slope

Why the Estimated Slope Varies

A regression slope summarizes how the mean response changes as the predictor increases. But a slope calculated from a sample is not usually identical to the population slope it aims to estimate. If we collected another sample under the same conditions, the responses would differ because of random error, and the estimated slope could change. Sampling variability describes this sample-to-sample variation.

The key idea is that slope precision depends on two features of the data: how much the responses vary around the regression line, and how widely the observed predictor values are spread. More response noise makes the slope harder to pin down. Greater predictor spread gives the data more leverage to distinguish a shallow slope from a steep one.

Definition: The sampling variability of the estimated slope is the variation in the sample slope \(b_1\) across repeated samples from the same population or under the same sampling model. Its standard deviation is the standard error of \(b_1\). When inference is conditional on observed predictor values, those values are held fixed while the responses vary across repetitions.

The Model and the Slope Formula

For simple linear regression, write the response for observation \(i\) as \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\). Here, \(x_i\) is the predictor value, \(\beta_0\) and \(\beta_1\) are the population intercept and slope, and \(\varepsilon_i\) is the error for that observation. To derive the sampling variability of \(b_1\), treat the observed \(x_i\) values as fixed and assume the errors are independent, have mean zero, and have a common variance \(\sigma^2\). The linear model conditions for slope inference also require a linear mean response; normal errors are needed for an exactly normal sampling distribution, but not for the variance calculation below.

Let \(\bar{x}\) be the sample mean of the predictor, and define the predictor sum of squares as \(S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2\). The ordinary least-squares sample slope can be written as a weighted sum of the responses:

b_1=\frac{\sum_{i=1}^{n}(x_i-\bar{x})Y_i}{S_{xx}}.

This expression makes the source of slope variability visible. Responses far from the predictor mean receive more weight in the numerator, while the denominator measures the total spread of the predictor values.

Deriving the Variance of the Sample Slope

Substitute the model \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\) into the slope formula. The constant intercept term drops out because the centered predictor values sum to zero. The remaining predictor term simplifies to \(\beta_1\), leaving the true slope plus a weighted combination of the errors.

\begin{aligned} b_1 &=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(\beta_0+\beta_1x_i+\varepsilon_i)}{S_{xx}}\\ &=\beta_1+\frac{\sum_{i=1}^{n}(x_i-\bar{x})\varepsilon_i}{S_{xx}}. \end{aligned}

The simplification uses \(\sum(x_i-\bar{x})=0\) and \(\sum(x_i-\bar{x})x_i=S_{xx}\). Since each error has mean zero, the expected value of the weighted error term is zero. Thus, conditional on the predictor values, the sample slope is unbiased for the population slope: \(E(b_1\mid x_1,\ldots,x_n)=\beta_1\).

To find its variance, use independence of the errors. The variance of a sum of independent random variables is the sum of their variances, and multiplying an error by a constant multiplies its variance by the square of that constant. Each error has variance \(\sigma^2\).

\begin{aligned} \operatorname{Var}(b_1\mid x_1,\ldots,x_n) &=\operatorname{Var}\left(\frac{\sum_{i=1}^{n}(x_i-\bar{x})\varepsilon_i}{S_{xx}}\right)\\ &=\frac{1}{S_{xx}^2}\sum_{i=1}^{n}(x_i-\bar{x})^2\operatorname{Var}(\varepsilon_i)\\ &=\frac{\sigma^2}{S_{xx}^2}\sum_{i=1}^{n}(x_i-\bar{x})^2\\ &=\frac{\sigma^2}{S_{xx}}. \end{aligned}

Taking the square root gives the standard deviation of the sample slope. Since the population error standard deviation \(\sigma\) is generally unknown, regression inference estimates it with the residual standard deviation \(s\). The estimated standard error of the slope is therefore:

Result: Conditional on the observed predictor values, the variance of the sample slope is \(\sigma^2/S_{xx}\), and its standard deviation is \(\sigma/\sqrt{S_{xx}}\). In practice, the estimated standard error is \(SE(b_1)=s/\sqrt{S_{xx}}\), where \(s\) is the regression residual standard deviation. The standard error has the same units as the slope: response units per predictor unit.

Under normally distributed errors, the weighted sum of errors in the derivation is normally distributed, so \(b_1\), conditional on the predictors, is normally distributed with mean \(\beta_1\) and variance \(\sigma^2/S_{xx}\). Without normality, the variance result still follows from the stated error assumptions, although the exact sampling distribution need not be normal.

Worked Example: Reading the Formula in a Small Design

Suppose a hypothetical experiment uses four fixed predictor values, \(1,2,3,4\) units. Assume the response errors are independent, have mean zero, and have standard deviation \(\sigma=3\) response units. Find the standard deviation of the sample slope under this model.

The predictor mean is \(\bar{x}=(1+2+3+4)/4=2.5\). Its sum of squares is \(S_{xx}=(1-2.5)^2+(2-2.5)^2+(3-2.5)^2+(4-2.5)^2=5\) predictor-units squared. Therefore,

\operatorname{SD}(b_1\mid x) =\frac{\sigma}{\sqrt{S_{xx}}} =\frac{3}{\sqrt{5}} \approx 1.342.

Across repeated responses at these same four predictor values, the sample slope has a standard deviation of about \(1.342\) response units per predictor unit. This is a model-based standard deviation using the known \(\sigma\); with real data, the estimated standard error substitutes the residual standard deviation \(s\).

How Predictor Spread Affects Precision

The denominator \(S_{xx}\) is the total squared distance of the predictor values from their mean. Holding residual variability constant, a larger \(S_{xx}\) means a smaller slope standard error. This explains why observations spread across a useful range of predictor values can estimate a slope more precisely than observations clustered near the center.

The relevant quantity is not the range alone. \(S_{xx}\) accounts for every predictor value and its distance from the mean. A single extreme value can contribute greatly to \(S_{xx}\), but relying heavily on an extreme observation may make the fitted slope sensitive to that observation. Spread improves the formula-based precision under the model, but it does not remove the need to assess whether the model conditions are reasonable.

Worked Example: Widening the Predictor Values

Compare two hypothetical designs, each with four observations and residual standard deviation \(s=2\) response units. In Design A, the predictor values are \(-2,-1,1,2\) units. In Design B, each value is halved: \(-1,-0.5,0.5,1\) units. Find and compare the estimated slope standard errors.

Both predictor means are zero. For Design A, \(S_{xx}=(-2)^2+(-1)^2+1^2+2^2=10\). For Design B, \(S_{xx}=(-1)^2+(-0.5)^2+0.5^2+1^2=2.5\). Thus:

SE_A(b_1)=\frac{2}{\sqrt{10}}\approx 0.6325, \qquad SE_B(b_1)=\frac{2}{\sqrt{2.5}}\approx 1.265.

Design A has twice the predictor distances from the mean, so its \(S_{xx}\) is four times as large. Its estimated standard error is half as large. The slope is estimated more precisely in Design A, assuming the same residual standard deviation and valid model conditions. The units of both standard errors are response units per predictor unit.

How Sample Size Affects Precision

Adding observations can increase \(S_{xx}\), but the effect depends on where the added predictor values lie. If the new observations follow a similar distribution across the predictor range, the total sum of squares generally grows with sample size. More precisely, \(S_{xx}\) is the sum of squared deviations for all observations, so its size depends jointly on the number of observations and how dispersed their predictor values are.

For a simple illustration, suppose predictor values are evenly divided between 0 and 1. Their mean is 0.5, so each contributes \((0.5)^2=0.25\) to \(S_{xx}\). With this balanced layout, \(S_{xx}=n/4\). If the residual standard deviation stays the same, multiplying the sample size by four multiplies \(S_{xx}\) by four and divides the slope standard error by two.

Worked Example: Increasing Sample Size Across the Same Range

Consider two hypothetical designs with predictor values split evenly between 0 and 1 predictor unit. The first has \(n=8\), and the second has \(n=32\). Assume both have residual standard deviation \(s=4\) response units and satisfy the regression model conditions. Calculate their estimated slope standard errors.

For \(n=8\), there are four observations at each endpoint. Each contributes \(0.25\), so \(S_{xx}=8(0.25)=2\). For \(n=32\), \(S_{xx}=32(0.25)=8\). Therefore:

SE_{n=8}(b_1)=\frac{4}{\sqrt{2}}\approx 2.828, \qquad SE_{n=32}(b_1)=\frac{4}{\sqrt{8}}\approx 1.414.

The larger design has four times the \(S_{xx}\), and its estimated standard error is half as large. In this example, the observations were added across the same predictor range rather than clustered at one value. If observations were added very close to the predictor mean, the increase in \(S_{xx}\), and therefore the gain in precision, could be smaller.

What the Standard Error Tells You

The estimated standard error describes the typical scale of the sample slope's sampling variation under the regression model. It is not the typical residual size for individual responses. Residual standard deviation \(s\) measures response variation around the fitted line; \(SE(b_1)\) translates that response noise into uncertainty about the slope by dividing by \(\sqrt{S_{xx}}\).

This distinction matters in practice. A dataset may have substantial residual scatter but still estimate the slope reasonably precisely if predictor values are widely spread. Conversely, responses may lie close to the fitted line, yet a narrow predictor range can leave the slope uncertain. Precision is determined by both factors, not by residual scatter or sample size alone.

Common Mistakes

Using \(s\) alone as the slope standard error. The residual standard deviation is in response units, whereas the slope standard error must be in response units per predictor unit. Divide \(s\) by \(\sqrt{S_{xx}}\).
Assuming that any increase in \(n\) guarantees a large precision gain. Added observations help most when they contribute meaningful predictor spread. Many observations clustered together may add little to \(S_{xx}\).
Confusing predictor range with \(S_{xx}\). The range uses only the minimum and maximum. \(S_{xx}\) uses all predictor values and their distances from the mean.
Calling the standard error the error in the fitted slope. The actual difference \(b_1-\beta_1\) is not known for a particular sample. The standard error describes the sampling variation expected under the model.

Key takeaway: Under the simple linear regression model, \(\operatorname{Var}(b_1\mid x)=\sigma^2/S_{xx}\), so the estimated standard error is \(s/\sqrt{S_{xx}}\). Lower residual variation and greater predictor spread make the estimated slope more precise; sample size helps insofar as it increases useful predictor spread.

Check Your Understanding

Use the model and results in this tutorial to answer the following questions.

For fixed predictor values, what does the sampling variability of the estimated slope describe?
If the residual standard deviation stays constant and \(S_{xx}\) becomes four times as large, what happens to the slope standard error? Explain using its formula.
Why does the variance derivation require independent errors with a common variance?
Two studies have the same \(s\), but one has a larger \(S_{xx}\). Which study has the smaller estimated slope standard error, and why?
Why might adding many observations near the predictor mean yield less improvement in slope precision than adding observations across the predictor range?