Deriving the Least-Squares Slope

Why Derive the Slope?

A least-squares regression line is chosen to make the squared prediction errors as small as possible. The fitted slope is not an arbitrary feature of a graph: it is the value that minimizes a specific function of the observed pairs. Deriving it shows why the slope depends on how \(x\) and \(y\) vary together and on how much the explanatory variable \(x\) varies on its own.

Here, \(x_i\) is the explanatory-variable value and \(y_i\) is the response-variable value for observation \(i\), for \(i=1,\ldots,n\). Write a candidate line as \(\hat{y}=a+bx\), where \(a\) is its intercept and \(b\) its slope. The residual for observation \(i\) is \(y_i-(a+bx_i)\).

This is a mathematical optimization of the observed data. To carry it out, assume the data consist of paired numerical values and that all values are finite. A unique least-squares slope also requires that the observed \(x\)-values are not all identical. No probability model, normal distribution, or independence assumption is needed just to derive or calculate this descriptive slope.

Deriving the Slope

Because both the intercept and slope can affect the sum of squared errors, first account for the intercept. For a fixed value of \(b\), differentiate \(Q(a,b)\) with respect to \(a\) and set the derivative equal to zero:

Rearranging this equation gives the intercept that minimizes the sum of squared errors for that fixed slope:

where \(\bar{x}\) and \(\bar{y}\) are the sample means of the explanatory and response values. Substituting this intercept into each residual gives

Thus, after choosing the best intercept for each \(b\), the sum of squared errors is a function of the slope alone. Define \(S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2\), \(S_{xy}=\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})\), and \(S_{yy}=\sum_{i=1}^{n}(y_i-\bar{y})^2\). Expanding the squared residuals gives

Differentiate with respect to \(b\). At a minimum the derivative is zero, so \(2bS_{xx}-2S_{xy}=0\). If \(S_{xx}>0\), solving gives the least-squares slope. The expression below also verifies that it is the unique minimum: completing the square shows that \(Q(b)\) is a positive multiple of a squared distance from this slope, plus a constant.

If all the \(x_i\) values are identical, then \(S_{xx}=0\), and the formula cannot be used. In that case, changing the slope can be offset by changing the intercept, so the data do not determine a unique slope. There is no variation in the explanatory variable with which to compare changes in the response.

Covariance and Variation in \(x\)

The numerator \(S_{xy}\) measures how the two variables deviate from their respective means together. It is positive when observations with above-average \(x\) tend to have above-average \(y\), and negative when above-average \(x\) tends to accompany below-average \(y\). The denominator \(S_{xx}\) measures the total squared deviations of the explanatory values from their mean.

The sample covariance and sample variance of \(x\) use the same divisor, \(n-1\):

Consequently, their common divisor cancels in the slope formula. This gives a compact interpretation: the least-squares slope is the sample covariance of \(x\) and \(y\), divided by the sample variance of \(x\).

The numerator has units of \(x\) times units of \(y\), while the denominator has units of \(x\) squared. Their ratio therefore has units of \(y\) per unit of \(x\), as a slope should. The sign of the slope matches the sign of the covariance because \(S_{xx}\) is positive. A positive slope means the fitted response increases as \(x\) increases; a negative slope means it decreases.

The slope is directional: it describes the fitted response \(y\) in relation to the explanatory variable \(x\). Reversing the roles of \(x\) and \(y\) changes which variable appears in the denominator, and generally produces a different slope. This formula also agrees with the correlation-based expression \(b_1=r(s_y/s_x)\), since \(s_{xy}=rs_xs_y\) and \(s_x^2\) is in the denominator.

Common Mistake: Using the Wrong Variable in the Denominator

A common error is to divide the covariance by the variance of the response variable and call the result the slope for predicting \(y\) from \(x\). The correct formula has \(x\)-variation in the denominator: \(b_1=s_{xy}/s_x^2\). The explanatory variable is the quantity whose variation is used to describe the change in the fitted response. If the goal instead is to predict \(x\) from \(y\), the roles reverse and the denominator becomes \(s_y^2\); that is a different regression line.

Another tempting mistake is to think that the denominator’s divisor \(n-1\) should remain in the slope. It appears in both the sample covariance and sample variance, so it cancels exactly. Either use the centered sums directly or use the ratio \(s_{xy}/s_x^2\); both yield the same slope.