Deriving the Least-Squares Coefficients

From a Fitting Rule to Coefficient Formulas

A fitting rule tells us what makes one line preferable to another; a coefficient formula tells us how to calculate that line. In least squares, we choose the intercept and slope to make the sum of the squared vertical discrepancies between the observed responses and the line as small as possible. Differentiating that sum gives two equations whose solution provides the ordinary least-squares coefficients.

The calculation is an optimization of the observed data. It does not, by itself, require a probability model for the responses or a normal distribution of errors. A unique slope does require that the observed predictor values are not all identical. That condition will appear directly in the algebra.

Definition: For paired observations $(x_i,y_i)$, $i=1,\ldots,n$, a candidate straight line has intercept $b_0$ and slope $b_1$, so its fitted value at $x_i$ is $b_0+b_1x_i$. The ordinary least-squares coefficients are the values $b_0$ and $b_1$ that minimize the sum of squared residuals $$ Q(b_0,b_1)=\sum_{i=1}^{n}\left[y_i-(b_0+b_1x_i)\right]^2. $$ The minimizing values are conventionally denoted $\hat{\beta}_0$ and $\hat{\beta}_1$, respectively.

Here, $b_0$ and $b_1$ are temporary candidate values while we minimize $Q$; $\hat{\beta}_0$ and $\hat{\beta}_1$ are the values that achieve the minimum. The expression $y_i-(b_0+b_1x_i)$ is the observed response minus the candidate fitted value. Squaring makes negative and positive discrepancies contribute positively and gives larger discrepancies more weight.

Differentiating the Objective

To find the minimizing coefficients, differentiate $Q$ separately with respect to the intercept and the slope. At an interior minimum, both partial derivatives must be zero. Differentiating with respect to $b_0$, each summand contributes $-2[y_i-(b_0+b_1x_i)]$, because the derivative of the expression inside the square with respect to $b_0$ is $-1$. Thus,

\frac{\partial Q}{\partial b_0} =-2\sum_{i=1}^{n}\left[y_i-(b_0+b_1x_i)\right]=0.

Divide by $-2$, expand the sum, and collect the terms involving $b_0$ and $b_1$. This gives the first equation:

\sum_{i=1}^{n}y_i =nb_0+b_1\sum_{i=1}^{n}x_i.

For the slope derivative, the derivative of each squared term includes an additional factor of $x_i$, since the candidate fitted value changes by $x_i$ units when $b_1$ changes by one unit. Therefore,

\frac{\partial Q}{\partial b_1} =-2\sum_{i=1}^{n}x_i\left[y_i-(b_0+b_1x_i)\right]=0.

Expanding this equation produces the second equation. Together, these are called the normal equations for the simple least-squares line:

\begin{aligned} nb_0+b_1\sum_{i=1}^{n}x_i&=\sum_{i=1}^{n}y_i,\\ b_0\sum_{i=1}^{n}x_i+b_1\sum_{i=1}^{n}x_i^2&=\sum_{i=1}^{n}x_iy_i. \end{aligned}

The second equation follows by distributing $x_i$ across the terms inside the brackets: the intercept term sums to $b_0\sum x_i$, and the slope term sums to $b_1\sum x_i^2$. The right-hand side is $\sum x_iy_i$.

Solving for the Coefficients

The first normal equation can be rearranged to express the intercept in terms of the slope:

b_0=\frac{\sum_{i=1}^{n}y_i-b_1\sum_{i=1}^{n}x_i}{n}.

Substitute this expression into the second normal equation and collect the terms involving $b_1$. Multiplying through by $n$ gives

b_1\left[n\sum_{i=1}^{n}x_i^2-\left(\sum_{i=1}^{n}x_i\right)^2\right] =n\sum_{i=1}^{n}x_iy_i-\left(\sum_{i=1}^{n}x_i\right)\left(\sum_{i=1}^{n}y_i\right).

If the bracket multiplying $b_1$ is positive, divide by it to obtain the slope. Substituting that slope into the rearranged first equation gives the intercept:

Theorem: If the predictor values are not all identical, the ordinary least-squares coefficients are $$ \hat{\beta}_1= \frac{n\sum_{i=1}^{n}x_iy_i-\left(\sum_{i=1}^{n}x_i\right)\left(\sum_{i=1}^{n}y_i\right)} {n\sum_{i=1}^{n}x_i^2-\left(\sum_{i=1}^{n}x_i\right)^2}, \qquad \hat{\beta}_0= \frac{\sum_{i=1}^{n}y_i-\hat{\beta}_1\sum_{i=1}^{n}x_i}{n}. $$

Why This Solution Is a Minimum

Setting the derivatives to zero finds a stationary point, but we also need to know that it is a minimum. The objective $Q$ is a quadratic function of $b_0$ and $b_1$. Its matrix of second derivatives is

\begin{pmatrix} \frac{\partial^2 Q}{\partial b_0^2}&\frac{\partial^2 Q}{\partial b_0\partial b_1}\\ \frac{\partial^2 Q}{\partial b_1\partial b_0}&\frac{\partial^2 Q}{\partial b_1^2} \end{pmatrix} = 2\begin{pmatrix} n&\sum x_i\\ \sum x_i&\sum x_i^2 \end{pmatrix}.

The first diagonal entry is positive because $n>0$. The determinant of this matrix is $4D$, where

D=n\sum_{i=1}^{n}x_i^2-\left(\sum_{i=1}^{n}x_i\right)^2 =\sum_{i<j}(x_i-x_j)^2.

The last equality follows by expanding the pairwise differences: each $x_i^2$ appears in $n-1$ pairs, and each product $x_ix_j$ appears with coefficient $-2$. This equals $n\sum x_i^2-(\sum x_i)^2$. Every term $(x_i-x_j)^2$ is nonnegative, so $D>0$ exactly when at least two predictor values differ. In that case, the matrix of second derivatives is positive definite: its first diagonal entry and its determinant are both positive. Consequently, the stationary point is the unique global minimum of $Q$. If all the $x_i$ are identical, $D=0$, and the data do not determine a unique slope and intercept separately.

Applying the Formulas

Worked Example: A Short Dataset With Three Observations

Suppose a hypothetical dataset records a predictor $x$, in hours, and a response $y$, in points, for three observations: $(1,2)$, $(2,3)$, and $(3,5)$. Calculate the least-squares coefficients.

First, the predictor values vary, so a unique solution is possible. The required sums are $\sum x_i=1+2+3=6$, $\sum y_i=2+3+5=10$, $\sum x_i^2=1+4+9=14$, and $\sum x_iy_i=2+6+15=23$. With $n=3$, the denominator is

$$ 3(14)-6^2=42-36=6. $$

The slope is

\hat{\beta}_1 =\frac{3(23)-(6)(10)}{6} =\frac{69-60}{6} =1.5\text{ points per hour}.

The intercept is

\hat{\beta}_0 =\frac{10-(1.5)(6)}{3} =\frac{1}{3}\text{ point}.

The fitted line is $\hat{y}=0.333+1.5x$, with the intercept and fitted response measured in points. The slope describes a fitted increase of 1.5 points for each additional hour of the predictor; this calculation alone does not establish a causal effect.

Worked Example: Keeping Track of the Sums

For a second hypothetical dataset, let the paired values be $(0,10)$, $(1,12)$, $(2,11)$, and $(3,15)$. The predictor is measured in sessions and the response in units. Find the coefficients from the raw sums.

There are $n=4$ observations. Direct addition gives $\sum x_i=6$, $\sum y_i=48$, and $\sum x_i^2=0+1+4+9=14$. The cross-products are $0(10)=0$, $1(12)=12$, $2(11)=22$, and $3(15)=45$, so $\sum x_iy_i=79$. The predictor values are not all equal. The denominator and slope are

D=4(14)-6^2=56-36=20, \qquad \hat{\beta}_1 =\frac{4(79)-6(48)}{20} =\frac{316-288}{20} =1.4\text{ units per session}.

Then

\hat{\beta}_0 =\frac{48-(1.4)(6)}{4} =\frac{39.6}{4} =9.9\text{ units}.

The fitted equation is $\hat{y}=9.9+1.4x$. The intercept is the fitted response at zero sessions, and the slope is the fitted change in response per additional session. These are descriptions of the fitted line, not claims that the response must change by exactly 1.4 units for every observation.

Worked Example: Checking the Intercept After Finding the Slope

Consider the hypothetical pairs $(2,7)$, $(4,8)$, and $(6,12)$, where $x$ is distance in kilometres and $y$ is time in minutes. Calculate both coefficients and verify that the intercept uses the first normal equation.

Here $n=3$, $\sum x_i=2+4+6=12$, $\sum y_i=7+8+12=27$, and $\sum x_i^2=4+16+36=56$. The cross-product sum is $\sum x_iy_i=14+32+72=118$. Since the predictor values differ, the denominator is positive:

$$ D=3(56)-12^2=168-144=24. $$

Thus,

\hat{\beta}_1 =\frac{3(118)-(12)(27)}{24} =\frac{354-324}{24} =1.25\text{ minutes per kilometre}.

The intercept is

\hat{\beta}_0 =\frac{27-(1.25)(12)}{3} =\frac{12}{3} =4\text{ minutes}.

As a check, the first normal equation requires $n\hat{\beta}_0+\hat{\beta}_1\sum x_i=\sum y_i$. Substitution gives $3(4)+(1.25)(12)=12+15=27$, which matches $\sum y_i$. The fitted line is $\hat{y}=4+1.25x$, with time in minutes.

Assumptions, Interpretation, and a Common Pitfall

For calculating these coefficients, the required inputs are paired numerical observations and a straight-line candidate $b_0+b_1x$. The algebraic optimization does not require normally distributed errors, constant error variance, or a causal relationship. Those conditions may matter for other regression tasks, but they are not needed to derive or compute the least-squares minimizer here. The key condition for a unique pair of coefficients is that the predictor values are not all the same.

Units provide a useful check on interpretation: $\hat{\beta}_0$ has the units of $y$, while $\hat{\beta}_1$ has units of $y$ per unit of $x$. The intercept is the line's fitted response when $x=0$; if zero is outside the relevant range, that numerical value may have little practical meaning. Neither coefficient, on its own, says that changing $x$ causes a change in $y$.

Pitfall: differentiating only with respect to the slope. The slope equation alone contains both $b_0$ and $b_1$, so it cannot generally determine both coefficients. Set both partial derivatives to zero and solve the two normal equations together.
Pitfall: dropping the predictor factor in the slope derivative. Differentiating the fitted value $b_0+b_1x_i$ with respect to $b_1$ gives $x_i$, not 1. Omitting this factor changes the second normal equation and the resulting slope.
Pitfall: dividing when the denominator is zero. If every predictor value is identical, $D=0$. The slope formula is then undefined; the data cannot distinguish a slope from a corresponding change in intercept.
Pitfall: confusing a least-squares description with causation. The formulas choose a line to minimize squared discrepancies in the observed data. They do not, by themselves, show that manipulating the predictor would change the response.

The central derivation is compact: define the squared-error objective, differentiate once for each coefficient, set both derivatives to zero, and solve the resulting pair of equations. Keeping the sums and their units explicit makes the calculation easier to audit and helps catch arithmetic or interpretation errors.

Check Your Understanding

Use the coefficient derivation and conditions above to answer these questions.

Write the least-squares objective $Q(b_0,b_1)$ for $n$ paired observations.
Why does differentiating with respect to $b_1$ introduce a factor of $x_i$ in each summand?
For $n=3$, $\sum x_i=6$, $\sum y_i=10$, $\sum x_i^2=14$, and $\sum x_iy_i=23$, calculate the denominator and slope.
What condition on the observed predictor values ensures a unique least-squares intercept and slope?
Why does minimizing the sum of squared residuals not, by itself, establish a causal effect of the predictor?