Deriving the Least-Squares Normal Equations

From Squared Error to Two Equations

For a fixed set of paired observations, least squares chooses the line whose residuals have the smallest sum of squares. The key step is to treat the intercept and slope as two separate variables in an objective function, differentiate with respect to each one, and set both derivatives equal to zero. This produces the normal equations.

The derivation is algebraic. It does not require the observations to have come from a particular probability distribution, nor does it require assumptions about the regression errors. Those assumptions matter for statistical inference, but not for finding the line that minimizes the squared errors in a given data set.

Definition: Given fixed observations $(x_1,y_1),\ldots,(x_n,y_n)$, a candidate line with intercept $a$ and slope $b$ predicts $a+bx_i$ at predictor value $x_i$. Its residual at observation $i$ is $e_i=y_i-(a+bx_i)$. The residual sum of squares is $$ Q(a,b)=\sum_{i=1}^{n}\left[y_i-(a+bx_i)\right]^2. $$ The least-squares intercept and slope minimize $Q(a,b)$ over the possible values of $a$ and $b$.

Here $x_i$ and $y_i$ are the observed data, while $a$ and $b$ are the quantities being chosen. In particular, when differentiating, each observation is held fixed. The same residual expression is used throughout: observed response minus fitted response.

Differentiating with Respect to Each Coefficient

The objective is a sum of squared residuals. For the derivative with respect to $a$, each residual changes at rate $-1$, because $e_i=y_i-a-bx_i$. For the derivative with respect to $b$, each residual changes at rate $-x_i$. Applying the chain rule to each squared residual gives the two partial derivatives.

\frac{\partial Q}{\partial a} =2\sum_{i=1}^{n} e_i\frac{\partial e_i}{\partial a} =-2\sum_{i=1}^{n} e_i =-2\sum_{i=1}^{n}\left[y_i-(a+bx_i)\right], $$ $$ \frac{\partial Q}{\partial b} =2\sum_{i=1}^{n} e_i\frac{\partial e_i}{\partial b} =-2\sum_{i=1}^{n} x_i e_i =-2\sum_{i=1}^{n}x_i\left[y_i-(a+bx_i)\right].

At a minimum that occurs at a finite pair $(a,b)$, both partial derivatives must be zero. Dividing each equation by $-2$ gives the residual form of the normal equations. Expanding the sums gives the equivalent form commonly used for calculations.

Theorem: The least-squares normal equations for a line with an intercept are $$ \sum_{i=1}^{n}\left[y_i-(a+bx_i)\right]=0, \qquad \sum_{i=1}^{n}x_i\left[y_i-(a+bx_i)\right]=0. $$ Equivalently, $$ na+b\sum_{i=1}^{n}x_i=\sum_{i=1}^{n}y_i, \qquad a\sum_{i=1}^{n}x_i+b\sum_{i=1}^{n}x_i^2=\sum_{i=1}^{n}x_i y_i. $$

The first equation comes from varying the intercept while holding the slope fixed. The second comes from varying the slope while holding the intercept fixed. Both conditions are required: satisfying just one derivative condition does not, in general, minimize the objective over both coefficients.

Why the Derivative Conditions Give a Minimum

Setting derivatives to zero identifies stationary points, but a stationary point is not automatically a minimum for every possible objective. Here the squared-error objective is a convex quadratic function of $a$ and $b$. Its second-derivative matrix is

\nabla^2 Q(a,b) = 2 \begin{pmatrix} n & \sum_{i=1}^{n}x_i\\ \sum_{i=1}^{n}x_i & \sum_{i=1}^{n}x_i^2 \end{pmatrix}.

For any numbers $r$ and $s$, the quadratic form associated with this matrix is $2\sum_{i=1}^{n}(r+s x_i)^2$, which cannot be negative. Thus the matrix is positive semidefinite, and $Q$ is convex. Any point satisfying both normal equations is therefore a global minimum.

There is a unique minimizing pair when the observed predictor values are not all identical. The determinant of the second-derivative matrix is

4\left(n\sum_{i=1}^{n}x_i^2-\left(\sum_{i=1}^{n}x_i\right)^2\right) = 4n\sum_{i=1}^{n}(x_i-\bar{x})^2, \qquad \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i.

If the $x_i$ values vary, then $\sum (x_i-\bar{x})^2>0$, so the determinant is positive and the matrix is positive definite. If every $x_i$ equals the same constant $c$, the data cannot distinguish an intercept from a slope: every prediction is $a+bc$. The normal equations still describe the minimizers, but they do not identify a unique intercept-slope pair.

Worked Example: Deriving Equations for Four Observations

A hypothetical calibration study records four input-output pairs: $(0,2),(1,4),(2,5),(3,8)$. Use ordinary least squares with a line $a+bx$ to derive the normal equations. Do not solve them for $a$ and $b$.

Method. Differentiate the residual sum of squares, then evaluate the sums in the expanded normal equations. The observed predictor values are not all equal, so the equations have a unique minimizing pair.

The objective for these observations is $$ Q(a,b)=(2-a)^2+(4-a-b)^2+(5-a-2b)^2+(8-a-3b)^2. $$ The required sums are $$ n=4,\qquad \sum x_i=0+1+2+3=6,\qquad \sum y_i=2+4+5+8=19. $$

For the squared predictor values, $$ \sum x_i^2=0^2+1^2+2^2+3^2=0+1+4+9=14. $$ For the predictor-response products, $$ \sum x_i y_i=(0)(2)+(1)(4)+(2)(5)+(3)(8)=0+4+10+24=38. $$

Substituting these values into the expanded equations gives $$ 4a+6b=19,\qquad 6a+14b=38. $$ These are the normal equations for this data set. As a check, the first is obtained from $\sum e_i=0$, and the second from $\sum x_i e_i=0$, with the sums above. Common mistake avoided: the slope equation weights each residual by its predictor value; it is not another unweighted sum of residuals.

Worked Example: Repeated Predictor Values

A hypothetical monitoring system produces the observations $(2,7),(2,9),(5,8)$, where $x$ is an input setting and $y$ is a measured response. Derive the two normal equations for a line with an intercept.

Method. Use the same derivatives, counting each observation separately even when two predictor values are equal. Repeated predictor values do not change the form of the equations.

The objective is $$ Q(a,b)=(7-a-2b)^2+(9-a-2b)^2+(8-a-5b)^2. $$ There are $n=3$ observations, and $$ \sum x_i=2+2+5=9,\qquad \sum y_i=7+9+8=24. $$ Also, $$ \sum x_i^2=2^2+2^2+5^2=4+4+25=33, $$ and $$ \sum x_i y_i=(2)(7)+(2)(9)+(5)(8)=14+18+40=72. $$

The first expanded normal equation is $$ 3a+9b=24. $$ The second is $$ 9a+33b=72. $$ Since the predictor values include both 2 and 5, they are not all identical; the objective has a unique minimizing pair. No coefficient calculation is needed to derive these equations. Common mistake avoided: the two observations with $x=2$ are both included in every sum; repeated values do not mean that one observation is discarded.

Worked Example: Checking the Slope Derivative Carefully

A hypothetical study records $(1,3),(1,5),(2,4),(4,9),(4,11)$. Derive the normal equations and verify the slope derivative directly from the residual expression.

Method. First calculate the four data summaries in the expanded equations. Then use the chain rule to check that the slope derivative includes a factor of $x_i$ for every residual.

The number of observations is $n=5$. The predictor and response sums are $$ \sum x_i=1+1+2+4+4=12,\qquad \sum y_i=3+5+4+9+11=32. $$ The squared predictor sum is $$ \sum x_i^2=1+1+4+16+16=38. $$ The product sum is $$ \sum x_i y_i=(1)(3)+(1)(5)+(2)(4)+(4)(9)+(4)(11) =3+5+8+36+44=96. $$

Substitution gives the equations $$ 5a+12b=32,\qquad 12a+38b=96. $$ For a direct derivative check, differentiating each squared residual with respect to $b$ gives $$ \frac{\partial Q}{\partial b} =-2\left[(1)(3-a-b)+(1)(5-a-b)+(2)(4-a-2b)+(4)(9-a-4b)+(4)(11-a-4b)\right]. $$ Collecting terms inside the brackets gives $96-12a-38b$, so $$ \frac{\partial Q}{\partial b}=-2(96-12a-38b). $$ Setting this derivative to zero is equivalent to $12a+38b=96$, as required. Common mistake avoided: omitting the predictor multiplier would produce the wrong derivative and would fail this direct check.

Common Derivative Trap

A frequent error is to differentiate each residual with respect to the slope as though its derivative were $-1$. That is the derivative with respect to the intercept. Since $e_i=y_i-a-bx_i$, the slope derivative is $\partial e_i/\partial b=-x_i$; the predictor value must remain in the sum.

For example, in the four-observation data set above, the incorrect slope derivative $-2\sum e_i$ would reproduce the intercept condition $4a+6b=19$. It would not give the correct slope condition. The correct derivative is $-2\sum x_i e_i$, which gives $6a+14b=38$. A quick check is to differentiate one term: $\partial (y_i-a-bx_i)^2/\partial b=-2x_i(y_i-a-bx_i)$.

Another way to avoid sign errors is to keep the residual definition fixed. With $e_i=y_i-(a+bx_i)$, the derivative conditions are $\sum e_i=0$ and $\sum x_i e_i=0$. If residuals are instead defined as fitted minus observed, both residual sums change sign, but the zero conditions remain equivalent. Mixing the two definitions partway through a calculation can produce inconsistent signs.

Scope and Use of the Equations

The equations apply to ordinary, unweighted least squares for a straight line that includes an intercept. They follow from minimizing the sum of squared vertical residuals in the observed response. Changing the objective, applying observation weights, or requiring the line to pass through the origin changes the differentiation conditions; the equations here should not be transferred to those problems without deriving the relevant objective first.

The normal equations are useful because they convert the minimization problem into two simultaneous linear equations in the intercept and slope. Their derivation also provides a reliable check on later calculations: the fitted coefficients must make the sum of residuals and the predictor-weighted sum of residuals equal to zero. This tutorial derives those conditions; calculating the coefficients from them is a separate step.

Key takeaway: Differentiate the residual sum of squares with respect to each coefficient separately. The intercept derivative gives $\sum e_i=0$; the slope derivative gives $\sum x_i e_i=0$. Expanding those conditions produces the two normal equations.

Check Your Understanding

For each problem, derive the requested condition or equations from ordinary least squares. Do not solve for the intercept and slope unless asked.

A hypothetical sensor records $(0,1),(1,2),(3,6)$. Calculate $n,\sum x_i,\sum y_i,\sum x_i^2,\sum x_i y_i$, then state both normal equations. (Answer: $3a+4b=9$ and $4a+10b=20$.)
A hypothetical laboratory records $(2,3),(2,7),(5,8)$. Retain both observations with predictor 2, calculate the required sums, and state the normal equations. (Answer: $3a+9b=18$ and $9a+33b=60$.)
For observations $(1,4),(3,4),(5,10),(5,12)$, calculate the sums needed for the expanded equations and state the equations. (Answer: $4a+14b=30$ and $14a+60b=126$.)
A hypothetical instrument produces $(0,5),(2,6),(4,11),(6,14)$. Derive the normal equations and state whether the data identify a unique minimizing intercept-slope pair. (Answer: $4a+12b=36$, $12a+56b=140$; yes, because the predictor values are not all identical.)
For observations $(1,2),(2,5),(4,7)$, test the candidate $a=1,b=2$ by calculating both residual sums $\sum e_i$ and $\sum x_i e_i$. Is the candidate a least-squares stationary point? (Answer: the residuals are $-1,0,-2$, so $\sum e_i=-3$ and $\sum x_i e_i=-9$; no, since neither normal-equation condition is satisfied.)

What You'll Learn