Deriving the Ordinary Least Squares Normal Equations

Why the Normal Equations Matter

In multiple regression, ordinary least squares (OLS) chooses coefficients to make the sum of squared differences between observed and fitted responses as small as possible. The normal equations turn that minimization problem into a system of linear equations. Deriving them makes clear which matrices are involved, why the signs take their particular form, and how to check a proposed coefficient vector.

The derivation is algebraic: it uses the design matrix and the response vector, not a probability distribution for the errors. For the usual regression interpretation, one may write the model conditionally on the observed predictors as \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\), with errors having conditional mean zero. Additional assumptions are needed for standard errors and inference, but are not needed to obtain the OLS normal equations.

Definition: Let \(\mathbf{y}\) be an \(n\times 1\) response vector, let \(\mathbf{X}\) be an \(n\times p\) design matrix, and let \(\boldsymbol{\beta}\) be a \(p\times 1\) coefficient vector. The residual sum of squares at a candidate coefficient vector \(\boldsymbol{\beta}\) is \[ \operatorname{RSS}(\boldsymbol{\beta})=(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^\top(\mathbf{y}-\mathbf{X}\boldsymbol{\beta}). \] The OLS estimate minimizes this quantity over coefficient vectors. The normal equations are \(\mathbf{X}^\top\mathbf{X}\hat{\boldsymbol{\beta}}=\mathbf{X}^\top\mathbf{y}\).

Deriving the Normal Equations

The design matrix has one row per observation and one column per coefficient. If the model includes an intercept, the first column is typically a column of ones. If there is no intercept, that column is absent. The dimensions are important: \(\mathbf{X}\boldsymbol{\beta}\) is \(n\times1\), while both \(\mathbf{X}^\top\mathbf{X}\) and \(\mathbf{X}^\top\mathbf{y}\) are \(p\times p\) and \(p\times1\), respectively.

Write the RSS as a scalar-valued function of the coefficient vector and expand it. Since a scalar equals its transpose, the two cross-product terms in the expansion are equal. Differentiating with respect to the vector gives the gradient below.

\begin{aligned} \operatorname{RSS}(\boldsymbol{\beta}) &=(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^\top (\mathbf{y}-\mathbf{X}\boldsymbol{\beta})\\ &=\mathbf{y}^\top\mathbf{y} -\mathbf{y}^\top\mathbf{X}\boldsymbol{\beta} -\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{y} +\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\\ &=\mathbf{y}^\top\mathbf{y} -2\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{y} +\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta},\\ \nabla_{\boldsymbol{\beta}}\operatorname{RSS}(\boldsymbol{\beta}) &=-2\mathbf{X}^\top\mathbf{y} +2\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}. \end{aligned}

For the quadratic derivative, \(\mathbf{X}^\top\mathbf{X}\) is symmetric, so the gradient of \(\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\) is \(2\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\). At a minimum of a differentiable function, its gradient is zero. Setting this gradient to zero and dividing by 2 yields the normal equations.

Define the objective.
Use \(\operatorname{RSS}(\boldsymbol{\beta})=(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^\top(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})\).

Expand and differentiate.
The gradient is \(-2\mathbf{X}^\top\mathbf{y}+2\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\).

Set the gradient to zero.
Rearranging gives \(\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}=\mathbf{X}^\top\mathbf{y}\).

Why this stationary point minimizes RSS: For any coefficient vector \(\boldsymbol{\beta}\) satisfying the normal equations and any change \(\mathbf{h}\), expansion around \(\boldsymbol{\beta}\) gives \[ \operatorname{RSS}(\boldsymbol{\beta}+\mathbf{h}) =\operatorname{RSS}(\boldsymbol{\beta}) +\mathbf{h}^\top\mathbf{X}^\top\mathbf{X}\mathbf{h}, \] because the linear term is zero by the normal equations. Moreover, \[ \mathbf{h}^\top\mathbf{X}^\top\mathbf{X}\mathbf{h} =(\mathbf{X}\mathbf{h})^\top(\mathbf{X}\mathbf{h})\geq 0. \] Thus no change in the coefficient vector can decrease RSS. This establishes that a solution to the normal equations is a minimizer. Whether that solution is unique is a separate question.

Worked Matrix Calculations

Each equation in \(\mathbf{X}^\top\mathbf{X}\hat{\boldsymbol{\beta}}=\mathbf{X}^\top\mathbf{y}\) is obtained from the observed design and response. Solve the system, then substitute the coefficients back into the equations. When useful, computing fitted values and RSS provides an additional arithmetic check.

Worked Example: A Two-Coefficient Model Without an Intercept

A hypothetical analyst uses two predictors and fits a model with no intercept to three observations. The design and response are \(\mathbf{X}=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}\) and \(\mathbf{y}=(5,1,4)^\top\). Find the OLS coefficient vector, fitted values, and RSS.

Method. Form the normal equations. The first column represents the first predictor and the second column the second predictor; neither is a constant intercept column.

Calculate the two matrix products.

\mathbf{X}^\top\mathbf{X} =\begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad \mathbf{X}^\top\mathbf{y} =\begin{pmatrix}5+4\\1+4\end{pmatrix} =\begin{pmatrix}9\\5\end{pmatrix}.

Writing \(\hat{\boldsymbol{\beta}}=(\hat{\beta}_1,\hat{\beta}_2)^\top\), the equations are \(2\hat{\beta}_1+\hat{\beta}_2=9\) and \(\hat{\beta}_1+2\hat{\beta}_2=5\). Twice the second equation is \(2\hat{\beta}_1+4\hat{\beta}_2=10\). Subtracting the first equation gives \(3\hat{\beta}_2=1\), so \(\hat{\beta}_2=1/3\). The first equation then gives \(2\hat{\beta}_1+1/3=9\), hence \(\hat{\beta}_1=13/3\).

Coefficients and fitted values. Therefore \(\hat{\boldsymbol{\beta}}=(13/3,1/3)^\top\), and \[ \hat{\mathbf{y}}=\mathbf{X}\hat{\boldsymbol{\beta}} =(13/3,1/3,14/3)^\top. \] The residual vector is \(\mathbf{y}-\hat{\mathbf{y}}=(2/3,2/3,-2/3)^\top\). Thus \[ \operatorname{RSS}=(2/3)^2+(2/3)^2+(-2/3)^2=4/9+4/9+4/9=4/3. \]

Check. Multiplying the proposed coefficients by \(\mathbf{X}^\top\mathbf{X}\) gives \((26/3+1/3,\;13/3+2/3)^\top=(9,5)^\top\), as required. The common mistake avoided here is accepting coefficients without substituting them into both normal equations.

Worked Example: An Intercept and One Predictor

For three hypothetical policy records, let the predictor values be \(x=(0,1,2)\) and the responses be \(y=(1,2,2.5)\). Fit the model with an intercept, then calculate and check its RSS.

Build the design matrix. The intercept column consists of ones, so \(\mathbf{X}=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}\). The matrix products are \[ \mathbf{X}^\top\mathbf{X} =\begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad \mathbf{X}^\top\mathbf{y} =\begin{pmatrix}1+2+2.5\\0(1)+1(2)+2(2.5)\end{pmatrix} =\begin{pmatrix}5.5\\7\end{pmatrix}. \]

Solve the equations. They are \(3\hat{\beta}_0+3\hat{\beta}_1=5.5\) and \(3\hat{\beta}_0+5\hat{\beta}_1=7\). Subtract the first from the second: \(2\hat{\beta}_1=1.5\), so \(\hat{\beta}_1=0.75\). Substitution gives \(3\hat{\beta}_0+2.25=5.5\), hence \(\hat{\beta}_0=3.25/3=13/12\). The fitted equation is \(\hat{y}=13/12+(3/4)x\).

Calculate RSS and check. At \(x=0,1,2\), the fitted values are \(13/12,\;22/12=11/6,\;31/12\). The residuals are \(-1/12,\;2/12=1/6,\;-1/12\). Therefore \[ \operatorname{RSS}=1/144+1/36+1/144=6/144=1/24. \] For the normal-equation check, the first left side is \(3(13/12)+3(3/4)=13/4+9/4=5.5\); the second is \(3(13/12)+5(3/4)=13/4+15/4=7\). Both match the right sides. The intercept must be represented by its own column; omitting it would fit a different model.

Worked Example: Two Predictors and an Intercept

A hypothetical claim study has four observations, an intercept, and two predictors. The columns are specified by \[ \mathbf{X}=\begin{pmatrix}1&-1&-1\\1&-1&1\\1&1&-1\\1&1&1\end{pmatrix}, \qquad \mathbf{y}=\begin{pmatrix}10\\6\\12\\12\end{pmatrix}. \] Find the coefficients and RSS.

Form the normal equations. The intercept column sums to 4, and each predictor column sums to 0. Each predictor has squared entries summing to 4, while their cross-product is \((-1)(-1)+(-1)(1)+(1)(-1)+(1)(1)=0\). Consequently, \[ \mathbf{X}^\top\mathbf{X} =\begin{pmatrix}4&0&0\\0&4&0\\0&0&4\end{pmatrix}. \] The other product is \[ \mathbf{X}^\top\mathbf{y} =\begin{pmatrix} 10+6+12+12\\ -10-6+12+12\\ -10+6-12+12 \end{pmatrix} =\begin{pmatrix}40\\8\\-4\end{pmatrix}. \]

Solve and verify. The equations are \(4\hat{\beta}_0=40\), \(4\hat{\beta}_1=8\), and \(4\hat{\beta}_2=-4\). Thus \(\hat{\boldsymbol{\beta}}=(10,2,-1)^\top\). The fitted values are \(10-2+1=9\), \(10-2-1=7\), \(10+2+1=13\), and \(10+2-1=11\), or \((9,7,13,11)^\top\). Subtracting from the observed responses gives residuals \((1,-1,-1,1)^\top\), so \(\operatorname{RSS}=1+1+1+1=4\). Finally, multiplying the diagonal matrix \(\mathbf{X}^\top\mathbf{X}\) by the coefficient vector gives \((40,8,-4)^\top=\mathbf{X}^\top\mathbf{y}\). The same check works regardless of whether the predictor columns happen to be uncorrelated in this sample.

Common Exam Traps

The most frequent errors arise before solving the system: using the wrong sign in the gradient, mixing up matrix dimensions, or silently changing whether the model has an intercept. A useful discipline is to write the RSS first, check the dimensions of every product, and verify the answer in the normal equations.

Reversing the sign. The gradient is \(-2\mathbf{X}^\top\mathbf{y}+2\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\). Setting it equal to zero gives \(\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}=\mathbf{X}^\top\mathbf{y}\), not \(-\mathbf{X}^\top\mathbf{y}\) on the right.
Using \(\mathbf{X}\mathbf{X}^\top\) in the coefficient equations. That product is \(n\times n\); the coefficient system is \(p\times p\), so it uses \(\mathbf{X}^\top\mathbf{X}\).
Forgetting an intercept. A model with an intercept has a column of ones in \(\mathbf{X}\). Leaving it out changes the model and generally changes every estimated coefficient.
Assuming normal errors are needed for the derivation. The minimization and differentiation are algebraic. Normality concerns inference in suitable settings, not the normal-equation calculation itself.

A proposed estimate should satisfy the equations after substitution. This is a direct check on the coefficient calculation, although it does not by itself establish that the coefficients are the only solution.

Check Your Understanding

For each problem, form or use the normal equations and check the requested result.

An intercept-only model is fitted to responses \(4,7,10\). Find the estimated intercept and RSS. (Answer: \(\hat{\beta}_0=7\), \(\operatorname{RSS}=18\).)
For a model without an intercept, \(\mathbf{X}=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}\) and \(\mathbf{y}=(3,5,7)^\top\). Find the coefficient vector from the normal equations. (Answer: \(\hat{\boldsymbol{\beta}}=(8/3,14/3)^\top\).)
With an intercept, use \(x=(1,2,3)\) and \(y=(4,5,8)\). Find the coefficients and RSS. (Answer: \(\hat{\beta}_0=5/3,\ \hat{\beta}_1=2,\ \operatorname{RSS}=2/3\).)
For an intercept and two predictors, let \(\mathbf{X}=\begin{pmatrix}1&-1&-1\\1&-1&1\\1&1&-1\\1&1&1\end{pmatrix}\) and \(\mathbf{y}=(5,1,7,7)^\top\). Find the coefficient vector and RSS. (Answer: \(\hat{\boldsymbol{\beta}}=(5,2,-1)^\top,\ \operatorname{RSS}=4\).)

What You'll Learn