From Residuals to a Quadratic Objective
In ordinary least squares, a design matrix \(X\) records the predictor values, a response vector \(y\) records the outcomes, and a coefficient vector \(\beta\) determines the fitted values \(X\beta\). The least-squares objective measures the squared length of the residual vector \(y-X\beta\). Differentiating this objective gives a direct route from the data and model to the normal equations.
Throughout, take \(X\) to be a fixed real \(m\times p\) matrix, \(y\in\mathbb{R}^m\), and \(\beta\in\mathbb{R}^p\). There are no assumptions that \(m\geq p\) or that the columns of \(X\) are linearly independent. Write \(\|v\|^2=v'v\) for the squared Euclidean norm of a vector \(v\).
The residual vector at \(\beta\) is \(r(\beta)=y-X\beta\). Expanding the objective gives $$ f(\beta)=y'y-2\beta'X'y+\beta'X'X\beta. $$ The matrix \(X'X\) is symmetric and positive semidefinite: for every \(v\in\mathbb{R}^p\), \(v'X'Xv=(Xv)'(Xv)=\|Xv\|^2\geq 0\). This quadratic form will control both the curvature of the objective and whether its minimizer is unique.
Gradient and Normal Equations
The differential makes the matrix dimensions and the factor of two explicit. For an increment \(d\beta\), the differential of the residual is \(d(y-X\beta)=-X\,d\beta\). Applying the product rule to the squared norm, $$ df=2(y-X\beta)'(-X\,d\beta)=-2(y-X\beta)'X\,d\beta. $$ Since this is a scalar, it can be written as \(df=\big[-2X'(y-X\beta)\big]'d\beta\). Therefore the gradient, expressed as a \(p\)-vector, is $$ \nabla f(\beta)=-2X'(y-X\beta)=2X'X\beta-2X'y. $$ Differentiating once more gives the Hessian \(\nabla^2 f(\beta)=2X'X\), which is positive semidefinite.
Here is a complete justification. First, the gradient vanishes at \(\hat{\beta}\) exactly when \(X'X\hat{\beta}=X'y\). To show that such a vector exists even if \(X\) is rank deficient, consider the null space of \(X'X\). For any \(v\), $$ v'X'Xv=\|Xv\|^2, $$ so \(X'Xv=0\) implies \(Xv=0\), and \(Xv=0\) implies \(X'Xv=0\). Thus \(\operatorname{null}(X'X)=\operatorname{null}(X)\). Because \(X'X\) is symmetric, its range is the orthogonal complement of its null space. For every \(v\in\operatorname{null}(X)\), \(v'X'y=(Xv)'y=0\). Hence \(X'y\) is orthogonal to \(\operatorname{null}(X'X)\), so it belongs to the range of \(X'X\). There is therefore at least one \(\hat{\beta}\) solving the normal equations.
Now suppose \(\hat{\beta}\) solves them, and take any \(h\in\mathbb{R}^p\). Expanding the objective at \(\hat{\beta}+h\) gives $$ f(\hat{\beta}+h)=\|y-X\hat{\beta}-Xh\|^2 =f(\hat{\beta})-2h'X'(y-X\hat{\beta})+h'X'Xh. $$ The normal equations imply \(X'(y-X\hat{\beta})=0\), so $$ f(\hat{\beta}+h)-f(\hat{\beta})=h'X'Xh=\|Xh\|^2\geq 0. $$ Thus every solution of the normal equations is a global minimizer. Conversely, a differentiable function on all of \(\mathbb{R}^p\) must have zero gradient at any minimum, so every minimizer satisfies the normal equations.
For uniqueness, if \(X\) has full column rank, then \(Xh=0\) only for \(h=0\). Consequently, for every nonzero \(h\), \(h'X'Xh=\|Xh\|^2>0\). The objective strictly increases away from a minimizer, so there can be only one. If \(X\) is rank deficient, choose a nonzero \(v\) with \(Xv=0\). Whenever \(\hat{\beta}\) is a solution, \(\hat{\beta}+v\) has the same fitted values and the same objective value; the minimizer is not unique. This proves the stated rank condition. \(\square\)
The equation \(X'(y-X\hat{\beta})=0\) also says that the residual at a least-squares solution is orthogonal to every column of \(X\). Indeed, its \(j\)th entry is the inner product of the \(j\)th column of \(X\) with the residual. This orthogonality is a consequence of the normal equations, not an additional assumption.
Worked Examples
Worked Example: One Coefficient and a Correctly Computed Residual Sum
Let \(X=(1,1,1)'\) and \(y=(1,2,4)'\). Find the least-squares coefficient and the minimum objective value.
Here \(X'X=1+1+1=3\) and \(X'y=1+2+4=7\). The normal equation is \(3\hat{\beta}=7\), so $$ \hat{\beta}=\frac{7}{3}=2.333\ldots. $$
The fitted vector is \((7/3,7/3,7/3)'\), and the residual vector is \((-4/3,-1/3,5/3)'\). Therefore $$ f(\hat{\beta})=\left(-\frac43\right)^2+\left(-\frac13\right)^2+\left(\frac53\right)^2 =\frac{16+1+25}{9}=\frac{42}{9}=\frac{14}{3}\approx 4.667. $$ As a check, the residuals sum to \((-4-1+5)/3=0\), which is the residual-orthogonality condition for the single column of ones.
Worked Example: Solving a Two-Coefficient Problem
Take $$ X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, \qquad y=\begin{pmatrix}1\\2\\2\end{pmatrix}. $$ Find the least-squares coefficient vector and verify the residual orthogonality.
The required products are $$ X'X=\begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad X'y=\begin{pmatrix}5\\6\end{pmatrix}. $$ Thus the normal equations are \(3\beta_1+3\beta_2=5\) and \(3\beta_1+5\beta_2=6\). Subtracting the first equation from the second gives \(2\beta_2=1\), so \(\hat{\beta}_2=1/2\). Substitution into the first gives \(3\hat{\beta}_1+3/2=5\), hence \(\hat{\beta}_1=7/6\). Therefore \(\hat{\beta}=(7/6,1/2)'\).
The fitted values are \(7/6\), \(7/6+1/2=5/3\), and \(7/6+1=13/6\). The residuals are \((-1/6,1/3,-1/6)'\), and their squared sum is $$ \frac{1}{36}+\frac{1}{9}+\frac{1}{36}=\frac{6}{36}=\frac16. $$ The residual inner product with the first column is \(-1/6+1/3-1/6=0\). With the second column it is \(0(-1/6)+1(1/3)+2(-1/6)=0\). Both normal-equation conditions hold.
Worked Example: Rank Deficiency and Multiple Minimizers
Let $$ X=\begin{pmatrix}1&1\\2&2\\3&3\end{pmatrix}, \qquad y=\begin{pmatrix}1\\2\\2\end{pmatrix}. $$ The columns of \(X\) are identical, so \(X\) does not have full column rank. Find all least-squares coefficient vectors.
For \(\beta=(\beta_1,\beta_2)'\), the fitted vector depends only on \(\gamma=\beta_1+\beta_2\): it is \((\gamma,2\gamma,3\gamma)'\). The objective as a function of this sum is $$ (1-\gamma)^2+(2-2\gamma)^2+(2-3\gamma)^2 =9\gamma^2-22\gamma+9. $$ Its derivative is \(18\gamma-22\), so the minimizing sum is \(\hat{\gamma}=22/18=11/9\), not \(11/14\); equivalently, the normal equation in the one-dimensional fitted direction gives \((1^2+2^2+3^2)\gamma=1(1)+2(2)+3(2)\), or \(14\gamma=11\). Thus \(\hat{\gamma}=11/14\).
The direct derivative above must be consistent with the expanded objective: the coefficient of \(\gamma^2\) is \(1+4+9=14\), and the linear coefficient is \(-2(1+4+6)=-22\). Hence the correct expansion is \(14\gamma^2-22\gamma+9\), whose derivative \(28\gamma-22=0\) again gives \(\hat{\gamma}=11/14\). The objective at that value is $$ 14\left(\frac{11}{14}\right)^2-22\left(\frac{11}{14}\right)+9 =\frac{121}{14}-\frac{121}{7}+9=\frac{5}{14}. $$ Every vector satisfying \(\beta_1+\beta_2=11/14\) is a minimizer. For example, \((11/14,0)'\) and \((0,11/14)'\) have identical fitted values and objective \(5/14\). The minimizer is therefore not unique.
What the Rank Condition Means
Full column rank means that no column of \(X\) is an exact linear combination of the others. When this holds, \(X'X\) is positive definite and invertible, so the normal equations can be written as $$ \hat{\beta}=(X'X)^{-1}X'y. $$ This formula is valid only under full column rank. When the columns are dependent, the normal equations still have solutions, but \(X'X\) is singular and the inverse formula is unavailable. The fitted vector \(X\hat{\beta}\) is nevertheless the same for all least-squares solutions: the theorem's perturbation argument shows that any two solutions differ by a vector \(v\) with \(Xv=0\).
A frequent mistake is to differentiate the objective and immediately write \(\hat{\beta}=(X'X)^{-1}X'y\) without checking whether that inverse exists. The correction is to first state the normal equations, which are valid with or without full rank, and then use the inverse formula only when the columns of \(X\) are linearly independent. Another common error is to confuse a zero gradient with a minimum in an arbitrary nonlinear problem. Here the Hessian \(2X'X\) is positive semidefinite, and the exact expansion \(f(\hat{\beta}+h)-f(\hat{\beta})=\|Xh\|^2\) proves global minimality directly.
Check Your Understanding
Answer each question by referring to the objective, its normal equations, or the rank condition.
- For a design matrix with \(m\) rows and \(p\) columns, what are the dimensions of \(X'X\), \(X'y\), and \(\nabla f(\beta)\)?
- Show from the gradient formula that a stationary point satisfies \(X'(y-X\hat{\beta})=0\).
- Why does a solution to the normal equations minimize the objective even if \(X'X\) is singular?
- What condition on \(X\) makes the coefficient vector unique, and why?
- If \(v\neq 0\) satisfies \(Xv=0\), explain why \(\hat{\beta}+v\) has the same objective value as a least-squares solution \(\hat{\beta}\).