Deriving OLS From Least Squares

From a Fitting Criterion to an Estimator

Ordinary least squares (OLS) begins with a precise fitting rule: choose coefficients that make the observed outcomes as close as possible, in squared distance, to the values predicted by a linear combination of the regressors. Writing that rule in matrix notation reveals both the familiar OLS formula and the condition under which the coefficients are uniquely determined.

This is an algebraic derivation. It requires no probability model, normality assumption, or claim that the fitted relationship is causal. The data must be arranged consistently, and the minimization problem must have a solution; uniqueness of the coefficient vector requires an additional rank condition.

Definition: Let \(y\) be an \(n \times 1\) vector of observed outcomes, \(X\) an \(n \times k\) matrix of observed regressors, and \(b\) a \(k \times 1\) candidate coefficient vector. The residual vector at \(b\) is \(e(b)=y-Xb\). The sample sum of squared residuals is \(SSR(b)=e(b)'e(b)=(y-Xb)'(y-Xb)\). An OLS estimate is any vector \(\hat{\beta}\) that minimizes \(SSR(b)\) over \(b \in \mathbb{R}^k\).

The columns of \(X\) are the regressors included in the fit. If a constant term is wanted, a column of ones must be included in \(X\); it is not automatically present. The \(i\)th fitted value is the \(i\)th component of \(Xb\), and the corresponding residual is the observed value minus that fitted value.

Expanding the Objective and Deriving the Normal Equations

For any candidate \(b\), expand the quadratic objective. Because \(y'Xb\) is a scalar, it equals its transpose, \(b'X'y\), so the two cross terms are equal.

$$ SSR(b)=(y-Xb)'(y-Xb)=y'y-2b'X'y+b'X'Xb. $$

The gradient with respect to the \(k\)-vector \(b\) is \(-2X'y+2X'Xb\). At an interior minimum over all of \(\mathbb{R}^k\), this gradient must be zero. Thus any minimizer satisfies the normal equations:

X'X\hat{\beta}=X'y.

The normal equations are not merely a first-order recipe that might identify a maximum. The objective is a convex quadratic: its Hessian is \(2X'X\), and for every vector \(a\), \(a'X'Xa=(Xa)'(Xa)\geq 0\). Therefore any solution to the normal equations is a global minimizer. The following identity gives a direct justification.

SSR(b)=SSR(\hat{\beta})+(b-\hat{\beta})'X'X(b-\hat{\beta}) \quad\text{whenever}\quad X'X\hat{\beta}=X'y.

To verify the identity, put \(d=b-\hat{\beta}\) and expand \(SSR(\hat{\beta}+d)\). The terms that are linear in \(d\) are \(2d'(X'X\hat{\beta}-X'y)\), which equal zero by the normal equations. The remaining difference is \(d'X'Xd=(Xd)'(Xd)\geq 0\). This proves both that a solution minimizes the objective and that every change in the fitted vector from \(X\hat{\beta}\) cannot lower the sum of squared residuals.

When Does the OLS Formula Give a Unique Estimate?

If \(X'X\) is invertible, the normal equations have exactly one solution. In that case the OLS formula is

\hat{\beta}=(X'X)^{-1}X'y.

For any vector \(a\), \(a'X'Xa=(Xa)'(Xa)\). Consequently \(X'X\) is invertible exactly when \(Xa=0\) implies \(a=0\), which is the condition that the columns of \(X\) are linearly independent. In matrix terminology, \(X\) must have full column rank \(k\). Necessarily \(k\leq n\), but having no more columns than rows is not by itself sufficient: the columns must also avoid exact linear dependence.

If \(X\) does not have full column rank, the inverse in the formula does not exist and the coefficient vector is not unique. Least-squares minimizers still exist: they correspond to coefficient vectors whose fitted vector is the projection of \(y\) onto the column space of \(X\), a finite-dimensional subspace. The fitted vector is unique, even when more than one coefficient vector produces it. For example, adding a vector \(a\) with \(Xa=0\) to a minimizing coefficient vector leaves every fitted value unchanged.

Worked Example: Two Regressors and the OLS Formula

Consider hypothetical data with two regressors and no constant term. The rows of \(X\) and the observed outcomes are

Observation	\(x_1\)	\(x_2\)	\(y\)
1	1	0	1
2	0	1	1
3	1	1	2.5

Here \(n=3\) and \(k=2\). The two columns are linearly independent, so the coefficient estimate will be unique. Compute the cross-products:

X'X= \begin{pmatrix} 2&1\\ 1&2 \end{pmatrix}, \qquad X'y= \begin{pmatrix} 1+2.5\\ 1+2.5 \end{pmatrix} = \begin{pmatrix} 3.5\\ 3.5 \end{pmatrix}.

The determinant of \(X'X\) is \(2(2)-1(1)=3\), which is nonzero. Its inverse and the coefficient estimate are

(X'X)^{-1}=\frac{1}{3} \begin{pmatrix} 2&-1\\ -1&2 \end{pmatrix}, \qquad \hat{\beta}=(X'X)^{-1}X'y =\frac{1}{3} \begin{pmatrix} 2&-1\\ -1&2 \end{pmatrix} \begin{pmatrix} 3.5\\ 3.5 \end{pmatrix} = \begin{pmatrix} 7/6\\ 7/6 \end{pmatrix}.

The fitted values are obtained by multiplying \(X\hat{\beta}\), not by adding an unlisted intercept:

X\hat{\beta}= \begin{pmatrix} 1&0\\ 0&1\\ 1&1 \end{pmatrix} \begin{pmatrix} 7/6\\ 7/6 \end{pmatrix} = \begin{pmatrix} 7/6\\ 7/6\\ 7/3 \end{pmatrix} \approx \begin{pmatrix} 1.167\\ 1.167\\ 2.333 \end{pmatrix}.

Subtracting these fitted values from \(y\) gives residuals \((-1/6,-1/6,1/6)'\). Thus \(SSR(\hat{\beta})=1/36+1/36+1/36=1/12\), or about \(0.08333\). As a check on the minimization, \(X'e=0\): each regressor column has zero inner product with the residual vector, as required by the normal equations.

Worked Example With an Intercept

The next hypothetical data set relates an outcome \(y\), measured in outcome units, to a regressor \(x\), measured in predictor units. This fit includes an intercept, represented by the first column of ones in \(X\).

Worked Example: Solving the Normal Equations

Suppose the three observed pairs are \((x,y)=(0,1),(1,2),(2,2)\). Then

X= \begin{pmatrix} 1&0\\ 1&1\\ 1&2 \end{pmatrix}, \qquad y= \begin{pmatrix} 1\\ 2\\ 2 \end{pmatrix}.

The determinant of \(X'X\) below is \(6\), so \(X\) has full column rank and the solution is unique. The normal equations and their solution are

X'X= \begin{pmatrix} 3&3\\ 3&5 \end{pmatrix}, \qquad X'y= \begin{pmatrix} 5\\ 6 \end{pmatrix}, \qquad (X'X)^{-1}=\frac{1}{6} \begin{pmatrix} 5&-3\\ -3&3 \end{pmatrix}, $$ $$ \hat{\beta} =\frac{1}{6} \begin{pmatrix} 5&-3\\ -3&3 \end{pmatrix} \begin{pmatrix} 5\\ 6 \end{pmatrix} = \begin{pmatrix} 7/6\\ 1/2 \end{pmatrix}.

The estimated intercept is \(7/6\) outcome units, and the fitted slope is \(1/2\) outcome unit per predictor unit. The fitted values are \((7/6,5/3,13/6)'\). The residuals are \((-1/6,1/3,-1/6)'\), and \(SSR=1/36+1/9+1/36=1/6\). Their sum is zero, and their inner product with the regressor values is \(0(-1/6)+1(1/3)+2(-1/6)=0\), verifying both normal equations.

Worked Example With Linearly Dependent Regressors

Worked Example: Why Coefficients May Not Be Unique

Suppose three hypothetical observations have \(y=(1,2,4)'\), and the two columns of \(X\) are a column of ones and a column of twos:

X= \begin{pmatrix} 1&2\\ 1&2\\ 1&2 \end{pmatrix}.

The second column is twice the first, so \(X\) is not full column rank. Here \(X'X=\begin{pmatrix}3&6\\6&12\end{pmatrix}\), whose determinant is \(36-36=0\), and there is no inverse. The normal equations are

\begin{pmatrix} 3&6\\ 6&12 \end{pmatrix} \begin{pmatrix} b_1\\ b_2 \end{pmatrix} = \begin{pmatrix} 7\\ 14 \end{pmatrix}, \qquad\text{so}\qquad b_1+2b_2=\frac{7}{3}.

Every pair satisfying this equation gives the same constant fitted value \(7/3\) for all three observations. For example, \((b_1,b_2)=(7/3,0)\) and \((1,2/3)\) both produce that fitted vector. The residuals are \((-4/3,-1/3,5/3)'\), so the minimum sum of squared residuals is \(16/9+1/9+25/9=14/3\). The coefficient allocation between the two identical-in-information columns is not determined, even though the best fitted vector is.

What the Derivation Does—and Does Not—Establish

Least squares is a criterion for choosing coefficients from a given \(X\) and \(y\). The derivation establishes the algebraic minimizer and the role of the rank condition; it does not, by itself, establish that a coefficient is unbiased, consistent, statistically significant, or causal. Those conclusions require additional assumptions and arguments. In particular, the calculation of the minimizer does not require that the conditional mean of \(y\) be exactly linear in the regressors.

The normal equations also have a geometric reading: the fitted vector is the closest vector in the column space of \(X\) to \(y\), and the residual is orthogonal to every column of \(X\). That reading is useful as a check, but the computational formula still depends on the actual design matrix. If two or more columns encode the same linear direction, the fitted vector can be determined while the individual coefficients cannot.

Common Mistakes

Assuming an intercept is automatic. It is present only if a column of ones is included in \(X\). Always inspect the design matrix before interpreting the fitted specification.
Using \((X'X)^{-1}\) without checking rank. The inverse formula applies only when the columns of \(X\) are linearly independent. If they are dependent, the normal equations may still have solutions, but the coefficient vector is not unique.
Treating the normal equations as statistical assumptions. They are the first-order conditions for the sample least-squares problem, not assumptions about how the data were generated.
Confusing a unique fitted vector with unique coefficients. With dependent columns, multiple coefficient vectors can produce exactly the same fitted values. Do not interpret an individual coefficient as determined when the design does not identify it.

Key takeaway: OLS minimizes \((y-Xb)'(y-Xb)\), which yields the normal equations \(X'X\hat{\beta}=X'y\). If \(X\) has full column rank, \(\hat{\beta}=(X'X)^{-1}X'y\) is unique. Without full column rank, a minimizing fitted vector still exists, but its coefficient representation is not unique.

Check Your Understanding

Use the objective, the normal equations, and the rank condition to answer these questions.

For an \(n \times k\) design matrix \(X\), write the sample sum of squared residuals as a function of \(b\).
What condition on the columns of \(X\) makes \((X'X)^{-1}\) exist?
If \(X\) has linearly dependent columns, can a least-squares fitted vector still be unique? Explain briefly.
A researcher wants an intercept in the regression. What must be included in \(X\)?
Why do the normal equations alone not establish that an OLS coefficient is unbiased or causal?