Uniqueness of the Full-Rank Least-Squares Estimate

Why Uniqueness Matters

In multiple regression, least squares chooses coefficient values that minimize the sum of squared differences between observed responses and fitted responses. For the coefficients to have a unique least-squares estimate, the predictors must provide enough independent information to distinguish their effects. The relevant condition is full column rank of the design matrix.

This is an algebraic condition on the observed design matrix, not a distributional assumption about the errors. Full column rank makes the cross-product matrix invertible and gives one coefficient vector that minimizes the residual sum of squares. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Let \(X\) be an \(n \times p\) design matrix, with \(n\) observations and \(p\) columns, and let \(y\) be the \(n \times 1\) response vector. The matrix \(X\) has full column rank if its \(p\) columns are linearly independent. Equivalently, \(Xa=0\) implies \(a=0\) for every \(p \times 1\) vector \(a\). In particular, full column rank requires \(n \ge p\).

The coefficient vector is denoted by \(\beta\), and a candidate estimate by \(b\). The least-squares criterion is the residual sum of squares \(Q(b)=(y-Xb)^\top(y-Xb)\). Its minimizer is the ordinary least-squares estimate \(\hat{\beta}\). The normal equations, derived in the earlier material on ordinary least-squares normal equations, are \(X^\top X\hat{\beta}=X^\top y\).

Why Full Column Rank Gives an Invertible Matrix

The key fact is that \(X^\top X\) is invertible exactly when the columns of \(X\) are linearly independent. To see why, consider any \(p \times 1\) vector \(a\). Then

a^\top X^\top X a=(Xa)^\top(Xa)=\lVert Xa\rVert^2.

A squared length is nonnegative, and it equals zero only when \(Xa=0\). If \(X\) has full column rank, \(Xa=0\) forces \(a=0\). Therefore \(a^\top X^\top X a>0\) for every nonzero \(a\). This property is called positive definiteness. A positive definite square matrix has no nonzero vector in its null space and is therefore invertible.

Applying the inverse of \(X^\top X\) to the normal equations gives the estimate:

\hat{\beta}=(X^\top X)^{-1}X^\top y.

This formula requires full column rank. If that condition fails, the inverse in the formula does not exist, and the formula cannot be used. There may still be least-squares fitted values, but the coefficient vector need not be unique; the detailed analysis of rank deficiency is a separate topic.

Theorem: If \(X\) has full column rank, then \(X^\top X\) is invertible and the least-squares coefficient vector is unique. It is \(\hat{\beta}=(X^\top X)^{-1}X^\top y\). No normality or constant-variance assumption on regression errors is needed for this algebraic uniqueness result.

Why the Least-Squares Minimizer Is Unique

Invertibility gives a solution to the normal equations. To show that this solution is not merely one candidate but the unique minimizer, expand the criterion around it. From the normal equations, \(X^\top X\hat{\beta}=X^\top y\). For any other coefficient vector \(b\), write \(b=\hat{\beta}+d\), where \(d=b-\hat{\beta}\). Then

\begin{aligned} Q(b) &=(y-X\hat{\beta}-Xd)^\top(y-X\hat{\beta}-Xd)\\ &=Q(\hat{\beta})-2d^\top X^\top(y-X\hat{\beta})+d^\top X^\top Xd\\ &=Q(\hat{\beta})+d^\top X^\top Xd. \end{aligned}

The middle term is zero because the normal equations imply \(X^\top(y-X\hat{\beta})=0\). When \(X\) has full column rank, positive definiteness gives \(d^\top X^\top Xd>0\) whenever \(d\ne0\). Thus every \(b\ne\hat{\beta}\) has \(Q(b)>Q(\hat{\beta})\). That strict inequality establishes uniqueness.

Check the design.
Identify the \(n \times p\) matrix \(X\), including any intercept column, and establish that its columns are linearly independent.

Form the cross-products.
Calculate \(X^\top X\) and \(X^\top y\), checking dimensions: they are \(p \times p\) and \(p \times 1\), respectively.

Solve the normal equations.
When \(X\) has full column rank, solve \(X^\top X\hat{\beta}=X^\top y\), or use the inverse formula.

Verify the result.
Substitute the calculated coefficients into the normal equations and check that both sides agree.

Worked Matrix Calculations

Worked Example: An Intercept and One Predictor

A hypothetical study has three observations. Its design matrix and response vector are

X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, \qquad y=\begin{pmatrix}2\\4\\5\end{pmatrix}.

Method. Calculate the cross-products, check full column rank, and solve the normal equations. This example avoids the mistake of assuming that an intercept makes the design full rank automatically; the columns still need to be checked.

The columns of \(X\) are \((1,1,1)^\top\) and \((0,1,2)^\top\). They are not scalar multiples, so they are linearly independent. Now,

X^\top X=\begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad X^\top y=\begin{pmatrix}2+4+5\\0(2)+1(4)+2(5)\end{pmatrix} =\begin{pmatrix}11\\14\end{pmatrix}.

The determinant of \(X^\top X\) is \(3(5)-3(3)=6\), which is nonzero. Its inverse and the resulting estimate are

(X^\top X)^{-1} =\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}, \qquad \hat{\beta} =\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix} \begin{pmatrix}11\\14\end{pmatrix} =\begin{pmatrix}13/6\\9/6\end{pmatrix} =\begin{pmatrix}13/6\\3/2\end{pmatrix}.

Check. Multiplying back gives \(3(13/6)+3(3/2)=11\) and \(3(13/6)+5(3/2)=14\), exactly the two entries of \(X^\top y\). The estimate satisfies the normal equations, and full rank makes it the unique least-squares estimate.

Worked Example: A Full-Rank Matrix with a Larger Coefficient

For a hypothetical three-observation data set, take

X=\begin{pmatrix}1&2\\1&1\\1&0\end{pmatrix}, \qquad y=\begin{pmatrix}0\\-2\\21\end{pmatrix}.

Method. Form \(X^\top X\) and \(X^\top y\), then solve. The common arithmetic trap here is to compute \(140-39\) as \(11\); the correct difference is \(101\).

The two columns are independent: if the second were a multiple of the first, its entries would be equal, but they are \(2,1,0\). The cross-products are

X^\top X= \begin{pmatrix} 1+1+1&2+1+0\\ 2+1+0&2^2+1^2+0^2 \end{pmatrix} =\begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad X^\top y= \begin{pmatrix}0-2+21\\2(0)+1(-2)+0(21)\end{pmatrix} =\begin{pmatrix}19\\-2\end{pmatrix}.

The determinant is \(15-9=6\), so the inverse exists. Multiplication gives

\hat{\beta} =\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix} \begin{pmatrix}19\\-2\end{pmatrix} =\frac{1}{6}\begin{pmatrix}5(19)-3(-2)\\-3(19)+3(-2)\end{pmatrix} =\begin{pmatrix}101/6\\-63/6\end{pmatrix} =\begin{pmatrix}101/6\\-21/2\end{pmatrix}.

The first numerator is \(95+6=101\), so \(101/6=16.8333\ldots\). Rounding the final estimates to two decimal places gives \(\hat{\beta}=(16.83,-10.50)^\top\). For a check, \(3(101/6)+3(-21/2)=19\), while \(3(101/6)+5(-21/2)=-2\). Both entries agree with \(X^\top y\).

Worked Example: Three Independent Columns

A hypothetical four-observation regression uses an intercept and two predictors:

X=\begin{pmatrix} 1&0&0\\ 1&1&0\\ 1&0&1\\ 1&1&1 \end{pmatrix}, \qquad y=\begin{pmatrix}2\\3\\5\\8\end{pmatrix}.

Method. Check that all three columns are independent, then solve the three normal equations. The trap to avoid is checking only that each pair of columns differs; the required condition is independence of the entire set of columns.

Suppose \(a_0(1,1,1,1)^\top+a_1(0,1,0,1)^\top+a_2(0,0,1,1)^\top=0\). The first row gives \(a_0=0\); the second then gives \(a_1=0\), and the third gives \(a_2=0\). Thus the columns are independent. Direct calculation gives

X^\top X=\begin{pmatrix}4&2&2\\2&2&1\\2&1&2\end{pmatrix}, \qquad X^\top y=\begin{pmatrix}18\\11\\13\end{pmatrix}.

The determinant of \(X^\top X\) is \(4(4-1)-2(4-2)+2(2-4)=4\), so the matrix is invertible. The equations are \(4b_0+2b_1+2b_2=18\), \(2b_0+2b_1+b_2=11\), and \(2b_0+b_1+2b_2=13\). Dividing the first equation by 2 gives \(2b_0+b_1+b_2=9\). Subtracting this from the second gives \(b_1=2\); subtracting it from the third gives \(b_2=4\). Substitution into \(2b_0+b_1+b_2=9\) gives \(2b_0+6=9\), hence \(b_0=3/2\).

Check. The first normal-equation left side is \(4(3/2)+2(2)+2(4)=18\). The second is \(2(3/2)+2(2)+4=11\), and the third is \(2(3/2)+2+2(4)=13\). Therefore \(\hat{\beta}=(3/2,2,4)^\top\), and it is unique.

What Uniqueness Does—and Does Not—Tell You

Full column rank guarantees one coefficient vector minimizing the observed sum of squared residuals. It does not, by itself, establish that the fitted model is scientifically appropriate, that the coefficients are precise, or that an estimate is unbiased. Those statistical properties involve the regression model and its assumptions, such as the conditional mean and error variance. In particular, normal errors are not needed to establish uniqueness.

In calculations, rank and arithmetic are separate checks. A nonzero determinant of \(X^\top X\) verifies invertibility when the matrix is small enough to calculate directly. For a larger design, the defining condition remains that no nonzero linear combination of the columns equals the zero vector. Once full rank is established, verify the computed estimate by substituting it into \(X^\top X\hat{\beta}=X^\top y\). That check can catch a sign error or an incorrect numerator before it is carried into interpretation.

Key takeaway: Full column rank makes \(a^\top X^\top Xa=\lVert Xa\rVert^2\) positive for every nonzero \(a\). Consequently, \(X^\top X\) is invertible, the normal equations have one solution, and the least-squares criterion has a unique minimizer.

Check Your Understanding

For each problem, use the rank condition and normal equations where needed.

A design matrix has \(n=8\) observations and \(p=5\) columns, and its columns are linearly independent. Is \(X^\top X\) invertible, and how many least-squares coefficient vectors minimize the criterion? (Answer: Yes; exactly one.)
For \(X^\top X=\begin{pmatrix}4&2\\2&3\end{pmatrix}\) and \(X^\top y=(10,7)^\top\), solve the normal equations. (Answer: \(\hat{\beta}=(2,1)^\top\).)
A matrix \(X\) has two identical columns. Can it have full column rank? Does the inverse formula for \(\hat{\beta}\) apply? (Answer: No; the columns are dependent, so \(X^\top X\) is singular and the inverse formula does not apply.)
For \(X^\top X=\begin{pmatrix}3&1\\1&2\end{pmatrix}\) and \(X^\top y=(8,5)^\top\), determine whether the solution is unique and calculate it. (Answer: Unique; \(\hat{\beta}=(11/5,7/5)^\top\).)
If \(X\) has full column rank but the errors are not normally distributed, does the least-squares coefficient vector remain unique? (Answer: Yes; uniqueness follows from the design matrix's full column rank, not from normality.)

What You'll Learn