Why Rank Matters in Multiple Regression
A multiple regression model can contain several predictors, but the data must provide distinct information about their effects if we are to estimate every coefficient uniquely. If one design-matrix column is an exact linear combination of other columns, changing some coefficients together leaves every fitted value unchanged. Least squares can still find the best-fitting values for the observations, but it cannot choose a unique coefficient vector from all the possibilities.
This issue is called rank deficiency. It is an exact algebraic problem, not merely a warning that predictors are strongly related. The key task is to determine which columns are redundant, then describe what remains identifiable: often certain combinations of coefficients and the fitted values, but not the individual coefficients.
Write the regression model in matrix form as \(y=X\beta+\varepsilon\), where \(y\) is the \(n\)-vector of responses, \(X\) is the design matrix, and \(\beta\) is the \(p\)-vector of coefficients. The least-squares estimate minimizes the residual sum of squares \(\|y-X\beta\|^2\). The algebraic question here does not require a probability distribution for \(\varepsilon\): it concerns whether distinct coefficient vectors can produce the same \(X\beta\).
Redundant Columns and the Intercept
An intercept contributes a column of ones to \(X\). A predictor that is constant across all observations is therefore redundant with the intercept. Likewise, if one predictor is a constant multiple of another, their columns cannot separately determine their effects. More generally, a relationship involving several predictors can make a column redundant even when no two columns are identical.
To test a proposed relation, check it row by row across the observed data. If, for example, a predictor column \(z\) satisfies \(z=2x\) for every observation, then the corresponding terms in the linear predictor obey \(\beta_1x+\beta_2z=(\beta_1+2\beta_2)x\). The data can identify the combined slope \(\beta_1+2\beta_2\), not the two slopes individually.
Worked Example: A Predictor That Is Twice Another Predictor
A hypothetical insurer models three annual claim-cost observations using an intercept and two recorded predictors. The predictor values are \(x=(0,1,2)\), \(z=(0,2,4)\), and the response values are \(y=(1,3,4)\). Find the least-squares fitted values and describe all coefficient vectors that produce them.
Method and rank check. The model is \(y_i=\beta_0+\beta_1x_i+\beta_2z_i+\varepsilon_i\). Since \(z_i=2x_i\) in every row, the design matrix has columns \(\mathbf{1},x,2x\), so its rank is 2 rather than 3. Rewrite the linear predictor as \(\beta_0+\gamma x_i\), where \(\gamma=\beta_1+2\beta_2\).
Fit the identifiable simple regression. The sample means are \(\bar{x}=(0+1+2)/3=1\) and \(\bar{y}=(1+3+4)/3=8/3\). The centered sums are \(\sum_i(x_i-\bar{x})^2=1+0+1=2\) and \(\sum_i(x_i-\bar{x})(y_i-\bar{y})=5/3+0+4/3=3\). Thus the fitted slope is \(3/2\), and the fitted intercept is \(8/3-(3/2)(1)=7/6\).
Recover the coefficient family. The original model must have \(\beta_0=7/6\) and \(\beta_1+2\beta_2=3/2\). Let \(\beta_2=t\), where \(t\) can be any real number. Then every least-squares coefficient vector is \((\beta_0,\beta_1,\beta_2)=(7/6,\,3/2-2t,\,t)\).
Check. Each such vector gives \(\beta_0+\beta_1x+\beta_2z=7/6+(3/2-2t)x+2tx=7/6+(3/2)x\). The residuals are \((-1/6,1/3,-1/6)\), so the residual sum of squares is \(1/36+1/9+1/36=1/6\). The common mistake this avoids is reporting separate, uniquely estimated slopes for predictors whose columns are exact multiples.
Why the Coefficients Are Nonunique
The normal equations for least squares are \(X^\mathsf{T}X\widehat{\beta}=X^\mathsf{T}y\). When \(X\) has full column rank, \(X^\mathsf{T}X\) is invertible and the unique solution is \((X^\mathsf{T}X)^{-1}X^\mathsf{T}y\). When columns are dependent, \(X^\mathsf{T}X\) is singular, so that inverse formula does not apply. The normal equations still have at least one least-squares solution; there may be infinitely many.
The reason is direct: \(X(\widehat{\beta}+cv)=X\widehat{\beta}+cXv=X\widehat{\beta}\). The vector \(v\) is a nonzero direction in the null space of \(X\): moving the coefficients in that direction changes the coefficient vector but not its predictions on the observed design rows.
Conversely, if two coefficient vectors give the same fitted vector, their difference is in this null space: \(X\beta=X\widetilde{\beta}\) implies \(X(\beta-\widetilde{\beta})=0\). Thus the ambiguity is precisely the set of coefficient changes that the design matrix cannot detect.
Fitted Values Can Still Be Unique
Nonunique coefficients do not imply nonunique least-squares fitted values. The fitted vector is the projection of \(y\) onto the column space of \(X\). A projection onto a fixed subspace is unique, even if the columns used to describe that subspace are redundant. Different coefficient vectors can therefore describe the same point in the column space.
This distinction is useful in practice. A model may give a definite fitted value for every observed row, and a definite residual sum of squares, even though its individual coefficients cannot be determined. But a coefficient interpretation that claims one redundant predictor has a particular effect, holding the others fixed, is not supported by the observed design.
Worked Example: An Intercept and a Constant Predictor
In a hypothetical portfolio study, a response \(y\) is modeled using an intercept and predictor \(x\). The three observed predictor values are \(x=(5,5,5)\), and the responses are \(y=(4,6,8)\). Determine the fitted values, residual sum of squares, and coefficient constraint.
Method and rank check. The design columns are \(\mathbf{1}=(1,1,1)^\mathsf{T}\) and \(x=(5,5,5)^\mathsf{T}=5\mathbf{1}\). The rank is 1, not 2. Every linear predictor is constant across the observations: \(\beta_0+5\beta_1\).
Find the best constant fit. The mean response is \(\bar{y}=(4+6+8)/3=6\). The least-squares constant is 6, so the fitted vector is \((6,6,6)\). The residuals are \((-2,0,2)\), and the residual sum of squares is \(4+0+4=8\).
Describe all solutions. The coefficients must satisfy \(\beta_0+5\beta_1=6\). For any real \(t\), choosing \(\beta_1=t\) and \(\beta_0=6-5t\) gives the same fitted values. For example, \((6,0)\) and \((1,1)\) both give fitted values of 6 for all three rows.
Check. In both example coefficient vectors the fitted value is \(6+5(0)=6\) or \(1+5(1)=6\), respectively. The common mistake this avoids is treating a constant predictor as if it provided information distinct from the intercept.
A Linear Relation Involving Several Columns
Redundancy need not be obvious from a pair of identical columns. With an intercept present, one predictor can equal a constant minus another predictor. In that case, the intercept and the two predictors are dependent as a group. The fitted values may be uniquely determined by an intercept and one effective slope, while the original three coefficients remain ambiguous.
Worked Example: Two Predictors Whose Sum Is Constant
A hypothetical model uses an intercept and predictors \(x\) and \(z\), with observed rows \((x,z)=(0,3),(1,2),(2,1)\) and responses \(y=(2,5,4)\). Determine the rank, fitted values, and all least-squares coefficient vectors.
Method and dependence check. In every row, \(z=3-x\), so the \(z\) column equals three times the intercept column minus the \(x\) column. The three columns have rank 2. Rewrite the linear predictor as \(\beta_0+\beta_1x+\beta_2z=(\beta_0+3\beta_2)+(\beta_1-\beta_2)x\).
Fit against \(x\). The means are \(\bar{x}=1\) and \(\bar{y}=11/3\). The centered sums are \(\sum_i(x_i-\bar{x})^2=2\) and \(\sum_i(x_i-\bar{x})(y_i-\bar{y})=5/3+0+1/3=2\). Therefore the effective slope is \(2/2=1\), and the effective intercept is \(11/3-1=8/3\).
State the original coefficient constraints. Matching the effective coefficients gives \(\beta_0+3\beta_2=8/3\) and \(\beta_1-\beta_2=1\). Setting \(\beta_2=t\), every solution is \((\beta_0,\beta_1,\beta_2)=(8/3-3t,\,1+t,\,t)\).
Check. The fitted values at \(x=0,1,2\) are \(8/3,11/3,14/3\). Subtracting the responses gives residuals \(2/3,-4/3,2/3\), whose squared sum is \(4/9+16/9+4/9=8/3\). The common mistake this avoids is checking only whether two predictor columns are identical, rather than checking whether a column is a combination of several others, including the intercept.
Common Mistake: Inverting a Singular Matrix
A frequent error is to use the full-rank formula even after discovering dependent columns. For the first worked example, \(z=2x\), so the columns of \(X\) are dependent and \(X^\mathsf{T}X\) is singular. Writing \(\widehat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y\) is not a valid calculation: the inverse does not exist. It is also wrong to select a convenient-looking coefficient vector and call it the unique OLS estimate without explaining that other vectors give exactly the same fit.
The correction is to state the identifiable combinations and the set of solutions. In that example, the fitted intercept is \(7/6\), and the identifiable slope combination is \(\beta_1+2\beta_2=3/2\). One could impose an extra convention to select one solution, but that convention would not make the separate slopes identifiable from these data.
Exact dependence should also be distinguished from predictors that are merely highly related. Exact dependence makes the coefficient solution nonunique. Strong but imperfect dependence does not by itself make the design rank deficient, though it can make coefficient estimates sensitive to the data. This tutorial concerns the exact case.
Check Your Understanding
For each problem, identify the relevant dependence and state exactly what the design permits you to determine.
- A model has an intercept and predictor values \(x=(4,4,4)\), with responses \(y=(3,5,7)\). What is the rank, and what equation must the coefficients satisfy for the least-squares fit? (Answer: Rank 1; \(\beta_0+4\beta_1=5\), giving fitted values \((5,5,5)\).)
- In a model with an intercept, predictor columns satisfy \(z=-2x\) in every row. What nonzero coefficient direction leaves fitted values unchanged, and what slope combination is identifiable? (Answer: A null direction is \((0,2,1)^\mathsf{T}\); the identifiable slope combination is \(\beta_1-2\beta_2\).)
- A design has columns \(\mathbf{1},x,z\), where \(z=1+x\) in every row. State its rank if \(x\) is not constant, and rewrite the linear predictor using \(x\). (Answer: Rank 2; \((\beta_0+\beta_2)+(\beta_1+\beta_2)x\).)
- Four design columns are \(c_1=(1,1,1,1)^\mathsf{T}\), \(c_2=(0,1,2,3)^\mathsf{T}\), and \(c_3=(1,3,5,7)^\mathsf{T}\). Find their rank and a nonzero null vector. (Answer: Rank 2; \((-1,-2,1)^\mathsf{T}\), since \(-c_1-2c_2+c_3=0\).)
- A model uses an intercept and predictors \(x,z\), with \(z=3x\), \(x=(0,1,2)\), and \(y=(1,2,5)\). Find the identifiable intercept and slope combination from the least-squares fit. (Answer: The intercept is \(2/3\), and \(\beta_1+3\beta_2=2\); the fitted values are \((2/3,8/3,14/3)\).)