Why the Starting Fit Matters
Backward elimination begins with a specified full model and removes predictors from it in stages. That starting model must be fit before the procedure can compare its coefficients or other fit summaries with those of reduced models. A key feasibility question is therefore not just how many predictors are present, but whether the observed predictor columns contain enough independent information to estimate their coefficients separately.
The number of predictors gives a quick necessary check, but it is not the whole check. The intercept is also a model parameter, and even a model with fewer parameters than observations can fail to have uniquely determined coefficients if one column is an exact linear combination of others.
Here, “uniquely fit” refers to uniquely determined ordinary least-squares coefficients. A least-squares minimizer still exists when the design is rank deficient, and its fitted values are unique; it is the coefficient vector that may not be. This distinction matters because conventional backward elimination interprets individual coefficients, and often their standard errors or tests. If those coefficients are not separately identified, such steps are not well-defined without an additional convention.
The Rank Condition for a Unique Fit
Write the linear regression model in matrix form as \(y=X\beta+\varepsilon\), where \(y\) is the vector of observed responses, \(\beta\) contains the intercept and predictor coefficients, and \(\varepsilon\) is the error vector. Ordinary least squares chooses a vector \(b\) to minimize the residual sum of squares \(\operatorname{RSS}(b)=(y-Xb)'(y-Xb)\). The rank condition concerns the observed design matrix \(X\); it does not require a particular probability distribution for the errors.
This formula applies when \(X\) has full column rank, so that \(X'X\) is invertible. In particular, the number of columns cannot exceed the number of rows: \(p+1\le n\). This count is necessary, not sufficient. Even when there are enough observations, exact dependence among the columns can still prevent full rank.
Why Full Column Rank Gives Uniqueness
The least-squares criterion is a quadratic function of the coefficient vector. Differentiating it with respect to \(b\) gives the normal equations. Full column rank ensures that these equations have exactly one solution.
Expanding and differentiating \((y-Xb)'(y-Xb)\) gives the normal equations \(X'Xb=X'y\) at a least-squares minimum.
For any vector \(a\), \(a'X'Xa=(Xa)'(Xa)=\lVert Xa\rVert^2\). If \(X\) has full column rank, \(Xa\ne 0\) for every nonzero \(a\), so this quantity is positive and \(X'X\) is invertible.
The normal equations then have the single solution \(\hat{\beta}=(X'X)^{-1}X'y\).
If \(X\) is rank deficient, there is a nonzero vector \(a\) such that \(Xa=0\). For any coefficient vector \(b\), \(X(b+ca)=Xb\) for every scalar \(c\). Thus moving along that direction leaves the fitted values unchanged. A least-squares solution exists, but the data cannot distinguish all coefficient vectors that give those same fitted values.
Worked Example: Verifying a Full-Rank Starting Model
A hypothetical pricing analyst has \(n=4\) observations and proposes two predictors, \(x_1\) and \(x_2\), plus an intercept. The design matrix and response vector are
Method. Check the rank and, if full, solve the least-squares equations. There are three columns (intercept and two predictor columns) and four observations, so the necessary count \(p+1\le n\) holds: \(3\le4\).
The columns are independent: if \(c_0+c_1x_1+c_2x_2=0\) in every row, the first row gives \(c_0=0\), the second gives \(c_1=0\), and the third gives \(c_2=0\). Therefore \(X\) has rank 3 and the coefficients are uniquely estimable.
The normal-equation components are
Solving \((X'X)\hat{\beta}=X'y\), the candidate \(\hat{\beta}=(2,3,2)'\) gives the first equation \(4(2)+2(3)+2(2)=18\), the second \(2(2)+2(3)+2=12\), and the third \(2(2)+3+2(2)=11\). Because the solution is unique, \(\hat{\beta}_0=2\), \(\hat{\beta}_1=3\), and \(\hat{\beta}_2=2\).
Check. The fitted values are \(2, 2+3=5, 2+2=4,\) and \(2+3+2=7\), exactly the four observed responses. The residuals are all zero. This starting model is feasible for ordinary backward elimination in the coefficient-uniqueness sense. The trap avoided is treating the predictor count alone as the rank check; here both the count and column independence were verified.
Rank Deficiency: Predictors That Cannot Be Separated
Rank deficiency can arise even when \(p+1\le n\). For example, a predictor may be constant and therefore collinear with the intercept, or two predictors may be exact multiples of each other. In such cases, the model may identify a combined contribution while failing to identify each coefficient individually. Whether even that combined contribution is identifiable depends on the other columns, including the intercept.
Worked Example: Two Exactly Proportional Predictors
In a hypothetical study with three observations, the predictor values satisfy \(x_1=(0,1,2)'\), \(x_2=(0,2,4)'=2x_1\), and the response is \(y=(1,4,7)'\). Determine which parameters the data identify.
Method. Substitute the observed predictor relationship into the model \(y_i=\beta_0+\beta_1x_{1i}+\beta_2x_{2i}+\varepsilon_i\). Since \(x_2=2x_1\), the fitted value at observation \(i\) is \(\beta_0+(\beta_1+2\beta_2)x_{1i}\).
The first observation has \(x_1=0\) and response 1, so the exact-fit equations require \(\beta_0=1\). At the second observation, \(x_1=1\) and \(y=4\), so \(\beta_1+2\beta_2=3\). At the third, \(x_1=2\) and \(y=7\), giving \(1+2(\beta_1+2\beta_2)=7\), which confirms the same combined slope of 3.
Because \(x_1\) varies across the observations, it is not collinear with the intercept. The intercept and the combined slope \(\beta_1+2\beta_2\) are identifiable. But the individual slopes are not: for any value \(t\), \(\beta_2=t\) and \(\beta_1=3-2t\) produce the same combined slope. For example, \((\beta_1,\beta_2)=(3,0)\) and \((1,1)\) both give the fitted values \((1,4,7)'\).
Check. The design has three columns but rank 2, since its second and third columns are proportional. Its fitted values are uniquely determined here, but its two predictor coefficients are not. Conventional backward elimination cannot assign a unique separate coefficient test to each of these exactly aliased predictors. The mistake avoided is to say that proportional predictors always identify a combined slope without checking that the varying predictor is distinct from the intercept.
More Predictors Than Observations
If there are more predictors than observations, then there are more design-matrix columns than rows even before considering any further dependencies. With an intercept, the relevant comparison is between \(p+1\) and \(n\). If \(p+1>n\), the columns cannot all be linearly independent, so a unique ordinary least-squares coefficient vector is impossible.
Worked Example: A Starting Model With More Predictors Than Observations
A hypothetical monitoring study has \(n=2\) observations and proposes \(p=3\) predictors. Its design matrix and responses are
Method. First count all parameters: the three predictors plus the intercept give four columns, but there are only two rows. Since \(\operatorname{rank}(X)\le2<4\), the design cannot have full column rank.
For an exact fit, the first row requires \(\beta_0=2\). The second requires \(\beta_0+\beta_1+\beta_2=7\), hence \(\beta_1+\beta_2=5\). The coefficient \(\beta_3\) does not appear in either row and is unrestricted. For instance, both \((\beta_0,\beta_1,\beta_2,\beta_3)=(2,5,0,0)\) and \((2,0,5,9)\) give fitted values \((2,7)'\). There are infinitely many least-squares coefficient vectors.
Check. The residual sum of squares can be zero, but that does not make the coefficients unique. Backward elimination cannot begin with a conventional unique full-model coefficient vector. The trap avoided is interpreting a perfect fit as evidence that the model parameters are individually estimable.
Practical Feasibility Checks and Common Traps
Before starting backward elimination, check both the column count and the structure of the design matrix. Full column rank is a property of the actual observed predictor values, not just the names of the variables. A model may pass a count check and still contain exact dependencies.
| Check | What it establishes | What it does not establish |
|---|---|---|
| \(p+1\le n\) | The number of design columns does not exceed the number of observations. | It does not rule out exact dependence among columns. |
| \(\operatorname{rank}(X)=p+1\) | The ordinary least-squares coefficient vector is unique. | It does not by itself verify other modeling assumptions used for inference. |
| Software returns a fitted model | A computational procedure produced some fit representation. | It does not guarantee that every named coefficient is uniquely estimable; software may omit or reparameterize dependent columns. |
Common exam trap: forget the intercept. Suppose \(n=5\) and \(p=5\). The incorrect check \(p\le n\) passes. But the design has \(p+1=6\) columns, and \(\operatorname{rank}(X)\le5<6\), so full column rank is impossible. The correction is to count the intercept as a parameter and require \(p+1\le n\); then separately check for exact linear dependence.
A related distinction is that near dependence is not the same as exact rank deficiency. Very similar predictor columns may lead to unstable coefficient estimates, but if they are not exactly dependent, the matrix can still have full rank and a unique ordinary least-squares solution. The feasibility issue addressed here is exact uniqueness; it should not be confused with the broader question of how stable a fit is.
Check Your Understanding
For each situation, decide whether the stated information guarantees a unique ordinary least-squares coefficient vector, and identify the relevant condition.
- A model has \(n=12\) observations and \(p=10\) predictors plus an intercept. The design is known to have full column rank. Is the coefficient vector unique? (Answer: Yes; there are \(p+1=11\) columns, and the stated rank is 11.)
- A model has \(n=8\), \(p=8\), and an intercept. Can its design have full column rank? (Answer: No; it has 9 columns, so its rank is at most 8.)
- A model has \(n=6\), two predictors and an intercept; the second predictor is exactly twice the first, and the first predictor varies across observations. What is the rank, and what can be identified about the two slopes? (Answer: The rank is 2; the intercept and the combined slope \(\beta_1+2\beta_2\) can be identified, but the two slopes cannot be identified separately.)
- A model has \(n=5\) and \(p=3\), but one predictor is constant at 4 for all observations and the model includes an intercept. Is full column rank possible? (Answer: No; the constant predictor column is four times the intercept column, so the four columns are dependent.)
- A rank-deficient model has several coefficient vectors that all produce the same fitted values. Which quantity is nonunique: the least-squares fitted values or the coefficient vector? (Answer: The coefficient vector is nonunique; the least-squares fitted values are unique.)