Why Constrained Least Squares Is Easy to Misread
In a linear model, constrained least squares minimizes the residual sum of squares subject to a specified set of coefficient equations. The calculations can look routine, but errors often arise before minimization: an analyst may count dependent equations as independent, infer a change in fitted values from a change in coefficients, or impose a restriction that the true parameter does not satisfy. These are different problems, and they have different consequences.
Let \(y\in\mathbb{R}^n\) be the observed response vector, \(X\) an \(n\times p\) design matrix, and \(\beta\in\mathbb{R}^p\) the coefficient vector. A system of linear restrictions has the form \(C\beta=d\), where \(C\) has \(p\) columns. The constrained least-squares problem is to minimize \(\|y-X\beta\|^2\) over the coefficients satisfying that system.
These definitions separate three questions: whether the constraints are feasible, whether an equation changes the feasible coefficient set, and whether the feasible coefficients produce different fitted values. A constrained least-squares solution exists when \(S\) is nonempty: in finite dimensions, \(M\) is then a nonempty closed affine set, so a closest fitted vector to \(y\) exists. The fitted vector is unique, although more than one coefficient vector can produce it.
Dependent Constraint Rows: Check Feasibility First
A row of \(C\) may be a linear combination of other rows. That dependence alone does not determine whether its equation is redundant. The right-hand sides matter when the remaining equations have a solution; if the remaining system is already inconsistent, deleting one equation may leave the feasible set empty either way.
Proof. If \(S_0\) is nonempty, every \(\beta\in S_0\) satisfies \(c'\beta=a'C_0\beta=a'd_0\). If \(d_c=a'd_0\), every such \(\beta\) also satisfies the deleted equation, so adding it does not change \(S_0\). If \(d_c\ne a'd_0\), no \(\beta\in S_0\) satisfies the deleted equation; adding it changes the nonempty set \(S_0\) to the empty set. Thus, when \(S_0\) is nonempty, redundancy is equivalent to equality of the corresponding right-hand sides. If \(S_0\) is empty, both before and after adding the equation the feasible set is empty, so the equation is redundant by definition. \(\square\)
Worked Example: A Dependent Row and an Inconsistent Remainder
Consider the three equations \(\beta=0\), \(\beta=1\), and \(\beta=2\). The last row of the constraint matrix is a copy of the first row, but its right-hand side differs. Is the last equation redundant?
After deleting the last equation, the first two require \(\beta=0\) and \(\beta=1\) simultaneously. Hence \(S_0=\varnothing\). Adding \(\beta=2\) cannot make that system feasible, so the full system also has feasible set \(\varnothing\). Under the definition above, the last equation is redundant because deleting it leaves the feasible set unchanged.
The usual right-hand-side check does not apply here: \(S_0\) is empty. By contrast, if the remaining system were just \(\beta=0\), then it would be consistent, and adding the dependent equation \(\beta=2\) would change its feasible set from \(\{0\}\) to the empty set. That equation would not be redundant.
Worked Example: Duplicate Consistent Restrictions
Minimize \((\beta_1-2)^2+\beta_2^2\) subject to both \(\beta_1+\beta_2=1\) and \(2\beta_1+2\beta_2=2\). The second equation is twice the first and has the matching right-hand side, so it is redundant.
Using the first equation, set \(\beta_2=1-\beta_1\). The objective becomes \((\beta_1-2)^2+(1-\beta_1)^2\). Its derivative is \(2(\beta_1-2)-2(1-\beta_1)=4\beta_1-6\), which is zero at \(\hat\beta_1=3/2\). The second derivative is \(4>0\), so this is the minimum. Then \(\hat\beta_2=1-3/2=-1/2\), and the minimized objective is \((3/2-2)^2+(-1/2)^2=1/4+1/4=1/2\).
Keeping the duplicate equation does not change the feasible set or this answer. It does, however, make the rows of the constraint matrix linearly dependent. A calculation that requires an inverse of a matrix built as if all restriction rows were independent can therefore fail even though the constrained least-squares problem itself is well-defined.
Coefficient Restrictions Are Not Fitted-Value Restrictions
A restriction is written in terms of \(\beta\), but its effect on predictions depends on \(X\). Two different feasible coefficient sets can map to the same set of fitted vectors under multiplication by \(X\). This is especially important when \(X\) is rank deficient: distinct coefficient vectors can represent the same fitted values.
Proof. Under a given feasible set \(S\), each admissible coefficient vector \(\beta\) produces the fitted vector \(X\beta\). Thus the full collection of possible fits is, by definition, \(XS\). For another feasible set \(S_1\), the possible fits are \(XS_1\). The fitted-value models are identical exactly when \(XS=XS_1\). In particular, \(S=S_1\) implies \(XS=XS_1\), but distinct sets can also have identical images under \(X\) when their differences lie in directions mapped to zero or when their images otherwise coincide. \(\square\)
Worked Example: A Coefficient Restriction That Does Not Change the Fits
Let \(X=\begin{pmatrix}1&1\\2&2\end{pmatrix}\) and observe \(y=(1,3)'\). Without restrictions, every fitted vector has the form \((\gamma,2\gamma)'\), where \(\gamma=\beta_1+\beta_2\). Now impose the coefficient restriction \(\beta_1=0\). Under that restriction \(\gamma=\beta_2\) can still be any real number, so the fitted-value set remains \(\{(\gamma,2\gamma)':\gamma\in\mathbb{R}\}\), exactly the unrestricted set.
For these data, the least-squares objective as a function of \(\gamma\) is \((1-\gamma)^2+(3-2\gamma)^2\). Its derivative is \(-2(1-\gamma)-4(3-2\gamma)=10\gamma-14\), so the minimizing value is \(\hat\gamma=14/10=1.4\). The fitted vector is \((1.4,2.8)'\). With the restriction, one minimizing coefficient vector is \((0,1.4)'\); without it, every vector summing to \(1.4\) gives the same fit.
Now change the restriction to \(\beta_1+\beta_2=0\). It forces \(\gamma=0\), so the only fitted vector is \((0,0)'\). This restriction does change the fitted-value model. The distinction is determined by the image \(X S\), not by how restrictive a coefficient equation looks on its own.
When a Restriction Excludes the True Parameter
A mathematically solvable constrained problem need not describe the true mean. In the model \(y=X\beta_0+\varepsilon\), suppose \(E[\varepsilon]=0\). If the imposed system \(C\beta=d\) does not contain \(\beta_0\), the restricted model excludes the true coefficient vector. Least squares then projects toward the restricted fitted-value set rather than, in general, toward the true mean \(X\beta_0\).
Proof. Since \(M\) is a nonempty affine subset of a finite-dimensional Euclidean space, it is closed and has a unique nearest point to each \(y\). Minimizing \(\|y-X\beta\|^2\) over \(\beta\in S\) is equivalent to minimizing \(\|y-m\|^2\) over \(m\in M\), so the constrained fitted vector is \(P_M(y)\). Write \(M=m_0+L\), where \(m_0\in M\) and \(L\) is its direction subspace. The projection is \(P_M(y)=m_0+P_L(y-m_0)\), where \(P_L\) is a linear orthogonal projection. Taking expectations, which is valid because this is an affine function of an integrable random vector, gives \(E[P_M(y)]=m_0+P_L(E[y]-m_0)=P_M(\mu)\). The bias relative to \(\mu=X\beta_0\) is therefore \(P_M(\mu)-\mu\). A projection equals its target exactly when the target lies in the set being projected onto. \(\square\)
Worked Example: A Restriction That Excludes the True Mean
Let \(X=I_2\), let the true coefficient vector be \(\beta_0=(1,2)'\), and impose the false restriction \(\beta_1=0\). Suppose \(y=\beta_0+\varepsilon\), with \(E[\varepsilon]=(0,0)'\). The restricted fitted-value set is \(M=\{(0,t)':t\in\mathbb{R}\}\). The true mean \((1,2)'\) is not in \(M\), and its projection onto \(M\) is \((0,2)'\). Thus the expected restricted fit has bias \((0,2)'-(1,2)'=(-1,0)'\).
For observed data \(y=(1.2,1.7)'\), the restriction forces the first fitted coordinate to zero, while the second can equal the observed value. The constrained estimate is \(\hat\beta=(0,1.7)'\), with residual vector \((1.2,0)'\) and residual sum of squares \(1.2^2=1.44\). The unrestricted fit is \(y\) itself and has residual sum of squares zero. The positive constrained residual is not an algebraic error: it is the cost of excluding the true mean direction from the restricted model.
Common Errors and a Practical Diagnostic
- Wrong: “The left-hand side is a linear combination of other rows, so the equation is redundant only if the right-hand sides match.” Correction: First check whether the system after deletion is consistent. If it is, matching right-hand sides is necessary and sufficient; if it is inconsistent, deletion leaves the feasible set empty regardless.
- Wrong: “A coefficient restriction must reduce the available fitted values.” Correction: Compare the sets \(XS\). As the rank-deficient example shows, a restriction can select fewer coefficient vectors while leaving every possible fit unchanged.
- Wrong: “The constrained minimizer is unbiased because it minimizes squared error.” Correction: Minimizing the sample objective does not make a false restriction correct. If the true mean is outside the restricted fitted-value set, its projection generally differs from the true mean.
- Wrong: “A set of equations always defines a usable constrained model.” Correction: Check feasibility. An empty feasible set gives no constrained least-squares solution.
A reliable diagnostic is to check the constraints in this order: establish whether the full system is feasible; when investigating a possibly redundant row, check consistency of the remaining system; identify the feasible coefficient set; map that set through \(X\) to find the actual fitted-value model; and finally ask whether the true mean is in that model. This separates algebraic redundancy, changes in the model space, and misspecification.
Check Your Understanding
Use the feasible-set and fitted-value distinctions to answer each question.
- If a constraint row is a linear combination of the remaining rows and the remaining system is consistent, what condition on the right-hand sides makes the row redundant?
- The system remaining after a constraint is deleted is already inconsistent. What are the feasible sets before and after adding that constraint?
- Why can two different feasible coefficient sets define the same restricted model of fitted values?
- In the example with \(X=I_2\) and restriction \(\beta_1=0\), what is the expected restricted fitted vector, and how does it differ from the true mean?
- What should be checked before interpreting a constrained least-squares estimate when a proposed restriction may be false?