A Multi-Restriction Least-Squares Calculation

Why Work Through a Multi-Restriction Calculation?

For a single linear restriction, the constrained estimate can be understood as a correction to the unrestricted least-squares estimate. With several restrictions, the same idea applies, but the corrections must account for how the restrictions interact. A numerical calculation makes each part visible: the restriction violations, the joint correction, the resulting fitted values and residuals, and the multipliers in the constrained normal equations.

Throughout, consider a design matrix $X$ with full column rank and a matrix $C$ whose rows encode independent linear restrictions. We minimize the residual sum of squares subject to $C\beta=d$. The coefficient vector is $\beta$, the unrestricted estimate is $\hat{\beta}$, and the restricted estimate is $\hat{\beta}_R$.

Definition: Given $y\in\mathbb{R}^n$, a full-column-rank matrix $X\in\mathbb{R}^{n\times p}$, and $C\in\mathbb{R}^{q\times p}$ with full row rank, the restricted least-squares estimate minimizes $(y-X\beta)'(y-X\beta)$ over all $\beta\in\mathbb{R}^p$ satisfying $C\beta=d$. The $q$ restrictions are independent because $C$ has row rank $q$.

The Joint Correction Formula

Let $A=X'X$, $b=X'y$, and $\hat{\beta}=A^{-1}b$, the unrestricted least-squares estimate. Because $X$ has full column rank, $A$ is positive definite and invertible. Define the restriction discrepancy $g=C\hat{\beta}-d$, and the $q\times q$ matrix $M=CA^{-1}C'$. The restricted estimate and its multipliers can be obtained by solving one system of $q$ equations.

Theorem: If $X$ has full column rank, $C$ has full row rank, and the restriction $C\beta=d$ is feasible, then $M= C(X'X)^{-1}C'$ is invertible. With the Lagrangian defined as $\frac12(y-X\beta)'(y-X\beta)+\lambda'(C\beta-d)$, the restricted estimate and multiplier vector are $$ \hat{\beta}_R=\hat{\beta}-A^{-1}C'M^{-1}(C\hat{\beta}-d), \qquad \hat{\lambda}=M^{-1}(C\hat{\beta}-d). $$

The formula is a joint adjustment, not a separate correction for each row of $C$. The matrix $M$ accounts for the relationships among the restrictions in the geometry defined by $A=X'X$.

Worked Calculation

Worked Example: Two Independent Restrictions in a Full-Rank Design

Take the four-observation, three-parameter design and response $$ X= \begin{pmatrix} 1&0&0\\ 0&1&0\\ 0&0&1\\ 1&1&1 \end{pmatrix}, \qquad y= \begin{pmatrix} 2\\1\\3\\4 \end{pmatrix}. $$ Impose the two restrictions $\beta_1+\beta_2=1$ and $\beta_2+\beta_3=2$. The first three rows of $X$ are the three coordinate vectors, so its columns are linearly independent. The restriction matrix also has independent rows: $$ C=\begin{pmatrix}1&1&0\\0&1&1\end{pmatrix}, \qquad d=\begin{pmatrix}1\\2\end{pmatrix}. $$

1. Calculate the unrestricted estimate. Direct multiplication gives $$ A=X'X= \begin{pmatrix}2&1&1\\1&2&1\\1&1&2\end{pmatrix} =I+\mathbf{1}\mathbf{1}', \qquad b=X'y= \begin{pmatrix}6\\5\\7\end{pmatrix}. $$ Here $A^{-1}=I-\frac14\mathbf{1}\mathbf{1}'$. Since the entries of $b$ sum to $18$, $$ \hat{\beta}=A^{-1}b =b-\frac{18}{4}\begin{pmatrix}1\\1\\1\end{pmatrix} =\begin{pmatrix}1.5\\0.5\\2.5\end{pmatrix}. $$

2. Find the restriction discrepancies and joint correction. The unrestricted estimate gives $C\hat{\beta}=(2,3)'$, so $g=C\hat{\beta}-d=(1,1)'$. Also, $$ A^{-1}C'= \begin{pmatrix} 0.5&-0.5\\ 0.5&0.5\\ -0.5&0.5 \end{pmatrix}, \qquad M=CA^{-1}C'= \begin{pmatrix}1&0\\0&1\end{pmatrix}. $$ Thus $\hat{\lambda}=M^{-1}g=(1,1)'$, and the coefficient adjustment is $$ A^{-1}C'\hat{\lambda} = \begin{pmatrix}0\\1\\0\end{pmatrix}. $$ Therefore $$ \hat{\beta}_R= \begin{pmatrix}1.5\\0.5\\2.5\end{pmatrix} - \begin{pmatrix}0\\1\\0\end{pmatrix} = \begin{pmatrix}1.5\\-0.5\\2.5\end{pmatrix}. $$ Check both restrictions: $1.5+(-0.5)=1$, and $(-0.5)+2.5=2$.

3. Calculate fitted values and residuals. Multiplying by $X$ gives $$ \hat{y}_R=X\hat{\beta}_R= \begin{pmatrix}1.5\\-0.5\\2.5\\3.5\end{pmatrix}, \qquad e_R=y-\hat{y}_R= \begin{pmatrix}0.5\\1.5\\0.5\\0.5\end{pmatrix}. $$ The residual sum of squares is $0.5^2+1.5^2+0.5^2+0.5^2=3$. The multiplier equation can also be checked directly: $A\hat{\beta}_R=(5,3,6)'$, so $b-A\hat{\beta}_R=(1,2,1)'=C'\hat{\lambda}$.

Why the Formula Works

The formula follows from the constrained normal equations. It is useful to establish both that the matrix being inverted is nonsingular and that the resulting estimate is the unique minimizer.

Proof: First, for any nonzero $z\in\mathbb{R}^q$, full row rank of $C$ implies $C'z\ne0$. Since $A^{-1}$ is positive definite, $$ z'Mz=(C'z)'A^{-1}(C'z)>0. $$ Hence $M$ is positive definite and invertible. Now define $\hat{\lambda}=M^{-1}(C\hat{\beta}-d)$ and $\hat{\beta}_R=\hat{\beta}-A^{-1}C'\hat{\lambda}$. Then $$ C\hat{\beta}_R=C\hat{\beta}-CA^{-1}C'\hat{\lambda} =C\hat{\beta}-M\hat{\lambda}=d, $$ so the estimate satisfies the restrictions. As $A\hat{\beta}=b$, it also satisfies $A\hat{\beta}_R-b+C'\hat{\lambda}=0$, the stationarity equation for the stated Lagrangian.

To verify optimality, take any other feasible $\beta=\hat{\beta}_R+h$. Feasibility implies $Ch=0$. For $Q(\beta)=\frac12(y-X\beta)'(y-X\beta)$, expansion around $\hat{\beta}_R$ gives $$ Q(\hat{\beta}_R+h)-Q(\hat{\beta}_R) =h'(A\hat{\beta}_R-b)+\frac12h'Ah =-(Ch)'\hat{\lambda}+\frac12h'Ah =\frac12h'Ah. $$ Because $A$ is positive definite, this difference is positive whenever $h\ne0$. Thus $\hat{\beta}_R$ is the unique constrained minimizer. $\square$

Checking the Cost of the Restrictions

The same calculation gives a useful check on the residual sum of squares. The restriction discrepancy $g$ determines how much the residual sum of squares increases relative to the unrestricted fit. This follows from the joint correction, and provides an independent numerical check on a calculation.

Result: Under the theorem's assumptions, the increase in residual sum of squares is $$ (y-X\hat{\beta}_R)'(y-X\hat{\beta}_R) -(y-X\hat{\beta})'(y-X\hat{\beta}) =g'M^{-1}g, \qquad g=C\hat{\beta}-d. $$ Derivation: Let $\delta=\hat{\beta}-\hat{\beta}_R=A^{-1}C'M^{-1}g$. The unrestricted residual $y-X\hat{\beta}$ is orthogonal to every column of $X$, so its cross-product with $X\delta$ is zero. The increase is therefore $$ \delta'A\delta =g'M^{-1}CA^{-1}AA^{-1}C'M^{-1}g =g'M^{-1}g. $$

Worked Example: Verify the Residual-Sum-of-Squares Increase

For the design and restrictions just considered, the unrestricted fitted vector is $X\hat{\beta}=(1.5,0.5,2.5,4.5)'$. Its residual vector is $(0.5,0.5,0.5,-0.5)'$, whose squared entries sum to $1$. The restricted residual sum of squares was $3$, so the increase is $3-1=2$.

The formula gives the same answer. Here $g=(1,1)'$ and $M=I$, hence $g'M^{-1}g=(1,1)(1,1)'=2$. Agreement verifies both the restricted fit and the restriction correction.

A Second Restriction Geometry

The matrix $M$ need not be diagonal. When it is not, the restrictions interact through the design, and using separate one-at-a-time corrections can give the wrong answer. The next calculation illustrates how the joint matrix handles this interaction.

Worked Example: Requiring All Three Coefficients to Be Equal

Keep $X$ and $y$ from the first example, but now impose $\beta_1-\beta_2=0$ and $\beta_2-\beta_3=0$: $$ C=\begin{pmatrix}1&-1&0\\0&1&-1\end{pmatrix}, \qquad d=\begin{pmatrix}0\\0\end{pmatrix}. $$ The unrestricted estimate remains $(1.5,0.5,2.5)'$. The discrepancy is $g=(1,-2)'$. Since both rows of $C$ sum to zero, multiplication by $A^{-1}=I-\frac14\mathbf{1}\mathbf{1}'$ leaves these rows unchanged. Consequently, $$ M=CA^{-1}C'= \begin{pmatrix}2&-1\\-1&2\end{pmatrix}, \qquad M^{-1}=\frac13\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$ The multipliers are $$ \hat{\lambda}=M^{-1}g =\frac13\begin{pmatrix}2&1\\1&2\end{pmatrix} \begin{pmatrix}1\\-2\end{pmatrix} =\begin{pmatrix}0\\-1\end{pmatrix}. $$ Here $A^{-1}C'\hat{\lambda}=(0,-1,1)'$, so $$ \hat{\beta}_R=(1.5,0.5,2.5)'-(0,-1,1)'=(1.5,1.5,1.5)'. $$ Both restrictions hold. The fitted values are $(1.5,1.5,1.5,4.5)'$, and the residuals are $(0.5,-0.5,1.5,-0.5)'$. Their squared entries sum to $3$. As a further check, $b-A\hat{\beta}_R=(0,-1,1)'=C'\hat{\lambda}$, as required by the multiplier equation.

Interpreting and Using the Multipliers

The multiplier vector records the correction needed to reconcile the unrestricted normal equations with the restrictions. Under the Lagrangian convention used here, the equations are $$ A\hat{\beta}_R-b+C'\hat{\lambda}=0, \qquad C\hat{\beta}_R=d. $$ Thus $C'\hat{\lambda}=b-A\hat{\beta}_R$. The multipliers are not generally equal to the coefficient adjustments: the coefficient adjustment is $A^{-1}C'\hat{\lambda}$. Their numerical values also depend on how the restrictions are written. For example, multiplying a restriction equation by a nonzero constant changes its multiplier, even though it does not change the set of feasible coefficients or the fitted values.

A practical calculation should therefore verify several distinct things: the restrictions hold, the multiplier equations hold, and the fitted values and residuals agree with multiplication by $X$ and subtraction from $y$. In a full-column-rank design, the restricted coefficient vector is unique. Its residuals need not be orthogonal to every column of $X$, as in unrestricted least squares; instead, the stationarity equations imply orthogonality to feasible coefficient directions. If $Ch=0$, then $h'X'e_R=0$.

Common Pitfall

Correcting each restriction independently. Separate single-restriction updates generally ignore dependence induced by the design. Use the full matrix $M=CA^{-1}C'$, then solve the joint system.
Using the wrong multiplier scale. With the Lagrangian $\frac12\mathrm{RSS}+\lambda'(C\beta-d)$, the stationarity equation contains $C'\lambda$. If the Lagrangian uses the full residual sum of squares instead, its multiplier is twice as large.
Checking coefficients but not the fit. A vector can satisfy the restrictions and still be incorrectly calculated. Verify the multiplier equations and compute $X\hat{\beta}_R$ and $y-X\hat{\beta}_R$ explicitly.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

Why is $CA^{-1}C'$ positive definite when $A$ is positive definite and $C$ has full row rank?
For the first numerical example, what is the restriction discrepancy $C\hat{\beta}-d$, and why is it not itself the coefficient correction?
In the second restriction example, verify the residual sum of squares by adding the four squared residuals.
Why can the restricted residual vector fail to be orthogonal to every column of $X$, while remaining orthogonal to feasible coefficient directions?
How would the numerical multipliers change if one restriction equation were multiplied by $3$, even though the restricted coefficients and fitted values did not change?