Parameterizing Linear Constraints on Regression Coefficients

Why Reparameterize a Constrained Model?

A linear restriction on regression coefficients describes which coefficient matrices are admissible. For example, a scientific model may require two predictors to have equal slopes, or require a specified sum of slopes for each response. One approach is to optimize over all coefficient matrices while imposing equations at every step. A more direct approach is to choose coordinates that describe only admissible matrices. This tutorial develops that approach and shows how it turns constrained least squares into an ordinary regression problem.

Let $Y$ be an $n\times q$ response matrix, $X$ an $n\times p$ design matrix, and $B$ a $p\times q$ coefficient matrix. The mean model is $E[Y]=XB$, when the expectation exists. The restriction will be $LB=C$, where $L$ is a known $r\times p$ matrix and $C$ is a known $r\times q$ matrix. The algebraic parameterization below does not require a probability model or a full-rank design matrix.

Definition: A constraint $LB=C$ is consistent if at least one $p\times q$ matrix $B$ satisfies it. A particular solution is any such matrix $B_0$. The null space of $L$ is $\mathcal N(L)=\{v\in\mathbb R^p:Lv=0\}$. A matrix $T$ whose columns form a basis of $\mathcal N(L)$ is a null-space basis matrix.

The Null-Space Parameterization

Write $s=\operatorname{rank}(L)$ and $d=p-s$. The null space of $L$ has dimension $d$, so a basis matrix $T$ has size $p\times d$ and full column rank. If $d=0$, the null space contains only the zero vector and there are no free coefficient coordinates.

Theorem: The constraint $LB=C$ is consistent exactly when every column of $C$ belongs to the column space of $L$. If it is consistent, choose any $B_0$ satisfying $LB_0=C$, and let the columns of $T$ be a basis of $\mathcal N(L)$. Then the complete set of solutions is $$ B=B_0+TA,\qquad A\in\mathbb R^{d\times q}. $$

Proof. The equation $LB=C$ separates by columns: for each column $c_j$ of $C$, it requires a vector $b_j$ such that $Lb_j=c_j$. Such a vector exists exactly when $c_j$ is in the column space of $L$, proving the consistency condition. Now suppose a particular solution $B_0$ exists. For any other solution $B$, subtract the equations $LB=C$ and $LB_0=C$. This gives $L(B-B_0)=0$. Thus each column of $B-B_0$ belongs to $\mathcal N(L)$, and because the columns of $T$ form a basis, there is a matrix $A$ such that $B-B_0=TA$. Conversely, for every $A$, $$ L(B_0+TA)=LB_0+(LT)A=C+0=C. $$ The parameterization therefore contains every feasible matrix and only feasible matrices. $\square$

If $L$ has redundant rows, its rank $s$ is smaller than $r$; the dimension of the free coefficient space is still $p-s$, not $p-r$. Redundancy does not change the null space. When the constraint is consistent, one convenient particular solution is $B_0=L^{+}C$, where $L^{+}$ is the Moore–Penrose pseudoinverse. Indeed, consistency means the columns of $C$ lie in the column space of $L$, so $LL^{+}C=C$. A different particular solution is equally valid; it changes the coordinates $A$, but not the set of feasible $B$ matrices.

Worked Example: Equal Coefficient Rows Across Two Predictors

Suppose $p=3$, $q=2$, and the two response outcomes must have equal coefficient rows for predictors 2 and 3. The restriction is $LB=0$, with $L=(0,1,-1)$. Write a general coefficient matrix as $B=(b_{ij})$. The equation says $b_{2j}-b_{3j}=0$ for each response $j$, so row 2 must equal row 3.

A particular solution is $B_0=0$. The null space consists of vectors $(v_1,v_2,v_3)'$ with $v_2=v_3$. Two independent vectors spanning it are $(1,0,0)'$ and $(0,1,1)'$, so take $$ T=\begin{pmatrix}1&0\\0&1\\0&1\end{pmatrix}. $$ For an arbitrary $2\times2$ matrix $A=(a_{ij})$, the parameterization gives $$ B=TA=\begin{pmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\\a_{21}&a_{22}\end{pmatrix}. $$ The second and third rows agree, as required. Conversely, every $B$ with those two rows equal has this form: take its first row as row 1 of $A$, and its common second/third row as row 2 of $A$.

Worked Example: A Nonzero Constraint for Each Outcome

Let $p=2$, $q=2$, $L=(1,1)$, and $C=(2,-1)$. The constraint requires the two entries in each column of $B$ to sum to the corresponding entry of $C$. A particular solution and a null-space basis are $$ B_0=\begin{pmatrix}2&-1\\0&0\end{pmatrix}, \qquad T=\begin{pmatrix}1\\-1\end{pmatrix}, $$ since $LB_0=C$ and $LT=0$. Therefore every constrained matrix is $$ B=B_0+TA =\begin{pmatrix}2+a_1&-1+a_2\\-a_1&-a_2\end{pmatrix}, \qquad A=\begin{pmatrix}a_1&a_2\end{pmatrix}. $$ The first column sums to $(2+a_1)+(-a_1)=2$; the second sums to $(-1+a_2)+(-a_2)=-1$. The two coordinates in $A$ are the remaining free choices, one for each outcome.

The same $T$ is used for both outcome columns because the restriction uses the same $L$. The particular solution supplies the required column sums, while $TA$ changes the coefficient matrix only in directions that leave those sums unchanged.

Turning the Constraint into a Regression

Under the matrix-valued linear model $Y=XB+U$, substitute $B=B_0+TA$. The result is $$ Y=XB_0+(XT)A+U. $$ The fixed term $XB_0$ is an offset: subtract it from the response, and the remaining model is a regression with response $Y-XB_0$, design matrix $XT$, and coefficient matrix $A$: $$ Y-XB_0=(XT)A+U. $$ Thus the restriction changes the regression coordinates and design. It does not require a separate constrained optimization once the parameterization has been constructed.

Result: Minimizing the residual sum of squares over all $B$ satisfying $LB=C$ is equivalent to minimizing $$ \lVert Y-XB_0-(XT)A\rVert_F^2 $$ over $A\in\mathbb R^{d\times q}$, and then setting $\hat B=B_0+T\hat A$. Here $\lVert M\rVert_F^2=\operatorname{tr}(M'M)$ is the sum of squared entries of $M$.

Derivation. The feasible matrices are exactly $B_0+TA$, by the theorem. For each such matrix, the residual is $$ Y-XB=Y-X(B_0+TA)=Y-XB_0-(XT)A. $$ Taking the sum of squared entries gives exactly the displayed objective. Since the parameterization neither excludes a feasible matrix nor includes an infeasible one, minimizing over $A$ gives the same minimum residual sum of squares as minimizing over constrained $B$. This is an equality of optimization problems; it does not assume that either minimizer is unique.

When $XT$ has full column rank, the ordinary least-squares coefficient matrix in the transformed regression is $$ \hat A=((XT)'(XT))^{-1}(XT)'(Y-XB_0), \qquad \hat B=B_0+T\hat A. $$ The inverse exists precisely when the columns of $XT$ are linearly independent. If $XT$ is rank deficient, least-squares fitted values still exist, but $\hat A$ need not be unique. Different minimizing values of $A$ can represent the same fitted value $XT A$; the resulting $B$ matrices are feasible, though they need not be identical. This distinction is important when the original design is rank deficient or when the restrictions leave directions that the design cannot distinguish.

Worked Example: Fitting a Constrained Regression by Hand

Take one response, three observations, and $$ X=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}, \qquad Y=\begin{pmatrix}2\\0\\3\end{pmatrix}, \qquad b_1+b_2=1. $$ Here $L=(1,1)$, $C=1$, $B_0=(1,0)'$, and $T=(1,-1)'$. Write $B=B_0+Ta=(1+a,-a)'$. The transformed response and design are $$ Y-XB_0=\begin{pmatrix}1\\0\\2\end{pmatrix}, \qquad XT=\begin{pmatrix}1\\-1\\0\end{pmatrix}. $$ The least-squares estimate of the single free coordinate is $$ \hat a=\frac{(XT)'(Y-XB_0)}{(XT)'(XT)} =\frac{1(1)+(-1)(0)+0(2)}{1^2+(-1)^2+0^2} =\frac12. $$ Thus $\hat B=(3/2,-1/2)'$, which satisfies $3/2-1/2=1$. The fitted values and residual sum of squares are $$ X\hat B=\begin{pmatrix}3/2\\-1/2\\1\end{pmatrix}, \qquad \lVert Y-X\hat B\rVert_F^2 =\left(\frac12\right)^2+\left(\frac12\right)^2+2^2 =\frac92. $$ As a direct check, imposing $b_2=1-b_1$ gives fitted values $(b_1,1-b_1,1)'$. The residual sum of squares is $(2-b_1)^2+(b_1-1)^2+4$. Its derivative is $-2(2-b_1)+2(b_1-1)=4b_1-6$, which vanishes at $b_1=3/2$, giving the same constrained fit.

Choosing and Checking a Parameterization

In practical work, a null-space basis can be found by solving $Lv=0$, by row reduction, or by a numerical null-space routine. The basis need not be orthonormal; it only needs to span the null space and have linearly independent columns. An orthonormal basis can improve numerical conditioning, but it does not change the set of feasible coefficient matrices. Before fitting, check the dimensions: $B_0$ is $p\times q$, $T$ is $p\times d$, and $A$ is $d\times q$, so $TA$ and $B_0$ are both $p\times q$. Also verify $LB_0=C$ and $LT=0$.

If a constraint is inconsistent, no reparameterization can make it feasible. For instance, with $L=(1,1)$, the two rows of a proposed $C$ must each be attainable as sums of the corresponding entries in a column of $B$. For this $L$, every scalar is attainable, so any $C$ with the correct dimensions is consistent. More generally, the column-space condition in the theorem is essential; it should be checked rather than assumed.

Common Pitfall and Why the Parameterization Matters

A common error is to use an arbitrary solution $B_0$ and then add unrestricted coefficient changes $D$, writing $B=B_0+D$. That generally violates the constraint, because $L(B_0+D)=C+LD$, which equals $C$ only when $LD=0$. The correction is to restrict every change to the null space: write $D=TA$, so $LD=LTA=0$. The opposite error is to choose too few null-space directions. Then some feasible matrices are missed, and the resulting fit is more constrained than the stated model.

This reparameterization is useful whenever a model is specified by exact linear coefficient relationships. It converts those relationships into a lower-dimensional set of free coordinates while preserving the model’s feasible fitted values. It also makes the source of non-uniqueness transparent: the reduced regression has a unique coefficient estimate when $XT$ has full column rank, but the constraints themselves do not guarantee that rank condition. The topic here is the algebraic construction and least-squares equivalence; inference about restrictions requires additional results.

Check Your Understanding

Use the dimensions and null-space argument to reason through each question.

For $L$ of size $r\times p$ and rank $s$, how many columns does a null-space basis matrix $T$ have?
State the condition on $C$ that makes $LB=C$ consistent.
If $LB_0=C$ and $LT=0$, explain why every matrix $B_0+TA$ satisfies the constraint.
In the transformed regression, identify the response matrix, design matrix, and coefficient matrix.
What can fail when $XT$ does not have full column rank, and what remains true about least-squares fitted values?