Choosing Independent Restrictions for an F Test

Why Independent Restrictions Matter

A general linear hypothesis may be written as several equations about the regression coefficients. The number of written equations is not always the number of distinct conditions: one equation may be a repeat or a linear combination of others. An F test must count the distinct restrictions, not merely the rows written down. Otherwise, its numerator degrees of freedom and its statistic can be wrong.

The task is to replace a redundant system with an equivalent system of independent equations. Equivalence requires care: the coefficient equations and their right-hand sides must both agree. A redundant system can even be inconsistent, in which case its null hypothesis has no parameter values satisfying it.

Model and Rank Reduction

Use the full-rank normal linear model $y=X\beta+\varepsilon$, where $y$ is an $n$-vector, $X$ is a known $n\times p$ matrix of rank $p$, and $\beta$ is a $p$-vector. Assume $\varepsilon\sim N_n(0,\sigma^2 I_n)$, with $\sigma^2>0$, and $n>p$. The least-squares estimator is $\hat{\beta}=(X'X)^{-1}X'y$. Consider the restrictions $R\beta=r$, where $R$ is an $m\times p$ matrix and $r$ is an $m$-vector.

Definition: The number of independent restrictions is $q=\operatorname{rank}(R)$. A full-row-rank basis for the restrictions is a $q\times p$ matrix $B$ whose rows are linearly independent and span the row space of $R$. When the right-hand side is compatible with the same row relationships, the system $R\beta=r$ is equivalent to $B\beta=c$ for a corresponding $q$-vector $c$.

One direct way to choose $B$ is to select $q$ independent rows of $R$. Every row of $R$ is then a linear combination of the selected rows. Consequently, there is an $m\times q$ matrix $A$ such that $R=AB$. Because the selected rows occur among the rows of $R$, $A$ has full column rank. The right-hand side must obey the same row relationships: there must be a vector $c$ such that $r=Ac$.

Theorem: Suppose $R=AB$, where $B$ has $q$ independent rows and $A$ has full column rank. The system $R\beta=r$ is equivalent to $B\beta=c$ if and only if $r=Ac$. If no such $c$ exists, $R\beta=r$ is inconsistent.

To verify the claim when $r=Ac$, any $\beta$ satisfying $B\beta=c$ satisfies $R\beta=AB\beta=Ac=r$. Conversely, if $R\beta=r$, then $A(B\beta-c)=0$. Since $A$ has full column rank, its null space contains only the zero vector, so $B\beta=c$. If $r$ is not in the column space of $A$, there is no $c$ with $r=Ac$; moreover, $R\beta=AB\beta$ must always lie in that column space. Thus no $\beta$ can satisfy $R\beta=r$. $\square$

Worked Example: Selecting a Basis and Checking Compatibility

Consider three proposed restrictions on $\beta=(\beta_1,\beta_2,\beta_3)'$: $ \beta_1+\beta_2=3$, $2\beta_1+2\beta_2=6$, and $\beta_1-\beta_3=0$. In matrix form,

R= \begin{pmatrix} 1&1&0\\ 2&2&0\\ 1&0&-1 \end{pmatrix}, \qquad r= \begin{pmatrix}3\\6\\0\end{pmatrix}.

The second row of $R$ is twice the first, and the third row is independent of the first. Thus $q=\operatorname{rank}(R)=2$. Select the first and third rows:

B= \begin{pmatrix}1&1&0\\1&0&-1\end{pmatrix}, \qquad A= \begin{pmatrix}1&0\\2&0\\0&1\end{pmatrix}.

Multiplication gives $AB=R$. Taking $c=(3,0)'$ gives $Ac=(3,6,0)'=r$, so the original system is equivalent to just $B\beta=c$. The second written equation adds no new condition. The F test therefore has numerator degrees of freedom $q=2$, not $m=3$.

The F Statistic from the Independent Basis

For a full-row-rank restriction matrix $B$, define $M=(X'X)^{-1}$ and $Q=BMB'$. Since $M$ is positive definite and $B$ has independent rows, $Q$ is positive definite and invertible. Let $SSE_U$ be the residual sum of squares from the unrestricted least-squares fit, and let $SSE_R$ be the minimum residual sum of squares subject to $B\beta=c$.

Theorem: If $B$ has $q$ independent rows, then $$ SSE_R-SSE_U=(B\hat{\beta}-c)'(BMB')^{-1}(B\hat{\beta}-c). $$ When $s^2=SSE_U/(n-p)$, the general linear hypothesis F statistic is $$ F=\frac{(B\hat{\beta}-c)'(BMB')^{-1}(B\hat{\beta}-c)}{q s^2} =\frac{(SSE_R-SSE_U)/q}{SSE_U/(n-p)}. $$

To derive the result, write the residual sum of squares at any coefficient vector as $SSE_U+(\beta-\hat{\beta})'X'X(\beta-\hat{\beta})$. This identity follows by expanding $\lVert y-X\beta\rVert^2$ around $\hat{\beta}$; the cross term is zero because $X'(y-X\hat{\beta})=0$. Put $d=B\hat{\beta}-c$. The constraint on $h=\beta-\hat{\beta}$ is $Bh=-d$. Minimize $h'X'Xh$ subject to that constraint using a multiplier vector $\lambda$. The stationarity equation is $X'Xh+B'\lambda=0$, so $h=-MB'\lambda$. Applying the constraint gives $-BMB'\lambda=-d$, hence $\lambda=Q^{-1}d$ and $h=-MB'Q^{-1}d$. At this minimizing value,

h'X'Xh =d'Q^{-1}BMB'Q^{-1}d =d'Q^{-1}d.

This is the increase in residual sum of squares under the restriction, proving the stated identity. $\square$

Worked Example: Computing the F Statistic After Reduction

Use the restrictions in the preceding calculation, with $B$ and $c$ as given there. Suppose $n=8$, $p=3$, $X'X=I_3$, $\hat{\beta}=(1,2,0)'$, and $s^2=2$. These summaries are compatible with the model: a design with orthonormal columns can have this fitted coefficient vector and an orthogonal residual vector with squared length $10$, giving $SSE_U=10$ and $s^2=10/(8-3)=2$.

Here $M=I_3$, and the observed restriction discrepancy and its covariance matrix without the factor $\sigma^2$ are

d=B\hat{\beta}-c= \begin{pmatrix}1+2-3\\1-0-0\end{pmatrix} = \begin{pmatrix}0\\1\end{pmatrix}, \qquad Q=BB'= \begin{pmatrix}2&1\\1&2\end{pmatrix}.

The determinant of $Q$ is $4-1=3$, so $Q^{-1}=\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. Consequently,

d'Q^{-1}d=\frac{2}{3}, \qquad F=\frac{2/3}{2(2)}=\frac{1}{6}.

There are $q=2$ independent restrictions and $n-p=5$ residual degrees of freedom. Under the null, the reference distribution is $F_{2,5}$. Using three degrees of freedom just because $R$ has three rows would not reflect the two distinct conditions.

Why the Choice of Basis Does Not Change the Test

There may be many ways to write the same independent restrictions. If $T$ is any invertible $q\times q$ matrix, then $B_*=TB$ and $c_*=Tc$ express the same hypothesis: multiplying $B\beta=c$ by $T$ is reversible. The statistic is also unchanged. Indeed, with $d=B\hat{\beta}-c$ and $Q=BMB'$, the transformed discrepancy is $Td$, and the transformed matrix is $TQT'$. Its inverse is $T'^{-1}Q^{-1}T^{-1}$. Therefore,

(Td)'(TQT')^{-1}(Td) =d'T'T'^{-1}Q^{-1}T^{-1}Td =d'Q^{-1}d.

This invariance is why selecting a convenient independent basis is legitimate; it does not privilege one particular set of independent equations.

Worked Example: Verifying Invariance Under a Change of Basis

Continue with $d=(0,1)'$ and $Q=\begin{pmatrix}2&1\\1&2\end{pmatrix}$ from the calculation above. Choose $T=\begin{pmatrix}1&1\\0&1\end{pmatrix}$, whose determinant is $1$, so it is invertible. The new discrepancy is $Td=(1,1)'$, and direct multiplication gives

TQT'= \begin{pmatrix}6&3\\3&2\end{pmatrix}, \qquad (TQT')^{-1}= \frac{1}{3} \begin{pmatrix}2&-3\\-3&6\end{pmatrix}.

The new quadratic form is $(1,1)\frac{1}{3}\begin{pmatrix}2&-3\\-3&6\end{pmatrix}(1,1)'=2/3$, equal to the original $d'Q^{-1}d=2/3$. The transformed equations have different coefficients, but they give both the same null hypothesis and the same F statistic.

Exact Calibration and Common Pitfalls

Under the null $B\beta=c$, the discrepancy $B\hat{\beta}-c=B(\hat{\beta}-\beta)$ is normal with mean zero and covariance $\sigma^2 Q$. Thus its quadratic form divided by $\sigma^2$ has a chi-squared distribution with $q$ degrees of freedom. The unrestricted residual sum of squares divided by $\sigma^2$ has a chi-squared distribution with $n-p$ degrees of freedom. These quantities are independent: the coefficient estimation error depends on the projection of $\varepsilon$ onto the column space of $X$, while the residual depends on its orthogonal projection. For normal errors, orthogonal projections are independent. Their ratio gives $F\sim F_{q,n-p}$ under the null. This exact calibration requires the stated normal linear model and positive residual degrees of freedom.

Worked Example: Detecting Inconsistent Redundancy

Suppose the proposed restrictions are $\beta_1=0$ and $2\beta_1=1$. The restriction matrix has rank one, and selecting its first row gives $B=(1)$ and $A=(1,2)'$, so $R=AB$. If a compatible right-hand side existed, it would have to satisfy $r=Ac=(c,2c)'$. But the stated right-hand side is $r=(0,1)'$. Its first entry forces $c=0$, while its second forces $2c=1$, which is impossible. There is no $\beta$ satisfying both equations. Keeping only the first equation would change the hypothesis; the stated system does not define a valid null for an F test.

A common mistake is to count the rows of $R$ as the numerator degrees of freedom even when some rows are dependent. That uses $m$ where the test requires $q=\operatorname{rank}(R)$. A second mistake is to discard a redundant-looking equation without checking its right-hand side. A row can be redundant in $R$ while its corresponding value in $r$ makes the complete system inconsistent. First verify compatibility, then reduce the restrictions and use the reduced rank in the F statistic.

Key takeaway: Reduce a consistent restriction system to $B\beta=c$ with $B$ full row rank. The number of independent restrictions $q$, not the number of written equations, determines the numerator degrees of freedom. Equivalent full-rank bases preserve the hypothesis and the F statistic.

Check Your Understanding

Use the rank, compatibility, and statistic arguments above to answer these questions.

If a restriction matrix has five rows but rank three, how many numerator degrees of freedom does its consistent exact F test use?
For $R=AB$ with $A$ full column rank, why does $R\beta=r$ imply $B\beta=c$ when $r=Ac$?
What must be checked about $r$ before dropping dependent rows of $R$?
Why is $BMB'$ invertible when $B$ has full row rank and $M=(X'X)^{-1}$?
How does an invertible change of basis $B_*=TB$, $c_*=Tc$ affect the restriction hypothesis and its quadratic form?