Partial F Tests as Quadratic-Form Tests

Why the Two Forms of the Test Agree

A partial F test compares two nested regression models: an unrestricted model and a smaller model obtained by imposing linear restrictions. It can be computed from the models’ residual sums of squares, or from the unrestricted coefficient estimates and the restrictions they are being tested against. These are not merely two related procedures. Under the conditions below, their numerators are algebraically identical.

The coefficient-based expression is a quadratic form because restrictions can have different scales and the coefficient estimates can be correlated. Its weighting matrix accounts for both. The result is that the partial F statistic is precisely the general linear hypothesis statistic for the restrictions defining the smaller model.

Models, Restrictions, and the Partial F Statistic

Consider the full-rank linear model $y=X\beta+\varepsilon$, where $y$ is an $n$-vector, $X$ is a known $n\times p$ matrix of rank $p$, and $\beta$ is a $p$-vector. Write $\hat{\beta}=(X'X)^{-1}X'y$ for the unrestricted least-squares estimator, and define the unrestricted residual sum of squares as $\mathrm{RSS}_U=(y-X\hat{\beta})'(y-X\hat{\beta})$.

Definition: Let $C$ be a $q\times p$ matrix of rank $q$, and let $d$ be a specified $q$-vector. The restricted model is the set of mean vectors $X\beta$ for which $C\beta=d$. Its residual sum of squares is $\mathrm{RSS}_R=\min_{C\beta=d}(y-X\beta)'(y-X\beta)$. The unrestricted and restricted models are nested, and their difference in parameter dimension is $q$.

Because $X$ has full column rank and $C$ has full row rank, $A=C(X'X)^{-1}C'$ is positive definite and therefore invertible. The general linear hypothesis quadratic form is

Q=(C\hat{\beta}-d)'[C(X'X)^{-1}C']^{-1}(C\hat{\beta}-d).

In the usual partial F test, the residual mean square from the unrestricted model is $\mathrm{RSS}_U/(n-p)$. Thus the two statistics to be compared are

F_{\mathrm{partial}}=\frac{(\mathrm{RSS}_R-\mathrm{RSS}_U)/q}{\mathrm{RSS}_U/(n-p)} \qquad\text{and}\qquad F_{\mathrm{GLH}}=\frac{Q/q}{\mathrm{RSS}_U/(n-p)}.

The denominator is the same in both expressions. The key algebraic fact is that their numerators before division by $q$ are equal.

Deriving the Quadratic-Form Identity

Theorem: In the full-rank model above, if $C$ has rank $q$, then $$ \mathrm{RSS}_R-\mathrm{RSS}_U =(C\hat{\beta}-d)'[C(X'X)^{-1}C']^{-1}(C\hat{\beta}-d). $$ Consequently, $F_{\mathrm{partial}}=F_{\mathrm{GLH}}$ whenever $n>p$ and $\mathrm{RSS}_U>0$.

To prove the identity, first note that the unrestricted least-squares normal equations give $X'(y-X\hat{\beta})=0$. For any other coefficient vector $b$, expand its residual sum of squares around $\hat{\beta}$. The cross term vanishes by the normal equations, leaving

(y-Xb)'(y-Xb) =\mathrm{RSS}_U+(b-\hat{\beta})'X'X(b-\hat{\beta}).

Therefore, finding the restricted fit amounts to minimizing $(b-\hat{\beta})'X'X(b-\hat{\beta})$ subject to $Cb=d$. Set $g=C\hat{\beta}-d$ and $\delta=b-\hat{\beta}$. The constraint becomes $C\delta=-g$. Let $G=X'X$, which is positive definite. The Lagrangian for this constrained minimization is $\delta'G\delta+2\lambda'(C\delta+g)$, with multiplier vector $\lambda$. Differentiating with respect to $\delta$ gives $G\delta+C'\lambda=0$, so $\delta=-G^{-1}C'\lambda$. Applying the constraint yields

-CG^{-1}C'\lambda=-g, \qquad \lambda=(CG^{-1}C')^{-1}g.

Thus the minimizing displacement is $\delta=-G^{-1}C'(CG^{-1}C')^{-1}g$. Substituting it into the minimized quadratic distance gives

\begin{aligned} \delta'G\delta &=g'(CG^{-1}C')^{-1}CG^{-1}GG^{-1}C'(CG^{-1}C')^{-1}g\\ &=g'(CG^{-1}C')^{-1}g. \end{aligned}

By the expansion of the residual sum of squares, this minimized distance is $\mathrm{RSS}_R-\mathrm{RSS}_U$. Replacing $G$ by $X'X$ and $g$ by $C\hat{\beta}-d$ proves the identity. Dividing both expressions by $q$ and by the common residual mean square proves equality of the two F statistics. $\square$

Worked Examples

Worked Example: Testing Two Coefficients in an Orthogonal Design

Let $n=5$, $p=3$, and take the columns of $X$ to be $x_1=(1,1,1,1,1)'$, $x_2=(1,-1,0,0,0)'$, and $x_3=(0,0,1,-1,0)'$. Let $y=2x_1+3x_2-x_3+z$, where $z=(1,1,-1,-1,0)'$. These columns are mutually orthogonal, and $z$ is orthogonal to each of them.

Since $x_1'x_1=5$ and $x_2'x_2=x_3'x_3=2$, the unrestricted coefficient estimates are $\hat{\beta}=(2,3,-1)'$. The unrestricted residual is $z$, so $\mathrm{RSS}_U=z'z=4$, with $n-p=2$ residual degrees of freedom. Test $H_0:\beta_2=\beta_3=0$, using $C=\begin{pmatrix}0&1&0\\0&0&1\end{pmatrix}$, $d=(0,0)'$, and $q=2$.

For the quadratic-form calculation, $C\hat{\beta}-d=(3,-1)'$, while $C(X'X)^{-1}C'=\operatorname{diag}(1/2,1/2)$. Its inverse is $\operatorname{diag}(2,2)$, giving $Q=2(3^2)+2((-1)^2)=20$. Hence the general linear hypothesis statistic is $(20/2)/(4/2)=5$.

For the partial F calculation, the restricted fit retains only $x_1$. Orthogonality means its residual contains $3x_2-x_3+z$, so $\mathrm{RSS}_R=9(2)+1(2)+4=24$. Therefore $\mathrm{RSS}_R-\mathrm{RSS}_U=24-4=20=Q$, and $F_{\mathrm{partial}}=(20/2)/(4/2)=5$, as required.

Worked Example: Nonzero Restrictions and Correlated Restriction Estimates

Let $X$ be the first three coordinate vectors in $\mathbb{R}^5$, and let $y=(1,2,-1,\sqrt{3},0)'$. Then $X'X=I_3$, $\hat{\beta}=(1,2,-1)'$, and $\mathrm{RSS}_U=3$. Consider the two restrictions $\beta_1+\beta_2=1$ and $\beta_2+\beta_3=0$. Here $C=\begin{pmatrix}1&1&0\\0&1&1\end{pmatrix}$, $d=(1,0)'$, and $q=2$.

The restriction departure is $g=C\hat{\beta}-d=(2,1)'$. The restrictions are not orthogonal in coefficient space: $CC'=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, whose inverse is $\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. Multiplication gives $(CC')^{-1}g=(1,0)'$, and therefore $Q=g'(CC')^{-1}g=2$.

For a direct check of the restricted fit, the displacement formula gives $\delta=-C'(CC')^{-1}g=(-1,-1,0)'$. The restricted estimate is consequently $\hat{\beta}+\delta=(0,1,-1)'$, which satisfies both restrictions. Its squared displacement from the unrestricted fit is $\delta'\delta=2$, so $\mathrm{RSS}_R=\mathrm{RSS}_U+2=5$. The partial F statistic is $((5-3)/2)/(3/2)=2/3$, matching $(Q/2)/(3/2)=2/3$.

Worked Example: Why Correlated Predictors Change the Weighting

Let $n=5$ and take the three design columns to be $x_1=e_1$, $x_2=e_2$, and $x_3=e_2+e_3$, where $e_i$ is the $i$th coordinate vector. Set $y=x_2+2x_3+\sqrt{3}e_4=(0,3,2,\sqrt{3},0)'$. The unrestricted estimate is $\hat{\beta}=(0,1,2)'$, the unrestricted residual sum of squares is $3$, and the residual degrees of freedom are $5-3=2$.

Test the two restrictions $\beta_2=\beta_3=0$. The columns $x_2$ and $x_3$ are not orthogonal: their Gram matrix is $\begin{pmatrix}1&1\\1&2\end{pmatrix}$, with inverse $\begin{pmatrix}2&-1\\-1&1\end{pmatrix}$. For $C=\begin{pmatrix}0&1&0\\0&0&1\end{pmatrix}$, $d=(0,0)'$, the departure is $g=(1,2)'$, and the covariance-weighting matrix in the quadratic form is the inverse of that Gram matrix, namely $\begin{pmatrix}1&1\\1&2\end{pmatrix}$. Thus $Q=(1,2)\begin{pmatrix}1&1\\1&2\end{pmatrix}(1,2)'=13$.

Under the restrictions, only $x_1$ remains. Since $x_1'y=0$, its fitted coefficient is zero and $\mathrm{RSS}_R=y'y=9+4+3=16$. Hence $\mathrm{RSS}_R-\mathrm{RSS}_U=16-3=13=Q$. Both calculations give $F=(13/2)/(3/2)=13/3$. The cross-product between predictors is why simply adding the squared coefficient estimates, $1^2+2^2=5$, would give the wrong numerator.

Exact F Calibration Under the Null

The algebraic equality does not itself require a probability model. To use the statistic as an exact F test, suppose additionally that $\varepsilon\sim N(0,\sigma^2 I_n)$, that $\sigma^2>0$, and that the null hypothesis $C\beta=d$ holds. With $n>p$, the exact null distribution is $F_{q,n-p}$.

Theorem: Under the Gaussian spherical-error assumptions and $C\beta=d$, the partial F statistic has an $F_{q,n-p}$ distribution.

For a proof, let $S=\{Xb:Cb=0\}$, a linear subspace of $\operatorname{col}(X)$ with dimension $p-q$. The restricted mean vectors form an affine subspace $\mathcal{M}_R=\{Xb:Cb=d\}$ whose directions are $S$. There is a unique component $w_0$ in $W=\operatorname{col}(X)\cap S^\perp$ shared by all vectors in $\mathcal{M}_R$: each restricted mean can be written as an element of $S$ plus $w_0$. The space $W$ has dimension $q$.

Under the null, the true mean $\mu=X\beta$ belongs to $\mathcal{M}_R$. Orthogonal projection onto the unrestricted space gives $\mathrm{RSS}_U=\|P_{\operatorname{col}(X)^\perp}\varepsilon\|^2$. The extra residual sum of squares imposed by the restrictions is the squared distance from the fitted unrestricted point to $\mathcal{M}_R$, namely $\mathrm{RSS}_R-\mathrm{RSS}_U=\|P_W\varepsilon\|^2$. Because the errors are Gaussian with covariance $\sigma^2 I_n$, orthogonal coordinates of $\varepsilon/\sigma$ are independent standard normal variables. Thus $(\mathrm{RSS}_R-\mathrm{RSS}_U)/\sigma^2\sim\chi^2_q$ and $\mathrm{RSS}_U/\sigma^2\sim\chi^2_{n-p}$. The two projections lie in orthogonal subspaces, so these variables are independent. Their ratio after division by their respective degrees of freedom has the $F_{q,n-p}$ distribution. $\square$

Common Pitfall and Practical Meaning

A common mistake is to replace the quadratic form by an unweighted sum of squared departures, such as $(C\hat{\beta}-d)'(C\hat{\beta}-d)$. That expression ignores the scale and dependence of the estimated restrictions. The correct weighting is $[C(X'X)^{-1}C']^{-1}$, which adjusts for their covariance. As the third example shows, ignoring predictor correlation can change the numerator substantially.

The identity is useful because it lets the same test be understood in two ways. The residual-sum-of-squares view asks how much fit is lost when the restrictions are imposed. The quadratic-form view asks how far the unrestricted estimates are from those restrictions, measured in the uncertainty scale of the estimated restrictions. They give the same answer, provided the restricted model is exactly the set $C\beta=d$ and $C$ has rank $q$. For a rank-deficient design, the coefficient representation must first be handled through estimable restrictions or an identifiable parameterization; the displayed inverse formula should not be used without its stated rank conditions.

Check Your Understanding

Use the definitions and derivation above to answer these questions.

What condition on $C$ makes $C(X'X)^{-1}C'$ invertible in the full-rank model?
Why does minimizing the restricted residual sum of squares reduce to minimizing a quadratic distance from $\hat{\beta}$?
For $C\hat{\beta}-d=g$, explain why the weighting matrix in the quadratic form cannot generally be replaced by the identity matrix.
Under the exact Gaussian null, what are the numerator and denominator degrees of freedom of the partial F distribution?
If the restricted and unrestricted residual sums of squares are 16 and 3, respectively, and $q=2$, what is the quadratic-form value $Q$?