Constructing the General Linear Hypothesis F Statistic

From Estimated Restrictions to a Test Statistic

A general linear hypothesis asks whether several linear combinations of regression coefficients take specified values. The estimate of those combinations gives a vector of discrepancies from the proposed values. But the size of a discrepancy is meaningful only relative to its uncertainty: one unit of discrepancy is more substantial when its standard error is small, and correlated discrepancies must be assessed together. The general linear hypothesis F statistic makes this adjustment and then uses the residual mean square to account for the unknown error variance.

The construction below is exact under a fixed-design Gaussian linear model. Its ingredients are the restriction vector, the covariance matrix determined by the design, and the residual mean square. The key is to use the covariance matrix to form a quadratic distance, rather than to add squared discrepancies without regard to their scales or dependence.

Model and conditions: Let $Y=X\beta+\varepsilon$, where $X$ is a fixed $n\times p$ matrix of full column rank, $n>p$, and $\varepsilon\sim N_n(0,\sigma^2 I_n)$ for $\sigma^2>0$. Let $L$ be a $q\times p$ matrix of full row rank, with $1\le q\le p$, and let $r$ be a specified $q$-vector. We test $H_0:L\beta=r$. These conditions make all $q$ restrictions estimable and linearly independent.

The ordinary least-squares estimator is $\hat{\beta}=(X'X)^{-1}X'Y$. Its covariance matrix is $\sigma^2(X'X)^{-1}$. Define the estimated restriction vector $d=L\hat{\beta}-r$, and define the design-based covariance matrix $C=L(X'X)^{-1}L'$. Under the stated conditions, $C$ is positive definite, so it can be inverted.

Definition: The general linear hypothesis F statistic is $$ F=\frac{(L\hat{\beta}-r)'[L(X'X)^{-1}L']^{-1}(L\hat{\beta}-r)/q}{\operatorname{MSE}}, \qquad \operatorname{MSE}=\frac{\operatorname{SSE}}{n-p}, $$ where $\operatorname{SSE}=(Y-X\hat{\beta})'(Y-X\hat{\beta})$. Under $H_0$, this statistic has the exact $F$ distribution with $q$ and $n-p$ degrees of freedom.

The numerator before division by $q$ is a covariance-adjusted squared distance. Since $\operatorname{Cov}(d)=\sigma^2C$, its scale is $\sigma^2$; dividing by the residual mean square, which estimates that same variance, produces a dimensionless statistic. The proof below establishes that the resulting ratio has the stated finite-sample distribution, not merely an approximate one.

Why the Statistic Has an Exact F Distribution

We derive the distribution under the null. Write $P_X=X(X'X)^{-1}X'$ for the projection onto the column space of $X$, and $M_X=I_n-P_X$ for the residual projection. Then $\hat{\beta}-\beta=(X'X)^{-1}X'\varepsilon$, while $\operatorname{SSE}=\varepsilon'M_X\varepsilon$. These formulas make it possible to establish the numerator distribution and its independence from the denominator.

Theorem: Under the model and rank conditions above, if $H_0:L\beta=r$ is true, then $$ F=\frac{d'C^{-1}d/q}{\operatorname{MSE}}\sim F_{q,n-p}, \qquad d=L\hat{\beta}-r,\quad C=L(X'X)^{-1}L'. $$

Proof. Under $H_0$, the restriction discrepancy has mean zero: $$ d=L(\hat{\beta}-\beta)=L(X'X)^{-1}X'\varepsilon. $$ It is multivariate normal, and direct covariance calculation gives $$ \operatorname{Cov}(d)=L(X'X)^{-1}X'\,(\sigma^2 I_n)\,X(X'X)^{-1}L'=\sigma^2 C. $$ Because $C$ is positive definite, let $C^{-1/2}$ denote its symmetric inverse square root. Then $Z=C^{-1/2}d/\sigma$ has a $q$-dimensional normal distribution with mean zero and covariance $I_q$. Its components are independent standard normal variables. By the definition of a chi-square distribution as a sum of squared independent standard normals, $$ \frac{d'C^{-1}d}{\sigma^2}=Z'Z\sim\chi^2_q. $$

For the denominator, $M_X$ is a symmetric idempotent matrix of rank $n-p$, and $\operatorname{SSE}=\varepsilon'M_X\varepsilon$. The Gaussian spherical error assumption implies that an orthogonal change of coordinates can diagonalize this projection into $n-p$ ones and $p$ zeros. In those coordinates, $\operatorname{SSE}/\sigma^2$ is the sum of squares of $n-p$ independent standard normal variables. Thus $$ \frac{\operatorname{SSE}}{\sigma^2}\sim\chi^2_{n-p}. $$

It remains to establish independence. Both $d$ and $M_X\varepsilon$ are linear transformations of the jointly normal vector $\varepsilon$. Their cross-covariance is $$ \operatorname{Cov}(d,M_X\varepsilon) =L(X'X)^{-1}X'(\sigma^2 I_n)M_X =\sigma^2L(X'X)^{-1}X'M_X=0, $$ because $X'M_X=0$. Jointly normal vectors with zero cross-covariance are independent. Therefore $d$, and hence $d'C^{-1}d$, is independent of $\operatorname{SSE}$. Finally, using $\operatorname{MSE}=\operatorname{SSE}/(n-p)$, the statistic is the ratio of two independent chi-square variables divided by their respective degrees of freedom: $$ F=\frac{[d'C^{-1}d/\sigma^2]/q}{[\operatorname{SSE}/\sigma^2]/(n-p)}\sim F_{q,n-p}. $$ This proves the result. $\square$

At significance level $\alpha$, the exact upper-tail test rejects $H_0$ when the observed statistic exceeds the $1-\alpha$ quantile of $F_{q,n-p}$. The numerator degrees of freedom here are $q$ because the conditions specify $q$ independent restrictions; the denominator degrees of freedom are the residual degrees of freedom $n-p$.

Worked Calculations

Worked Example: A Covariance-Adjusted Restriction Vector

Suppose two independent restrictions have estimated discrepancy $d=(4,3)'$, design-based covariance matrix $C$ satisfying $$ C^{-1}=\frac{1}{9}\begin{pmatrix}5&-1\\-1&2\end{pmatrix}, $$ and residual mean square $\operatorname{MSE}=1$. Compute the general linear hypothesis statistic.

There are $q=2$ restrictions. First calculate the quadratic form, keeping both off-diagonal terms: $$ d'C^{-1}d =\frac{1}{9}\left[5(4^2)+2(-1)(4)(3)+2(3^2)\right] =\frac{1}{9}(80-24+18) =\frac{74}{9}. $$

Divide by the number of restrictions and by the residual mean square: $$ F=\frac{(74/9)/2}{1}=\frac{37}{9}\approx4.11. $$

The statistic is compared with an $F$ distribution having 2 numerator degrees of freedom and $n-p$ denominator degrees of freedom, where $n-p$ comes from the fitted model. No significance decision can be made without that residual degrees of freedom (and a chosen significance level).

Worked Example: Constructing the Statistic From a Small Regression

Consider a regression with an intercept and slope, four observations, and design rows $(1,x_i)$ for $x=(-1.5,-0.5,0.5,1.5)$. The observed responses are $Y=(1.5,0.5,1.5,4.5)'$. Test $H_0:\beta=(0,0)'$ against the general alternative.

The design matrix has $X'X=\operatorname{diag}(4,5)$. The response can be written as $Y=X(2,1)'+e$, where $e=(1,-1,-1,1)'$. Indeed, $X'e=0$, since the entries of $e$ sum to zero and $$ (-1.5)(1)+(-0.5)(-1)+(0.5)(-1)+(1.5)(1)=0. $$ So the least-squares estimate is $\hat{\beta}=(2,1)'$, and the residual sum of squares is $e'e=4$. With $n=4$ and $p=2$, $\operatorname{MSE}=4/(4-2)=2$.

Here $L=I_2$, $r=(0,0)'$, and $d=(2,1)'$. The design-based covariance matrix is $$ C=(X'X)^{-1}=\begin{pmatrix}1/4&0\\0&1/5\end{pmatrix}, \qquad C^{-1}=\begin{pmatrix}4&0\\0&5\end{pmatrix}. $$ Thus $d'C^{-1}d=4(2^2)+5(1^2)=21$. Since $q=2$, $$ F=\frac{21/2}{2}=\frac{21}{4}=5.25. $$

Under the null this has an $F_{2,2}$ distribution. The calculation illustrates that the restriction discrepancies are weighted by the inverse design-based covariance before the result is divided by $q\operatorname{MSE}$.

Worked Example: Restrictions With Correlated Estimates

Suppose a fitted model gives $\hat{\beta}=(3,1)'$, and the restrictions are $\beta_1+\beta_2=4$ and $\beta_1-\beta_2=0$. Thus $$ L=\begin{pmatrix}1&1\\1&-1\end{pmatrix}, \qquad r=\begin{pmatrix}4\\0\end{pmatrix}. $$ Assume the design-based covariance matrix of the estimated restrictions is $C=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, and the residual mean square is 3.

The estimated restriction vector and inverse covariance matrix are $$ d=L\hat{\beta}-r =\begin{pmatrix}4\\2\end{pmatrix}-\begin{pmatrix}4\\0\end{pmatrix} =\begin{pmatrix}0\\2\end{pmatrix}, \qquad C^{-1}=\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}. $$ Therefore $$ d'C^{-1}d=\frac{1}{3}\left[2(0^2)-2(0)(2)+2(2^2)\right]=\frac{8}{3}. $$ With $q=2$, the statistic is $$ F=\frac{(8/3)/2}{3}=\frac{4}{9}. $$

The off-diagonal entries of $C$ express dependence between the two estimated restrictions. The inverse covariance matrix accounts for that dependence when measuring their joint discrepancy. Under the null, the statistic has an $F_{2,n-p}$ distribution.

Common Pitfall and Interpretation

A common mistake is to use $d'd$ as the numerator and assume every restriction discrepancy is equally precise and independent. That is generally wrong: the coordinates of $d$ may have different variances, and they may be correlated. The correct quadratic form is $d'C^{-1}d$, where $C=L(X'X)^{-1}L'$. A second error is to divide this quadratic form by $q$ but forget to divide by $\operatorname{MSE}$. Without that studentization, the statistic still has the unknown scale $\sigma^2$ and does not have the stated F distribution.

The exact calibration also depends on the assumptions used in the proof. In particular, the fixed-design Gaussian model with spherical errors gives the chi-square residual sum of squares and its independence from the restriction estimate. The formula can still be computed in other settings, but the exact $F_{q,n-p}$ reference distribution does not follow merely from having a linear model.

Key takeaway: Form the discrepancy $d=L\hat{\beta}-r$, weight it by its design-based covariance through $C^{-1}$, divide by the number of independent restrictions, and studentize with residual mean square. Under the stated Gaussian model and a true null hypothesis, this produces an exact $F_{q,n-p}$ statistic.

Check Your Understanding

Check that you can explain the construction as well as carry out the arithmetic.

Why is the restriction discrepancy weighted by $C^{-1}$, rather than evaluated using $d'd$?
Under the null, what is the distribution of $d'C^{-1}d/\sigma^2$, and why?
What role does the residual mean square play in the statistic?
Which model assumptions ensure that the numerator and residual sum of squares are independent?
If $q=3$, $d'C^{-1}d=18$, and $\operatorname{MSE}=2$, what is the observed F statistic?

Constructing the General Linear Hypothesis F Statistic

What You'll Learn