Testing Linear Restrictions After Reparameterization

Why a Change of Coordinates Should Not Change a Test

A regression model can be written using different coefficient coordinates without changing the set of possible fitted values. For instance, the original coefficients might be replaced by sums and differences, or by coefficients with a more direct scientific interpretation. The numerical coordinates change, but a test about the same underlying restriction should not. The key is to transform the hypothesis along with the coefficients.

This tutorial considers the full-rank normal linear model and exact t and F tests for linear restrictions. It shows how to translate a hypothesis through an invertible coefficient transformation, then verifies algebraically that the test statistic is unchanged. Throughout, the model has \(n\) observations and \(p\) coefficients:

y=X\beta+\varepsilon,\qquad \varepsilon\sim N(0,\sigma^2 I_n),

where \(X\) is a fixed \(n\times p\) matrix of rank \(p\), \(\beta\in\mathbb{R}^p\), and \(n>p\). The error variance \(\sigma^2\) is unknown. Write \(\hat{\beta}=(X'X)^{-1}X'y\), \(s^2=\mathrm{RSS}/(n-p)\), and \(V=(X'X)^{-1}\).

Transforming Coefficients and Restrictions

Definition: An invertible reparameterization of the coefficient vector is a change of coordinates \(\gamma=T\beta\), where \(T\) is a known nonsingular \(p\times p\) matrix. Thus \(\beta=T^{-1}\gamma\), and the same mean vector can be written \(X\beta=XT^{-1}\gamma\). A linear hypothesis \(H_0:R\beta=r\) becomes \(H_0:RT^{-1}\gamma=r\).

The transformed design matrix is \(Z=XT^{-1}\), which has rank \(p\). Least squares in the transformed model gives \(\hat{\gamma}=T\hat{\beta}\): minimizing \(\|y-Z\gamma\|^2\) is the same optimization as minimizing \(\|y-X\beta\|^2\) after substituting \(\beta=T^{-1}\gamma\). The residuals, fitted values, residual sum of squares, and residual degrees of freedom are therefore unchanged.

A restriction is transformed by substituting \(\beta=T^{-1}\gamma\) into its original equation. It is not generally correct to use \(RT\). This distinction matters because the coefficient transformation \(\gamma=T\beta\) and the restriction transformation act in opposite directions.

For an exact joint test, suppose \(R\) is a \(q\times p\) matrix of rank \(q\), and \(r\in\mathbb{R}^q\). The exact F statistic is

F_R= \frac{(R\hat{\beta}-r)' \{R(X'X)^{-1}R'\}^{-1} (R\hat{\beta}-r)/q}{s^2}.

The rank condition ensures that the covariance matrix inside the inverse is nonsingular. For a single restriction, \(q=1\), the corresponding signed t statistic is

t_R= \frac{R\hat{\beta}-r} {s\sqrt{R(X'X)^{-1}R'}}, \qquad F_R=t_R^2.

Worked Examples

Worked Example: Testing Equality of Two Slopes in Sum-and-Difference Coordinates

Suppose \(\beta=(\beta_0,\beta_1,\beta_2)'\), and the hypothesis is \(H_0:\beta_1=\beta_2\). Use new coordinates \(\gamma=(\beta_0,\gamma_1,\gamma_2)'\), where \(\gamma_1=\beta_1+\beta_2\) and \(\gamma_2=\beta_1-\beta_2\). The transformation is invertible because \(\beta_1=(\gamma_1+\gamma_2)/2\) and \(\beta_2=(\gamma_1-\gamma_2)/2\). The equality restriction is exactly \(H_0:\gamma_2=0\).

Suppose the observed summaries are \(\hat{\beta}=(2,1.5,0.5)'\), \(V=\mathrm{diag}(1/12,1/4,1/4)\), \(s^2=4\), and \(n-p=9\). In the original coordinates, the contrast row is \(R=(0,1,-1)\). Its estimate is \(R\hat{\beta}=1.5-0.5=1\), and its variance factor is

RVR'=\frac14+\frac14=\frac12.

Thus \(t_R=1/(2\sqrt{1/2})=1/\sqrt{2}\), so \(F_R=t_R^2=1/2\), with 1 and 9 degrees of freedom. In the new coordinates, \(\hat{\gamma}_2=\hat{\beta}_1-\hat{\beta}_2=1\), and its variance factor is again \(1/2\). The transformed test therefore has the same \(t=1/\sqrt{2}\) and \(F=1/2\). The sign of a one-sided t statistic would depend on how the direction of the alternative is expressed, but the corresponding hypothesis must be transformed as well.

Worked Example: A Nonzero Restriction After Mixing Coordinates

Let \(\beta=(\beta_1,\beta_2)'\) and test \(H_0:\beta_1+2\beta_2=3\). Set \(\gamma_1=\beta_1+\beta_2\) and \(\gamma_2=\beta_1-\beta_2\), so \(\beta_1=(\gamma_1+\gamma_2)/2\) and \(\beta_2=(\gamma_1-\gamma_2)/2\). Substitution gives the transformed restriction \(H_0:(3/2)\gamma_1-(1/2)\gamma_2=3\).

Take \(\hat{\beta}=(1,2)'\), \(V=\begin{pmatrix}1&0.25\\0.25&2\end{pmatrix}\), and \(s^2=1.5\). In the original coordinates, \(R=(1,2)\), so \(R\hat{\beta}=5\) and

$$ RVR'=1+0.25(2)+0.25(2)+2(2^2)=10. $$

The signed t statistic is \((5-3)/\sqrt{1.5(10)}=2/\sqrt{15}\), and the one-restriction F statistic is \(4/15\).

In transformed coordinates, \(\hat{\gamma}=(3,-1)'\) and \(V_\gamma=TVT'=\begin{pmatrix}3.5&-1\\-1&2.5\end{pmatrix}\), where \(T=\begin{pmatrix}1&1\\1&-1\end{pmatrix}\). The transformed restriction row is \(C=(1.5,-0.5)\). Its estimate is \(C\hat{\gamma}=1.5(3)-0.5(-1)=5\), and its variance factor is

CV_\gamma C' =1.5^2(3.5)+2(1.5)(-0.5)(-1)+(-0.5)^2(2.5) =7.875+1.5+0.625=10.

Both the numerator contrast and its variance factor agree with the original calculation. The transformed test therefore gives the same \(t=2/\sqrt{15}\) and \(F=4/15\).

Worked Example: A Joint Test in Coordinates That Include the Restrictions

Let \(\beta=(\beta_1,\beta_2,\beta_3)'\), with the joint null hypothesis \(\beta_1+\beta_2=0\) and \(\beta_2+\beta_3=0\). Define \(\gamma_1=\beta_1+\beta_2\), \(\gamma_2=\beta_2+\beta_3\), and \(\gamma_3=\beta_3\). This is invertible: \(\beta_3=\gamma_3\), \(\beta_2=\gamma_2-\gamma_3\), and \(\beta_1=\gamma_1-\gamma_2+\gamma_3\). The null is now \(\gamma_1=\gamma_2=0\).

Suppose \(\hat{\beta}=(1,0,2)'\), \(V=I_3\), \(s^2=2\), and \(n-p=10\). The restriction matrix is \(R=\begin{pmatrix}1&1&0\\0&1&1\end{pmatrix}\), giving \(R\hat{\beta}=(1,2)'\) and \(RVR'=\begin{pmatrix}2&1\\1&2\end{pmatrix}\). Its inverse is \(\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}\), so the quadratic form in the numerator is

(1,2)\frac13 \begin{pmatrix}2&-1\\-1&2\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} =(1,2)\begin{pmatrix}0\\1\end{pmatrix}=2.

The F statistic is \((2/2)/2=0.5\), with 2 and 10 degrees of freedom. In the new coordinates, \((\hat{\gamma}_1,\hat{\gamma}_2)'=(1,2)'\), and the covariance matrix for those two coordinates is the same \(2\times2\) matrix \(RVR'\) calculated above. Testing the first two transformed coordinates against zero gives the same quadratic form and the same \(F=0.5\).

Why the Exact F Calibration Is Preserved

Theorem: Under the full-rank normal linear model, if \(R\) has rank \(q\), the statistic \(F_R\) has an exact \(F_{q,n-p}\) distribution when \(H_0:R\beta=r\) is true. Under any invertible reparameterization \(\gamma=T\beta\), the correctly transformed restriction \(RT^{-1}\gamma=r\) gives exactly the same observed statistic and the same null distribution.

To establish the exact calibration, let \(P_X=X(X'X)^{-1}X'\) and \(M_X=I_n-P_X\). The fitted-value estimation error is \(X(\hat{\beta}-\beta)=P_X\varepsilon\), the projection of the normal error onto the column space of \(X\); the residual is \(M_X\varepsilon\), its projection onto the orthogonal complement. These are projections in the \(n\)-dimensional observation space. Since \(P_XM_X=0\), the two projected errors have zero covariance. They are jointly normal, so they are independent.

The contrast \(R\hat{\beta}-r\) is normal with mean \(R\beta-r\) and covariance \(\sigma^2 RVR'\). Under the null, its standardized quadratic form

\frac{(R\hat{\beta}-r)'(RVR')^{-1}(R\hat{\beta}-r)}{\sigma^2}

has a chi-square distribution with \(q\) degrees of freedom. Also, \(\mathrm{RSS}=\varepsilon'M_X\varepsilon\), and \(M_X\) is an orthogonal projection of rank \(n-p\). Normality therefore gives \(\mathrm{RSS}/\sigma^2\sim\chi^2_{n-p}\). The independence of the two projected errors implies independence of these two chi-square quantities. Dividing each by its degrees of freedom gives the stated \(F_{q,n-p}\) distribution.

It remains to verify the coordinate invariance algebraically. The transformed estimator is \(\hat{\gamma}=T\hat{\beta}\), with covariance estimate factor \(V_\gamma=TVT'\). Put \(C=RT^{-1}\), the transformed restriction matrix. Then

C\hat{\gamma}-r=RT^{-1}T\hat{\beta}-r=R\hat{\beta}-r, \qquad CV_\gamma C'=RT^{-1}TVT'(T^{-1})'R'=RVR'.

Consequently the contrast vector, its covariance factor, the residual mean square, and the numerator degrees of freedom are identical in the two coordinate systems. Substitution into the F formula proves equality of the statistics, not merely equality of their distributions. For a single restriction, the signed t statistic is likewise identical when the same direction of the alternative is used. \(\square\)

Common Pitfall and Practical Significance

A common error is to transform the coefficients but leave the restriction row unchanged, or to multiply the row by \(T\) instead of \(T^{-1}\). For example, if \(\gamma=T\beta\), then the original equation \(R\beta=r\) becomes \(RT^{-1}\gamma=r\), because the substitution is \(\beta=T^{-1}\gamma\). A test built from an incorrectly transformed row may be a valid test of a different hypothesis, but it does not answer the original question.

The invariance is useful when a new parameterization makes the null hypothesis easier to state or interpret. A model may use sums and differences, centered variables, or scientifically meaningful combinations of coefficients. Provided the transformation is invertible and the restriction is translated correctly, the inferential conclusion does not depend on which coordinates are used. The exact result here relies on the stated normal linear model, full column rank, and a restriction matrix of rank \(q\); it is not a claim that arbitrary changes to the model or to the hypothesis preserve a test.

Key takeaway: If \(\gamma=T\beta\), then transform \(R\beta=r\) into \(RT^{-1}\gamma=r\). The contrast estimate and covariance transform together, leaving the exact t or F statistic unchanged.

Check Your Understanding

For each question, identify how the restriction changes and what must be compared to verify invariance.

If \(\gamma=T\beta\), what equation in \(\gamma\) represents the original hypothesis \(R\beta=r\)?
Why does an invertible reparameterization leave the residual sum of squares unchanged?
For a single restriction, which two quantities in the t statistic must be checked after transformation?
In the joint example, why is the covariance factor for the first two transformed coefficients not the \(2\times2\) identity matrix?
Which model assumptions give the exact \(F_{q,n-p}\) calibration, rather than just equality of the two calculated statistics?