Invariance Under Reparameterization

Why Invariance Matters

A multivariate linear model can be written in different coordinates without changing its underlying fitted mean model. Predictors may be rescaled or replaced by another basis, and responses may be recorded in different units or mixed through a nonsingular linear transformation. A sound test of the same scientific claim should not depend on which such coordinates are used.

The qualification is essential: changing coordinates and changing the hypothesis are not the same operation. If the response columns are mixed but the original coordinate contrast is used unchanged, the resulting test may concern a different outcome. This tutorial makes the distinction precise for general linear restrictions and the associated hypothesis and error matrices.

Model and Hypothesis Coordinates

Let \(Y\) be an \(n\times p\) response matrix, \(X\) an \(n\times q\) design matrix of full column rank, and \(B\) the \(q\times p\) coefficient matrix. The model for the mean is \(E[Y]=XB\). Write \(\hat B=(X'X)^{-1}X'Y\), \(P_X=X(X'X)^{-1}X'\), and \(E=Y'(I-P_X)Y\). Assume the response error covariance is positive definite; for the generalized-root discussion below, also assume \(E\) is positive definite.

A general hypothesis has the form \(CBM=\Theta\), where \(C\) is \(r\times q\) with full row rank, \(M\) is \(p\times s\) with full column rank, and \(\Theta\) is a specified \(r\times s\) matrix. The hypothesis sum-of-squares-and-cross-products matrix and the error matrix for these response contrasts are

H=(C\hat B M-\Theta)' \{C(X'X)^{-1}C'\}^{-1} (C\hat B M-\Theta), \qquad E_M=M'EM.

These matrices summarize the fitted departure from the restriction and the residual variation in the selected response contrasts. Classical multivariate criteria are built from the generalized eigenvalues of \((H,E_M)\), or equivalently the roots of \(\det(H-\lambda E_M)=0\). The transformations below preserve these roots when they preserve the hypothesis and change both matrices coherently.

Definition: A reparameterization is invariant for a test when it changes the coordinates used to describe the model or restrictions but leaves the tested set of mean structures unchanged. A test is not invariant to a transformation that changes that set, even if the transformed model itself describes the same data equally well.

Changing Predictor Coordinates

Let \(A\) be a nonsingular \(q\times q\) matrix and set \(X_*=XA\). Then the same mean matrix has coefficient matrix \(B_*=A^{-1}B\), since \(X_*B_*=XAA^{-1}B=XB\). Thus the re-expressed restriction is \(C A B_* M=\Theta\), not generally \(C B_* M=\Theta\).

Theorem: Under the predictor reparameterization \(X_*=XA\), \(B_*=A^{-1}B\), the restriction \(CBM=\Theta\) is exactly equivalent to \((CA)B_*M=\Theta\). The corresponding \(H\), \(E_M\), and generalized roots are unchanged.

Proof. The fitted values are unchanged because \(P_{X_*}=P_X\): substituting \(X_*=XA\) into the projection formula gives \(P_{X_*}=XA\{A'X'XA\}^{-1}A'X'=X(X'X)^{-1}X'\). Also, \(C A B_*M=C A A^{-1}BM=CBM\), so the restriction is identical.

For the hypothesis matrix, \(\hat B_*=A^{-1}\hat B\) and \[ (CA)(X_*'X_*)^{-1}(CA)'=C(X'X)^{-1}C', \] because \((A'X'XA)^{-1}=A^{-1}(X'X)^{-1}A^{-T}\). The fitted restriction difference is also identical: \((CA)\hat B_*M-\Theta=C\hat BM-\Theta\). Hence \(H_*=H\). Since the fitted projection is unchanged, the residual matrix \(E\) is unchanged and so is \(E_M\). The generalized roots therefore agree. \(\square\)

Worked Example: Replacing Predictors by a New Basis

Suppose \(q=p=2\), and the coefficient matrix in the original predictor coordinates is \(B=\begin{pmatrix}3&4\\1&2\end{pmatrix}\). The hypothesis is that the coefficient of predictor 1 for response 1 is zero: take \(C=(1,0)\), \(M=(1,0)'\), and \(\Theta=0\). The tested quantity is \(CBM=3\).

Use \(A=\begin{pmatrix}1&2\\0&1\end{pmatrix}\), so \(X_*=XA\). Its inverse is \(A^{-1}=\begin{pmatrix}1&-2\\0&1\end{pmatrix}\), and therefore \(B_*=A^{-1}B=\begin{pmatrix}1&0\\1&2\end{pmatrix}\).

The transformed restriction uses \(CA=(1,2)\). Direct calculation gives \((CA)B_*M=(1,2)\begin{pmatrix}1\\1\end{pmatrix}=3\), the same quantity as before. By contrast, testing \(CB_*M=0\) would test the first coefficient in the new basis, which is \(1\), not the original coefficient \(3\). The model coordinates changed; that unadjusted restriction did not preserve the claim.

Changing Response Coordinates

Let \(D\) be a nonsingular \(p\times p\) matrix and define \(Y_*=YD\). The coefficient matrix becomes \(B_*=BD\). To preserve the original response contrast \(BM\), use \(M_*=D^{-1}M\); then \(B_*M_*=BD D^{-1}M=BM\). This also keeps the target \(\Theta\) unchanged.

Theorem: Under \(Y_*=YD\), the hypothesis \(CBM=\Theta\) is represented by \(CB_*M_*=\Theta\), with \(M_*=D^{-1}M\). The hypothesis matrix and error matrix for this contrast are exactly the original \(H\) and \(E_M\).

Proof. The transformed design is still \(X\), so its coefficient estimate is \(\hat B_*=\hat BD\), and the transformed residual cross-product is \(E_*=D'ED\). The restriction difference is \(C\hat B_*M_*-\Theta=C\hat BDD^{-1}M-\Theta=C\hat BM-\Theta\). The predictor-side covariance factor \(C(X'X)^{-1}C'\) is unchanged, giving \(H_*=H\). For the error matrix, \(M_*'E_*M_*=M'D^{-T}D'ED D^{-1}M=M'EM=E_M\). Thus both matrices, and consequently their generalized roots, are unchanged. \(\square\)

Worked Example: Preserving a Response Contrast After Mixing Outcomes

Let \(B=\begin{pmatrix}2&5\\3&7\end{pmatrix}\). Take \(C=(0,1)\) to select predictor 2 and \(M=(0,1)'\) to select response 2. The tested coefficient is \(CBM=(0,1)\begin{pmatrix}5\\7\end{pmatrix}=7\).

Now use \(D=\begin{pmatrix}1&1\\0&1\end{pmatrix}\), whose inverse is \(D^{-1}=\begin{pmatrix}1&-1\\0&1\end{pmatrix}\). The transformed coefficients are \(B_*=BD=\begin{pmatrix}2&7\\3&10\end{pmatrix}\), and the contrast corresponding to the original response 2 is \(M_*=D^{-1}M=(-1,1)'\).

The transformed calculation is \(CB_*M_*=(0,1)\begin{pmatrix}2&7\\3&10\end{pmatrix}\begin{pmatrix}-1\\1\end{pmatrix}=-3+10=7\). It preserves the original claim. If one instead used the unadjusted contrast \(M=(0,1)'\), the result would be \(10\): the second transformed response is the sum of the two original responses, so that calculation tests a different coefficient.

Changing Restriction Coordinates

Restrictions themselves can be expressed in different coordinates. For nonsingular \(L\) of size \(r\times r\) and \(K\) of size \(s\times s\), the equation \(CBM=\Theta\) is equivalent to \((LC)B(MK)=L\Theta K\). The left transformation recombines the restriction equations; the right transformation changes the basis for the selected response contrasts.

Under this change, the restriction difference becomes \(L(C\hat BM-\Theta)K\), and the error matrix becomes \(K'E_MK\). Substitution into the definition of \(H\) gives \(H_*=K'HK\): the factors \(L\) cancel against the inverse of \(L\{C(X'X)^{-1}C'\}L'\). Thus \(H\) and \(E_M\) need not be entrywise equal, but they undergo the same congruence transformation on the response-contrast coordinates.

Result: If \(H_*=K'HK\) and \(E_*=K'E_MK\) for nonsingular \(K\), then the generalized roots are unchanged. Indeed, \(\det(H_*-\lambda E_*)=\det(K')\det(H-\lambda E_M)\det(K)=\det(K)^2\det(H-\lambda E_M)\). The roots of the determinant equation are therefore the same.

Worked Example: A Change of Basis for Two Restrictions

Suppose the hypothesis and error matrices are \(H=\begin{pmatrix}4&0\\0&1\end{pmatrix}\) and \(E_M=\begin{pmatrix}2&0\\0&1\end{pmatrix}\). Their generalized roots solve \((4-2\lambda)(1-\lambda)=0\), so they are \(2\) and \(1\).

Use the nonsingular restriction-coordinate change \(K=\begin{pmatrix}1&1\\0&1\end{pmatrix}\). Since \(K'=\begin{pmatrix}1&0\\1&1\end{pmatrix}\), direct multiplication gives \(H_*=K'HK=\begin{pmatrix}4&4\\4&5\end{pmatrix}\) and \(E_*=K'E_MK=\begin{pmatrix}2&2\\2&3\end{pmatrix}\).

Here \(\det(K)=1\), so \(\det(H_*-\lambda E_*)=\det(H-\lambda E_M)=(4-2\lambda)(1-\lambda)\). The roots remain \(2\) and \(1\), even though neither transformed matrix equals its original. This verifies invariance of root-based criteria under a change of restriction basis.

What Is Preserved—and What Is Not

These calculations separate coordinate changes from scientific changes. A nonsingular predictor transformation preserves the model space, provided the coefficient restrictions are transformed with it. A nonsingular response transformation preserves a selected original contrast when the contrast coordinates are adjusted by \(D^{-1}\). A nonsingular change of restriction coordinates preserves the null equation when its target is transformed along with it.

The same reasoning applies to simple rescaling. Changing units for an outcome is a nonsingular diagonal response transformation. A contrast stated in the old units must be translated into the new coordinates; otherwise its numerical coefficients describe a different contrast. Similarly, recoding predictors by a nonsingular linear transformation does not make an unchanged coefficient restriction refer to the same effect automatically.

Common pitfall: “The fitted model is unchanged, so every test is unchanged” is incorrect. The fitted mean space can be the same while the restriction being tested changes. After a transformation, write the transformed coefficient matrix and substitute it into the restriction. If the resulting equation is not algebraically equivalent to the original one, the test concerns a different claim.

In practice, check invariance in three steps: identify the coordinate transformation, express the original null hypothesis in the new coordinates, and compare the resulting \(H\) and error matrices or their generalized roots. This avoids mistaking a convenient recoding for a change in the underlying scientific question.

Check Your Understanding

For each question, distinguish a change of coordinates from a change in the tested restriction.

If \(X_*=XA\) for nonsingular \(A\), what coefficient matrix represents the same mean model, and how must \(C\) change in \(CBM=\Theta\)?
Under \(Y_*=YD\), what contrast matrix \(M_*\) preserves the original quantity \(BM\)?
Why can \(H\) and \(E_M\) change entrywise under a restriction-coordinate transformation while the generalized roots remain fixed?
For the response transformation in the worked example, what original response combination does the unadjusted second transformed response represent?
State one algebraic check that can reveal whether a proposed recoding has changed the null hypothesis.