Testing Predictor-Space Effects Across Responses

From Predictor Contrasts to a Joint Hypothesis

In a multivariate linear model, each predictor has a coefficient for each response. A scientific question about one predictor contrast therefore concerns a vector of coefficients, not just one number. For example, asking whether a treatment contrast matters across several outcomes means asking whether that contrast has zero coefficient for every outcome. The key step is to express the selected predictor contrasts as a subspace and translate that subspace into a matrix restriction.

Let $Y$ be an $n\times p$ response matrix and $X$ an $n\times q$ design matrix. The coefficient matrix $B$ is $q\times p$, so the multivariate linear model is $Y=XB+U$. Assume for now that $X$ has full column rank. A row vector $c^T$ selects a linear combination of predictor coefficients; the resulting coefficients across responses are the row vector $c^T B$.

Definition: Let $S$ be an $r$-dimensional subspace of predictor-coefficient contrast vectors in $\mathbb{R}^q$. If the columns of a full-column-rank matrix $A\in\mathbb{R}^{q\times r}$ form a basis for $S$, the hypothesis that all selected contrasts have zero coefficients across all $p$ responses is $$ H_0:A^T B=0_{r\times p}. $$ Equivalently, in the general linear hypothesis notation $C B M=\Theta$, take $C=A^T$, $M=I_p$, and $\Theta=0$.

The choice of basis does not define a different scientific question. Any other basis for the same subspace has the form $A^*=AR$, where $R$ is an invertible $r\times r$ matrix. Then $A^{*T}B=R^T A^T B$, which is zero exactly when $A^TB$ is zero. The row space of $C=A^T$, rather than the particular entries used to represent it, determines the restriction.

Constructing the Contrast Matrix

To build $C$, first write down what the predictor coefficients mean under the chosen coding of $X$. Then write each selected contrast as a vector in that coefficient coordinate system. Put those vectors in the rows of $C$. This avoids a common ambiguity: a contrast of group means and a contrast of treatment-coded coefficients need not have the same coordinates.

If the scientific question concerns a whole subspace, a basis is sufficient; listing every possible linear combination in that subspace would only add redundant restrictions. For a full-row-rank $C$, the number of independent restrictions is $r$. If a proposed matrix has dependent rows, remove redundancy or use its rank when describing the hypothesis.

Worked Example: Testing Two Polynomial Predictor Effects

Suppose the rows of $B$ correspond, in order, to an intercept, a linear dose term, and a quadratic dose term, and there are two responses. The question is whether dose has any linear or quadratic association with either response.

The selected predictor-contrast subspace is spanned by $a_1=(0,1,0)^T$ and $a_2=(0,0,1)^T$. Taking these as the columns of $A$ gives $$ A=\begin{pmatrix}0&0\\1&0\\0&1\end{pmatrix}, \qquad C=A^T=\begin{pmatrix}0&1&0\\0&0&1\end{pmatrix}. $$

Thus the null hypothesis is $$ CB= \begin{pmatrix} b_{\mathrm{dose},1}&b_{\mathrm{dose},2}\\ b_{\mathrm{dose}^2,1}&b_{\mathrm{dose}^2,2} \end{pmatrix} =0. $$ It comprises four scalar equalities: both dose coefficients and both quadratic dose coefficients are zero. The intercept coefficients are unrestricted.

Worked Example: Contrasts Among Three Group Means

Suppose the three rows of $B$ are the response-mean vectors for groups 1, 2, and 3 under a cell-means model. To test whether all three groups have the same mean vector across $p$ responses, choose the two adjacent differences $$ C=\begin{pmatrix}1&-1&0\\0&1&-1\end{pmatrix}. $$

Here $CB=0$ says that the group 1 mean vector equals the group 2 mean vector and that the group 2 mean vector equals the group 3 mean vector. By transitivity, all three mean vectors are equal. The two rows are linearly independent, so this tests a two-dimensional predictor-contrast subspace. For any response, there are two independent restrictions; across all responses, the matrix equality imposes $2p$ scalar restrictions.

The same question could be represented with the contrasts group 1 minus group 3 and group 2 minus group 3. Those rows give another basis for the same subspace, and therefore the same null hypothesis.

The Hypothesis Sum-of-Squares Matrix

Write $\widehat B=(X^TX)^{-1}X^TY$ for the unrestricted least-squares estimate, and set $V=(X^TX)^{-1}$. For a full-row-rank $C\in\mathbb{R}^{r\times q}$, the estimated selected contrasts are $D=C\widehat B$. Their covariance factor in predictor-coefficient space is $CVC^T$, an invertible $r\times r$ matrix. The hypothesis sum-of-squares matrix is $$ H=(C\widehat B)^T\bigl[C(X^TX)^{-1}C^T\bigr]^{-1}(C\widehat B). $$ It is a $p\times p$ matrix: its diagonal entries summarize fitted variation associated with the tested contrasts for individual responses, while its off-diagonal entries retain cross-response information.

Theorem: In the full-rank model, the difference between the unrestricted fitted sum-of-squares matrix and the fitted sum-of-squares matrix under $CB=0$ is $H$ as displayed above. In particular, $H$ is positive semidefinite and has rank at most $\min(r,p)$.

To prove the result, minimize the residual sum of squares subject to $CB=0$. Because $\widehat B$ is the unrestricted least-squares estimate, the residual criterion, up to a constant independent of $B$, is $$ \operatorname{tr}\{(B-\widehat B)^TX^TX(B-\widehat B)\}. $$ Let $S_C=CVC^T$. The constrained minimizer is $$ \widehat B_0=\widehat B-VC^TS_C^{-1}C\widehat B. $$ It satisfies the restriction since $C\widehat B_0=C\widehat B-S_CS_C^{-1}C\widehat B=0$. To verify it minimizes the criterion, consider any other feasible $B=\widehat B_0+K$, where $CK=0$. The cross term in the criterion expansion vanishes: $$ \operatorname{tr}\{K^TX^TX(\widehat B_0-\widehat B)\} =-\operatorname{tr}\{K^TC^TS_C^{-1}C\widehat B\}=0, $$ because $CK=0$. The remaining added term is $\operatorname{tr}(K^TX^TXK)\geq 0$. Thus $\widehat B_0$ is the constrained minimizer.

The fitted-value difference is $X(\widehat B-\widehat B_0)=XVC^TS_C^{-1}C\widehat B$. Since $X^TX=V^{-1}$, its cross-product is $$ \begin{aligned} \{X(\widehat B-\widehat B_0)\}^T\{X(\widehat B-\widehat B_0)\} &=(C\widehat B)^TS_C^{-1}CV C^TS_C^{-1}(C\widehat B)\\ &=(C\widehat B)^TS_C^{-1}(C\widehat B)=H. \end{aligned} $$ This proves the difference formula. For any vector $z\in\mathbb{R}^p$, $z^THz=(C\widehat Bz)^TS_C^{-1}(C\widehat Bz)\geq0$, so $H$ is positive semidefinite. Its rank cannot exceed the number of rows of $C$, $r$, or the number of responses, $p$. $\square$

Worked Example: Computing H for Two Selected Coefficients

Suppose there are three predictor coefficients and two responses. Test the second and third coefficients jointly, so $$ C=\begin{pmatrix}0&1&0\\0&0&1\end{pmatrix}. $$ Suppose $V=(X^TX)^{-1}=\operatorname{diag}(1/10,1/4,1/5)$, and the selected estimated coefficients are $$ D=C\widehat B=\begin{pmatrix}2&1\\-1&3\end{pmatrix}. $$

The contrast covariance factor and its inverse are $$ CVC^T=\begin{pmatrix}1/4&0\\0&1/5\end{pmatrix}, \qquad (CVC^T)^{-1}=\begin{pmatrix}4&0\\0&5\end{pmatrix}. $$ Therefore $$ H=D^T(CVC^T)^{-1}D =\begin{pmatrix}2&-1\\1&3\end{pmatrix} \begin{pmatrix}4&0\\0&5\end{pmatrix} \begin{pmatrix}2&1\\-1&3\end{pmatrix} =\begin{pmatrix}21&-7\\-7&49\end{pmatrix}. $$ For a check, its diagonal entries are $4(2^2)+5((-1)^2)=21$ and $4(1^2)+5(3^2)=49$; its off-diagonal entry is $4(2)(1)+5(-1)(3)=-7$. The determinant is $21(49)-(-7)^2=980>0$, consistent with positive definiteness in this example.

Sampling Interpretation and Testing

For a distributional interpretation, assume the rows of $U$ are independent $N_p(0,\Sigma)$, with $\Sigma$ positive definite. Define $P_X=X(X^TX)^{-1}X^T$, the projection onto the columns of $X$. The reduced model $CB=0$ has fitted values in a subspace of the unrestricted fitted-value space. If $N$ is a basis matrix for the null space of $C$, that reduced fitted-value space is the column space of $XN$. Let $P_0$ be its orthogonal projection. Then $$ H=Y^T(P_X-P_0)Y, \qquad E=Y^T(I-P_X)Y. $$ The first identity follows from the fitted-value difference just derived; the second is the unrestricted residual sum-of-squares matrix.

Because $X$ has rank $q$ and $C$ has rank $r$, $XN$ has rank $q-r$. Hence $P_X-P_0$ is a symmetric idempotent matrix of rank $r$. Also, $(P_X-P_0)(I-P_X)=0$, so the two projected error matrices are independent under the Gaussian model. Under $H_0$, their distributions are $$ H\sim W_p(\Sigma,r), \qquad E\sim W_p(\Sigma,n-q), $$ and they are independent. Here $W_p(\Sigma,d)$ denotes the $p$-dimensional Wishart distribution with scale matrix $\Sigma$ and $d$ degrees of freedom. This follows by choosing orthonormal bases for the two orthogonal projection spaces: the projected rows are independent $N_p(0,\Sigma)$ vectors, and their cross-products have the stated Wishart distributions.

The observed $H$ will generally not be zero even when the null hypothesis is true; sampling noise produces nonzero estimated contrasts. The null distribution describes how large this matrix can be by chance, relative to the residual matrix $E$. Classical multivariate procedures compare the hypothesis and error matrices through functions of their roots or determinants. The key construction here is the correct $H$: it reflects exactly the selected predictor subspace and incorporates the design through $C(X^TX)^{-1}C^T$.

Common Pitfalls

Testing only one response at a time. Separate tests do not state the joint null that every selected contrast is zero across all responses. The joint restriction is the matrix equation $CB=0$.
Confusing a coefficient coordinate with a scientific contrast. A coefficient row represents a particular effect only under a specified predictor coding. Translate the scientific contrast into that coding before forming $C$.
Keeping redundant rows as if they were new restrictions. Repeating a contrast or adding a linear combination of existing rows does not enlarge the tested subspace. The effective restriction count is $\operatorname{rank}(C)$.
Treating $H$ as the observed truth of the null. $H$ is a random sample matrix, not a yes-or-no indicator. Inference compares it with its sampling variation, represented by $E$.

Key takeaway: Choose a basis for the predictor-contrast subspace, place its transposed basis vectors in $C$, and test $CB=0$ across all responses. The resulting hypothesis matrix is $H=(C\widehat B)^T[C(X^TX)^{-1}C^T]^{-1}(C\widehat B)$.

Check Your Understanding

Work through these questions by identifying the contrast subspace before writing down a test.

In a model with an intercept, linear dose, and quadratic dose, what matrix $C$ tests the quadratic coefficient alone across all responses?
Why do two full-rank contrast matrices with the same row space specify the same null hypothesis?
For a full-row-rank $C$ with $r$ rows, how many scalar coefficient restrictions are imposed across $p$ responses?
What role does $C(X^TX)^{-1}C^T$ play in the hypothesis matrix?
Under the Gaussian null model, what are the degrees of freedom of $H$ and of the unrestricted residual matrix $E$?