Covariances of Linear Coefficient Summaries

Why Summarize a Coefficient Matrix?

A multivariate regression can estimate many coefficients at once: each predictor has a coefficient for each outcome. Often the scientific question concerns neither one coefficient nor the full matrix, but a particular comparison—for example, the difference between two predictors’ effects on one outcome, or a weighted combination of effects across several outcomes. Such summaries are useful only when their uncertainty is calculated on the same scale as the summary.

This tutorial develops the covariance of those linear summaries from the covariance of the estimated coefficient matrix. The key point is that a contrast across predictors and a contrast across outcomes both affect the variance. Their contributions combine through separate covariance factors.

Linear Summaries and Their Covariance

Use the multivariate linear model $Y=XB+U$, where $Y$ is an $n\times q$ response matrix, $X$ is an $n\times p$ design matrix, $B$ is a $p\times q$ coefficient matrix, and the rows of $U$ are independent with mean zero and covariance matrix $\Sigma$. Treat $X$ as fixed. For now, suppose $X$ has full column rank and write $G=(X'X)^{-1}$. Column-stacking vectorization, denoted by $\operatorname{vec}$, stacks the columns of a matrix in order.

A linear summary has the form $LBR$, where $L$ is an $a\times p$ matrix specifying combinations of predictors and $R$ is a $q\times b$ matrix specifying combinations of outcomes. The estimated summary is $L\hat B R$, an $a\times b$ matrix. Its vectorization is a vector of length $ab$, whose covariance matrix describes uncertainty and covariance among all its entries.

Theorem: Under the model above, the covariance of the estimated linear summary is $$ \operatorname{Cov}\!\left(\operatorname{vec}(L\hat B R)\right) = (R'\Sigma R)\otimes(LGL'). $$ Here $\otimes$ denotes the Kronecker product. In particular, for column vectors $l\in\mathbb{R}^p$ and $r\in\mathbb{R}^q$, the scalar summary $l'\hat B r$ has variance $$ \operatorname{Var}(l'\hat B r)=(l'Gl)(r'\Sigma r). $$

The formula is exact when $\Sigma$ is the true response covariance matrix. If $\Sigma$ is unknown, a standard error is obtained by substituting an estimate of $\Sigma$, such as the residual covariance estimate under the fitted model. The covariance formula itself does not require normal errors; the stated mean and covariance assumptions and independence across rows suffice.

Deriving the Summary Covariance

First derive the coefficient covariance, then apply the vectorization identity $\operatorname{vec}(ABC)=(C'\otimes A)\operatorname{vec}(B)$, for conformable matrices. The derivation makes explicit why the order of the covariance factors matters.

Write the coefficient error.
With $\hat B=(X'X)^{-1}X'Y$, we have $\hat B-B=(X'X)^{-1}X'U$.

Vectorize it.
Let $H=(X'X)^{-1}X'$. Then $\operatorname{vec}(\hat B-B)=(I_q\otimes H)\operatorname{vec}(U)$.

Use the row-error covariance.
Independence across rows and common covariance $\Sigma$ give $\operatorname{Cov}(\operatorname{vec}(U))=\Sigma\otimes I_n$.

Obtain the coefficient covariance.
Applying the covariance transformation rule gives $\operatorname{Cov}(\operatorname{vec}(\hat B))=\Sigma\otimes HH'=\Sigma\otimes G$.

Transform the summary.
Since $\operatorname{vec}(L\hat B R)=(R'\otimes L)\operatorname{vec}(\hat B)$, transform its covariance by left and right multiplication.

For completeness, the coefficient-covariance calculation in Step 4 follows by multiplying the matrices:

\begin{aligned} \operatorname{Cov}(\operatorname{vec}(\hat B)) &=(I_q\otimes H)(\Sigma\otimes I_n)(I_q\otimes H')\\ &=\Sigma\otimes(HH')\\ &=\Sigma\otimes(X'X)^{-1}. \end{aligned}

The covariance of the summary is therefore

\begin{aligned} \operatorname{Cov}(\operatorname{vec}(L\hat B R)) &=(R'\otimes L)(\Sigma\otimes G)(R\otimes L')\\ &=(R'\Sigma R)\otimes(LGL'). \end{aligned}

The second equality uses the mixed-product rule for Kronecker products. For a scalar summary, take $a=b=1$, $L=l'$, and $R=r$. The Kronecker product of the resulting two scalars is their product, giving $(l'Gl)(r'\Sigma r)$. This proves the theorem. $\square$

Worked Examples

Worked Example: A Contrast Across Predictors and Outcomes

Suppose there are two predictors and two outcomes, and the estimated coefficient matrix and covariance factors are $$ \hat B= \begin{pmatrix} 2&1\\ -0.5&3 \end{pmatrix}, \qquad G= \begin{pmatrix} 0.5&0.1\\ 0.1&0.4 \end{pmatrix}, \qquad \Sigma= \begin{pmatrix} 4&1.2\\ 1.2&9 \end{pmatrix}. $$ Consider the difference between the predictors’ coefficients, followed by the difference between the two outcomes. Thus $L=(1,-1)$ and $R=(1,-1)'$.

First calculate the estimated summary: $$ L\hat B=(2-(-0.5),\;1-3)=(2.5,-2), \qquad L\hat B R=2.5(1)+(-2)(-1)=4.5. $$ The predictor contrast factor is $$ LGL'=0.5+0.4-0.1-0.1=0.7, $$ and the outcome contrast factor is $$ R'\Sigma R=4+9-1.2-1.2=10.6. $$ Therefore the estimated variance and standard error are $$ \widehat{\operatorname{Var}}(L\hat B R)=0.7(10.6)=7.42, \qquad \operatorname{SE}(L\hat B R)=\sqrt{7.42}\approx2.724. $$ This calculation assumes the displayed $\Sigma$ is known; with an estimated response covariance, substitute that estimate in the same calculation.

Worked Example: Standard Errors for a Predictor’s Outcome Coefficients

Using the same matrices, select the second predictor’s coefficients for both outcomes. Set $L=(0,1)$ and $R=I_2$. The summary is the row vector $$ L\hat B R=(0,1) \begin{pmatrix} 2&1\\ -0.5&3 \end{pmatrix} =(-0.5,3). $$ Its vectorization is the column vector $(-0.5,3)'$. Here $LGL'=0.4$, so the covariance matrix is $$ \operatorname{Cov}\!\left(\operatorname{vec}(L\hat B)\right) =\Sigma\otimes 0.4 = \begin{pmatrix} 1.6&0.48\\ 0.48&3.6 \end{pmatrix}. $$ Thus the standard errors for the two coefficients are $\sqrt{1.6}\approx1.265$ and $\sqrt{3.6}\approx1.897$, respectively. The off-diagonal entry $0.48$ is their covariance; it is not part of either individual standard error. For the sum of the two coefficients, $R=(1,1)'$, the variance is $$ 0.4\bigl(4+9+2(1.2)\bigr)=0.4(15.4)=6.16, $$ and the standard error is $\sqrt{6.16}\approx2.482$.

Worked Example: Covariance Between Two Different Summaries

Let the first summary be $T_1=l_1'\hat B r_1$, where $l_1=(1,-1)'$ and $r_1=(1,-1)'$. Let the second be $T_2=l_2'\hat B r_2$, where $l_2=(0,1)'$ and $r_2=(1,1)'$. Using the same $G$ and $\Sigma$, their covariance is $$ \operatorname{Cov}(T_1,T_2) =(l_1'Gl_2)(r_1'\Sigma r_2). $$ Calculate each factor: $$ l_1'Gl_2=(1,-1) \begin{pmatrix}0.1\\0.4\end{pmatrix} =0.1-0.4=-0.3, \qquad r_1'\Sigma r_2=(1,-1) \begin{pmatrix}5.2\\10.2\end{pmatrix} =5.2-10.2=-5. $$ Hence $\operatorname{Cov}(T_1,T_2)=(-0.3)(-5)=1.5$. As checks, $T_1$ has variance $7.42$, as in the first example, while $$ \operatorname{Var}(T_2) =(l_2'Gl_2)(r_2'\Sigma r_2) =0.4(4+9+2.4)=6.16. $$ The covariance is positive because both contrast factors are negative; neither factor alone is the covariance of the two summaries.

Rank Deficiency and Estimable Summaries

The full-column-rank assumption gives the simplest expression for $G$. If $X$ is rank deficient, individual entries of $B$ may not be identifiable, but some linear summaries can still be estimable. In that case, use a symmetric generalized inverse $G$ of $X'X$, satisfying $(X'X)G(X'X)=X'X$. A row of $L$ is estimable when it lies in the row space of $X$; equivalently, for such a row, $LGX'X=L$. For an estimable $LBR$, the same covariance formula applies with this generalized inverse: $$ \operatorname{Cov}(\operatorname{vec}(L\hat B R)) =(R'\Sigma R)\otimes(LGL'). $$

To see why the coefficient estimate’s covariance factor reduces to $LGL'$, a generalized-inverse least-squares estimate has coefficient covariance factor $G(X'X)G$. For an estimable $L$, $$ LG(X'X)GL'=LG L', $$ because $LGX'X=L$ and, by symmetry, $X'XGL'=L'$. Thus the uncertainty of an estimable summary does not require treating nonestimable coefficients as though they were individually identified. The formula is for estimable summaries; it does not make a nonestimable coefficient contrast meaningful.

Common Mistakes and Practical Use

The covariance formula separates uncertainty due to predictor combinations from uncertainty due to outcome combinations. This is especially useful when a scientific question defines a contrast in both dimensions: the calculation can be checked factor by factor before the final product is formed. For a matrix-valued summary, the Kronecker formula also gives covariances among its different entries, not just separate standard errors.

Reversing the vectorization order. With column-stacking, $\operatorname{vec}(L\hat B R)=(R'\otimes L)\operatorname{vec}(\hat B)$. Using $L\otimes R'$ instead generally has incompatible dimensions or represents a different ordering.
Using the wrong covariance-factor order. The coefficient covariance is $\Sigma\otimes G$, not $G\otimes\Sigma$, under column-stacking. The summary covariance accordingly has outcome factor $R'\Sigma R$ first and predictor factor $LGL'$ second.
Adding the two scalar factors. For $l'\hat B r$, the variance is $(l'Gl)(r'\Sigma r)$, a product. Each factor is a variance contribution on its respective contrast scale.
Reporting a variance as a standard error. The diagonal entries of a summary covariance matrix are variances. Take their square roots to obtain standard errors.
Using the formula for a nonestimable contrast. In a rank-deficient design, check estimability of the rows of $L$ before interpreting $L\hat B R$.

Key takeaway: For a linear coefficient summary $L\hat B R$, its covariance is the Kronecker product of the outcome-contrast covariance and the predictor-contrast covariance. For one scalar contrast, multiply those two scalar factors; for multiple summaries, retain the full covariance matrix.

Check Your Understanding

Use the definitions and formulas above to answer these questions.

For $L$ of size $a\times p$, $\hat B$ of size $p\times q$, and $R$ of size $q\times b$, what are the dimensions of $L\hat B R$ and its vectorized covariance matrix?
Why does the covariance of $\operatorname{vec}(\hat B)$ have the order $\Sigma\otimes G$ under column-stacking?
If $l'Gl=0.25$ and $r'\Sigma r=8$, what are the variance and standard error of $l'\hat B r$?
For a matrix summary, what information is contained in the off-diagonal entries of its covariance matrix?
When $X$ is rank deficient, what condition must the rows of $L$ satisfy for the stated covariance formula to apply to $L\hat B R$?