Covariance of the Least-Squares Coefficient Matrix

Why Coefficient Covariance Has Two Components

In a multivariate regression, each predictor has a coefficient for every response. Uncertainty in those coefficients therefore has two sources: the design determines how precisely predictor effects can be separated, while covariance between responses determines how coefficient estimates for different outcomes vary together. The covariance of the full coefficient matrix combines these two structures in a single Kronecker product.

We derive that expression for a fixed, full-column-rank design and make its assumptions explicit. In particular, independence across cases is sufficient for the familiar formula, but it is not necessary: what the covariance calculation requires is a common within-case covariance and zero covariance between distinct cases.

Model, Vectorization, and Covariance Assumptions

Let \(Y\) be an \(n\times q\) response matrix, \(X\) a fixed \(n\times p\) design matrix of full column rank, and \(B\) a \(p\times q\) coefficient matrix. Write the multivariate linear model as \(Y=XB+E\), where \(E\) is the \(n\times q\) error matrix. The \(i\)th row of \(E\) is \(e_i'\), with \(e_i\) a \(q\)-dimensional column vector. Assume \(E[e_i]=0\).

We use column-wise vectorization: \(\operatorname{vec}(M)\) stacks the columns of a matrix \(M\) in order. Define \(A=(X'X)^{-1}\). The least-squares coefficient matrix is \(\hat B=AX'Y\), so that \(\hat B-B=AX'E\).

The order of the factors matters. Under column-wise vectorization, the \(q\times q\) response covariance \(\Sigma\) indexes blocks, and each block is an \(n\times n\) matrix describing the case dimension. Thus the block in response positions \(j,k\) is \(\Sigma_{jk}I_n\).

Main Result and Derivation

To establish the error covariance used in the theorem, partition \(\operatorname{vec}(E)\) into its \(q\) columns. The covariance block between columns \(j\) and \(k\) has \((i,l)\) entry \(\operatorname{Cov}(E_{ij},E_{lk})\). When \(i=l\), this entry is \(\Sigma_{jk}\); when \(i\ne l\), it is zero by the cross-case covariance condition. The entire block is therefore \(\Sigma_{jk}I_n\). Collecting all such blocks gives \(\operatorname{Cov}(\operatorname{vec}(E))=\Sigma\otimes I_n\).

Now let \(C=AX'\), a \(p\times n\) matrix. The vectorization identity \(\operatorname{vec}(CM)=(I_q\otimes C)\operatorname{vec}(M)\), for an \(n\times q\) matrix \(M\), gives \(\operatorname{vec}(\hat B-B)=(I_q\otimes C)\operatorname{vec}(E)\). For any fixed matrix \(K\) and random vector \(Z\), \(\operatorname{Cov}(KZ)=K\operatorname{Cov}(Z)K'\). Applying this rule and the mixed-product property of Kronecker products,

In the last equality, \(A=(X'X)^{-1}\) is symmetric and \(AX'XA=A(X'X)A=A\). This proves the stated formula. \(\square\)

The same steps also show what changes if the error covariance conditions do not hold. If \(\Omega=\operatorname{Cov}(\operatorname{vec}(E))\) is the actual covariance matrix, then

The Kronecker formula follows when \(\Omega=\Sigma\otimes I_n\); if the covariance conditions fail, the covariance will generally differ from that formula.

Reading the Formula Coefficient by Coefficient

The vector \(\operatorname{vec}(\hat B)\) lists the estimated coefficients for response 1 first, then those for response 2, and so on. Consequently, \(\Sigma\otimes A\) is a \(q\times q\) array of \(p\times p\) blocks. Its block in response positions \(j,k\) is \(\Sigma_{jk}A\). In particular, for predictor indices \(a,b\) and response indices \(j,k\),

This entry-wise statement follows directly by selecting entry \(a,b\) from block \(j,k\) of the Kronecker product. If \(j=k\), it gives the covariance between coefficients for the same response; if \(a=b\), it gives the covariance between coefficients on the same predictor for two responses. In particular, \(\operatorname{Var}(\hat B_{aj})=A_{aa}\Sigma_{jj}\).

The design contribution \(A\) describes predictor geometry. Collinear or poorly separated design columns tend to make entries of \((X'X)^{-1}\) large, raising coefficient variances. The response contribution \(\Sigma\) scales those variances by each outcome's error variance and determines how estimates across outcomes covary. A zero \(\Sigma_{jk}\) makes the corresponding cross-response coefficient covariance block zero, but it does not by itself assert independence.

Common Pitfall and Practical Interpretation

A common mistake is to say that dependence between cases automatically invalidates \(\operatorname{Cov}(\operatorname{vec}(E))=\Sigma\otimes I_n\). Dependence does not automatically mean nonzero covariance. The covariance calculation uses only the individual case covariance matrices and the cross-case covariances; dependent errors can still have zero cross-case covariances. Conversely, if cross-case covariances are nonzero, or the case covariance matrices differ, the stacked covariance generally is not \(\Sigma\otimes I_n\), and the general expression involving \(\Omega\) should be used instead.

A second pitfall is to read a covariance matrix as a complete description of the joint distribution. The theorem gives variances and covariances; without additional distributional assumptions, it does not establish normality or independence of the estimated coefficients. The formula is useful precisely because it separates design uncertainty from response covariance without requiring normal errors.