Why Transform a Covariance Matrix?
A random vector may record several measurements on the same unit. A new analysis might use scaled measurements, sums and differences, or other linear combinations of those coordinates. The transformed measurements are still random, and their variances and covariances depend on how the original coordinates vary together.
The central rule is compact: if \(X\) is transformed to \(AX+b\), its covariance matrix becomes \(A\operatorname{Cov}(X)A^{\mathsf T}\). The order of the factors matters, as do their dimensions. We derive the rule from the definition and use it in several calculations.
Let \(A\) be a fixed, nonrandom \(m\times p\) real matrix and let \(b\) be a fixed vector in \(\mathbb{R}^m\). Then \(Y=AX+b\) is an \(m\)-dimensional random vector. Its covariance matrix must therefore be \(m\times m\). The finite-second-moment condition ensures that the covariances in these matrices are finite.
Worked Example: A Two-Coordinate Transformation
Suppose \(X\) has mean \(\mu=(1,-2)^{\mathsf T}\) and covariance matrix $$ \Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}. $$ For $$ A=\begin{pmatrix}1&2\\-1&1\end{pmatrix}, \qquad b=\begin{pmatrix}3\\-2\end{pmatrix}, $$ find the mean and covariance matrix of \(Y=AX+b\).
The mean is \(E[Y]=A\mu+b\). First, $$ A\mu= \begin{pmatrix}1&2\\-1&1\end{pmatrix} \begin{pmatrix}1\\-2\end{pmatrix} = \begin{pmatrix}-3\\-3\end{pmatrix}, $$ so \(E[Y]=(0,-5)^{\mathsf T}\).
For covariance, the transformation rule gives \(A\Sigma A^{\mathsf T}\). The intermediate product and the final product are $$ A\Sigma= \begin{pmatrix}6&19\\-3&8\end{pmatrix}, \qquad A\Sigma A^{\mathsf T} = \begin{pmatrix}6&19\\-3&8\end{pmatrix} \begin{pmatrix}1&-1\\2&1\end{pmatrix} = \begin{pmatrix}44&13\\13&11\end{pmatrix}. $$ Thus \(Y\) has mean \((0,-5)^{\mathsf T}\) and covariance matrix \(\begin{pmatrix}44&13\\13&11\end{pmatrix}\). The shift \(b\) changes the mean but not the covariance.
The Transformation Rule
To prove the result, first calculate the mean. Since the entries of \(A\) and \(b\) are fixed, linearity of expectation gives \(E[AX+b]=A\mu+b\). Therefore, $$ (AX+b)-E[AX+b]=A(X-\mu). $$ Substitute this centered vector into the definition of covariance: $$ \operatorname{Cov}(AX+b) =E\!\left[A(X-\mu)(X-\mu)^{\mathsf T}A^{\mathsf T}\right]. $$ The matrix entries here are finite sums of terms involving the entries of \((X-\mu)(X-\mu)^{\mathsf T}\). Because the sums are finite and \(A\) is fixed, each entry can be evaluated by linearity of expectation: $$ \operatorname{Cov}(AX+b) =A\,E\!\left[(X-\mu)(X-\mu)^{\mathsf T}\right]A^{\mathsf T} =A\operatorname{Cov}(X)A^{\mathsf T}. $$ This proves the theorem. \(\square\)
The same argument gives an entry-by-entry interpretation. Write \(a_{ik}\) for the entry in row \(i\), column \(k\) of \(A\), and write \(\Sigma=\operatorname{Cov}(X)\). The \(i\)th coordinate of \(AX\) is \(\sum_{k=1}^{p}a_{ik}X_k\), so $$ \operatorname{Cov}(Y_i,Y_j) =\sum_{k=1}^{p}\sum_{l=1}^{p}a_{ik}a_{jl}\operatorname{Cov}(X_k,X_l) =\sum_{k=1}^{p}\sum_{l=1}^{p}a_{ik}\Sigma_{kl}a_{jl}. $$ Thus each transformed covariance combines the original variances and covariances, weighted by the coefficients used to form the two transformed coordinates.
Worked Example: Two Combinations of Three Coordinates
Let \(X=(X_1,X_2,X_3)^{\mathsf T}\) have covariance matrix $$ \Sigma= \begin{pmatrix} 1&0.5&-0.25\\ 0.5&4&1\\ -0.25&1&2 \end{pmatrix}, $$ and define \(Y=AX+b\), where $$ A=\begin{pmatrix}1&1&0\\0&-1&2\end{pmatrix}, \qquad b=\begin{pmatrix}2\\-1\end{pmatrix}. $$ The two coordinates are \(Y_1=X_1+X_2+2\) and \(Y_2=-X_2+2X_3-1\). Calculate their covariance matrix.
The fixed shifts do not affect covariance. First multiply \(\Sigma\) by \(A^{\mathsf T}\), or equivalently calculate \(A\Sigma\): $$ A\Sigma= \begin{pmatrix} 1.5&4.5&0.75\\ -1&-2&3 \end{pmatrix}. $$ Now multiply by $$ A^{\mathsf T}=\begin{pmatrix}1&0\\1&-1\\0&2\end{pmatrix}. $$ The entries are $$ (A\Sigma A^{\mathsf T})_{11}=1.5+4.5=6,\qquad (A\Sigma A^{\mathsf T})_{12}=-4.5+1.5=-3, $$ $$ (A\Sigma A^{\mathsf T})_{21}=-1-2=-3,\qquad (A\Sigma A^{\mathsf T})_{22}=2+6=8. $$ Consequently, $$ \operatorname{Cov}(Y)= \begin{pmatrix}6&-3\\-3&8\end{pmatrix}. $$ As a direct check, \(\operatorname{Var}(Y_1)=1+4+2(0.5)=6\). Also, \(\operatorname{Var}(Y_2)=4+4(2)-4(1)=8\), and \(\operatorname{Cov}(Y_1,Y_2)=-0.5-0.5-4+2=-3\). These agree with the matrix calculation.
Dimensions, Shifts, and Coordinate Changes
If \(\Sigma\) is \(p\times p\) and \(A\) is \(m\times p\), then \(A\Sigma\) is \(m\times p\), and \(A\Sigma A^{\mathsf T}\) is \(m\times m\). This dimension check is useful before doing any arithmetic. The transpose on the right is essential: it aligns the coefficients for the second transformed coordinate.
Several consequences follow directly from the formula. A fixed shift \(b\) has no effect on covariance. Multiplying every transformed coordinate by a constant \(c\) multiplies its covariance matrix by \(c^2\), since \((cA)\Sigma(cA)^{\mathsf T}=c^2A\Sigma A^{\mathsf T}\). More generally, different rows of \(A\) can scale, reorder, or combine coordinates in different ways.
Worked Example: Rescaling and Reordering Coordinates
Suppose \(X=(X_1,X_2)^{\mathsf T}\) has covariance matrix $$ \Sigma=\begin{pmatrix}4&-1\\-1&9\end{pmatrix}. $$ Let \(Y=(2X_2+5,\,3X_1-4)^{\mathsf T}\). Identify \(A\) and calculate \(\operatorname{Cov}(Y)\).
The coefficients of \(X_1\) and \(X_2\) in each coordinate give $$ A=\begin{pmatrix}0&2\\3&0\end{pmatrix}, \qquad Y=AX+\begin{pmatrix}5\\-4\end{pmatrix}. $$ The shift can be omitted from the covariance calculation. Multiplication gives $$ A\Sigma= \begin{pmatrix}-2&18\\12&-3\end{pmatrix}, \qquad A^{\mathsf T}=\begin{pmatrix}0&3\\2&0\end{pmatrix}, $$ and therefore $$ \operatorname{Cov}(Y)=A\Sigma A^{\mathsf T} = \begin{pmatrix}36&-6\\-6&36\end{pmatrix}. $$ The diagonal entries are \(4\operatorname{Var}(X_2)=36\) and \(9\operatorname{Var}(X_1)=36\). The off-diagonal entry is \(6\operatorname{Cov}(X_2,X_1)=6(-1)=-6\), confirming the matrix product.
Transformations can also be applied successively. If \(Y=AX+b\) and \(Z=CY+d\), where \(C\) and \(d\) are fixed and have compatible dimensions, then \(Z=(CA)X+(Cb+d)\). Its covariance is $$ \operatorname{Cov}(Z)=C\operatorname{Cov}(Y)C^{\mathsf T} =C(A\Sigma A^{\mathsf T})C^{\mathsf T} =(CA)\Sigma(CA)^{\mathsf T}. $$ This follows because \((CA)^{\mathsf T}=A^{\mathsf T}C^{\mathsf T}\). Either transforming in stages or multiplying the transformation matrices first gives the same covariance.
Common Mistake and Why the Rule Matters
A frequent mistake is to write \(A^{\mathsf T}\Sigma A\) for the covariance of \(AX\). If \(A\) is \(m\times p\) and \(\Sigma\) is \(p\times p\), that product is generally not even defined: \(A^{\mathsf T}\) is \(p\times m\), so \(A^{\mathsf T}\Sigma\) requires \(m=p\). Even when the dimensions happen to allow the multiplication, the expression generally describes a different arrangement of coefficients. For \(Y=AX+b\), start with the transformed coordinates and use \(A\Sigma A^{\mathsf T}\).
The rule applies to a fixed matrix \(A\). If the matrix itself is random, it cannot simply be taken outside the expectation in the proof; the displayed formula does not automatically follow. The rule also helps keep the roles of mean and covariance distinct: the mean transforms as \(A\mu+b\), while covariance transforms as \(A\Sigma A^{\mathsf T}\) and does not involve \(b\).
This calculation is useful whenever a statistical analysis changes coordinates or creates new measurements as linear combinations of existing ones. It gives the variances of the new coordinates on the diagonal and their pairwise covariances off the diagonal, while preserving all contributions from the dependence among the original coordinates.
Check Your Understanding
Use the transformation rule and the dimensions of the matrices to answer these questions.
- If \(X\) is \(p\)-dimensional and \(A\) is \(m\times p\), what are the dimensions of \(\operatorname{Cov}(X)\) and \(\operatorname{Cov}(AX+b)\)?
- Why does adding a fixed vector \(b\) change the mean of \(AX+b\) but not its covariance?
- For \(Y_1=X_1+X_2\), write \(\operatorname{Var}(Y_1)\) in terms of the variances and covariance of \(X_1\) and \(X_2\).
- What feature of the derivation fails if \(A\) is random rather than fixed?
- How can symmetry of a calculated transformed covariance matrix help identify an error?