Covariance Under Linear Transformations

Why Covariance Changes Under a Transformation

A vector of measurements can be recoded, rescaled, or combined into new variables. For example, two laboratory measurements might be converted into a sum and a difference, or measurements in different units might be standardized by multiplying by constants. The means change under these operations, but covariance changes according to a precise matrix rule. Knowing that rule lets us calculate uncertainty in the new coordinates without reconstructing the full distribution.

The rule does not require normality or independence. It requires only finite second moments, and it applies to discrete, continuous, or mixed random vectors. The earlier material on means and covariance from joint moments supplies the basic definitions used here.

Covariance Matrices and Cross-Covariance

Definition: Let $X$ be a random vector in $\mathbb{R}^p$ with finite second moments and mean $\mu_X=E[X]$. Its covariance matrix is $$ \operatorname{Cov}(X)=E\left[(X-\mu_X)(X-\mu_X)^T\right]. $$ For random vectors $X\in\mathbb{R}^p$ and $Y\in\mathbb{R}^q$ with finite second moments, define their cross-covariance matrix by $$ \operatorname{Cov}(X,Y)=E\left[(X-\mu_X)(Y-\mu_Y)^T\right]. $$ The covariance matrix is $p\times p$, and the cross-covariance matrix is $p\times q$.

The $(i,j)$ entry of $\operatorname{Cov}(X,Y)$ is $\operatorname{Cov}(X_i,Y_j)$. In particular, $\operatorname{Cov}(X,X)=\operatorname{Cov}(X)$, which is symmetric. Finite second moments ensure that these expectations exist: each product of centered components has finite absolute expectation by the Cauchy–Schwarz inequality.

For a fixed matrix $A$ and vector $a$, the affine transformation of $X$ is $U=AX+a$. If $A$ has $r$ rows, then $U\in\mathbb{R}^r$. The vector $a$ shifts every value by the same amount; $A$ forms linear combinations of the components.

Worked Example: A Sum and a Difference

Suppose $X=(X_1,X_2)^T$ has mean $(3,-1)^T$ and covariance $$ \Sigma_X= \begin{pmatrix} 4&1\\ 1&9 \end{pmatrix}. $$ Form $U=AX+a$, where $$ A= \begin{pmatrix} 1&2\\ -1&1 \end{pmatrix}, \qquad a= \begin{pmatrix} 2\\ 5 \end{pmatrix}. $$

The transformed vector has mean $A\mu_X+a$. The covariance calculation below depends on $A$, not on the mean or shift. First multiply: $$ A\Sigma_X= \begin{pmatrix} 1&2\\ -1&1 \end{pmatrix} \begin{pmatrix} 4&1\\ 1&9 \end{pmatrix} = \begin{pmatrix} 6&19\\ -3&8 \end{pmatrix}. $$ Then multiply by $A^T$: $$ \operatorname{Cov}(U)=A\Sigma_XA^T = \begin{pmatrix} 6&19\\ -3&8 \end{pmatrix} \begin{pmatrix} 1&-1\\ 2&1 \end{pmatrix} = \begin{pmatrix} 44&13\\ 13&11 \end{pmatrix}. $$

Thus the first transformed component has variance $44$, the second has variance $11$, and their covariance is $13$. The result is symmetric, as a covariance matrix must be.

The Affine Covariance Transformation Formula

Theorem: Let $X\in\mathbb{R}^p$ have finite second moments, let $A$ be a fixed $r\times p$ matrix, and let $a\in\mathbb{R}^r$ be fixed. For $U=AX+a$, $$ E[U]=A\mu_X+a, \qquad \operatorname{Cov}(U)=A\operatorname{Cov}(X)A^T. $$ The formula includes linear transformations by taking $a=0$.

Proof. Linearity of expectation gives $E[U]=E[AX+a]=A\mu_X+a$. Hence $$ U-E[U]=AX+a-(A\mu_X+a)=A(X-\mu_X). $$ Substitute this centered expression into the covariance definition: $$ \begin{aligned} \operatorname{Cov}(U) &=E\left[(U-E[U])(U-E[U])^T\right]\\ &=E\left[A(X-\mu_X)(X-\mu_X)^TA^T\right]\\ &=A\,E\left[(X-\mu_X)(X-\mu_X)^T\right]A^T\\ &=A\operatorname{Cov}(X)A^T. \end{aligned} $$ The matrices $A$ and $A^T$ are fixed, so they can be taken outside the expectation. The finite second-moment assumption ensures that the entries being averaged are integrable. $\square$

The dimensions provide a useful check: $A$ is $r\times p$, $\operatorname{Cov}(X)$ is $p\times p$, and $A^T$ is $p\times r$. The result is $r\times r$, the required size for the covariance of $U$. The shift $a$ disappears because subtracting the mean centers $U$ at $A\mu_X+a$, canceling the same fixed shift.

Cross-Covariance After Two Transformations

Two random vectors may be transformed separately, using different matrices and different shifts. Their cross-covariance follows the same centering argument, with one transformation on each side.

Theorem: Let $X\in\mathbb{R}^p$ and $Y\in\mathbb{R}^q$ have finite second moments. Let $A$ be a fixed $r\times p$ matrix, $B$ a fixed $s\times q$ matrix, and $a\in\mathbb{R}^r$, $b\in\mathbb{R}^s$ fixed. For $U=AX+a$ and $V=BY+b$, $$ \operatorname{Cov}(U,V)=A\operatorname{Cov}(X,Y)B^T. $$

Proof. Centering the transformed vectors gives $U-E[U]=A(X-\mu_X)$ and $V-E[V]=B(Y-\mu_Y)$. Therefore $$ \begin{aligned} \operatorname{Cov}(U,V) &=E\left[(U-E[U])(V-E[V])^T\right]\\ &=E\left[A(X-\mu_X)(Y-\mu_Y)^TB^T\right]\\ &=A\,E\left[(X-\mu_X)(Y-\mu_Y)^T\right]B^T\\ &=A\operatorname{Cov}(X,Y)B^T. \end{aligned} $$ The result has dimension $r\times s$, matching the dimensions of $U$ and $V$. Fixed shifts cancel on centering, and the finite second moments justify the expectations. $\square$

Worked Example: Cross-Covariance of Two Linear Combinations

Let $X_1,X_2,Z_3,Z_4$ be independent, mean-zero, variance-one random variables. Define $$ X= \begin{pmatrix}X_1\\X_2\end{pmatrix}, \qquad Y= \begin{pmatrix}2X_1+Z_3\\-X_1+3X_2+Z_4\end{pmatrix}. $$ Independence and unit variances give $$ \operatorname{Cov}(X,Y)= \begin{pmatrix} \operatorname{Cov}(X_1,Y_1)&\operatorname{Cov}(X_1,Y_2)\\ \operatorname{Cov}(X_2,Y_1)&\operatorname{Cov}(X_2,Y_2) \end{pmatrix} = \begin{pmatrix} 2&-1\\ 0&3 \end{pmatrix}. $$ For example, $\operatorname{Cov}(X_1,Y_2)=\operatorname{Cov}(X_1,-X_1+3X_2+Z_4)=-1$.

Now take $U=X_1+2X_2$ and $V=3Y_1-Y_2$, so the transformation matrices are $A=(1\ \ 2)$ and $B=(3\ \ -1)$. The cross-covariance formula gives $$ \begin{aligned} \operatorname{Cov}(U,V) &= \begin{pmatrix}1&2\end{pmatrix} \begin{pmatrix}2&-1\\0&3\end{pmatrix} \begin{pmatrix}3\\-1\end{pmatrix}\\ &= \begin{pmatrix}2&5\end{pmatrix} \begin{pmatrix}3\\-1\end{pmatrix} =6-5=1. \end{aligned} $$ Checking by expansion, $$ \operatorname{Cov}(U,V) =3(2)-(-1)+6(0)-2(3)=1, $$ in agreement. The shifts, if added to either transformed variable, would not alter this result.

Consequences for Variances and Valid Covariance Matrices

A useful special case is a scalar linear combination $c^TX$, where $c\in\mathbb{R}^p$. It is the transformation formula with a one-row matrix, so $$ \operatorname{Var}(c^TX)=c^T\operatorname{Cov}(X)c. $$ This identity also shows why every covariance matrix is positive semidefinite.

Result: If $\Sigma=\operatorname{Cov}(X)$, then for every fixed vector $c$, $c^T\Sigma c\geq 0$. Consequently, for every matrix $A$, the transformed covariance $A\Sigma A^T$ is positive semidefinite.

Proof. The scalar $c^T(X-\mu_X)$ has finite variance, and $$ c^T\Sigma c =c^TE\left[(X-\mu_X)(X-\mu_X)^T\right]c =E\left[\{c^T(X-\mu_X)\}^2\right] =\operatorname{Var}(c^TX)\geq 0. $$ For the transformed matrix, take any vector $d$ of the appropriate dimension. Then $$ d^T(A\Sigma A^T)d=(A^Td)^T\Sigma(A^Td)\geq 0, $$ by the first part. $\square$

This argument does not claim that the transformed covariance is always invertible. If rows of $A$ repeat or one transformed component is an exact linear combination of others, some directions may have zero variance. Positive semidefinite means all quadratic forms are nonnegative; it does not mean all are strictly positive.

Worked Example: Sum and Difference With a Shift

Suppose $X=(X_1,X_2)^T$ has covariance $$ \Sigma_X= \begin{pmatrix} 1&0.5\\ 0.5&4 \end{pmatrix}. $$ Set $U=X_1+X_2+7$ and $V=X_1-X_2-2$. The linear transformation matrix is $$ A= \begin{pmatrix} 1&1\\ 1&-1 \end{pmatrix}, $$ and the shifts are $(7,-2)^T$. First calculate $$ A\Sigma_X= \begin{pmatrix} 1.5&4.5\\ 0.5&-3.5 \end{pmatrix}. $$ Thus $$ \operatorname{Cov} \begin{pmatrix}U\\V\end{pmatrix} =A\Sigma_XA^T = \begin{pmatrix} 1.5&4.5\\ 0.5&-3.5 \end{pmatrix} \begin{pmatrix} 1&1\\ 1&-1 \end{pmatrix} = \begin{pmatrix} 6&-3\\ -3&4 \end{pmatrix}. $$

The same entries can be checked directly. The variance of the sum is $1+4+2(0.5)=6$; the variance of the difference is $1+4-2(0.5)=4$. Their covariance is $$ \operatorname{Cov}(X_1+X_2,X_1-X_2) =\operatorname{Var}(X_1)-\operatorname{Var}(X_2) =-3, $$ because the two cross-covariance terms cancel. The constants $7$ and $-2$ do not enter any of these calculations.

Common Mistakes and Practical Checks

Putting the transpose on the wrong side. For $U=AX$, the covariance is $A\Sigma_XA^T$, not $A^T\Sigma_XA$. Check dimensions: the correct expression is $r\times r$ when $A$ is $r\times p$.
Assuming the components must be independent. The formulas use the full covariance or cross-covariance matrix, including off-diagonal entries. Independence is not required.
Keeping affine shifts in the covariance formula. A fixed vector changes the mean but cancels when the transformed vector is centered.
Dropping cross terms in a variance calculation. For example, $\operatorname{Var}(X_1+X_2)$ includes $2\operatorname{Cov}(X_1,X_2)$. The matrix product automatically includes this contribution.

These transformations matter whenever variables are recoded or combined: uncertainty in a linear score, a difference between measurements, or a new coordinate system can be calculated directly from the original covariance structure. Before accepting a result, verify its dimensions, symmetry when it is a covariance matrix, and nonnegative variances on the diagonal.

Check Your Understanding

Use the formulas and the centering argument to answer these questions.

If $U=AX+a$, what assumptions are needed for the covariance formula, and why does $a$ not appear in it?
For $A$ of size $r\times p$ and $\Sigma_X$ of size $p\times p$, what are the dimensions of $A\Sigma_XA^T$?
Write the cross-covariance formula for $U=AX+a$ and $V=BY+b$, and state the dimensions of its result.
If $U=2X_1-X_2$, express $\operatorname{Var}(U)$ using the component variances and covariance.
Why can a transformed covariance matrix be singular even when the original covariance matrix is not?