Gaussian Vectors and Linear Combinations
Linear transformations of Gaussian vectors arise whenever several measurements, increments, or errors are combined. The transformed coordinates may be dependent, and the transformation may reduce the dimension of the random variation. The same calculation handles both ordinary covariance matrices and singular ones; singularity means that some nonzero linear combination of the coordinates has no random variation.
Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). A random vector \(X=(X_1,\ldots,X_d)^{\mathsf T}\) takes values in \(\mathbb{R}^d\). In this tutorial, a univariate normal variable is allowed to be degenerate: a variable with mean \(a\) and variance \(0\) equals \(a\) almost surely.
Every Gaussian vector has finite second moments because each coordinate is a scalar projection. Its covariance matrix is symmetric and positive semidefinite: for any \(u\in\mathbb{R}^d\), \(u^{\mathsf T}\Sigma u=\operatorname{Var}(u^{\mathsf T}X)\ge0\). No assumption of independence among the coordinates is part of the definition.
Consider a deterministic matrix \(A\in\mathbb{R}^{r\times d}\) and a deterministic vector \(b\in\mathbb{R}^r\). The affine transformation is \(Y=b+AX\), an \(\mathbb{R}^r\)-valued random vector. The key points are that each scalar projection of \(Y\) is a scalar projection of \(X\), plus a constant, and that covariance involves both the transformation and its transpose.
Proof. To verify that \(Y\) is Gaussian, take any \(c\in\mathbb{R}^r\). Then \(c^{\mathsf T}Y=c^{\mathsf T}b+(A^{\mathsf T}c)^{\mathsf T}X\). By the definition of a Gaussian vector, \((A^{\mathsf T}c)^{\mathsf T}X\) is normal; adding the constant \(c^{\mathsf T}b\) leaves it normal. If \(A^{\mathsf T}c=0\), this projection is constant, which is allowed by the definition. Thus every scalar projection of \(Y\) is normal. Linearity of expectation gives \(\mathbb{E}[Y]=b+A\mathbb{E}[X]=b+Am\). For the covariance, \(Y-\mathbb{E}[Y]=A(X-m)\), so $$ \operatorname{Cov}(Y)=\mathbb{E}\!\left[A(X-m)(X-m)^{\mathsf T}A^{\mathsf T}\right] =A\mathbb{E}\!\left[(X-m)(X-m)^{\mathsf T}\right]A^{\mathsf T} =A\Sigma A^{\mathsf T}. $$ The deterministic matrices can be taken outside the expectation, and the middle expectation is \(\Sigma\) by definition. This proves both the Gaussian property and the stated moment formulas. \(\square\)
The theorem gives a way to derive the transformed distribution: it is Gaussian, with the displayed mean and covariance. The covariance formula can also be checked coordinate by coordinate. If \(A\) has rows \(a_1^{\mathsf T},\ldots,a_r^{\mathsf T}\), then \(Y_i=b_i+a_i^{\mathsf T}X\), and the \((i,j)\)-entry of \(A\Sigma A^{\mathsf T}\) is \(a_i^{\mathsf T}\Sigma a_j=\operatorname{Cov}(Y_i,Y_j)\). The transpose is essential: \(A\Sigma\) alone is generally not symmetric and is not the covariance matrix.
Two-Dimensional Transformations
Worked Example: Combining Correlated Coordinates
Suppose \(X=(X_1,X_2)^{\mathsf T}\sim N_2(m,\Sigma)\), where \(m=(1,2)^{\mathsf T}\) and $$ \Sigma=\begin{bmatrix}2&2\\2&4\end{bmatrix}. $$ The covariance matrix is valid: it is symmetric, its first diagonal entry is \(2>0\), and its determinant is \(2\cdot4-2\cdot2=4>0\), so it is positive definite. Set $$ A=\begin{bmatrix}1&1\\1&-1\end{bmatrix},\qquad b=\begin{bmatrix}3\\-1\end{bmatrix}, \qquad Y=b+AX. $$
The mean is obtained by multiplying \(A\) by \(m\): $$ Am=\begin{bmatrix}1+2\\1-2\end{bmatrix} =\begin{bmatrix}3\\-1\end{bmatrix}, \qquad \mathbb{E}[Y]=b+Am=\begin{bmatrix}3+3\\-1-1\end{bmatrix} =\begin{bmatrix}6\\-2\end{bmatrix}. $$ For the covariance, first compute $$ A\Sigma= \begin{bmatrix} 2+2&2+4\\ 2-2&2-4 \end{bmatrix} = \begin{bmatrix}4&6\\0&-2\end{bmatrix}. $$ Since \(A^{\mathsf T}=A\), the next multiplication gives $$ \operatorname{Cov}(Y)=A\Sigma A^{\mathsf T} =\begin{bmatrix}4&6\\0&-2\end{bmatrix} \begin{bmatrix}1&1\\1&-1\end{bmatrix} =\begin{bmatrix}10&-2\\-2&2\end{bmatrix}. $$
Check by expanding the coordinates. Here \(Y_1=3+X_1+X_2\) and \(Y_2=-1+X_1-X_2\). Thus \(\operatorname{Var}(Y_1)=2+4+2(2)=10\), \(\operatorname{Var}(Y_2)=2+4-2(2)=2\), and \(\operatorname{Cov}(Y_1,Y_2)=\operatorname{Var}(X_1)-\operatorname{Var}(X_2)=2-4=-2\). These agree with the matrix calculation. In particular, the intermediate matrix \(A\Sigma=\begin{bmatrix}4&6\\0&-2\end{bmatrix}\) is not a covariance matrix: it is not symmetric. The final covariance is symmetric, as it must be.
Worked Example: A Singular Transformed Covariance
Let \(U\) and \(V\) be independent standard normal variables, and put \(X=(U,V)^{\mathsf T}\). Then \(X\) is Gaussian with mean \((0,0)^{\mathsf T}\) and covariance \(I_2=\begin{bmatrix}1&0\\0&1\end{bmatrix}\). Consider $$ A=\begin{bmatrix}1&1\\2&2\end{bmatrix},\qquad b=\begin{bmatrix}1\\-1\end{bmatrix}. $$ The transformed vector is \(Y=b+AX\). Its mean is \(b\), and its covariance is $$ A I_2 A^{\mathsf T}=AA^{\mathsf T} =\begin{bmatrix}1+1&2+2\\2+2&4+4\end{bmatrix} =\begin{bmatrix}2&4\\4&8\end{bmatrix}. $$
Writing \(R=U+V\), the coordinates are \(Y_1=1+R\) and \(Y_2=-1+2R\). Since \(\operatorname{Var}(R)=\operatorname{Var}(U)+\operatorname{Var}(V)=1+1=2\), their variances are \(2\) and \(4\cdot2=8\), and their covariance is \(2\cdot2=4\), confirming every entry. The covariance is singular: its determinant is \(2\cdot8-4\cdot4=0\). In fact \(Y_2=2Y_1-3\) for every outcome, so the random vector lies on a line rather than varying freely over the plane.
What Singularity Means
Proof. If \(\Gamma\) is singular, linear algebra gives a nonzero vector \(c\) with \(\Gamma c=0\). Therefore \(\operatorname{Var}(c^{\mathsf T}Y)=c^{\mathsf T}\Gamma c=0\). A finite-second-moment variable with variance zero equals its expectation almost surely, so \(c^{\mathsf T}Y=c^{\mathsf T}\mathbb{E}[Y]\) almost surely. Conversely, suppose this equality holds almost surely for some nonzero \(c\). Then \(c^{\mathsf T}(Y-\mathbb{E}[Y])=0\) almost surely. For every coordinate \(i\), $$ (\Gamma c)_i =\mathbb{E}\!\left[(Y_i-\mathbb{E}[Y_i])\,c^{\mathsf T}(Y-\mathbb{E}[Y])\right] =0. $$ Hence \(\Gamma c=0\) for a nonzero \(c\), which means \(\Gamma\) is singular. \(\square\)
For a Gaussian vector, this proposition identifies the precise effect of singularity: a nonzero linear combination of coordinates is constant almost surely. In the preceding example, \(c=(2,-1)^{\mathsf T}\) gives \(c^{\mathsf T}Y=2Y_1-Y_2=3\). Thus the singular covariance records the exact linear constraint. It does not mean that the vector fails to be Gaussian; degenerate normal projections are included in the definition.
Worked Example: A Nonsingular Transformation with Correlated Output
Suppose \(X\sim N_2(m,\Sigma)\), with \(m=(0,1)^{\mathsf T}\) and $$ \Sigma=\begin{bmatrix}1&-1\\-1&2\end{bmatrix}. $$ This matrix is positive definite because its first diagonal entry is \(1>0\) and its determinant is \(1\cdot2-(-1)(-1)=1>0\). Take $$ A=\begin{bmatrix}2&1\\1&0\end{bmatrix},\qquad b=\begin{bmatrix}-1\\2\end{bmatrix}. $$ First, $$ Am=\begin{bmatrix}2\cdot0+1\cdot1\\1\cdot0+0\cdot1\end{bmatrix} =\begin{bmatrix}1\\0\end{bmatrix}, \qquad \mathbb{E}[Y]=b+Am=\begin{bmatrix}0\\2\end{bmatrix}. $$
For the covariance, the intermediate product and final product are $$ A\Sigma= \begin{bmatrix}2-1&-2+2\\1&-1\end{bmatrix} =\begin{bmatrix}1&0\\1&-1\end{bmatrix}, \qquad A\Sigma A^{\mathsf T} =\begin{bmatrix}1&0\\1&-1\end{bmatrix} \begin{bmatrix}2&1\\1&0\end{bmatrix} =\begin{bmatrix}2&1\\1&1\end{bmatrix}. $$ This covariance has determinant \(2\cdot1-1\cdot1=1>0\), so it is nonsingular. To verify directly, the coordinates are \(Y_1=-1+2X_1+X_2\) and \(Y_2=2+X_1\). Their means are \(-1+2\cdot0+1=0\) and \(2+0=2\). Their variances are \(4(1)+2+4(-1)=2\) and \(1\), while their covariance is \(2\operatorname{Var}(X_1)+\operatorname{Cov}(X_2,X_1)=2-1=1\). This reproduces the calculated matrix.
Common Mistake and Practical Checks
Common mistake: using \(A\Sigma\) as the transformed covariance. For the first example, direct multiplication gives $$ A\Sigma= \begin{bmatrix}1&1\\1&-1\end{bmatrix} \begin{bmatrix}2&2\\2&4\end{bmatrix} =\begin{bmatrix}2+2&2+4\\2-2&2-4\end{bmatrix} =\begin{bmatrix}4&6\\0&-2\end{bmatrix}. $$ This matrix is nonsymmetric, so it cannot be a covariance matrix. The required second multiplication is by \(A^{\mathsf T}\), giving \(A\Sigma A^{\mathsf T}=\begin{bmatrix}10&-2\\-2&2\end{bmatrix}\). Another useful check is to calculate the variances of the transformed coordinates from their linear expressions, as in the examples; each must agree with the corresponding diagonal entry of \(A\Sigma A^{\mathsf T}\).
When applying the affine transformation rule, keep the roles of the inputs distinct: \(b\) changes the mean but not the covariance, while \(A\) changes both. A rank-deficient \(A\) can force a singular output covariance even if the input covariance is nonsingular. Conversely, a nonsingular covariance requires that no nonzero linear combination of the output coordinates be constant almost surely. These checks diagnose algebraic errors without requiring a density formula, which would not apply in the usual full-dimensional form to a singular Gaussian vector.
Check Your Understanding
For each question, state any assumptions you use and verify matrix calculations by multiplication or by expanding the transformed coordinates.
- If \(X\sim N_2(m,\Sigma)\) and \(Y=b+AX\), write the formulas for \(\mathbb{E}[Y]\) and \(\operatorname{Cov}(Y)\). Why does \(b\) not appear in the covariance?
- Let \(X\) have covariance \(I_2\), and let \(A=\begin{bmatrix}1&-1\\2&-2\end{bmatrix}\). Compute \(A I_2 A^{\mathsf T}\) and identify a nonzero linear combination of the output coordinates with zero variance.
- For the first worked example, calculate \(\operatorname{Var}(Y_1)\) directly from \(Y_1=3+X_1+X_2\). Which entries of \(\Sigma\) contribute?
- Explain why a zero variance for \(c^{\mathsf T}Y\), with \(c\ne0\), implies a constant linear relation almost surely.
- Can an affine transformation of a Gaussian vector remain Gaussian when its covariance matrix is singular? Justify your answer using the definition by scalar projections.