Affine Transformation Laws for Gaussian Vectors

Why Affine Transformations Matter

Many multivariate questions do not concern every coordinate of a Gaussian vector separately. A projection may retain only a chosen direction, a contrast may compare several measurements, and a linear map may combine or duplicate coordinates. These operations all have the same mathematical form: transform a vector linearly, then add a fixed offset.

The central fact is simple but powerful: an affine transformation of a Gaussian vector is again Gaussian. Its mean transforms by the same affine map, while its covariance transforms by multiplication on both sides. The result remains valid even when the transformation discards information and the new covariance is singular.

Definition: A random vector $X\in\mathbb R^p$ is Gaussian with mean $\mu\in\mathbb R^p$ and covariance matrix $\Sigma$ if it has the same distribution as $\mu+C Z$, where $Z\sim N_p(0,I_p)$ and $CC^T=\Sigma$. The matrix $\Sigma$ is symmetric and positive semidefinite; it need not be invertible. We write $X\sim N_p(\mu,\Sigma)$, including when this distribution is degenerate.

Let $A$ be an $m\times p$ matrix and $b\in\mathbb R^m$. The dimensions ensure that $AX+b$ is an $m$-dimensional random vector. The transformation can change both the location and the dimension of the distribution.

Theorem: If $X\sim N_p(\mu,\Sigma)$, $A\in\mathbb R^{m\times p}$, and $b\in\mathbb R^m$, then $Y=AX+b$ is Gaussian and $$ Y\sim N_m(A\mu+b,\ A\Sigma A^T). $$ The transformed covariance may be singular, even if $\Sigma$ is positive definite.

Deriving the Affine Transformation Law

Use the Gaussian representation $X=\mu+C Z$, where $Z\sim N_p(0,I_p)$ and $CC^T=\Sigma$. Substitution into $Y=AX+b$ gives

Y=A\mu+b+ACZ.

A linear transformation of a standard normal vector is Gaussian, including when its matrix is rank deficient. This establishes the Gaussian form of $Y$. Its expectation and covariance can also be calculated directly, as follows.

Calculate the mean.
Linearity of expectation gives $E[Y]=A E[X]+b=A\mu+b$.

Center the transformed vector.
$Y-E[Y]=A(X-\mu)$, since the fixed offset cancels.

Calculate the covariance.
By the definition of covariance, $\operatorname{Cov}(Y)=E[(Y-E[Y])(Y-E[Y])^T]=A\Sigma A^T$.

For completeness, the covariance calculation follows by substituting the centered expression and taking the constant matrices outside the expectation:

\begin{aligned} \operatorname{Cov}(Y) &=E\!\left[A(X-\mu)(X-\mu)^T A^T\right]\\ &=A E\!\left[(X-\mu)(X-\mu)^T\right]A^T\\ &=A\Sigma A^T. \end{aligned}

The representation also verifies the Gaussian claim explicitly: $Y=A\mu+b+ACZ$ is an affine image of a standard normal vector, and $(AC)(AC)^T=A\Sigma A^T$. Thus the stated mean and covariance agree with the representation. $\square$

For a scalar linear summary, take $A=c^T$ and let $b=d$. The result says that $c^T X+d$ is univariate normal with mean $c^T\mu+d$ and variance $c^T\Sigma c$. This includes contrasts, weighted sums, and projections onto a single direction.

Worked Examples

Worked Example: Projecting onto a Direction

Let $X\sim N_2(\mu,\Sigma)$, where $\mu=(2,-1)^T$ and $\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}$. Project $X$ orthogonally onto the line spanned by $u=(1,1)^T/\sqrt{2}$. Find the distribution of the projected vector.

The projection matrix is $P=uu^T=\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix}$. This is a linear transformation with $A=P$ and $b=0$. Its transformed mean is

P\mu =\frac12 \begin{pmatrix}1&1\\1&1\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} = \begin{pmatrix}1/2\\1/2\end{pmatrix}.

Because $P=uu^T$, its transformed covariance is $P\Sigma P=u(u^T\Sigma u)u^T$. The scalar in this expression is

u^T\Sigma u =\frac12(4+1+1+2)=4.

Therefore $P\Sigma P=4uu^T=\begin{pmatrix}2&2\\2&2\end{pmatrix}$, and $PX\sim N_2\!\left((1/2,1/2)^T,\begin{pmatrix}2&2\\2&2\end{pmatrix}\right)$. The two coordinates of $PX$ are identical, so its covariance has rank one and the vector lies on the projected line. Projection does not destroy Gaussianity; it produces a degenerate Gaussian distribution in the original two-dimensional space.

Worked Example: A Contrast of Three Measurements

Suppose $X=(X_1,X_2,X_3)^T\sim N_3(\mu,\Sigma)$, with $\mu=(10,12,15)^T$ and $\Sigma=\begin{pmatrix}4&1&0\\1&9&2\\0&2&16\end{pmatrix}$. Find the distribution of the contrast $D=X_1-X_2$.

Write the contrast as $D=AX$, where $A=\begin{pmatrix}1&-1&0\end{pmatrix}$. Its mean is $A\mu=10-12=-2$. To calculate its variance, multiply the row vector by the covariance matrix and then by its transpose:

A\Sigma = \begin{pmatrix}1&-1&0\end{pmatrix} \begin{pmatrix}4&1&0\\1&9&2\\0&2&16\end{pmatrix} = \begin{pmatrix}3&-8&-2\end{pmatrix}, $$ $$ A\Sigma A^T = \begin{pmatrix}3&-8&-2\end{pmatrix} \begin{pmatrix}1\\-1\\0\end{pmatrix} =3+8=11.

Thus $D\sim N_1(-2,11)$. The covariance between $X_1$ and $X_2$ matters: the variance is $4+9-2(1)=11$, not $4+9$. In general, a contrast's variance depends on both the component variances and their covariances.

Worked Example: A Rank-Deficient Map

Let $X\sim N_2(\mu,\Sigma)$, where $\mu=(1,-1)^T$ and $\Sigma=\begin{pmatrix}1&0\\0&4\end{pmatrix}$. Apply $A=\begin{pmatrix}1&2\\2&4\end{pmatrix}$, which has two rows but rank one. Find the distribution of $Y=AX$ and describe its degeneracy.

The mean is $A\mu=(1-2,\ 2-4)^T=(-1,-2)^T$. For the covariance, first compute

A\Sigma = \begin{pmatrix}1&2\\2&4\end{pmatrix} \begin{pmatrix}1&0\\0&4\end{pmatrix} = \begin{pmatrix}1&8\\2&16\end{pmatrix}, $$ $$ A\Sigma A^T = \begin{pmatrix}1&8\\2&16\end{pmatrix} \begin{pmatrix}1&2\\2&4\end{pmatrix} = \begin{pmatrix}17&34\\34&68\end{pmatrix}.

Consequently $Y\sim N_2\!\left((-1,-2)^T,\begin{pmatrix}17&34\\34&68\end{pmatrix}\right)$. The second row of $A$ is twice the first, so $Y_2=2Y_1$ for every outcome. Its covariance is singular, and the distribution is supported on the line $y_2=2y_1$; it has no ordinary two-dimensional density. It is nevertheless a well-defined Gaussian distribution.

Rank, Support, and Degeneracy

The source of degeneracy can be made precise. Choose a square root $\Sigma^{1/2}$ with $\Sigma^{1/2}(\Sigma^{1/2})^T=\Sigma$, and put $B=A\Sigma^{1/2}$. Then the transformed covariance is $BB^T$. Its rank equals the rank of $B$: indeed, for any vector $v$, $v^TBB^Tv=\|B^Tv\|^2$, so $BB^Tv=0$ exactly when $B^Tv=0$. Hence $BB^T$ and $B$ have the same rank.

It follows that the rank of $A\Sigma A^T$ is the rank of $A\Sigma^{1/2}$, which is at most the rank of $A$. The transformed distribution is nondegenerate in $\mathbb R^m$ exactly when $A\Sigma^{1/2}$ has rank $m$. When its rank is smaller, the distribution is concentrated on an affine subspace whose direction space is the range of $A\Sigma^{1/2}$. Thus a positive-definite source covariance alone does not guarantee a positive-definite transformed covariance: the map $A$ must retain $m$ independent directions.

Common Mistakes and Practical Interpretation

One common mistake is to transform covariance as $A^T\Sigma A$. For an $m\times p$ map and a $p\times p$ covariance matrix, the correct transformed covariance is $A\Sigma A^T$, an $m\times m$ matrix. The transpose on the right is also what makes the result symmetric and positive semidefinite.

Another mistake is to assume that a singular covariance means the transformed vector is not Gaussian. Singularity means that some directions have zero variance and the distribution is confined to a lower-dimensional affine subspace; the affine transformation law still applies. Conversely, adding $b$ changes the mean but not the covariance, because a fixed offset disappears when the vector is centered.

Affine transformations are useful because they let a single Gaussian model describe derived quantities without rebuilding the distribution from scratch. The same calculation handles projections, contrasts, and linear summaries. The key checks are the dimensions of $A$ and $b$, the transformed mean $A\mu+b$, and whether the rank of the transformed covariance matches the intended output dimension.

Check Your Understanding

Use the affine transformation law and the rank interpretation to answer these questions.

If $X\sim N_p(\mu,\Sigma)$ and $Y=AX+b$, what are the dimensions and formulas for the mean and covariance of $Y$?
Why does adding a fixed vector $b$ not change the covariance?
For a scalar contrast $c^TX$, which covariance terms contribute to its variance?
What does a rank-deficient transformed covariance imply about the support of the Gaussian distribution?
Can a singular Gaussian vector still be Gaussian? Explain what singularity says about its variability.