Gaussian Vectors as Affine Images of Standard Normals

Gaussian Vectors from Standard Normal Coordinates

A multivariate Gaussian distribution can be built from independent standard normal coordinates by shifting and combining them linearly. This construction is useful because it describes both ordinary Gaussian models, which have a density in the full space, and degenerate ones, which are confined to a lower-dimensional subspace. The dimensions and rank of the matrix used in the construction make that distinction explicit.

Let $Z$ be a $q$-dimensional standard normal vector: its $q$ coordinates are independent random variables, each with distribution $N(0,1)$. For a fixed vector $\mu\in\mathbb{R}^p$ and matrix $A\in\mathbb{R}^{p\times q}$, the vector $\mu+AZ$ takes values in $\mathbb{R}^p$.

Definition: A random vector $X\in\mathbb{R}^p$ is a Gaussian vector if it has the distribution of $\mu+AZ$, where $\mu\in\mathbb{R}^p$, $A\in\mathbb{R}^{p\times q}$ for some $q$, and $Z$ is a $q$-dimensional standard normal vector. Its mean parameter is $\mu$, and its covariance matrix is determined by $A$.

The number $q$ of standard normal coordinates need not equal the dimension $p$ of $X$. The matrix $A$ determines how the coordinates are combined; different choices of $A$ can describe the same Gaussian distribution. In particular, $A$ need not be invertible or even square.

Mean, Covariance, and Characteristic Function

The mean and covariance follow directly from the standard normal coordinates. Recall that $E[Z]=0$ and $\operatorname{Cov}(Z)=I_q$, where $I_q$ is the $q\times q$ identity matrix. Linearity of expectation gives the mean. Applying the covariance formula to $AZ$ gives the covariance.

Theorem: If $X=\mu+AZ$ with $Z$ a $q$-dimensional standard normal vector, then $$ E[X]=\mu,\qquad \operatorname{Cov}(X)=AA^T. $$ For $t\in\mathbb{R}^p$, its characteristic function is $$ \varphi_X(t)=E\!\left[\exp(i t^T X)\right] =\exp\!\left(i t^T\mu-\frac{1}{2}t^TAA^Tt\right). $$

To verify the moment formulas, take expectations coordinatewise. Since $A$ is fixed,

E[X]=E[\mu+AZ]=\mu+A E[Z]=\mu.

For covariance, subtract the mean and use $X-\mu=AZ$:

\begin{aligned} \operatorname{Cov}(X) &=E\!\left[(X-\mu)(X-\mu)^T\right]\\ &=E\!\left[AZZ^TA^T\right]\\ &=A E[ZZ^T]A^T\\ &=AA^T. \end{aligned}

For the characteristic function, use $t^TX=t^T\mu+(A^Tt)^TZ$. Independence of the coordinates of $Z$, together with the univariate standard normal characteristic function $E[\exp(isZ_j)]=\exp(-s^2/2)$, yields

\begin{aligned} \varphi_X(t) &=\exp(i t^T\mu)\prod_{j=1}^q \exp\!\left(-\frac{1}{2}\bigl((A^Tt)_j\bigr)^2\right)\\ &=\exp\!\left(i t^T\mu-\frac{1}{2}(A^Tt)^T(A^Tt)\right)\\ &=\exp\!\left(i t^T\mu-\frac{1}{2}t^TAA^Tt\right). \end{aligned}

These calculations prove the theorem. In particular, the characteristic function depends on $A$ only through $AA^T$. Since a characteristic function uniquely determines a probability distribution, Gaussian vectors with the same $\mu$ and covariance matrix have the same distribution. $\square$

Worked Example: A Scalar Gaussian from Two Normal Coordinates

Let $Z_1,Z_2$ be independent $N(0,1)$ random variables, and define $X=3+2Z_1-Z_2$. Identify the parameters of this Gaussian vector, and calculate its mean, variance, and characteristic function.

Write $Z=(Z_1,Z_2)^T$, $\mu=3$, and $A=(2,-1)$. Then $X=\mu+AZ$. The mean is $E[X]=\mu=3$. The covariance matrix is the scalar

$\operatorname{Var}(X)=AA^T=(2)(2)+(-1)(-1)=5.$

The characteristic function is therefore

\varphi_X(t)=\exp\!\left(3it-\frac{5t^2}{2}\right).

Thus $X\sim N(3,5)$, where the second parameter is the variance. The two latent coordinates contribute variances $2^2=4$ and $(-1)^2=1$; independence is what allows these contributions to add.

Rank Determines Whether the Law Is Degenerate

The covariance $AA^T$ is always positive semidefinite: for any $u\in\mathbb{R}^p$, $u^TAA^Tu$ is a squared length and cannot be negative. Whether it is positive definite depends on the rank of $A$.

Theorem: For $X=\mu+AZ$, the covariance $AA^T$ is positive definite if and only if $A$ has full row rank $p$. If $A$ has rank less than $p$, the law of $X$ is degenerate and is supported on the proper affine subspace $\mu+\operatorname{Range}(A)$.

For any $u\in\mathbb{R}^p$,

u^TAA^Tu=(A^Tu)^T(A^Tu)=\lVert A^Tu\rVert^2.

Consequently, $AA^T$ is positive definite exactly when $A^Tu=0$ implies $u=0$. This is equivalent to the rows of $A$ being linearly independent, or $\operatorname{rank}(A)=p$.

For every realization of $Z$, $AZ$ belongs to $\operatorname{Range}(A)$, so $X$ belongs to $\mu+\operatorname{Range}(A)$. If $A$ has rank below $p$, this affine subspace has dimension less than $p$. Such a subspace has $p$-dimensional Lebesgue measure zero, so a probability distribution concentrated there cannot have a density with respect to $p$-dimensional Lebesgue measure. If $A$ has full row rank, the law is nonsingular and does have a density. $\square$

For a nonsingular Gaussian vector, that density is

f_X(x)=\frac{1}{(2\pi)^{p/2}\det(AA^T)^{1/2}} \exp\!\left[-\frac{1}{2}(x-\mu)^T(AA^T)^{-1}(x-\mu)\right], \qquad x\in\mathbb{R}^p.

One way to see the density formula is to choose orthogonal coordinates in $\mathbb{R}^q$ so that $AQ=[C\ \ 0]$, with $C$ an invertible $p\times p$ matrix. Orthogonal transformations preserve the standard normal distribution, so $X$ has the same distribution as $\mu+CW$, where $W$ is a $p$-dimensional standard normal vector. The change-of-variables formula gives the density with covariance $CC^T=AA^T$, and $|\det C|=\det(AA^T)^{1/2}$.

Worked Example: A Nonsingular Two-Dimensional Gaussian

Let $Z=(Z_1,Z_2)^T$ have independent standard normal coordinates, and define

X= \begin{pmatrix}2\\-1\end{pmatrix} + \begin{pmatrix}1&1\\1&-1\end{pmatrix}Z.

Here $\mu=(2,-1)^T$, and multiplying the matrix by its transpose gives

AA^T= \begin{pmatrix}1&1\\1&-1\end{pmatrix} \begin{pmatrix}1&1\\1&-1\end{pmatrix} = \begin{pmatrix}2&0\\0&2\end{pmatrix}.

The covariance is positive definite: for nonzero $u=(u_1,u_2)^T$, $u^TAA^Tu=2u_1^2+2u_2^2>0$. The determinant of $A$ is $-2$, so $A$ has full row rank and this law is nonsingular. Its characteristic function is

\varphi_X(t)=\exp\!\left(i(2t_1-t_2)-(t_1^2+t_2^2)\right).

For example, the density’s normalizing constant is $1/(2\pi\sqrt{4})=1/(4\pi)$. The exponent is $-\tfrac12(x-\mu)^T(AA^T)^{-1}(x-\mu)=-\tfrac14\{(x_1-2)^2+(x_2+1)^2\}$, which verifies the full density directly.

Worked Example: A Degenerate Gaussian on a Line

Let $Z\sim N(0,1)$ and set $X=(1,-2)^T+(2,1)^TZ$. The matrix $A=(2,1)^T$ has rank one, less than the output dimension $p=2$. The mean and covariance are

E[X]=\begin{pmatrix}1\\-2\end{pmatrix}, \qquad \operatorname{Cov}(X)=AA^T= \begin{pmatrix}4&2\\2&1\end{pmatrix}.

The covariance is not positive definite: its determinant is $4-4=0$. More directly, the coordinates satisfy $X_1=1+2Z$ and $X_2=-2+Z$, hence $X_1-2X_2=5$ with probability one. The distribution is concentrated on that line, so it has no density over the whole plane. Its characteristic function, for $t=(t_1,t_2)^T$, is

\varphi_X(t)= \exp\!\left(i(t_1-2t_2)-\frac12(2t_1+t_2)^2\right).

This is a valid Gaussian vector despite the singular covariance matrix. Degeneracy means the distribution occupies a lower-dimensional set, not that the construction fails to define a probability law.

Common Mistake: Requiring a Density

A frequent mistake is to define a multivariate Gaussian only by the familiar density formula involving $(AA^T)^{-1}$, and then conclude that a singular covariance matrix cannot describe a Gaussian distribution. That formula requires a positive definite covariance, so it applies only to the nonsingular case. The affine-image definition also includes singular cases: when a linear combination of the output coordinates has no random variation, the law is concentrated on a lower-dimensional affine subspace.

The converse pitfall is to assume that the number of standard normal coordinates must equal the output dimension. A $p$-dimensional Gaussian vector may be constructed from $q$ independent standard normal coordinates for any suitable $q$. It is the rank of $A$, not merely the number of its columns, that determines whether the covariance is positive definite.

Key takeaway: In the representation $X=\mu+AZ$, the mean is $\mu$, the covariance is $AA^T$, and the characteristic function is $\exp(i t^T\mu-\tfrac12 t^TAA^Tt)$. Full row rank gives a nonsingular law; lower rank gives a degenerate law supported on a proper affine subspace.

Check Your Understanding

Use the affine-image representation to answer each question.

If $X=AZ+\mu$ and $Z$ has covariance $I_q$, what are $E[X]$ and $\operatorname{Cov}(X)$?
For a $p$-dimensional Gaussian vector, what rank condition on $A$ makes $AA^T$ positive definite?
Why can a Gaussian vector have a singular covariance matrix and still be a valid probability distribution?
Suppose $A$ is a $3\times 5$ matrix of rank two. What can you conclude about the support and density of $\mu+AZ$ in $\mathbb{R}^3$?
For $X=\mu+AZ$, write its characteristic function at $t\in\mathbb{R}^p$.

Gaussian Vectors as Affine Images of Standard Normals

What You'll Learn