Deriving the Nonsingular Gaussian Density

Why the Jacobian Matters

A standard normal vector has a simple density: its coordinates contribute a quadratic sum, and the density is largest at the origin. A nonsingular linear transformation can stretch, compress, rotate, and shear that distribution. To obtain the density after such a transformation, it is not enough to substitute the inverse transformation into the standard normal density. We must also account for how the transformation changes volume. That adjustment is the absolute value of a Jacobian determinant.

This derivation gives a direct route from independent standard normal coordinates to the familiar density of a nonsingular Gaussian vector. It also explains why the determinant of the covariance matrix appears in the normalizing constant, and why the inverse covariance matrix appears in the exponent.

Setup: Transforming a Standard Normal Vector

Let \(Z=(Z_1,\ldots,Z_p)^T\) have the standard \(p\)-dimensional normal distribution \(N_p(0,I_p)\), where \(I_p\) is the \(p\times p\) identity matrix. Its coordinates are independent standard normal random variables, so its density with respect to \(p\)-dimensional Lebesgue measure is

f_Z(z)=(2\pi)^{-p/2}\exp\left(-\frac{1}{2}z^Tz\right),\qquad z\in\mathbb{R}^p.

Choose a fixed vector \(\mu\in\mathbb{R}^p\) and a nonsingular \(p\times p\) matrix \(A\), and define \(X=\mu+AZ\). Nonsingularity ensures that this transformation is one-to-one and onto: every \(x\in\mathbb{R}^p\) comes from exactly one \(z\in\mathbb{R}^p\). Its inverse is \(z=A^{-1}(x-\mu)\).

Definition: A Gaussian vector \(X\) is nonsingular when its covariance matrix \(\Sigma\) is positive definite. In the transformation considered here, \(\Sigma=AA^T\) for a nonsingular matrix \(A\). The affine map \(z\mapsto \mu+Az\) is then invertible, so \(X\) has a density on all of \(\mathbb{R}^p\).

The change-of-variables theorem for densities states that if \(g:\mathbb{R}^p\to\mathbb{R}^p\) is a continuously differentiable bijection with a continuously differentiable inverse, then the density of \(g(Z)\) at \(x\) is \(f_Z(g^{-1}(x))|\det Dg^{-1}(x)|\). Here \(g(z)=\mu+Az\) is such a map, and its inverse derivative is the constant matrix \(A^{-1}\). Therefore the Jacobian factor is \(|\det(A^{-1})|=1/|\det A|\). The absolute value is essential: a transformation that reverses orientation still changes volume by a positive factor.

Deriving the Density

Apply the change-of-variables theorem with \(g^{-1}(x)=A^{-1}(x-\mu)\). Substituting this inverse into the standard normal density gives

f_X(x) =f_Z\bigl(A^{-1}(x-\mu)\bigr)\frac{1}{|\det A|} =(2\pi)^{-p/2}\frac{1}{|\det A|} \exp\left[-\frac{1}{2}(x-\mu)^T A^{-T}A^{-1}(x-\mu)\right].

To write this in terms of the covariance matrix, use the covariance transformation for \(X=\mu+AZ\): since \(E[Z]=0\) and \(\operatorname{Cov}(Z)=I_p\), \(E[X]=\mu\) and \(\operatorname{Cov}(X)=AA^T\). Set \(\Sigma=AA^T\). Because \(A\) is nonsingular, \(\Sigma\) is positive definite and

\Sigma^{-1}=A^{-T}A^{-1}, \qquad \det(\Sigma)=\det(A)\det(A^T)=(\det A)^2, \qquad \sqrt{\det(\Sigma)}=|\det A|.

Substituting these identities yields the density in mean-and-covariance form. A positive-definite covariance matrix has a nonsingular factor \(A\) with \(\Sigma=AA^T\); thus the formula applies to any nonsingular Gaussian vector.

Theorem: If \(X\) is a \(p\)-dimensional Gaussian vector with mean \(\mu\) and positive-definite covariance matrix \(\Sigma\), then its density is

f_X(x)=\frac{1}{(2\pi)^{p/2}\sqrt{\det(\Sigma)}} \exp\left[-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right], \qquad x\in\mathbb{R}^p.

Proof. Represent \(X\) as \(\mu+AZ\), where \(Z\sim N_p(0,I_p)\), \(A\) is nonsingular, and \(\Sigma=AA^T\). The affine map and inverse satisfy the hypotheses of the change-of-variables theorem. Its inverse Jacobian has absolute determinant \(1/|\det A|\), so substitution into \(f_Z\) gives the density displayed above. The identities \(\Sigma^{-1}=A^{-T}A^{-1}\) and \(\sqrt{\det(\Sigma)}=|\det A|\) put that expression in terms of \(\Sigma\), proving the result. In particular, the change of variables also confirms normalization: integrating the transformed density over \(x\) and substituting \(x=\mu+Az\) cancels the factor \(1/|\det A|\) against the volume factor \(|\det A|\), leaving \(\int_{\mathbb{R}^p} f_Z(z)\,dz=1\). \(\square\)

Worked Derivations

Worked Example: A One-Dimensional Transformation with Negative Scale

Let \(Z\sim N(0,1)\) and \(X=5-3Z\). Find the density of \(X\), paying attention to the sign of the scale factor.

The inverse transformation is \(z=(5-x)/3\), and its derivative with respect to \(x\) is \(-1/3\). The absolute Jacobian is therefore \(1/3\), not \(-1/3\). Substitution gives

f_X(x)=\frac{1}{\sqrt{2\pi}}\exp\left[-\frac{1}{2}\left(\frac{5-x}{3}\right)^2\right]\frac{1}{3} =\frac{1}{3\sqrt{2\pi}}\exp\left[-\frac{(x-5)^2}{18}\right].

This is the \(N(5,9)\) density. For a numerical check, at \(x=2\) the inverse value is \(z=1\), so the transformed density is \(f_Z(1)/3=e^{-1/2}/(3\sqrt{2\pi})\), exactly as the formula gives.

Worked Example: A Bivariate Gaussian Density from a Matrix

Let \(Z\sim N_2(0,I_2)\), and set

X=\begin{pmatrix}1\\-1\end{pmatrix} +\begin{pmatrix}2&0\\1&1\end{pmatrix}Z.

Write \(A=\begin{pmatrix}2&0\\1&1\end{pmatrix}\). Its determinant is \(2\), so the Jacobian factor for the inverse transformation is \(1/2\). Its covariance matrix and inverse are

\Sigma=AA^T=\begin{pmatrix}4&2\\2&2\end{pmatrix}, \qquad \det(\Sigma)=4, \qquad \Sigma^{-1}=\frac{1}{4}\begin{pmatrix}2&-2\\-2&4\end{pmatrix} =\begin{pmatrix}1/2&-1/2\\-1/2&1\end{pmatrix}.

For \(x=(x_1,x_2)^T\), let \(u=x_1-1\) and \(v=x_2+1\). Then \((x-\mu)^T\Sigma^{-1}(x-\mu)=u^2/2-uv+v^2\), and the square root of the covariance determinant is \(2\). Hence

f_X(x_1,x_2)=\frac{1}{4\pi} \exp\left[-\frac{1}{2}\left(\frac{u^2}{2}-uv+v^2\right)\right].

For example, at \(x=(2,0)^T\), \(u=v=1\), so the quadratic form is \(1/2-1+1=1/2\). The density there is \(e^{-1/4}/(4\pi)\). Directly, the inverse transform gives \(z=(1/2,1/2)^T\), whose squared length is \(1/2\); multiplying \(f_Z(1/2,1/2)=e^{-1/4}/(2\pi)\) by the inverse Jacobian \(1/2\) gives \(e^{-1/4}/(4\pi)\). This agrees with the covariance-form calculation.

Worked Example: A Three-Dimensional Shear and Scale

Let \(Z\sim N_3(0,I_3)\), \(\mu=(0,1,-1)^T\), and

A=\begin{pmatrix}1&0&0\\2&3&0\\0&1&2\end{pmatrix}, \qquad X=\mu+AZ.

The matrix is lower triangular, so \(\det(A)=1\cdot3\cdot2=6\). The covariance is \(\Sigma=AA^T=\begin{pmatrix}1&2&0\\2&13&3\\0&3&5\end{pmatrix}\), and \(\det(\Sigma)=36\). Consider \(x=(1,3,1)^T\). Then \(x-\mu=(1,2,2)^T\), and solving \(Az=x-\mu\) gives, row by row, \(z_1=1\), \(2z_1+3z_2=2\), hence \(z_2=0\), and \(z_2+2z_3=2\), hence \(z_3=1\). Thus \(z^Tz=2\). The transformed standard density is

f_X(1,3,1)=(2\pi)^{-3/2}\exp(-1)\frac{1}{6} =\frac{e^{-1}}{6(2\pi)^{3/2}}.

The same value follows from the covariance-form density: its determinant factor is \(1/\sqrt{36}=1/6\), and its quadratic form at this point is \(2\), since \(\Sigma^{-1}=A^{-T}A^{-1}\). This example shows how the determinant tracks total volume change while the transformed squared length determines the exponential term.

Why the Choice of Factor Does Not Matter

A given positive-definite covariance matrix can be written as \(AA^T\) in more than one way. The density must not depend on which such matrix is used. The following argument verifies this directly.

Result: If \(A_1\) and \(A_2\) are nonsingular and \(A_1A_1^T=A_2A_2^T=\Sigma\), then \(\mu+A_1Z\) and \(\mu+A_2Z\), for \(Z\sim N_p(0,I_p)\), have the same density.

Proof. Define \(Q=A_1^{-1}A_2\). Then

QQ^T=A_1^{-1}A_2A_2^TA_1^{-T} =A_1^{-1}A_1A_1^TA_1^{-T}=I_p.

Thus \(Q\) is orthogonal. An orthogonal transformation preserves squared length, since \((Q^Tz)^T(Q^Tz)=z^TQQ^Tz=z^Tz\), and has absolute determinant \(1\), since \((\det Q)^2=1\). Applying the change-of-variables formula to \(QZ\) therefore gives \(f_{QZ}(z)=f_Z(z)\). Also \(A_2=A_1Q\), so \(\mu+A_2Z=\mu+A_1(QZ)\). Since \(QZ\) has the same standard normal density as \(Z\), these two transformed vectors have the same distribution and density. \(\square\)

Common Pitfall and Practical Check

A common error is to substitute \(A^{-1}(x-\mu)\) into \(f_Z\) but omit the factor \(1/|\det A|\). That produces the right exponential shape but generally not a density: its integral is \(|\det A|\), rather than \(1\). Another error is to use \(\det A\) without an absolute value. If \(\det A<0\), a density would then be negative, which is impossible. The Jacobian in the density transformation is the absolute determinant of the derivative of the inverse map.

As a check on any derived formula, verify both pieces. The exponential must use the transformed squared distance, \((x-\mu)^T\Sigma^{-1}(x-\mu)\), and the leading factor must be \(1/((2\pi)^{p/2}\sqrt{\det(\Sigma)})\). The determinant term accounts for spread across the full \(p\)-dimensional space; replacing it with a product of diagonal entries is not valid in general.

Check Your Understanding

Try these questions by tracking the inverse map and its Jacobian.

For \(X=\mu+AZ\), what is the inverse transformation, and why is the density multiplier \(1/|\det A|\)?
If \(A\) is a nonsingular \(p\times p\) matrix and \(\Sigma=AA^T\), derive \(\det(\Sigma)=(\det A)^2\).
In the bivariate worked example, why would using \(1/\det(A)\) without absolute value be unsafe for a general nonsingular matrix?
For the three-dimensional example, solve \(Az=(1,2,2)^T\) and compute \(z^Tz\) without using the covariance-form expression.
Why does replacing \(A\) by another nonsingular factor of the same covariance matrix leave the resulting Gaussian density unchanged?