Whitening Transformations for Gaussian Vectors

Why Whiten a Gaussian Vector?

A nonsingular Gaussian vector may have coordinates with different variances and nonzero covariances. A whitening transformation removes both features at once: after centering and transforming, the covariance matrix is the identity. For a Gaussian vector, this also gives a standard normal vector, whose coordinates are independent standard normal random variables.

Whitening is useful for expressing a correlated Gaussian model in standard coordinates, comparing directions on a common scale, and understanding the geometry encoded by a covariance matrix. The transformation is generally not unique: once the covariance has been made the identity, any further orthogonal rotation preserves it.

Definition and Construction

Let $X$ be a $p$-dimensional Gaussian vector with mean vector $\mu=E[X]$ and nonsingular covariance matrix $\Sigma=\operatorname{Var}(X)$. Nonsingularity of a covariance matrix means that $\Sigma$ is positive definite. A whitening transformation is a linear map applied to the centered vector $X-\mu$.

Definition: A $p\times p$ matrix $W$ is a whitening matrix for $\Sigma$ if $$ W\Sigma W^T=I_p. $$ The corresponding whitened vector is $Z=W(X-\mu)$. It is a standard normal vector when $Z\sim N_p(0,I_p)$.

The eigenstructure of a positive definite covariance matrix supplies a direct construction. By the spectral theorem, there is an orthogonal matrix $U$ and a diagonal matrix $\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p)$ such that $\Sigma=U\Lambda U^T$, where every eigenvalue $\lambda_i$ is positive. Define $\Sigma^{-1/2}=U\operatorname{diag}(\lambda_1^{-1/2},\ldots,\lambda_p^{-1/2})U^T$. This is the symmetric inverse square root of $\Sigma$.

Theorem: If $X\sim N_p(\mu,\Sigma)$ with $\Sigma$ positive definite, then $$ Z=\Sigma^{-1/2}(X-\mu)\sim N_p(0,I_p). $$

Proof. The affine transformation rule for Gaussian vectors gives $Z$ a Gaussian distribution. Its mean is $E[Z]=\Sigma^{-1/2}(E[X]-\mu)=0$. Its covariance is

\operatorname{Var}(Z) =\Sigma^{-1/2}\Sigma(\Sigma^{-1/2})^T =\Sigma^{-1/2}\Sigma\Sigma^{-1/2} =I_p.

The last equality follows directly from the eigendecomposition: the product has eigenvalues $\lambda_i^{-1/2}\lambda_i\lambda_i^{-1/2}=1$ in every eigenvector direction. Thus $Z$ is Gaussian with mean zero and identity covariance, which is precisely $N_p(0,I_p)$. $\square$

Because a Gaussian vector with identity covariance has independent standard normal coordinates, whitening gives more than unit variances and zero pairwise covariances. The Gaussian assumption is essential to that conclusion; an arbitrary non-Gaussian vector can have identity covariance without having independent coordinates.

Worked Constructions

Worked Example: Whitening a Two-Dimensional Gaussian Vector

Suppose $X\sim N_2(\mu,\Sigma)$, where $\mu=(1,-1)^T$ and $\Sigma=\begin{pmatrix}4&2\\2&3\end{pmatrix}$. Construct the symmetric whitening matrix and verify the covariance.

The characteristic polynomial of $\Sigma$ is $(4-\lambda)(3-\lambda)-4=\lambda^2-7\lambda+8$? Checking the determinant directly gives $(4-\lambda)(3-\lambda)-4=\lambda^2-7\lambda+8$, whose roots are $(7\pm\sqrt{17})/2$. Thus the eigenvalues are not $5$ and $2$; an exact eigendecomposition is needed.

For a transparent construction, use the eigenvalues $\lambda_+=(7+\sqrt{17})/2$ and $\lambda_-=(7-\sqrt{17})/2$. Corresponding unit eigenvectors can be obtained by normalizing $(2,\lambda_+-4)^T$ and $(2,\lambda_--4)^T$, respectively. These eigenvectors are orthogonal because $\Sigma$ is symmetric. Let $U$ have these normalized vectors as columns. Then $\Sigma=U\operatorname{diag}(\lambda_+,\lambda_-)U^T$, and $$ W=U\operatorname{diag}(\lambda_+^{-1/2},\lambda_-^{-1/2})U^T. $$

To verify the result without expanding the radicals, use $U^TU=I_2$: $W\Sigma W^T =U\operatorname{diag}(\lambda_+^{-1/2},\lambda_-^{-1/2}) \operatorname{diag}(\lambda_+,\lambda_-) \operatorname{diag}(\lambda_+^{-1/2},\lambda_-^{-1/2})U^T =UI_2U^T=I_2$. Therefore $W(X-\mu)\sim N_2(0,I_2)$.

Worked Example: Whitening a Diagonal Covariance Matrix

Let $X\sim N_3(\mu,\Sigma)$, with $\Sigma=\operatorname{diag}(9,4,1)$. Find a whitening matrix and check each transformed variance.

The covariance is already diagonal, so its eigenvectors are the coordinate directions. Scale the centered coordinates by the reciprocal standard deviations: $$ W=\operatorname{diag}(1/3,1/2,1). $$ Then $$ W\Sigma W^T =\operatorname{diag}(1/3,1/2,1) \operatorname{diag}(9,4,1) \operatorname{diag}(1/3,1/2,1) =\operatorname{diag}(1,1,1)=I_3. $$

Consequently $W(X-\mu)\sim N_3(0,I_3)$. In coordinates, the first centered component is divided by $3$, the second by $2$, and the third is unchanged; these are exactly the standard deviations of the original coordinates.

Worked Example: Whitening a Correlated Standardized Pair

Suppose $X\sim N_2(0,\Sigma)$, where $\Sigma=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}$. Each coordinate already has variance one, but the coordinates are correlated. Find a whitening transformation.

The unit eigenvectors are $u_+=(1,1)^T/\sqrt{2}$ and $u_-=(1,-1)^T/\sqrt{2}$, with eigenvalues $1.6=8/5$ and $0.4=2/5$, respectively. Thus $U=(u_+,u_-)$, and a symmetric whitening matrix is $$ W=U\operatorname{diag}\left(\sqrt{5/8},\sqrt{5/2}\right)U^T. $$

The covariance in the eigenvector coordinates is $\operatorname{diag}(8/5,2/5)$. Applying $W$ scales those two coordinates by $\sqrt{5/8}$ and $\sqrt{5/2}$, so the resulting variances are $(5/8)(8/5)=1$ and $(5/2)(2/5)=1$, with covariance zero. Equivalently, $W\Sigma W^T=I_2$, and hence $WX\sim N_2(0,I_2)$.

Simply leaving both coordinates unchanged because their variances are already one would not whiten the vector: its covariance would still be $\Sigma$, with off-diagonal entries $0.6$.

Why Whitening Is Not Unique

The symmetric inverse square root is a natural choice, but it is not the only whitening matrix. If $Q$ is orthogonal, so $QQ^T=I_p$, then rotating a whitened vector by $Q$ preserves its identity covariance. In fact, these rotations describe all possible whitening matrices.

Theorem: For positive definite $\Sigma$, every matrix $W$ satisfying $W\Sigma W^T=I_p$ has the form $$ W=Q\Sigma^{-1/2} $$ for an orthogonal matrix $Q$. Conversely, every matrix of this form is a whitening matrix.

Proof. First, if $Q$ is orthogonal, then $(Q\Sigma^{-1/2})\Sigma(Q\Sigma^{-1/2})^T =Q(\Sigma^{-1/2}\Sigma\Sigma^{-1/2})Q^T =QI_pQ^T=I_p$.

For the converse, suppose $W\Sigma W^T=I_p$. Since the product has rank $p$, $W$ is invertible. Set $Q=W\Sigma^{1/2}$, where $\Sigma^{1/2}$ is the symmetric positive definite square root. Then $QQ^T=W\Sigma W^T=I_p$. A square matrix satisfying this identity is orthogonal, so $W=Q\Sigma^{-1/2}$. This proves both directions. $\square$

Worked Example: Two Different Whitening Matrices

For the diagonal covariance $\Sigma=\operatorname{diag}(9,4)$, one whitening matrix is $W_1=\operatorname{diag}(1/3,1/2)$. Let $Q=\begin{pmatrix}0&-1\\1&0\end{pmatrix}$, a 90-degree rotation. A second whitening matrix is $$ W_2=QW_1=\begin{pmatrix}0&-1/2\\1/3&0\end{pmatrix}. $$

Indeed, $W_2\Sigma W_2^T =Q(W_1\Sigma W_1^T)Q^T =QI_2Q^T=I_2$. The two maps give different standard normal coordinates, but both have the required standard normal distribution. The rotation changes the orientation of the whitened coordinate axes without changing their lengths or covariance.

Affine Form, Interpretation, and Common Pitfalls

A standard normal vector is centered. Thus, if the transformation is written as $AX+b$ rather than $W(X-\mu)$, it must also remove the mean. Taking $A=W$, the appropriate offset is $b=-W\mu$. The covariance condition $W\Sigma W^T=I_p$ alone does not make $WX$ have mean zero unless $W\mu=0$.

Geometrically, the eigenvectors of $\Sigma$ are the principal directions of variation. A displacement along an eigenvector with eigenvalue $\lambda_i$ is divided by $\sqrt{\lambda_i}$, its standard deviation in that direction. This rescales the covariance ellipsoid to a sphere; an orthogonal rotation can then choose any orientation for the whitened axes.

A frequent mistake is to standardize each original coordinate separately and assume that this removes dependence. That only sets the diagonal entries of the covariance to one. If off-diagonal covariances remain, the transformed covariance is not the identity. Whitening accounts for the full covariance matrix, including its eigenvectors and eigenvalues. Another error is to use $W^T\Sigma W$ as the condition without checking the transformation: for $Z=WX$, the covariance is $W\Sigma W^T$.

Key takeaway: For a nonsingular Gaussian vector, center by its mean and multiply by any $W$ satisfying $W\Sigma W^T=I_p$. The symmetric choice is $\Sigma^{-1/2}$, and every other choice is an orthogonal rotation of it.

Check Your Understanding

Use the definitions and results above to answer the following questions.

Why must every eigenvalue of the covariance matrix be positive for the symmetric inverse square root construction?
If $X\sim N_p(\mu,\Sigma)$, what is the mean of $\Sigma^{-1/2}X$, and what offset makes the transformed vector centered?
Explain why a Gaussian vector with identity covariance has independent coordinates, and why the same conclusion does not follow for an arbitrary vector.
If $W\Sigma W^T=I_p$ and $Q$ is orthogonal, verify that $QW$ is also a whitening matrix.
Why does dividing each coordinate by its own standard deviation fail to whiten a vector whose standardized coordinates remain correlated?