Quadratic Forms Under General Gaussian Covariance

Why Covariance Changes the Quadratic Form

A quadratic form $X'AX$ is easy to interpret when $X$ has independent standard-normal coordinates: diagonalizing the symmetric matrix $A$ reveals the weights applied to independent squared normals. With a general covariance matrix, however, the coordinates of $X$ need not be independent or have unit variance. The key is to first express $X$ in standard-normal coordinates. The matrix that then determines the weights is not $A$ itself, but the symmetric matrix $\Sigma^{1/2}A\Sigma^{1/2}$.

This reduction makes the effect of covariance explicit. In particular, even when $A$ is fixed, changing $\Sigma$ can change the eigenvalues and therefore the distribution of the quadratic form. We assume throughout that the covariance matrix is positive definite; singular covariance requires a separate treatment.

Standard-Normal Coordinates

Definition: Let $X$ be a $p$-dimensional Gaussian random vector with mean $\mu$ and positive-definite covariance matrix $\Sigma$, written $X\sim N_p(\mu,\Sigma)$. Let $\Sigma^{1/2}$ denote the unique symmetric positive-definite square root of $\Sigma$. Then $Z=\Sigma^{-1/2}(X-\mu)\sim N_p(0,I_p)$, and $X=\mu+\Sigma^{1/2}Z$.

A quadratic form depends only on the symmetric part of its matrix: for any real matrix $A$, $x'Ax=x'\{(A+A')/2\}x$, since the skew-symmetric part contributes zero. We therefore take $A$ to be symmetric without loss of generality. This assumption lets us use the spectral theorem to diagonalize the matrix arising after standardization.

Theorem: If $X\sim N_p(\mu,\Sigma)$, $\Sigma$ is positive definite, and $A$ is symmetric, define $B=\Sigma^{1/2}A\Sigma^{1/2}$ and $c=\Sigma^{1/2}A\mu$. For $Z=\Sigma^{-1/2}(X-\mu)$, $$ X'AX=Z'BZ+2c'Z+\mu'A\mu. $$ If $Q$ is orthogonal and $Q'BQ=\operatorname{diag}(\lambda_1,\ldots,\lambda_p)$, then with $W=Q'Z$ and $d=Q'c$, $$ X'AX=\sum_{i=1}^p\left(\lambda_i W_i^2+2d_iW_i\right)+\mu'A\mu, $$ where the $W_i$ are independent standard normal random variables. In the centered case $\mu=0$, this reduces to $\sum_{i=1}^p\lambda_iW_i^2$.

Proof. Substitute $X=\mu+\Sigma^{1/2}Z$ and use the symmetry of $A$ and $\Sigma^{1/2}$: $$ X'AX=(\mu+\Sigma^{1/2}Z)'A(\mu+\Sigma^{1/2}Z) =\mu'A\mu+2Z'\Sigma^{1/2}A\mu+Z'\Sigma^{1/2}A\Sigma^{1/2}Z. $$ The definitions of $B$ and $c$ give the first representation. Since $B$ is symmetric, the spectral theorem gives an orthogonal $Q$ with $Q'BQ=\operatorname{diag}(\lambda_1,\ldots,\lambda_p)$. An orthogonal transformation preserves the standard multivariate normal distribution, so $W=Q'Z\sim N_p(0,I_p)$, whose coordinates are independent standard normals. Substituting $Z=QW$ in the first representation gives the stated sum. When $\mu=0$, $c=0$ and the constant term is zero. $\square$

The matrix $B$ is symmetric even if $A$ and $\Sigma$ do not commute. Its real eigenvalues are the weights in the centered representation. A positive eigenvalue contributes a positive multiple of a one-degree-of-freedom chi-square variable; a negative eigenvalue contributes a negative multiple. A zero eigenvalue contributes nothing to the centered form. Thus the eigenvalues of $B$, not generally those of $A$, describe the effect of both the quadratic form and the covariance.

Worked Examples

Worked Example: A Diagonal Covariance and an Indefinite Form

Let $X\sim N_2(0,\Sigma)$, where $\Sigma=\operatorname{diag}(4,1)$, and take $A=\begin{pmatrix}1&1\\1&0\end{pmatrix}$. Find the weighted chi-square representation of $X'AX$.

The symmetric square root is $\Sigma^{1/2}=\operatorname{diag}(2,1)$. Consequently, $$ B=\Sigma^{1/2}A\Sigma^{1/2} =\begin{pmatrix}2&0\\0&1\end{pmatrix} \begin{pmatrix}1&1\\1&0\end{pmatrix} \begin{pmatrix}2&0\\0&1\end{pmatrix} =\begin{pmatrix}4&2\\2&0\end{pmatrix}. $$

The characteristic polynomial is $\det(B-\lambda I)=\lambda^2-4\lambda-4$. Its roots are $\lambda_1=2+2\sqrt{2}$ and $\lambda_2=2-2\sqrt{2}$. The second is negative because $\sqrt{2}>1$. The theorem therefore gives, for independent $W_1,W_2\sim N(0,1)$, $$ X'AX=(2+2\sqrt{2})W_1^2+(2-2\sqrt{2})W_2^2. $$

The negative weight is possible because $A$ is indefinite. The covariance has changed the weights: using the eigenvalues of $A$ instead would give the wrong representation.

Worked Example: Correlation Changes the Nonzero Weight

Let $X\sim N_2(0,\Sigma)$, where $\Sigma=\frac12\begin{pmatrix}5&3\\3&5\end{pmatrix}$, and let $A=\begin{pmatrix}1&0\\0&0\end{pmatrix}$. Find the distributional representation of $X'AX$.

The matrix $S=\frac12\begin{pmatrix}3&1\\1&3\end{pmatrix}$ is the symmetric positive-definite square root of $\Sigma$: direct multiplication gives $S^2=\frac12\begin{pmatrix}5&3\\3&5\end{pmatrix}=\Sigma$. Thus $$ B=SAS =\begin{pmatrix}9/4&3/4\\3/4&1/4\end{pmatrix}. $$ Its trace is $9/4+1/4=5/2$, and its determinant is $(9/4)(1/4)-(3/4)^2=0$. Its eigenvalues are therefore $5/2$ and $0$. It follows that $$ X'AX\ \stackrel{d}{=}\ \frac52 W_1^2 \ \sim\ \frac52\chi^2_1. $$

This is also directly checkable: $X'AX=X_1^2$, and $X_1\sim N(0,5/2)$, so $X_1=\sqrt{5/2}\,W_1$ in distribution. The matrix calculation reaches the same result while illustrating how covariance enters the eigenvalue.

Worked Example: A Nonzero Mean Adds Linear Terms

Let $X\sim N_2(\mu,I_2)$, with $\mu=(1,3)'$, and let $A=\operatorname{diag}(2,0)$. Express $X'AX$ in standard-normal coordinates and identify why the eigenvalues alone do not give its distribution.

Here $\Sigma^{1/2}=I_2$, so $B=A$, $c=A\mu=(2,0)'$, and $\mu'A\mu=2$. Write $Z=(Z_1,Z_2)'\sim N_2(0,I_2)$. Then $$ X'AX=2Z_1^2+4Z_1+2=2(Z_1+1)^2. $$

The eigenvalues of $B$ are $2$ and $0$, but the linear term $4Z_1$ remains because the mean is nonzero. In particular, this is not the centered law $2\chi^2_1$; it is $2$ times a noncentral chi-square variable with one degree of freedom and noncentrality parameter $1$. The mean coordinate in the zero-eigenvalue direction has no effect here because $A$ assigns that direction zero weight.

Consequences for Moments and Nonzero Means

In the centered case, the weighted chi-square representation gives simple moment formulas. These formulas are also a useful check on a computed matrix $B$.

Result: If $X\sim N_p(0,\Sigma)$, $\Sigma$ is positive definite, and $A$ is symmetric, then $$ E[X'AX]=\operatorname{tr}(B),\qquad \operatorname{Var}(X'AX)=2\operatorname{tr}(B^2), \quad B=\Sigma^{1/2}A\Sigma^{1/2}. $$ For general mean $\mu$, with $c=\Sigma^{1/2}A\mu$, the corresponding formulas are $$ E[X'AX]=\operatorname{tr}(B)+\mu'A\mu,\qquad \operatorname{Var}(X'AX)=2\operatorname{tr}(B^2)+4c'c. $$

Proof. Use the diagonal representation from the theorem. The $W_i$ are independent standard normals, so $E[W_i]=0$, $E[W_i^2]=1$, $E[W_i^3]=0$, and $E[W_i^4]=3$. Therefore $E[W_i^2]=1$, $\operatorname{Var}(W_i^2)=3-1=2$, and $\operatorname{Cov}(W_i^2,W_i)=0$. Taking expectations in the representation gives $E[X'AX]=\sum_i\lambda_i+\mu'A\mu=\operatorname{tr}(B)+\mu'A\mu$. For each coordinate, $$ \operatorname{Var}(\lambda_iW_i^2+2d_iW_i) =2\lambda_i^2+4d_i^2, $$ using the zero covariance just calculated. Independence lets us add these variances, yielding $2\sum_i\lambda_i^2+4\sum_i d_i^2=2\operatorname{tr}(B^2)+4c'c$. Setting $\mu=0$ gives the centered formulas. $\square$

For a nonzero mean, the diagonalized expression also shows precisely what the eigenvalues do and do not determine. The $\lambda_i$ give the quadratic weights, while $d_i$ give the linear terms in the corresponding normal coordinates. In fact, $c=B\Sigma^{-1/2}\mu$, so if $\lambda_i=0$, then $d_i=0$ in that eigendirection. For nonzero eigenvalues one may complete the square: $$ \lambda_iW_i^2+2d_iW_i =\lambda_i\left(W_i+\frac{d_i}{\lambda_i}\right)^2-\frac{d_i^2}{\lambda_i}. $$ This makes clear why a noncentral form generally needs information about the mean as well as the eigenvalues.

Common Mistake: Diagonalizing the Wrong Matrix

A common error is to diagonalize $A$ and use its eigenvalues as the weights, even when $\Sigma\ne I_p$. That reasoning silently assumes the coordinates of $X$ are already independent standard normals. In general, the correct sequence is to standardize $X$, form $B=\Sigma^{1/2}A\Sigma^{1/2}$, and diagonalize $B$. The covariance may alter the eigenvalues and can change the resulting weighted chi-square law. For nonzero $\mu$, even the eigenvalues of $B$ are not by themselves enough: the linear terms $2d_iW_i$ must also be accounted for.

Key takeaway: For a centered Gaussian vector with positive-definite covariance, the distribution of $X'AX$ is governed by the eigenvalues of the symmetric matrix $\Sigma^{1/2}A\Sigma^{1/2}$. For a noncentered vector, retain the additional linear and constant terms produced by the mean.

Check Your Understanding

Try these questions using standard-normal coordinates.

For $X\sim N_p(0,\Sigma)$ with positive-definite $\Sigma$, which matrix should be diagonalized to obtain the weights in $X'AX$?
If the eigenvalues of $\Sigma^{1/2}A\Sigma^{1/2}$ are $3$, $0$, and $-1$, write the centered quadratic form’s weighted chi-square representation.
Why can the eigenvalues of $A$ alone fail to determine the distribution when $\Sigma\ne I_p$?
For a nonzero mean, what terms in the standard-normal representation are absent in the centered case?
State the mean and variance of a centered Gaussian quadratic form in terms of $B=\Sigma^{1/2}A\Sigma^{1/2}$.