Means and Variances of Gaussian Quadratic Forms

Why These Moments Matter

A quadratic form $X'AX$ compresses a possibly high-dimensional Gaussian vector into one scalar. Its expectation describes its average size, while its variance measures how much it fluctuates. These calculations are useful in sampling theory, where sums of squares and related statistics are naturally quadratic forms. The formulas also show why the mean vector cannot generally be ignored: a nonzero mean contributes to both moments.

We derive the formulas directly from Gaussian moments. The result allows any positive semidefinite covariance matrix, including a singular one, and does not require the quadratic-form matrix to be positive semidefinite.

Setup and Main Formulas

Definition: Let $X$ be a $p$-dimensional Gaussian random vector with mean $E[X]=\mu$ and covariance matrix $\operatorname{Var}(X)=\Sigma$, where $\Sigma$ is positive semidefinite. For a real $p\times p$ matrix $A$, the scalar random variable $X'AX$ is a Gaussian quadratic form. The Gaussian vector may be degenerate if $\Sigma$ is singular.

Only the symmetric part of $A$ affects the quadratic form. Write $S=(A+A')/2$. The matrix $A-S$ is skew-symmetric, and for every real vector $x$, the scalar $x'(A-S)x$ equals its own transpose and also its negative, so it is zero. Thus $x'Ax=x'Sx$. We may therefore state the formulas for symmetric $A$; for an arbitrary $A$, use them with $S$ in place of $A$.

Theorem: If $X\sim N(\mu,\Sigma)$ and $A$ is a real symmetric $p\times p$ matrix, then $$ E[X'AX]=\operatorname{tr}(A\Sigma)+\mu'A\mu $$ and $$ \operatorname{Var}(X'AX)=2\operatorname{tr}(A\Sigma A\Sigma)+4\mu'A\Sigma A\mu. $$ These identities hold whether or not $\Sigma$ is invertible.

Here $\operatorname{tr}(M)$ denotes the trace of a square matrix $M$, the sum of its diagonal entries. The mean formula has a covariance contribution, $\operatorname{tr}(A\Sigma)$, and a contribution from the location of the Gaussian vector, $\mu'A\mu$. The variance has a centered quadratic-form contribution and an additional term that depends on the mean.

Derivation

Set $Z=X-\mu$, so that $E[Z]=0$, $\operatorname{Var}(Z)=\Sigma$, and $Z$ is a centered Gaussian vector. Expanding the quadratic form gives $$ X'AX=\mu'A\mu+2\mu'AZ+Z'AZ, $$ where symmetry of $A$ makes the two cross terms equal.

First, $E[Z'AZ]=\operatorname{tr}(A\Sigma)$. To see this, expand the quadratic form by entries: $Z'AZ=\sum_{i,j}A_{ij}Z_iZ_j$, and $E[Z_iZ_j]=\Sigma_{ij}$. Hence $E[Z'AZ]=\sum_{i,j}A_{ij}\Sigma_{ij}=\operatorname{tr}(A\Sigma)$, since both $A$ and $\Sigma$ are symmetric. Also $E[\mu'AZ]=0$. Taking expectations proves the stated mean.

For the variance, the constant $\mu'A\mu$ has no effect. The centered Gaussian distribution is symmetric about zero, so every third-order centered moment is zero. In particular, the covariance between the linear term $2\mu'AZ$ and the quadratic term $Z'AZ$ vanishes: each term in their product is a constant times a third-order centered moment. Therefore, $$ \operatorname{Var}(X'AX)=4\operatorname{Var}(\mu'AZ)+\operatorname{Var}(Z'AZ). $$ The variance of the linear term before multiplication by 2 is $\operatorname{Var}(\mu'AZ)=\mu'A\Sigma A\mu$, using symmetry of $A$.

It remains to compute the variance of $Z'AZ$. Because $\Sigma$ is positive semidefinite, there is a matrix $C$ such that $CC'=\Sigma$. If $U\sim N(0,I_p)$, then $CU$ has the same Gaussian distribution as $Z$, including when $\Sigma$ is singular. Let $H=C'AC$, which is symmetric. Then $Z'AZ$ has the same distribution as $U'HU$. By the spectral theorem, an orthogonal change of coordinates diagonalizes $H$; the transformed vector still has independent standard normal components. If $\lambda_1,\ldots,\lambda_p$ are the eigenvalues of $H$, the quadratic form consequently has the same distribution as $\sum_{i=1}^p\lambda_i U_i^2$. Since independent standard normal variables satisfy $\operatorname{Var}(U_i^2)=2$, $$ \operatorname{Var}(Z'AZ)=2\sum_{i=1}^p\lambda_i^2 =2\operatorname{tr}(H^2) =2\operatorname{tr}(A\Sigma A\Sigma). $$ The last equality follows by substituting $H=C'AC$ and using the cyclic property of trace. Combining this with the linear-term variance gives the theorem. $\square$

Worked Applications

Worked Example: A Central Projection Form

Let $X\sim N(0,I_4)$, and let $A$ be the diagonal matrix with diagonal entries $1,1,0,0$. Calculate the expectation and variance of $Q=X'AX$.

The matrix $A$ is symmetric and satisfies $A^2=A$. The form is $Q=X_1^2+X_2^2$. Its trace is $2$, and $\operatorname{tr}(A^2)=2$. Since $\mu=0$ and $\Sigma=I_4$, the mean formula gives $$ E[Q]=\operatorname{tr}(AI_4)+0=2. $$ For the variance, $A I_4 A I_4=A^2=A$, so $$ \operatorname{Var}(Q)=2\operatorname{tr}(A)=2(2)=4. $$ As a check, $X_1^2$ and $X_2^2$ are independent, each with mean $1$ and variance $2$, giving the same total mean $2$ and variance $4$.

Worked Example: A Noncentral Sum of Squares

Let $X\sim N(\mu,I_3)$, where $\mu=(1,-1,2)'$, and take $A=I_3$. Find the mean and variance of $Q=X'X$.

First calculate $\mu'\mu=1^2+(-1)^2+2^2=6$. The mean is $$ E[Q]=\operatorname{tr}(I_3)+\mu'\mu=3+6=9. $$ For the variance, $\operatorname{tr}(I_3 I_3 I_3 I_3)=3$, and $\mu'I_3 I_3 I_3\mu=\mu'\mu=6$. Therefore, $$ \operatorname{Var}(Q)=2(3)+4(6)=6+24=30. $$ The $4\mu'A\Sigma A\mu$ term is essential here: using only $2\operatorname{tr}(A\Sigma A\Sigma)$ would give $6$, not the correct variance $30$.

Worked Example: A Correlated Gaussian Vector

Suppose $X\sim N(\mu,\Sigma)$, where $$ \mu=(1,0)',\qquad \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix},\qquad A=\begin{pmatrix}1&0\\0&2\end{pmatrix}. $$ Calculate the mean and variance of $Q=X'AX$.

For the mean, $\mu'A\mu=1$, while $$ \operatorname{tr}(A\Sigma) =\operatorname{tr}\begin{pmatrix}4&1\\2&4\end{pmatrix} =8. $$ Thus $E[Q]=8+1=9$. For the first variance term, calculate $$ A\Sigma=\begin{pmatrix}4&1\\2&4\end{pmatrix}, \qquad (A\Sigma)^2=\begin{pmatrix}18&8\\16&18\end{pmatrix}, \qquad \operatorname{tr}(A\Sigma A\Sigma)=36. $$ For the second variance term, $$ A\Sigma A=\begin{pmatrix}4&2\\2&8\end{pmatrix}, \qquad \mu'A\Sigma A\mu=4. $$ It follows that $$ \operatorname{Var}(Q)=2(36)+4(4)=72+16=88. $$ The same mean can be checked entry by entry: $Q=X_1^2+2X_2^2$, so $E[Q]=(4+1)+2(2+0)=9$. This check also confirms that the location contribution in the mean is $1$.

Common Mistakes and Interpretation

The formulas separate the effects of covariance and location, but those effects must be combined using the correct matrices. A frequent mistake is to apply the centered formula to a Gaussian vector with nonzero mean. For $X\sim N(\mu,\Sigma)$, the expression $2\operatorname{tr}(A\Sigma A\Sigma)$ alone is the variance only when the mean contribution $4\mu'A\Sigma A\mu$ vanishes. In particular, a nonzero $\mu$ does not automatically make that contribution positive: its value depends on $A$, $\Sigma$, and $\mu$ together.

Using $A$ when it is not symmetric. Replace it by $S=(A+A')/2$ first. The skew-symmetric part contributes nothing to $X'AX$.
Dropping the mean term. The correct expectation is $\operatorname{tr}(A\Sigma)+\mu'A\mu$, not just $\operatorname{tr}(A\Sigma)$.
Confusing covariance with second moments. The covariance matrix describes $X-\mu$; the mean contribution must be added separately.
Assuming an inverse exists. The derivation uses a square root $C$ with $CC'=\Sigma$, not $\Sigma^{-1}$. The formulas therefore remain valid for singular covariance matrices.

For any symmetric $A$, the variance is the sum of the two displayed terms in the theorem. The first describes fluctuations of the centered quadratic part; the second describes the added variation from the linear term created when the quadratic form is expanded around $\mu$. This decomposition is often the most useful way to remember the calculation without losing track of its assumptions.

Check Your Understanding

Use the formulas and assumptions in this tutorial to answer the following questions.

Why does replacing a real matrix $A$ by $(A+A')/2$ leave $X'AX$ unchanged?
For $X\sim N(0,\Sigma)$ and symmetric $A$, state the mean and variance of $X'AX$.
If $X\sim N(\mu,I_p)$ and $A=I_p$, express the mean and variance of $X'X$ in terms of $p$ and $\mu'\mu$.
Which step in the variance derivation makes the covariance between the linear and quadratic terms zero?
Does the moment formula require $\Sigma$ to be invertible? Explain briefly.