Quadratic Forms With Singular Gaussian Covariance

Why Singular Covariance Changes the Calculation

A Gaussian vector with singular covariance is not spread throughout its ambient space. Its random variation is confined to a lower-dimensional affine subspace. A quadratic form can therefore respond to fewer directions than the vector has coordinates, and its degrees of freedom are determined by the directions the form actually sees.

The useful reduction is to represent the random part using independent standard normal coordinates on the covariance support. Once reduced, a centered quadratic form becomes an ordinary quadratic form in a standard normal vector. Its eigenvalues give the weights in its distribution. This makes both the distribution and the effective degrees of freedom explicit.

Reducing to the Gaussian Support

Definition: A Gaussian vector \(X\in\mathbb{R}^p\) with mean \(\mu\) and covariance matrix \(\Sigma\) is singular when \(\Sigma\) is positive semidefinite but has rank \(r<p\). Its support is the affine subspace \(\mu+\operatorname{col}(\Sigma)\), where \(\operatorname{col}(\Sigma)\) is the column space of \(\Sigma\).

By the spectral decomposition, write \(\Sigma=U_r\Lambda U_r'\), where \(U_r\) has \(r\) orthonormal columns and \(\Lambda\) is diagonal with the positive eigenvalues of \(\Sigma\). Define \(A=U_r\Lambda^{1/2}\). Then \(A\) has full column rank, \(\Sigma=AA'\), and

X\overset{d}{=}\mu+AZ,\qquad Z\sim N_r(0,I_r).

This representation expresses all random variation in \(r\) independent standard normal coordinates. It also shows why treating \(\Sigma\) as if it had an ordinary inverse is not valid: there are \(p-r\) directions with no random variation at all.

Worked Example: Finding the Effective Coordinates

Let \(X\) have mean \(\mu=(1,-1,2)'\) and covariance \(\Sigma=\operatorname{diag}(4,1,0)\). The rank is \(r=2\), and a factor is \(A=\begin{pmatrix}2&0\\0&1\\0&0\end{pmatrix}\), since \(AA'=\Sigma\). Thus

X\overset{d}{=} \begin{pmatrix}1\\-1\\2\end{pmatrix} + \begin{pmatrix}2&0\\0&1\\0&0\end{pmatrix} \begin{pmatrix}Z_1\\Z_2\end{pmatrix}, \qquad Z_1,Z_2\ \text{independent }N(0,1).

The third coordinate is always \(2\); it has no random variation. The support is the plane of points whose third coordinate equals \(2\). Although \(X\) has three coordinates, its random part has only two effective coordinates.

Distribution of a Centered Quadratic Form

Let \(B\) be a real symmetric \(p\times p\) matrix and consider the centered form \(Q=(X-\mu)'B(X-\mu)\). Substituting the support representation gives \(Q=Z'C Z\), where \(C=A'BA\) is an \(r\times r\) symmetric matrix. The reduction uses the matrix restricted to the random support, not \(B\) alone.

Theorem: Suppose \(X\sim N_p(\mu,\Sigma)\), where \(\Sigma\) is positive semidefinite of rank \(r\), and let \(A\) be any \(p\times r\) full-column-rank matrix such that \(AA'=\Sigma\). For any symmetric \(B\), let \(\lambda_1,\ldots,\lambda_r\) be the eigenvalues of \(A'BA\). Then

(X-\mu)'B(X-\mu)\ \overset{d}{=}\ \sum_{j=1}^r \lambda_j Z_j^2, \qquad Z_1,\ldots,Z_r\ \text{independent }N(0,1).

If \(B\) is positive semidefinite, every \(\lambda_j\geq 0\). In that case, the number of nonzero weights is \(\operatorname{rank}(A'BA)\), which may be smaller than \(r\). This is the effective number of random directions used by the form.

Proof. From \(X\overset{d}{=}\mu+AZ\), the form has the same distribution as \(Z'A'BAZ=Z'CZ\). Since \(C\) is real and symmetric, there is an orthogonal matrix \(V\) such that \(V'CV=\operatorname{diag}(\lambda_1,\ldots,\lambda_r)\). Set \(W=V'Z\). Orthogonal transformations preserve the standard multivariate normal distribution, so the coordinates of \(W\) are independent \(N(0,1)\). Therefore \(Z'CZ=W'\operatorname{diag}(\lambda_1,\ldots,\lambda_r)W=\sum_{j=1}^r\lambda_j W_j^2\), proving the result. \(\square\)

For a positive semidefinite \(B\), all the weights are nonnegative, so the distribution is a weighted sum of independent chi-square variables with one degree of freedom. It is an ordinary chi-square distribution only under additional restrictions on the weights. For example, if all the nonzero weights equal \(1\), the sum is chi-square with degrees of freedom equal to the number of those weights.

Worked Example: A Weighted Chi-Square Form

Use the vector from the preceding example and take \(B=I_3\). Then \(Q=(X-\mu)'(X-\mu)\), and

A'A= \begin{pmatrix}2&0\\0&1\end{pmatrix} \begin{pmatrix}2&0\\0&1\end{pmatrix} = \begin{pmatrix}4&0\\0&1\end{pmatrix}.

The reduced matrix has eigenvalues \(4\) and \(1\). Hence \(Q\overset{d}{=}4Z_1^2+Z_2^2\). This is not \(\chi^2_2\), because its weights are unequal. As checks, \(E[Q]=4+1=5\), and, using \(\operatorname{Var}(Z_j^2)=2\) and independence, \(\operatorname{Var}(Q)=4^2(2)+1^2(2)=34\).

When a Shifted Form Is Noncentral Chi-Square

A shifted form has the shape \(Q=(X-c)'B(X-c)\), with \(c\) not necessarily equal to \(\mu\). For positive semidefinite \(B\), its reduction yields a constant plus a weighted sum of noncentral chi-square terms. The constant matters: a shift in a direction with no random variation can contribute a fixed amount without increasing the degrees of freedom.

Result: Let \(d=\mu-c\), let \(A\) satisfy \(AA'=\Sigma\), and suppose \(B\) is symmetric and positive semidefinite. Set \(C=A'BA\) and \(b=A'Bd\). If the positive eigenvalues of \(C\) are \(\lambda_1,\ldots,\lambda_s\), choose orthonormal eigenvectors so that the coordinates of \(b\) in those directions are \(b_1,\ldots,b_s\). Then

(X-c)'B(X-c)\ \overset{d}{=}\ \kappa+\sum_{j=1}^s\lambda_j(Z_j+\delta_j)^2, \qquad \delta_j=\frac{b_j}{\lambda_j}, \qquad \kappa=d'Bd-\sum_{j=1}^s\frac{b_j^2}{\lambda_j}\geq 0.

Here \((Z_j+\delta_j)^2\) has a noncentral chi-square distribution with one degree of freedom and noncentrality parameter \(\delta_j^2\). The weights need not be equal, and the nonnegative constant \(\kappa\) need not be zero.

Proof. Write \(X-c\overset{d}{=}d+AZ\). Expanding gives \(Q=Z'CZ+2b'Z+d'Bd\). If \(v\) lies in the null space of \(C\), then \(0=v'Cv=(Av)'B(Av)\). Positive semidefiniteness of \(B\) implies \(B^{1/2}Av=0\), and hence \(v'b=(Av)'Bd=0\). Thus \(b\) has no component in the null space of \(C\); in the eigenbasis of \(C\), its coordinates in zero-eigenvalue directions are zero. In each positive-eigenvalue direction, complete the square:

\lambda_j Z_j^2+2b_jZ_j = \lambda_j\left(Z_j+\frac{b_j}{\lambda_j}\right)^2 -\frac{b_j^2}{\lambda_j}.

Summing these identities gives the stated representation. Finally, \(\kappa\geq0\): it is the minimum over \(z\in\mathbb{R}^r\) of \((d+Az)'B(d+Az)\). Indeed, completing the squares gives that minimum as \(d'Bd-\sum_j b_j^2/\lambda_j\), and every value of the quadratic form is nonnegative because \(B\) is positive semidefinite. \(\square\)

Worked Example: A Shifted Form on a Singular Support

Let \(X=(Z,0)'\), where \(Z\sim N(0,1)\), and take \(B=I_2\) and \(c=(1,0)'\). The covariance is \(\Sigma=\operatorname{diag}(1,0)\), so choose \(A=(1,0)'\). Here \(d=\mu-c=(-1,0)'\), \(C=A'BA=1\), and \(b=A'Bd=-1\). Therefore \(\lambda_1=1\), \(\delta_1=-1\), and \(\kappa=d'Bd-b^2/\lambda_1=1-1=0\). Directly,

(X-c)'(X-c)=(Z-1)^2\sim\chi^2_1(1).

The noncentrality is \(1\), and the degrees of freedom are \(1\), not \(2\): the second coordinate of \(X\) is fixed at zero and adds no random direction.

The Pseudoinverse Mahalanobis Distance

A particularly important form uses the Moore–Penrose pseudoinverse \(\Sigma^+\), which inverts the positive eigenvalues of \(\Sigma\) and leaves its null directions at zero. For a singular Gaussian vector, this produces a chi-square statistic with degrees of freedom equal to the covariance rank.

Theorem: If \(X\sim N_p(\mu,\Sigma)\), with \(\Sigma\) positive semidefinite of rank \(r\), then

(X-\mu)'\Sigma^+(X-\mu)\sim\chi^2_r.

Proof. Write \(\Sigma=U_r\Lambda U_r'\), with \(\Lambda\) diagonal and positive. Then \(\Sigma^+=U_r\Lambda^{-1}U_r'\) and \(A=U_r\Lambda^{1/2}\) is a factor of \(\Sigma\). Using \(X-\mu\overset{d}{=}AZ\),

(X-\mu)'\Sigma^+(X-\mu) \overset{d}{=}Z'A'\Sigma^+AZ =Z'I_rZ =\sum_{j=1}^r Z_j^2.

The sum of squares of \(r\) independent standard normal variables has the \(\chi^2_r\) distribution. \(\square\)

Worked Example: Checking the Pseudoinverse Degrees of Freedom

For \(\Sigma=\operatorname{diag}(9,4,0)\), the pseudoinverse is \(\Sigma^+=\operatorname{diag}(1/9,1/4,0)\), and the rank is \(2\). With \(X-\mu=(3Z_1,2Z_2,0)'\),

(X-\mu)'\Sigma^+(X-\mu) =\frac{(3Z_1)^2}{9}+\frac{(2Z_2)^2}{4} =Z_1^2+Z_2^2 \sim\chi^2_2.

The zero-variance third coordinate contributes neither a term nor a degree of freedom.

Effective Degrees of Freedom and a Common Pitfall

For a centered positive semidefinite quadratic form, the matrix \(A'BA\) summarizes how the form acts on the random support. Its rank counts the directions with nonzero weights. That rank is at most \(\operatorname{rank}(\Sigma)\), but it need not equal it: the quadratic form may ignore some random directions. Conversely, the ambient dimension \(p\) alone does not determine degrees of freedom.

Common mistake: “\(X\) has \(p\) coordinates, so the quadratic form has \(p\) degrees of freedom.” This counts coordinates without checking whether they vary independently or whether the form uses them.
Correction: Factor \(\Sigma=AA'\), form \(A'BA\), and examine its nonzero eigenvalues. For the pseudoinverse Mahalanobis distance, the result is \(\chi^2_r\), where \(r=\operatorname{rank}(\Sigma)\); for a general form, its weights and effective rank come from \(A'BA\).
Another pitfall: Applying the shifted-form completion formula to an indefinite \(B\) without checking its assumptions. The nonnegative constant and the zero-eigenvalue argument above rely on \(B\) being positive semidefinite.

For a centered form with general symmetric \(B\), the eigenvalue representation still holds, but some weights can be negative. For a shifted form with indefinite \(B\), linear terms in null directions of \(A'BA\) may remain; it need not reduce to a constant plus weighted noncentral chi-square terms. Check the definiteness condition before using that representation.

Check Your Understanding

Use the support reduction to answer each question.

If \(\Sigma\) is \(5\times5\) with rank \(3\), how many independent standard normal coordinates are needed to represent the random part of \(X\)?
For a centered form with \(B\) positive semidefinite, which matrix determines its nonzero weights?
Can the rank of \(A'BA\) be larger than the rank of \(\Sigma\)? Explain.
What are the distribution and degrees of freedom of \((X-\mu)'\Sigma^+(X-\mu)\) when \(\operatorname{rank}(\Sigma)=4\)?
Why does the shifted-form result require \(B\) to be positive semidefinite?