Conditioning Degenerate Gaussian Vectors on Linear Constraints

Why Singular Gaussian Conditioning Needs Care

A Gaussian vector need not have a density over the whole ambient space. If one component is an exact linear combination of others, or if some directions have zero variance, the distribution is concentrated on a lower-dimensional affine set. Conditioning on a linear measurement must respect that restriction: some measurement values are possible, while others cannot occur at all.

For a nonsingular measurement covariance, the familiar conditional Gaussian formulas use an ordinary matrix inverse. With singular covariance, an ordinary inverse does not exist. The Moore–Penrose generalized inverse supplies the corresponding formula, but it does not make impossible constraints possible. The support condition is part of the result.

The support statement follows by representing \(X\) as \(\mu+\Sigma^{1/2}Z\), where \(Z\) is a standard Gaussian vector: the possible directions away from \(\mu\) are exactly \(\operatorname{Range}(\Sigma^{1/2})=\operatorname{Range}(\Sigma)\). The measurement \(Y\) is therefore supported on \(H\mu+\operatorname{Range}(C)\). In particular, a proposed value \(y\) is compatible with the model precisely when \(y-H\mu\in\operatorname{Range}(C)\).

For a symmetric positive semidefinite matrix \(C\), its Moore–Penrose inverse \(C^+\) is defined by diagonalizing \(C\): replace each positive eigenvalue by its reciprocal and leave each zero eigenvalue at zero. Thus \(CC^+=C^+C\) is the orthogonal projector onto \(\operatorname{Range}(C)\). When \(C\) is nonsingular, \(C^+=C^{-1}\).

The Conditional Law Under a Linear Constraint

An exact event \(Y=y\) often has probability zero even when \(y\) is in the support. Conditioning at such a value is understood through a regular conditional distribution: a family of conditional laws that gives the usual conditional probabilities when averaged over the distribution of \(Y\). For a Gaussian model, the formula below provides a natural Gaussian version of that family at every point of the support. Outside the support, a conditional law is not determined by the model.

To prove the formulas, write \(\delta=Y-H\mu\), \(G=\Sigma H^{\mathsf T}C^+\), and define the residual $$ R=X-\mu-G(Y-H\mu). $$ Since \(C^+C\) projects onto \(\operatorname{Range}(C)\), the support condition gives \(CC^+\delta=\delta\). It follows that \(H(\mu+G\delta)=H\mu+\delta=y\): the proposed conditional mean satisfies the constraint.

The residual and \(Y\) are jointly Gaussian because they are affine transformations of \(X\). Their cross-covariance is zero. Indeed, \(\operatorname{Range}(H\Sigma)\subseteq\operatorname{Range}(C)\): writing \(A=H\Sigma^{1/2}\), these ranges are \(\operatorname{Range}(AA^{\mathsf T})=\operatorname{Range}(A)\) and \(\operatorname{Range}(A\Sigma^{1/2})\subseteq\operatorname{Range}(A)\). Therefore \(C^+C H\Sigma=H\Sigma\), and $$ \operatorname{Cov}(R,Y)=\Sigma H^{\mathsf T}-\Sigma H^{\mathsf T}C^+C=0. $$ The standard Gaussian block-independence result says that jointly Gaussian vectors with zero cross-covariance are independent. Thus \(R\) is independent of \(Y\), so the conditional law of \(X\) is the law of \(\mu+G(y-H\mu)+R\). Its conditional mean is \(\mu+G(y-H\mu)\). Directly taking the covariance of \(R\) gives $$ \operatorname{Cov}(R)=\Sigma-GH\Sigma-\Sigma H^{\mathsf T}G^{\mathsf T}+GCG^{\mathsf T} =\Sigma-\Sigma H^{\mathsf T}C^+H\Sigma. $$ Here \(C^+= (C^+)^{\mathsf T}\) and \(C^+CC^+=C^+\), both immediate from the eigenvalue definition. This proves the conditional mean and covariance formulas.

It remains to verify the support claim. Let \(S=\Sigma^{1/2}\) and \(B=HS\). Then \(C=BB^{\mathsf T}\), and the conditional covariance \(K\) can be written $$ K=S\bigl[I-B^{\mathsf T}(BB^{\mathsf T})^+B\bigr]S. $$ The bracketed matrix is the orthogonal projector onto \(\ker(B)\), as follows by applying the eigenvalue definition of the generalized inverse to a singular value decomposition of \(B\). Consequently, \(\operatorname{Range}(K)=\operatorname{Range}(\Sigma)\cap\ker(H)\). The conditional mean belongs to \(\mu+\operatorname{Range}(\Sigma)\) and satisfies \(Hx=y\). A Gaussian distribution with covariance \(K\) is supported on its mean plus \(\operatorname{Range}(K)\). These facts show that its support is exactly the stated intersection. \(\square\)

Worked Applications

Interpreting the Support and Avoiding a Common Error

A singular conditional covariance is not a failure of the formula. It records directions that remain deterministic after conditioning. More generally, the conditional support consists precisely of points that both satisfy the original linear restrictions encoded by \(\Sigma\) and obey \(Hx=y\). Redundant rows of \(H\) are allowed; the generalized inverse accounts for them without pretending they provide independent information.

A common mistake is to substitute an ordinary inverse when \(C\) is singular, or to apply the generalized-inverse formula at an impossible value and call the result a conditional law. For example, in the first worked example the vector \((2,3)^{\mathsf T}\) is not in the support of \(Y\). The correction is to check \(y-H\mu\in\operatorname{Range}(C)\) first. If that condition fails, \(Y=y\) is incompatible with the model; the formula does not define a meaningful conditional distribution there.