Conditioning on Singular Gaussian Blocks

Why Singular Gaussian Blocks Need Special Treatment

The usual conditional Gaussian formulas involve the inverse of the covariance matrix of the observed block. That inverse does not exist when some observed coordinates are redundant or have zero variance. The singularity is not a defect in the model: it says that the observed vector is confined to a lower-dimensional affine set. The conditional law is well defined for observations on that set, and its formula uses the Moore–Penrose inverse.

Let $X$ be a $p$-dimensional Gaussian vector and $Y$ a $q$-dimensional Gaussian vector. Write their joint mean and covariance in block form as follows:

\begin{pmatrix}X\\Y\end{pmatrix} \sim N\left( \begin{pmatrix}\mu_X\\\mu_Y\end{pmatrix}, \begin{pmatrix}A&C\\C^T&B\end{pmatrix} \right),

where $A=\operatorname{Var}(X)$, $B=\operatorname{Var}(Y)$, and $C=\operatorname{Cov}(X,Y)$. The full block matrix is positive semidefinite, while $B$ is allowed to be singular.

Support and the Moore–Penrose Inverse

For a symmetric positive semidefinite matrix $B$, its range $\operatorname{Range}(B)$ is the set of possible directions of variation of a centered Gaussian vector with covariance $B$. Thus $Y$ lies in the affine set $\mu_Y+\operatorname{Range}(B)$ with probability one. This affine set is the support of $Y$.

Definition: If $B=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_q)Q^T$, where $Q$ is orthogonal and $\lambda_i\geq 0$, its Moore–Penrose inverse is $$ B^+=Q\operatorname{diag}(\lambda_1^+,\ldots,\lambda_q^+)Q^T, \qquad \lambda_i^+= \begin{cases} 1/\lambda_i,&\lambda_i>0,\\ 0,&\lambda_i=0. \end{cases} $$ It inverts $B$ on $\operatorname{Range}(B)$ and acts as zero on $\operatorname{Null}(B)$. In particular, $BB^+=B^+B$ is the orthogonal projection onto $\operatorname{Range}(B)$.

Before applying this inverse to a conditional formula, it is important that the cross-covariance $C$ be compatible with the null directions of $B$. Joint positive semidefiniteness guarantees precisely this compatibility.

Result: For the joint Gaussian block above, $C=C B^+B$ and $C^T=BB^+C^T$. Equivalently, every row of $C$ belongs to $\operatorname{Range}(B)$.

To verify this, take any $v\in\operatorname{Null}(B)$. The scalar $v^T(Y-\mu_Y)$ has variance $v^TBv=0$, so it equals zero almost surely. Therefore its covariance with $X$ is zero, giving $Cv=0$. Hence $\operatorname{Null}(B)\subseteq\operatorname{Null}(C)$, which means the rows of $C$ are orthogonal to $\operatorname{Null}(B)$. For symmetric $B$, that orthogonal complement is $\operatorname{Range}(B)$, proving the stated identities.

Worked Example: A Scalar Given a Redundant Observation

Let $U$ and $V$ be independent $N(0,1)$ variables, and set $X=U+V$ and $Y=(U,2U)^T$. The second coordinate of $Y$ repeats the first up to a factor of two. Find the law of $X$ given $Y=(t,2t)^T$.

The mean vectors are zero. Direct covariance calculations give $A=2$, $B=uu^T$ for $u=(1,2)^T$, and $C=(1,2)$. Since $u^Tu=5$, the matrix $B$ has eigenvalue $5$ in the direction $u$ and eigenvalue zero in its orthogonal direction. Its Moore–Penrose inverse is

B^+=\frac{uu^T}{25} =\frac{1}{25}\begin{pmatrix}1&2\\2&4\end{pmatrix}.

The conditional mean coefficient and covariance are

CB^+=\begin{pmatrix}\frac15&\frac25\end{pmatrix}, \qquad CB^+C^T=1, \qquad A-CB^+C^T=2-1=1.

For $y=(t,2t)^T$, the conditional mean is $\frac15t+\frac25(2t)=t$. Therefore $X\mid Y=(t,2t)^T\sim N(t,1)$. This agrees with the construction: observing $Y$ reveals $U=t$, while the independent $V$ remains $N(0,1)$.

The Conditional Gaussian Formula on the Support

Theorem: Suppose $(X^T,Y^T)^T$ is jointly Gaussian with the block mean and covariance above, and $B$ may be singular. For every $y\in\mu_Y+\operatorname{Range}(B)$, a conditional Gaussian law is $$ X\mid Y=y\ \sim\ N\left(\mu_X+CB^+(y-\mu_Y),\ A-CB^+C^T\right). $$ The conditional covariance may itself be singular. The formula specifies a regular conditional distribution on the support; values assigned outside the support do not affect that conditional distribution.

Here is a derivation that also explains why the Moore–Penrose inverse appears. Define the centered observation $D=Y-\mu_Y$, the matrix $L=CB^+$, and the residual

R=X-\mu_X-LD.

First, the compatibility identities give

\operatorname{Cov}(R,D)=C-LB=C-CB^+B=0.

The covariance expansion must include both cross terms. In full, it is

\operatorname{Var}(R)=A-LC^T-CL^T+LBL^T.

Substitute $L=CB^+$. Since $B^+$ is symmetric and $B^+BB^+=B^+$, the terms become $LC^T=CB^+C^T$, $CL^T=CB^+C^T$, and $LBL^T=CB^+BB^+C^T=CB^+C^T$. Thus

\operatorname{Var}(R)=A-CB^+C^T-CB^+C^T+CB^+C^T =A-CB^+C^T.

The pair $(R,D)$ is jointly Gaussian because it is an affine transformation of $(X,Y)$. Its cross-covariance is zero. To see directly that the two components are independent here, the characteristic function of a jointly Gaussian vector is the exponential of its linear mean term minus one half of its covariance quadratic form. With zero cross-covariance, that quadratic form separates into an $R$ term and a $D$ term; the characteristic function therefore factors into the marginal characteristic functions. Hence $R$ and $D$ are independent.

Finally, $X=\mu_X+LD+R$. Given $Y=y$, the value of $D$ is $y-\mu_Y$, while the independent residual $R$ retains its Gaussian law with mean zero and covariance $A-CB^+C^T$. This yields the displayed conditional mean and covariance. The derivation also shows that this covariance is positive semidefinite: it is the covariance of $R$. Since $Y$ belongs to its support almost surely, the formula gives a version of the conditional law on every support value. $\square$

Worked Example: A Vector Given a Repeated Coordinate

Let $U,V,W$ be independent $N(0,1)$ variables. Set $X=(U+V,\,2U+W)^T$ and $Y=(U,U)^T$. Find the conditional distribution of $X$ given $Y=(t,t)^T$.

The covariance blocks are

A=\begin{pmatrix}2&2\\2&5\end{pmatrix}, \qquad B=\begin{pmatrix}1&1\\1&1\end{pmatrix}, \qquad C=\begin{pmatrix}1&1\\2&2\end{pmatrix}.

The matrix $B$ has eigenvalue $2$ along $(1,1)^T$ and zero along $(1,-1)^T$, so $B^+=\frac14\begin{pmatrix}1&1\\1&1\end{pmatrix}$. Multiplication gives

CB^+=\begin{pmatrix}\frac12&\frac12\\1&1\end{pmatrix}, \qquad CB^+C^T=\begin{pmatrix}1&2\\2&4\end{pmatrix}.

The support of $Y$ consists of vectors $(t,t)^T$. At such a value the conditional mean is $(t,2t)^T$, and the conditional covariance is

A-CB^+C^T =\begin{pmatrix}1&0\\0&1\end{pmatrix}.

Thus $X\mid Y=(t,t)^T\sim N((t,2t)^T,I_2)$. The construction checks the result: $Y$ reveals $U=t$, leaving the independent $V$ and $W$ as the two unit-variance residuals.

Worked Example: Conditioning on a Singular Observation of a Sum

Let $U,V$ be independent $N(0,1)$ variables, take $X=(U,V)^T$, and observe $Y=(U+V,\,2U+2V)^T$. Find the conditional distribution of $X$ when $Y=(s,2s)^T$.

Here $A=I_2$. With $u=(1,2)^T$, the covariance blocks are $B=2uu^T$ and $C=\begin{pmatrix}1&2\\1&2\end{pmatrix}$. The nonzero eigenvalue of $B$ is $2u^Tu=10$, so

B^+=\frac{uu^T}{50} =\frac1{50}\begin{pmatrix}1&2\\2&4\end{pmatrix}.

Each row of $C$ is $(1,2)$. Multiplying gives $CB^+=\begin{pmatrix}1/10&1/5\\1/10&1/5\end{pmatrix}$, and then

CB^+C^T=\begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix}, \qquad A-CB^+C^T =\begin{pmatrix}1/2&-1/2\\-1/2&1/2\end{pmatrix}.

The conditional mean at $y=(s,2s)^T$ is $(s/2,s/2)^T$. Therefore $X\mid Y=(s,2s)^T\sim N((s/2,s/2)^T,\begin{pmatrix}1/2&-1/2\\-1/2&1/2\end{pmatrix})$. The covariance is singular: conditional on the observed sum, the two components fluctuate in opposite directions, so their sum has zero conditional variance.

Support Values, Impossible Values, and a Common Pitfall

In the last example, $Y$ can only equal $(s,2s)^T$ for some real $s$. An input such as $y=(1,3)^T$ is outside the support and has probability zero under the model. The theorem does not assert a meaningful conditional law for such an impossible observation. A regular conditional distribution is only determined almost surely, so its values outside the support may be assigned arbitrarily.

A common mistake is to treat $B^+$ as though it were an ordinary inverse and apply the formula to any vector $y-\mu_Y$. The resulting algebraic expression can be evaluated off the support, but that does not make it the conditional distribution for an observation the model cannot produce. Check first that $y-\mu_Y\in\operatorname{Range}(B)$. On the support, the formula accounts for exactly the identifiable directions; null directions contain no random variation to condition on.

Key takeaway: For a singular Gaussian observation, restrict the conditional formula to the affine support of $Y$. The Moore–Penrose inverse inverts the covariance on its range, and joint positive semidefiniteness ensures the cross-covariance uses only those same directions.

Check Your Understanding

Use the support and conditional formula to answer these questions.

If $B$ has rank $r<q$, what is the dimension of the affine support of $Y$?
Why does $v\in\operatorname{Null}(B)$ imply $Cv=0$ for a jointly Gaussian block with positive semidefinite joint covariance?
For $B=\begin{pmatrix}4&0\\0&0\end{pmatrix}$, find $B^+$ and describe $\operatorname{Range}(B)$.
In the repeated-coordinate example with $Y=(U,U)^T$, why is $y=(t_1,t_2)^T$ a possible observation only when $t_1=t_2$?
What random vector is used in the proof to isolate the part of $X$ remaining after its linear dependence on $Y$ is removed?

Conditioning on Singular Gaussian Blocks

What You'll Learn