Gaussian Conditional Distributions

Why Gaussian Conditioning Has a Matrix Formula

For jointly Gaussian vectors, conditioning changes the mean in a way that is linear in the observed value, while the conditional covariance does not depend on that value. When the conditioning vector has a nonsingular covariance matrix, the familiar formula uses an ordinary matrix inverse. The same structure persists when that covariance matrix is singular, but the inverse must be replaced by the Moore–Penrose pseudoinverse, and the possible conditioning values lie on a lower-dimensional affine set.

This topic is useful whenever a multivariate normal model includes exact linear constraints, redundant coordinates, or observations with no noise in some directions. The singular case is not a separate exception to Gaussian conditioning: it is the same projection-and-residual calculation, carried out on the subspace where the conditioning vector can vary.

Joint Gaussian Vectors and Covariance Blocks

A random vector $W$ in $\mathbb{R}^r$ is Gaussian if every linear combination of its coordinates is a one-dimensional normal random variable, allowing a constant random variable as a degenerate normal. Its mean is denoted by $\mu_W=\mathbb{E}[W]$, and its covariance matrix is $\operatorname{Cov}(W)=\mathbb{E}[(W-\mu_W)(W-\mu_W)^{\mathsf T}]$. A Gaussian vector may have a singular covariance matrix.

Let $X$ take values in $\mathbb{R}^d$ and $Y$ in $\mathbb{R}^k$, and suppose the pair $(X,Y)$ is jointly Gaussian. Write its mean and covariance in blocks:

\mathbb{E} \begin{bmatrix} X\\Y \end{bmatrix} = \begin{bmatrix} \mu_X\\\mu_Y \end{bmatrix}, \qquad \operatorname{Cov} \begin{bmatrix} X\\Y \end{bmatrix} = \begin{bmatrix} \Sigma_{XX}&\Sigma_{XY}\\ \Sigma_{YX}&\Sigma_{YY} \end{bmatrix}, \qquad \Sigma_{YX}=\Sigma_{XY}^{\mathsf T}.

For a symmetric positive semidefinite matrix $C$, its Moore–Penrose pseudoinverse $C^+$ acts as the ordinary inverse on the range of $C$ and as zero on the null space of $C$. In particular, $C^+$ is symmetric, $C^+CC^+\!=C^+$, and $C^+C$ is the orthogonal projection onto $\operatorname{Range}(C)$.

Definition: The possibly degenerate Gaussian law with mean $m$ and positive semidefinite covariance matrix $S$ is denoted by $\mathcal{N}(m,S)$. Its characteristic function at $t$ is $\exp(i t^{\mathsf T}m-\tfrac12t^{\mathsf T}St)$. If $S$ is singular, this law is supported on a lower-dimensional affine subspace.

The Conditional Gaussian Formula

Theorem: Suppose $(X,Y)$ is jointly Gaussian with the block means and covariances above. Set $C=\Sigma_{YY}$ and $B=\Sigma_{XY}$. Then a regular conditional distribution of $X$ given $Y=y$ is $$ \mathcal{L}(X\mid Y=y) = \mathcal{N}\!\left( \mu_X+B C^+(y-\mu_Y), \ \Sigma_{XX}-B C^+B^{\mathsf T} \right). $$ This formula gives a version for every $y\in\mathbb{R}^k$; only its values on the support of $Y$ affect the conditional distribution. That support is $\mu_Y+\operatorname{Range}(C)$.

Proof. First, $B$ vanishes on the null space of $C$. Indeed, if $v\in\operatorname{Null}(C)$, then $$ \mathbb{E}\!\left[(v^{\mathsf T}(Y-\mu_Y))^2\right]=v^{\mathsf T}Cv=0. $$ Thus $v^{\mathsf T}(Y-\mu_Y)=0$ almost surely, and consequently $Bv=\mathbb{E}[(X-\mu_X)v^{\mathsf T}(Y-\mu_Y)]=0$. Since $C^+C$ projects onto $\operatorname{Range}(C)=\operatorname{Null}(C)^\perp$, this gives $BC^+C=B$. Define the residual $$ Z=X-\mu_X-BC^+(Y-\mu_Y). $$ It is jointly Gaussian with $Y$, and its cross-covariance with $Y$ is $$ \operatorname{Cov}(Z,Y)=B-BC^+C=0. $$ Uncorrelated jointly Gaussian vectors are independent: their joint characteristic function has no cross term and factors into the product of their marginal characteristic functions. Hence $Z$ and $Y$ are independent. The covariance of $Z$ is $$ \begin{aligned} \operatorname{Cov}(Z) &=\Sigma_{XX}-BC^+B^{\mathsf T}-BC^+B^{\mathsf T} +BC^+CC^+B^{\mathsf T}\\ &=\Sigma_{XX}-BC^+B^{\mathsf T}, \end{aligned} $$ where $C^+CC^+=C^+$ was used in the second equality. This matrix is positive semidefinite because it is the covariance matrix of $Z$. Rearranging the definition of $Z$ yields $$ X=\mu_X+BC^+(Y-\mu_Y)+Z. $$ The residual $Z$ is independent of $Y$, so after conditioning on $Y=y$, the first two terms are fixed and the remaining term has the unchanged law of $Z$. This gives the stated Gaussian conditional law. Finally, $Y-\mu_Y$ lies in $\operatorname{Range}(C)$ almost surely: every vector in $\operatorname{Null}(C)$ has zero-variance inner product with $Y-\mu_Y$, and the null space has a finite basis. The affine support is therefore $\mu_Y+\operatorname{Range}(C)$. The displayed kernel is a measurable translated Gaussian law; its values outside this support do not affect the conditional-law identity. $\square$

When $C$ is invertible, $C^+=C^{-1}$, and the theorem reduces to the usual inverse-matrix formula. When $C$ is singular, the expression still makes sense because the cross-covariance $B$ is necessarily zero in every direction of $\operatorname{Null}(C)$. The pseudoinverse therefore inverts only the directions in which $Y$ actually varies.

Worked Calculations

Worked Example: Scalar Conditioning

Suppose $Y\sim\mathcal{N}(1,4)$, and let $X=2Y+\varepsilon$, where $\varepsilon\sim\mathcal{N}(0,9)$ is independent of $Y$. The pair $(X,Y)$ is jointly Gaussian. Its means are $\mu_Y=1$ and $\mu_X=2$. Its covariance blocks are

\Sigma_{YY}=4,\qquad \Sigma_{XY}=\operatorname{Cov}(2Y+\varepsilon,Y)=2\cdot4=8, \qquad \Sigma_{XX}=4\cdot4+9=25.

The conditional mean and variance are therefore

\begin{aligned} \mathbb{E}[X\mid Y=y] &=2+\frac{8}{4}(y-1)=2y,\\ \operatorname{Var}(X\mid Y=y) &=25-\frac{8^2}{4}=9. \end{aligned}

This agrees directly with the defining relation $X=2Y+\varepsilon$: once $Y=y$ is observed, $X=2y+\varepsilon$, and the independent noise still has mean $0$ and variance $9$.

Worked Example: A Two-Dimensional Matrix Calculation

Let $Y\sim\mathcal{N}(0,I_2)$, let $E\sim\mathcal{N}(0,\operatorname{diag}(1,4))$ be independent of $Y$, and set

X=MY+E, \qquad M= \begin{bmatrix} 1&2\\ -1&1 \end{bmatrix}.

Here $\Sigma_{YY}=I_2$ and $\Sigma_{XY}=M$. Direct multiplication gives

\Sigma_{XX} =MM^{\mathsf T}+\operatorname{diag}(1,4) = \begin{bmatrix} 5&1\\ 1&2 \end{bmatrix} + \begin{bmatrix} 1&0\\ 0&4 \end{bmatrix} = \begin{bmatrix} 6&1\\ 1&6 \end{bmatrix}.

Since $I_2^+=I_2$, the conditional covariance is

\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^+\Sigma_{YX} = \begin{bmatrix}6&1\\1&6\end{bmatrix} - MM^{\mathsf T} = \begin{bmatrix}1&0\\0&4\end{bmatrix}.

For the observed value $y=(2,-1)^{\mathsf T}$, the conditional mean is

My= \begin{bmatrix} 1&2\\-1&1 \end{bmatrix} \begin{bmatrix}2\\-1\end{bmatrix} = \begin{bmatrix}0\\-3\end{bmatrix}.

Thus $X\mid Y=(2,-1)^{\mathsf T}$ has law $\mathcal{N}((0,-3)^{\mathsf T},\operatorname{diag}(1,4))$. The calculation also follows from the construction: after observing $Y=y$, only the independent vector $E$ remains random.

Worked Example: Conditioning on a Singular Gaussian Vector

Let $U,V$ be independent standard normal random variables, and define $Y=(U,2U)^{\mathsf T}$ and $X=3U+V$. The second coordinate of $Y$ is always twice the first, so $Y$ has a singular covariance matrix. The means are zero, and the covariance blocks are

C=\Sigma_{YY} = \begin{bmatrix}1&2\\2&4\end{bmatrix}, \qquad B=\Sigma_{XY} = \begin{bmatrix}3&6\end{bmatrix}, \qquad \Sigma_{XX}=10.

The matrix $C$ has range spanned by $(1,2)^{\mathsf T}$ and eigenvalue $5$ on that range. Its pseudoinverse is

C^+=\frac{1}{25} \begin{bmatrix}1&2\\2&4\end{bmatrix}.

Indeed, $C C^+=C/5$, the projection onto the span of $(1,2)^{\mathsf T}$. The conditional mean coefficient and variance are

BC^+ = \begin{bmatrix}3&6\end{bmatrix} \frac{1}{25} \begin{bmatrix}1&2\\2&4\end{bmatrix} = \begin{bmatrix}3/5&6/5\end{bmatrix}, \qquad 10-BC^+B^{\mathsf T}=10-9=1.

Every possible conditioning value has the form $y=(u,2u)^{\mathsf T}$. At such a value, the conditional mean is

\begin{bmatrix}3/5&6/5\end{bmatrix} \begin{bmatrix}u\\2u\end{bmatrix} =\frac35u+\frac{12}{5}u=3u.

Consequently $X\mid Y=(u,2u)^{\mathsf T}\sim\mathcal{N}(3u,1)$. The observation determines $U=u$, but it reveals nothing about the independent variable $V$, which accounts for the remaining variance $1$.

Singularity, Support, and Interpretation

A singular conditioning covariance means that some linear combinations of the coordinates of $Y$ are deterministic. It does not mean that the conditional formula is undefined. The pseudoinverse ignores those deterministic directions, while the affine support specifies which values can actually be observed. For example, in the preceding calculation a vector $y=(y_1,y_2)^{\mathsf T}$ with $y_2\ne2y_1$ has probability zero under the law of $Y$.

A conditional distribution is defined up to changes on conditioning values that occur with probability zero. The theorem provides a convenient formula on all of $\mathbb{R}^k$, but only its values on $\mu_Y+\operatorname{Range}(C)$ are determined by the joint law. In a particularly extreme case, if $Y$ is deterministic, then $C=0$, $C^+=0$, and $B=0$; the formula gives the unconditional law of $X$ at the sole possible conditioning value.

The conditional covariance $\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^+\Sigma_{YX}$ is always positive semidefinite, as the proof identifies it as the covariance of a residual vector. It may itself be singular. Thus conditioning can leave exact linear constraints on $X$, and a valid conditional Gaussian law need not have a density with respect to full-dimensional Lebesgue measure.

Common pitfall: Do not substitute an ordinary inverse when $\Sigma_{YY}$ is singular, and do not treat values outside the affine support of $Y$ as possible observations. Use the Moore–Penrose pseudoinverse and keep track of the support.

Check Your Understanding

Use the covariance-block formula and the support condition to answer these questions.

What replaces $\Sigma_{YY}^{-1}$ in the conditional Gaussian formula when $\Sigma_{YY}$ is singular?
Why must $\Sigma_{XY}v=0$ whenever $v$ belongs to the null space of $\Sigma_{YY}$?
For $Y=(U,2U)^{\mathsf T}$, which observed values are in the support of $Y$?
In the two-dimensional matrix example, what is the conditional mean when $y=(1,1)^{\mathsf T}$?
Why is the conditional covariance positive semidefinite, even when it is singular?