Common Errors in Gaussian Conditioning

Why Gaussian Conditioning Formulas Are Easy to Misuse

Gaussian conditioning has compact matrix formulas, but a misplaced transpose or an unexamined inverse can make a plausible-looking calculation wrong. Even when every matrix is square, the result may be nonsymmetric or fail to be a covariance matrix. A separate error is to report the marginal covariance of a vector as its conditional covariance.

The safest practice is to write down the block dimensions before multiplying, and to check that a proposed conditional covariance is symmetric and positive semidefinite. The formulas below use \(X\in\mathbb{R}^p\) for the vector being predicted and \(Y\in\mathbb{R}^q\) for the vector being conditioned on.

The Correct Block Orientation

Let \((X,Y)\) be jointly Gaussian, with means \(\mu_X,\mu_Y\) and covariance matrix partitioned as follows. The first index on a covariance block names the variable in its rows.

\operatorname{Cov} \begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix}, \qquad \Sigma_{YX}=\Sigma_{XY}^{T}.

Here \(\Sigma_{XX}\) is \(p\times p\), \(\Sigma_{YY}\) is \(q\times q\), and \(\Sigma_{XY}\) is \(p\times q\). Suppose first that \(\Sigma_{YY}\) is invertible. The conditional formulas are

Theorem: If \((X,Y)\) is jointly Gaussian and \(\Sigma_{YY}\) is invertible, then

E[X\mid Y=y] = \mu_X+\Sigma_{XY}\Sigma_{YY}^{-1}(y-\mu_Y), \qquad \operatorname{Cov}(X\mid Y=y) = \Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}.

The matrix multiplying \(y-\mu_Y\), called the gain matrix here, has size \(p\times q\). The conditional covariance has size \(p\times p\). These dimension checks are immediate ways to reject a formula with the blocks reversed.

To justify the result, define \(A=\Sigma_{XY}\Sigma_{YY}^{-1}\) and the residual \(R=X-\mu_X-A(Y-\mu_Y)\). Its cross-covariance with \(Y\) is zero:

\operatorname{Cov}(R,Y) =\Sigma_{XY}-A\Sigma_{YY} =\Sigma_{XY}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YY} =0.

The pair \((R,Y)\) is jointly Gaussian because it is an affine transformation of a jointly Gaussian vector. For jointly Gaussian vectors, zero cross-covariance implies independence. Thus \(R\) is independent of \(Y\), has mean zero, and has covariance

\begin{aligned} \operatorname{Cov}(R) &=\Sigma_{XX}-A\Sigma_{YX}-\Sigma_{XY}A^T +A\Sigma_{YY}A^T\\ &=\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}. \end{aligned}

In the second equality, symmetry of \(\Sigma_{YY}\) gives \(A^T=\Sigma_{YY}^{-1}\Sigma_{YX}\). Since \(X=\mu_X+A(Y-\mu_Y)+R\) and \(R\) is independent of \(Y\), conditioning on \(Y=y\) fixes the middle term and leaves the distribution of \(R\) unchanged. This proves both formulas. \(\square\)

The covariance reduction also has a useful check. Because \(\Sigma_{YY}\) is positive definite, its inverse is positive definite. For every \(a\in\mathbb{R}^p\),

a^T\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}a = (\Sigma_{YX}a)^T\Sigma_{YY}^{-1}(\Sigma_{YX}a) \geq 0.

Therefore \(\Sigma_{XX}-\operatorname{Cov}(X\mid Y)\) is positive semidefinite: conditioning cannot increase covariance in this setting. The conditional covariance itself must also be symmetric and positive semidefinite, since it is the covariance of the residual \(R\).

Worked Example: Reversing the Cross-Covariance Block

Let \(X=(X_1,X_2)^T\) and let \(Y\) be scalar, with zero means and covariance blocks

\Sigma_{XX}= \begin{pmatrix}4&1\\1&3\end{pmatrix}, \qquad \Sigma_{XY}= \begin{pmatrix}1\\2\end{pmatrix}, \qquad \Sigma_{YY}=2.

The gain is \(2\times1\), as required to multiply the scalar observation. The conditional mean is

E[X\mid Y=y] = \begin{pmatrix}1/2\\1\end{pmatrix}y.

For the conditional covariance, the rightmost block must be \(\Sigma_{YX}=\Sigma_{XY}^T=(1\ \ 2)\). Hence

\begin{aligned} \operatorname{Cov}(X\mid Y=y) &= \begin{pmatrix}4&1\\1&3\end{pmatrix} - \begin{pmatrix}1/2\\1\end{pmatrix} \begin{pmatrix}1&2\end{pmatrix}\\ &= \begin{pmatrix}4&1\\1&3\end{pmatrix} - \begin{pmatrix}1/2&1\\1&2\end{pmatrix} = \begin{pmatrix}7/2&0\\0&1\end{pmatrix}. \end{aligned}

The result is symmetric and positive definite. Treating \(\Sigma_{YX}\) as though it were another \(2\times1\) column would make the product dimensions inconsistent; silently using a column in its place is not a valid multiplication.

Transpose Errors That Survive Dimension Checks

When \(p=q\), a block-orientation error can pass a superficial dimension check because both cross-covariance blocks are square. But \(\Sigma_{XY}\) and \(\Sigma_{YX}\) are transposes, not interchangeable copies. In the covariance formula, the correct product is \(\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\), which is symmetric. Replacing the final block by \(\Sigma_{XY}\) can produce a nonsymmetric matrix.

Worked Example: A Missing Transpose Produces an Invalid Covariance

Take zero-mean two-dimensional vectors with

\Sigma_{XX}=I_2,\qquad \Sigma_{YY}=I_2,\qquad \Sigma_{XY}=A= \begin{pmatrix}0.3&0\\0.2&0.4\end{pmatrix}.

This is a valid positive definite joint covariance: the squared Frobenius norm of \(A\) is \(0.3^2+0.2^2+0.4^2=0.29<1\), so the largest singular value of \(A\) is less than one. The Schur complement \(I_2-A^TA\) is therefore positive definite, establishing positivity of the block covariance.

The correct conditional mean shift for \(Y=y\) is \(Ay\). For \(y=(1,-1)^T\), it is

Ay= \begin{pmatrix}0.3\\-0.2\end{pmatrix}.

The conditional covariance is

\begin{aligned} I_2-AA^T &= \begin{pmatrix}1&0\\0&1\end{pmatrix} - \begin{pmatrix}0.09&0.06\\0.06&0.20\end{pmatrix}\\ &= \begin{pmatrix}0.91&-0.06\\-0.06&0.80\end{pmatrix}. \end{aligned}

If the last \(A^T\) is omitted, the result becomes \(I_2-AA\), where

I_2-AA = \begin{pmatrix}0.91&0\\-0.14&0.84\end{pmatrix}.

This matrix is not symmetric, so it cannot be a covariance matrix. The error is exposed even though all the matrices in the mistaken product are \(2\times2\). The symmetry check catches what the dimension check alone cannot.

Singular Conditioning Blocks

An ordinary inverse is not available if \(\Sigma_{YY}\) is singular. That does not automatically make Gaussian conditioning undefined: a singular Gaussian vector is supported on a lower-dimensional affine subspace. Its conditional distribution is specified for conditioning values in that support. In this case the Moore–Penrose pseudoinverse, denoted \(\Sigma_{YY}^{+}\), replaces the inverse in the block formulas:

Key result: For jointly Gaussian \((X,Y)\) with possibly singular \(\Sigma_{YY}\), and \(y-\mu_Y\) in the range of \(\Sigma_{YY}\), the conditional mean and covariance are

\begin{aligned} E[X\mid Y=y] &=\mu_X+\Sigma_{XY}\Sigma_{YY}^{+}(y-\mu_Y),\\ \operatorname{Cov}(X\mid Y=y) &=\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{+}\Sigma_{YX}. \end{aligned}

The support condition matters: values outside \(\mu_Y+\operatorname{Range}(\Sigma_{YY})\) have probability zero under the Gaussian law, and the conditional distribution there is not determined by conditioning on an event the model can produce. Do not simply write an ordinary inverse or pretend that every \(y\) is possible.

Worked Example: Conditioning on a Singular Pair

Let \(Z\) and \(\varepsilon\) be independent \(N(0,1)\) variables, and set \(X=Z+\varepsilon\), \(Y=(Z,2Z)^T\). Then

\Sigma_{XX}=2,\qquad \Sigma_{XY}=(1\ \ 2),\qquad \Sigma_{YY}= \begin{pmatrix}1&2\\2&4\end{pmatrix}.

The determinant of \(\Sigma_{YY}\) is \(4-4=0\), so its ordinary inverse does not exist. Its pseudoinverse is

\Sigma_{YY}^{+} = \frac{1}{25} \begin{pmatrix}1&2\\2&4\end{pmatrix},

as follows from the rank-one form \(vv^T\), where \(v=(1,2)^T\) and \((vv^T)^+=vv^T/(v^Tv)^2=vv^T/25\). The support of \(Y\) consists of \(y=(y_1,y_2)^T\) with \(y_2=2y_1\). For such \(y\), the conditional mean is

\begin{aligned} \Sigma_{XY}\Sigma_{YY}^{+}y &=\frac{1}{25}(1\ \ 2) \begin{pmatrix}1&2\\2&4\end{pmatrix} \begin{pmatrix}y_1\\y_2\end{pmatrix}\\ &=\frac{1}{5}(y_1+2y_2)=y_1. \end{aligned}

The conditional variance is

2-(1\ \ 2)\frac{1}{25} \begin{pmatrix}1&2\\2&4\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} =2-1=1.

This agrees with the construction: observing \(Y\) reveals \(Z=y_1\), leaving only the independent noise \(\varepsilon\) in \(X\).

Marginal Covariance Is Not Conditional Covariance

The marginal covariance \(\Sigma_{XX}\) describes variation in \(X\) before \(Y\) is observed. The conditional covariance describes the remaining variation after observing \(Y\). For an invertible \(\Sigma_{YY}\), their difference is \(\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\), which is positive semidefinite. They agree only when the cross-covariance block is zero.

Worked Example: The Marginal Variance Overstates Remaining Uncertainty

Let \(X\) and \(Y\) be jointly Gaussian with zero means, unit marginal variances, and correlation \(0.8\). Thus \(\Sigma_{XX}=\Sigma_{YY}=1\) and \(\Sigma_{XY}=\Sigma_{YX}=0.8\). The conditional mean and variance are

E[X\mid Y=y]=0.8y, \qquad \operatorname{Var}(X\mid Y=y)=1-(0.8)(1)(0.8)=0.36.

The marginal variance of \(X\) is \(1\), not \(0.36\). Using \(1\) as the conditional variance ignores information in the observation. Conversely, \(0.36\) is the residual uncertainty after conditioning; it is not the variance of \(X\) before observing \(Y\).

A Practical Error Check

Before accepting a Gaussian conditioning calculation, verify the following in order:

Label block dimensions.
Write \(\Sigma_{XY}\) as \(p\times q\) and \(\Sigma_{YX}=\Sigma_{XY}^T\) as \(q\times p\).

Check the inverse assumption.
Use \(\Sigma_{YY}^{-1}\) only when \(\Sigma_{YY}\) is invertible. If it is singular, use the pseudoinverse formula on the support.

Check the output.
The mean shift must be \(p\)-dimensional; the conditional covariance must be \(p\times p\), symmetric, and positive semidefinite.

Keep the two covariances distinct.
Compare \(\Sigma_{XX}\), the marginal covariance, with the Schur-complement expression for conditional covariance.

A common mistake is to reason, “the covariance of \(X\) is \(\Sigma_{XX}\), so that is also its covariance given \(Y\).” The first statement describes the marginal distribution; it does not account for information supplied by \(Y\). The correction is to subtract the covariance explained by the cross-blocks. Another common mistake is to transpose a block until the product dimensions work. The correction is to retain the row-variable/column-variable definition of each block, then let the dimensions determine the multiplication.

Key takeaway: For conditioning \(X\) on \(Y\), the cross-block order is fixed: \(\Sigma_{XY}\) appears next to the conditional observation, and \(\Sigma_{YX}\) closes the covariance product. Check symmetry, positive semidefiniteness, invertibility, and whether the quantity sought is marginal or conditional.

Check Your Understanding

For each question, identify the error or justify the proposed calculation.

If \(X\in\mathbb{R}^3\) and \(Y\in\mathbb{R}^2\), what are the dimensions of \(\Sigma_{XY}\), the gain matrix, and \(\operatorname{Cov}(X\mid Y=y)\)?
Why is a nonsymmetric matrix an immediate warning that a proposed conditional covariance formula is wrong?
What must be checked before using \(\Sigma_{YY}^{-1}\) in a Gaussian conditioning formula?
In the singular example, which possible values of \(y=(y_1,y_2)^T\) lie in the support of \(Y\), and why?
For scalar jointly Gaussian \(X,Y\) with covariance \(\operatorname{Cov}(X,Y)=0\), how do the marginal and conditional variances of \(X\) compare?