Tutorials › Statistical Theory I - Probability and Distribution Theory › Vector Gaussian Conditioning With Matrix Calculations

Home  /  Tutorials  /  Gaussian Conditioning and Linear Maps

Gaussian Conditioning and Linear Maps · Tutorial 728 of 994

Vector Gaussian Conditioning With Matrix Calculations

Learn to calculate and verify the full conditional Gaussian law by working through the matrix inverse, gain, mean, and Schur complement.

Advanced 10 min read

What You'll Learn

  • Identify the response, predictor, and covariance blocks in a joint Gaussian model.
  • Calculate the conditional mean using the cross-covariance and predictor covariance.
  • Compute the conditional covariance as a Schur complement.
  • Check matrix dimensions and verify conditional covariance calculations.
  • Interpret the conditional law at a specified predictor value.

From Scalar Conditioning to Vector Calculations

When both a response and its predictors are vectors, conditioning still gives a Gaussian distribution, but the scalar regression coefficient becomes a matrix. The calculation is not just a matter of replacing numbers with matrices: block order, transpose, and multiplication order all matter. A small error in any of these can change every entry of the answer.

This tutorial works through the full calculation for bivariate response and predictor vectors. The central tasks are to identify the covariance blocks, invert the predictor covariance, compute the conditional mean at an observed predictor value, and subtract the appropriate Schur-complement term to obtain the conditional covariance.

Setup: Let \(Y\in\mathbb{R}^p\) be a response vector and \(X\in\mathbb{R}^q\) a predictor vector. Suppose their joint distribution is Gaussian, with mean and covariance
$$ \begin{pmatrix}Y\\X\end{pmatrix} \sim N\left( \begin{pmatrix}\mu_Y\\\mu_X\end{pmatrix}, \begin{pmatrix} \Sigma_{YY}&\Sigma_{YX}\\ \Sigma_{XY}&\Sigma_{XX} \end{pmatrix} \right), \qquad \Sigma_{XY}=\Sigma_{YX}^{\mathsf T}. $$
Here \(\Sigma_{XX}\) is assumed positive definite, so it is invertible. The block \(\Sigma_{YX}\) has \(p\) rows and \(q\) columns.

The Conditional Gaussian Formula

For a specified predictor value \(x\in\mathbb{R}^q\), the conditional distribution of \(Y\) has a mean vector and a covariance matrix. The order of the matrices in both formulas follows from their dimensions: the cross-covariance \(\Sigma_{YX}\) maps predictor changes to response changes.

Theorem: Under the setup above, a version of the conditional distribution of \(Y\) given \(X=x\) is Gaussian, with
$$ Y\mid X=x\sim N\left( \mu_Y+\Sigma_{YX}\Sigma_{XX}^{-1}(x-\mu_X),\ \Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} \right). $$
The formula applies at every \(x\) using the Gaussian conditional distribution defined by the model; since \(\Sigma_{XX}\) is positive definite, \(X\) has a density on \(\mathbb{R}^q\). The conditional covariance does not depend on the particular observed value \(x\).

To justify the formula, define the \(p\times q\) matrix \(B=\Sigma_{YX}\Sigma_{XX}^{-1}\), and form the residual vector \(R=Y-\mu_Y-B(X-\mu_X)\). Its cross-covariance with \(X\) is

$$ \operatorname{Cov}(R,X) =\Sigma_{YX}-B\Sigma_{XX} =\Sigma_{YX}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XX} =0. $$

The pair \((R,X)\) is jointly Gaussian because it is an affine transformation of the jointly Gaussian pair \((Y,X)\). For jointly Gaussian vectors, zero cross-covariance implies independence: their joint characteristic function has exponent determined by the covariance matrix, and a block-diagonal covariance makes that characteristic function the product of the two marginal characteristic functions. Thus \(R\) and \(X\) are independent.

The mean of \(R\) is zero. Its covariance is obtained by expanding the covariance of the residual:

$$ \begin{aligned} \operatorname{Var}(R) &=\Sigma_{YY}-B\Sigma_{XY}-\Sigma_{YX}B^{\mathsf T} +B\Sigma_{XX}B^{\mathsf T}\\ &=\Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}. \end{aligned} $$

In the last equality, \(\Sigma_{XX}\) and its inverse are symmetric, so \(B^{\mathsf T}=\Sigma_{XX}^{-1}\Sigma_{XY}\). The two negative terms are each \(\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}\), and the final positive term is one copy of that matrix. Finally, \(Y=\mu_Y+B(X-\mu_X)+R\). Since \(R\) is independent of \(X\), conditioning on \(X=x\) fixes the first two terms and leaves the Gaussian residual distribution unchanged. This gives both the conditional mean and covariance in the theorem. \(\square\)

Worked Bivariate Calculations

Each example uses the same sequence: calculate \(\Sigma_{XX}^{-1}\), multiply to obtain the regression matrix \(B\), evaluate the conditional mean, and calculate the Schur complement for the conditional covariance. Keeping intermediate matrices visible makes arithmetic and dimensions easier to check.

Worked Example: A Nondiagonal Predictor Covariance

Suppose \(Y=(Y_1,Y_2)^{\mathsf T}\) and \(X=(X_1,X_2)^{\mathsf T}\) have means \(\mu_Y=(1,-2)^{\mathsf T}\), \(\mu_X=(0,1)^{\mathsf T}\), and covariance blocks

$$ \Sigma_{YY}=\begin{pmatrix}5&1\\1&4\end{pmatrix},\quad \Sigma_{YX}=\begin{pmatrix}2&1\\-1&2\end{pmatrix},\quad \Sigma_{XX}=\begin{pmatrix}4&1\\1&3\end{pmatrix}. $$

Find the conditional distribution at \(x=(2,0)^{\mathsf T}\). First, the determinant of \(\Sigma_{XX}\) is \(4\cdot3-1\cdot1=11\), so

$$ \Sigma_{XX}^{-1}=\frac1{11}\begin{pmatrix}3&-1\\-1&4\end{pmatrix}. $$

Multiplying in the required order gives

$$ B=\Sigma_{YX}\Sigma_{XX}^{-1} =\frac1{11}\begin{pmatrix}2&1\\-1&2\end{pmatrix} \begin{pmatrix}3&-1\\-1&4\end{pmatrix} =\frac1{11}\begin{pmatrix}5&2\\-5&9\end{pmatrix}. $$

The predictor deviation is \(x-\mu_X=(2,-1)^{\mathsf T}\). Therefore

$$ B(x-\mu_X) =\frac1{11}\begin{pmatrix}5&2\\-5&9\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} =\frac1{11}\begin{pmatrix}10-2\\-10-9\end{pmatrix} =\begin{pmatrix}8/11\\-19/11\end{pmatrix}. $$

Adding this to \(\mu_Y\) gives conditional mean \((19/11,-41/11)^{\mathsf T}\). For the covariance, calculate the subtracted term:

$$ \begin{aligned} \Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} &=\frac1{11}\begin{pmatrix}5&2\\-5&9\end{pmatrix} \begin{pmatrix}2&-1\\1&2\end{pmatrix}\\ &=\frac1{11}\begin{pmatrix}12&-1\\-1&23\end{pmatrix}. \end{aligned} $$

Thus the conditional covariance is \(\begin{pmatrix}5&1\\1&4\end{pmatrix} -\frac1{11}\begin{pmatrix}12&-1\\-1&23\end{pmatrix} =\frac1{11}\begin{pmatrix}43&12\\12&21\end{pmatrix}\). It is symmetric, and its determinant is \((43\cdot21-12^2)/121=759/121>0\), consistent with a positive definite conditional covariance.

Worked Example: Diagonal Predictor Covariance

Let \(\mu_Y=(0,1)^{\mathsf T}\), \(\mu_X=(1,-1)^{\mathsf T}\), and

$$ \Sigma_{YY}=\begin{pmatrix}5&1\\1&4\end{pmatrix},\quad \Sigma_{YX}=\begin{pmatrix}1&2\\2&-1\end{pmatrix},\quad \Sigma_{XX}=\begin{pmatrix}2&0\\0&5\end{pmatrix}. $$

At \(x=(3,2)^{\mathsf T}\), the inverse and regression matrix are

$$ \Sigma_{XX}^{-1}=\begin{pmatrix}1/2&0\\0&1/5\end{pmatrix}, \qquad B=\Sigma_{YX}\Sigma_{XX}^{-1} =\begin{pmatrix}1/2&2/5\\1&-1/5\end{pmatrix}. $$

Since \(x-\mu_X=(2,3)^{\mathsf T}\),

$$ B(x-\mu_X) =\begin{pmatrix}1/2&2/5\\1&-1/5\end{pmatrix} \begin{pmatrix}2\\3\end{pmatrix} =\begin{pmatrix}1+6/5\\2-3/5\end{pmatrix} =\begin{pmatrix}11/5\\7/5\end{pmatrix}. $$

The conditional mean is therefore \((11/5,12/5)^{\mathsf T}\). For the covariance reduction, multiply the cross-covariance by its transpose with the inverse between them:

$$ \begin{aligned} \Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} &=\begin{pmatrix}1&2\\2&-1\end{pmatrix} \begin{pmatrix}1/2&0\\0&1/5\end{pmatrix} \begin{pmatrix}1&2\\2&-1\end{pmatrix}\\ &=\begin{pmatrix}13/10&3/5\\3/5&11/5\end{pmatrix}. \end{aligned} $$

Subtracting from \(\Sigma_{YY}\) gives conditional covariance \(\begin{pmatrix}37/10&2/5\\2/5&9/5\end{pmatrix}\). Its determinant is \((37/10)(9/5)-(2/5)^2=333/50-8/50=13/2>0\); its diagonal entries are positive, so it is positive definite.

Worked Example: Identity Cross-Covariance

Take \(\mu_Y=(4,-1)^{\mathsf T}\), \(\mu_X=(-1,2)^{\mathsf T}\), with

$$ \Sigma_{YY}=\begin{pmatrix}3&1\\1&2\end{pmatrix},\quad \Sigma_{YX}=\begin{pmatrix}1&0\\0&1\end{pmatrix},\quad \Sigma_{XX}=\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$

The predictor covariance has determinant \(3\), and

$$ \Sigma_{XX}^{-1}=\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}, \qquad B=\Sigma_{YX}\Sigma_{XX}^{-1} =\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}. $$

For \(x=(0,1)^{\mathsf T}\), the deviation is \((1,-1)^{\mathsf T}\), so

$$ B(x-\mu_X)=\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =\begin{pmatrix}1\\-1\end{pmatrix}. $$

The conditional mean is \((5,-2)^{\mathsf T}\). Because \(\Sigma_{YX}\) is the identity, the covariance reduction is just \(\Sigma_{XX}^{-1}\), yielding

$$ \Sigma_{Y\mid x} =\begin{pmatrix}3&1\\1&2\end{pmatrix} -\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix} =\begin{pmatrix}7/3&4/3\\4/3&4/3\end{pmatrix}. $$

Its determinant is \(28/9-16/9=4/3>0\), and its diagonal entries are positive. The conditional covariance is thus positive definite.

Checks That Prevent Matrix Errors

The mean calculation is a matrix version of linear prediction. The deviation \(x-\mu_X\) is a \(q\)-vector; \(B\) is \(p\times q\); and the resulting adjustment is a \(p\)-vector that can be added to \(\mu_Y\). The covariance calculation does not involve \(x\): it describes the remaining uncertainty in the response after conditioning.

The conditional covariance must be symmetric. Its symmetry follows because \(\Sigma_{XX}^{-1}\) is symmetric and \(\Sigma_{XY}=\Sigma_{YX}^{\mathsf T}\). In numerical work, also check that the diagonal entries are nonnegative and, for a \(2\times2\) positive definite result, that the determinant is positive. Such checks do not replace correct algebra, but they can reveal a transposed block or multiplication mistake.

Common pitfall: Multiplying matrices entry by entry, reversing the cross-covariance block, or applying an inverse to the wrong side. The correct mean adjustment is \(\Sigma_{YX}\Sigma_{XX}^{-1}(x-\mu_X)\), not \(\Sigma_{XX}^{-1}\Sigma_{YX}(x-\mu_X)\). The latter generally has incompatible dimensions. Check the dimensions before calculating: \(p\times q\), then \(q\times q\), then \(q\times1\), producing \(p\times1\).

A second common error is to subtract \(\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{YX}\) when the last factor should be \(\Sigma_{XY}\), the transpose block. The covariance correction must be \(p\times p\) and symmetric. Writing out all three factors before multiplying helps preserve both the block order and the dimensions.

The formulas are useful whenever a Gaussian model separates variables into a response block and an observed predictor block. They provide the exact conditional distribution, not only a point prediction: the mean gives the response expected at the specified predictor value, while the covariance quantifies the remaining uncertainty jointly across response components.

Check Your Understanding

For each problem, keep the order of the covariance blocks explicit.

  1. For \(Y\in\mathbb{R}^2\) and \(X\in\mathbb{R}^2\), what are the dimensions of \(\Sigma_{YX}\), \(\Sigma_{XX}^{-1}\), and the conditional mean adjustment?
  2. Why does the conditional covariance in the Gaussian formula not depend on the observed value \(x\)?
  3. If \(\Sigma_{YX}=\begin{pmatrix}1&0\\0&1\end{pmatrix}\) and \(\Sigma_{XX}=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), calculate \(\Sigma_{YX}\Sigma_{XX}^{-1}\).
  4. What symmetry check should hold for \(\Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}\), and which block relation guarantees it?
  5. In Worked Example 2, verify the first component of the mean adjustment by calculating the first row of \(B\) dotted with \(x-\mu_X\).

Previous

Next

Back to Tutorials