Conditional Mean and Covariance in Block Form

Why Block Conditioning Matters

In a multivariate model, the variables of interest are often divided into two groups: a target block whose values are unknown and an observed block whose values are known. A clinical study, for example, might use a patient's recorded measurements to predict a future biomarker. The Gaussian conditional formulas turn the joint mean and covariance of these blocks into a complete distribution for the unknown block, not just a point prediction.

The central practical task is to keep the blocks in the correct order. The conditional mean uses the covariance between the target and observed blocks, multiplied by the inverse covariance of the observed block. The conditional covariance does not depend on the particular observed value, although the conditional mean generally does.

Definition: Let $X_1$ be a $p$-dimensional target vector and $X_2$ a $q$-dimensional observed vector. Suppose their joint distribution is multivariate Gaussian with mean and covariance $$ \begin{pmatrix}X_1\\X_2\end{pmatrix} \sim N_{p+q}\left( \begin{pmatrix}\mu_1\\\mu_2\end{pmatrix}, \begin{pmatrix} \Sigma_{11}&\Sigma_{12}\\ \Sigma_{21}&\Sigma_{22} \end{pmatrix} \right). $$ Here $\Sigma_{12}=\operatorname{Cov}(X_1,X_2)$ and $\Sigma_{21}=\Sigma_{12}^{\mathsf T}$. If the joint covariance is positive definite, then $\Sigma_{22}$ is invertible, and for an observed value $X_2=x_2$, $$ X_1\mid X_2=x_2\sim N_p\left( \mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2), \ \Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} \right). $$

The first parameter is the conditional mean vector; the second is the conditional covariance matrix. The dimensions help prevent errors: $\Sigma_{12}$ is $p\times q$, $\Sigma_{22}^{-1}$ is $q\times q$, and $x_2-\mu_2$ is $q\times1$. Their product is a $p\times1$ adjustment to $\mu_1$. The conditional covariance is $p\times p$.

The statement conditions on the observed value $x_2$, rather than treating $X_2=x_2$ as an event with positive probability. For continuous variables, an individual point usually has probability zero; the conditional distribution is specified by the Gaussian conditional density. The positive-definiteness assumption ensures that the joint density and the required inverse matrices are well defined.

Deriving the Block Formula

The block formula is the partitioned version of the Gaussian conditional law obtained by completing the square. The derivation below makes explicit how the blocks determine the result. Set $$ C=\Sigma_{12}\Sigma_{22}^{-1}, \qquad A=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}. $$ The matrix $A$ will be the conditional covariance.

Result: Under the assumptions in the definition, the conditional density of $X_1$ given $X_2=x_2$ is the density of a Gaussian vector with mean $\mu_1+C(x_2-\mu_2)$ and covariance $A$.

To verify this, first factor the joint covariance matrix:

\begin{pmatrix} \Sigma_{11}&\Sigma_{12}\\ \Sigma_{21}&\Sigma_{22} \end{pmatrix} = \begin{pmatrix}I&C\\0&I\end{pmatrix} \begin{pmatrix}A&0\\0&\Sigma_{22}\end{pmatrix} \begin{pmatrix}I&0\\C^{\mathsf T}&I\end{pmatrix}.

Multiplying the right-hand side gives bottom-right block $\Sigma_{22}$, top-right block $C\Sigma_{22}=\Sigma_{12}$, bottom-left block $\Sigma_{21}$, and top-left block $A+C\Sigma_{22}C^{\mathsf T}=\Sigma_{11}$. Since the joint covariance is positive definite, this factorization implies that $A$ is positive definite. The triangular factors have determinant one, so the determinant of the joint covariance is $|A||\Sigma_{22}|$.

Write $u=x_1-\mu_1$ and $v=x_2-\mu_2$. Inverting the triangular factors shows that the quadratic form in the joint Gaussian density is

\begin{pmatrix}u\\v\end{pmatrix}^{\mathsf T} \begin{pmatrix}\Sigma_{11}&\Sigma_{12}\\\Sigma_{21}&\Sigma_{22}\end{pmatrix}^{-1} \begin{pmatrix}u\\v\end{pmatrix} = (u-Cv)^{\mathsf T}A^{-1}(u-Cv)+v^{\mathsf T}\Sigma_{22}^{-1}v.

Thus the joint density factors into a term proportional to $\exp\{-\tfrac12 v^{\mathsf T}\Sigma_{22}^{-1}v\}$, which depends only on $x_2$, and a term proportional to $\exp\{-\tfrac12 (u-Cv)^{\mathsf T}A^{-1}(u-Cv)\}$, which, as a function of $x_1$, is a Gaussian density with mean $\mu_1+Cv$ and covariance $A$. The determinant factorization supplies the matching normalizing constants. Dividing the joint density by the marginal density of $X_2$ therefore gives the stated conditional density. $\square$

Worked Block Calculations

Worked Example: Conditioning a Two-Variable Target Block

Suppose $X_1=(X_{11},X_{12})^{\mathsf T}$ is a two-dimensional target and $X_2$ is a scalar observation, with $$ \begin{pmatrix}X_1\\X_2\end{pmatrix} \sim N_3\left( \begin{pmatrix}10\\20\\5\end{pmatrix}, \begin{pmatrix} 4&1&2\\ 1&9&3\\ 2&3&4 \end{pmatrix} \right). $$ Find the conditional distribution of $X_1$ when $X_2=7$.

The blocks are $\mu_1=(10,20)^{\mathsf T}$, $\mu_2=5$, $\Sigma_{11}=\begin{pmatrix}4&1\\1&9\end{pmatrix}$, $\Sigma_{12}=\begin{pmatrix}2\\3\end{pmatrix}$, and $\Sigma_{22}=4$. The conditional mean is $$ \begin{pmatrix}10\\20\end{pmatrix} + \begin{pmatrix}2\\3\end{pmatrix}\frac{1}{4}(7-5) = \begin{pmatrix}10\\20\end{pmatrix} + \begin{pmatrix}1\\1.5\end{pmatrix} = \begin{pmatrix}11\\21.5\end{pmatrix}. $$

For the conditional covariance, calculate the adjustment explicitly: $$ \begin{pmatrix}2\\3\end{pmatrix}\frac{1}{4} \begin{pmatrix}2&3\end{pmatrix} = \begin{pmatrix}1&1.5\\1.5&2.25\end{pmatrix}. $$ Therefore, $$ \Sigma_{1\mid2} = \begin{pmatrix}4&1\\1&9\end{pmatrix} - \begin{pmatrix}1&1.5\\1.5&2.25\end{pmatrix} = \begin{pmatrix}3&-0.5\\-0.5&6.75\end{pmatrix}. $$ The conditional covariance is positive definite: its top-left entry is $3>0$, and its determinant is $3(6.75)-0.5^2=20>0$. Hence the answer is a valid bivariate Gaussian distribution with the mean and covariance just calculated.

Worked Example: Conditioning on a Two-Dimensional Observation

Let $X_1$ be a scalar target and $X_2=(X_{21},X_{22})^{\mathsf T}$ a two-dimensional observed block. Suppose $$ \begin{pmatrix}X_1\\X_2\end{pmatrix} \sim N_3\left( \begin{pmatrix}50\\0\\0\end{pmatrix}, \begin{pmatrix} 16&2&-1\\ 2&4&0\\ -1&0&1 \end{pmatrix} \right). $$ Find the conditional distribution when $X_2=(2,-1)^{\mathsf T}$.

Here $\Sigma_{12}=(2,-1)$ and $\Sigma_{22}=\begin{pmatrix}4&0\\0&1\end{pmatrix}$, so $\Sigma_{22}^{-1}=\begin{pmatrix}1/4&0\\0&1\end{pmatrix}$. The mean adjustment is $$ (2,-1) \begin{pmatrix}1/4&0\\0&1\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} = (0.5,-1)\begin{pmatrix}2\\-1\end{pmatrix} =1+1=2. $$ Thus $E[X_1\mid X_2=(2,-1)^{\mathsf T}]=50+2=52$.

The covariance adjustment is $$ (2,-1) \begin{pmatrix}1/4&0\\0&1\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} =2. $$ Since $\Sigma_{11}=16$, the conditional variance is $16-2=14$. The result is $X_1\mid X_2=(2,-1)^{\mathsf T}\sim N(52,14)$. This covariance matrix is valid because the variance $14$ is positive; the joint covariance is positive definite as well, since its corresponding conditional variance and $\Sigma_{22}$ are both positive definite.

Conditional Prediction Distributions

When the target block is a future quantity, its conditional Gaussian law is a prediction distribution. The conditional mean gives the center of the distribution, while the conditional covariance describes uncertainty about the future quantity after the observed block is known. It is a distribution, not merely a point prediction.

A prediction may also concern an affine summary of the target block, such as a total, contrast, or weighted score. The following result gives its conditional distribution directly.

Result: If $Z=a+BX_1$, where $a$ is a fixed vector and $B$ is a fixed matrix, then, conditional on $X_2=x_2$, $$ Z\mid X_2=x_2\sim N\left( a+B\{\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2)\}, \ B\{\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\}B^{\mathsf T} \right). $$ The resulting Gaussian distribution may be degenerate if $B$ does not have full row rank.

To prove the result, the conditional law of $X_1$ is Gaussian by the block formula. Applying the affine transformation $z=a+Bx_1$ to its conditional density (or, equivalently, to its Gaussian characteristic function) gives mean $a+B E[X_1\mid X_2=x_2]$ and covariance $B\operatorname{Var}(X_1\mid X_2=x_2)B^{\mathsf T}$. Substitution of the block conditional mean and covariance gives the displayed expression. This argument also covers a singular covariance for $Z$, for which the Gaussian law is supported on a lower-dimensional affine set. $\square$

Worked Example: A Conditional Prediction Interval

Suppose a future scalar outcome $Y$ and two recorded measurements $W=(W_1,W_2)^{\mathsf T}$ satisfy $$ \begin{pmatrix}Y\\W\end{pmatrix} \sim N_3\left( \begin{pmatrix}5\\0\\0\end{pmatrix}, \begin{pmatrix} 10&2&3\\ 2&4&0\\ 3&0&9 \end{pmatrix} \right). $$ Find the prediction distribution for $Y$ when $W=(2,3)^{\mathsf T}$, and give an approximate central 95% prediction interval.

The observed-block inverse is $\Sigma_{22}^{-1}=\operatorname{diag}(1/4,1/9)$. The conditional mean is $$ 5+(2,3) \begin{pmatrix}1/4&0\\0&1/9\end{pmatrix} \begin{pmatrix}2\\3\end{pmatrix} = 5+(0.5,1/3)\begin{pmatrix}2\\3\end{pmatrix} =5+1+1=7. $$ The conditional variance is $$ 10-(2,3) \begin{pmatrix}1/4&0\\0&1/9\end{pmatrix} \begin{pmatrix}2\\3\end{pmatrix} =10-(1+1)=8. $$ Thus $Y\mid W=(2,3)^{\mathsf T}\sim N(7,8)$. Its standard deviation is $\sqrt{8}\approx2.828$. Using $1.96$ standard deviations for a central 95% interval gives $$ 7\pm1.96\sqrt{8} \approx 7\pm5.544 $$ or approximately $(1.46,12.54)$. This interval describes uncertainty about the future outcome conditional on the recorded measurements.

Common Mistakes and Practical Checks

The covariance blocks are not interchangeable. In the conditional mean, the factor is $\Sigma_{12}\Sigma_{22}^{-1}$, not $\Sigma_{21}\Sigma_{22}^{-1}$; the latter generally has incompatible dimensions. For the conditional covariance, the subtraction is $\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$, which is $p\times p$, as required.

Another common error is to replace the conditional covariance by the covariance of the observed block, or to assume it depends on the observed value. The observed value changes the conditional mean through $x_2-\mu_2$, but it does not appear in the conditional covariance formula. Nor should the conditional variance be confused with the squared standard deviation: in a univariate Gaussian notation $N(\mu,\sigma^2)$, the second argument is the variance.

A useful calculation check is to verify dimensions at every multiplication and confirm that the resulting conditional covariance is symmetric. Under the stated positive-definiteness assumption it is also positive definite. If a computed matrix is not symmetric, or a computed variance is negative, check the block ordering, transpose, and matrix multiplication before interpreting the result.

Key takeaway: To condition a Gaussian target block on an observed block, adjust the target mean by $\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2)$ and subtract $\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ from its covariance. Keep the block dimensions visible; they encode the correct order of the calculation.

Check Your Understanding

Use the block formulas and show the matrix dimensions or calculations that support your answers.

In the conditional mean formula, what are the dimensions of $\Sigma_{12}$, $\Sigma_{22}^{-1}$, and $x_2-\mu_2$ when $X_1$ has dimension $p$ and $X_2$ has dimension $q$?
Suppose $X_1$ is scalar, $\Sigma_{11}=9$, $\Sigma_{12}=(1,2)$, and $\Sigma_{22}=\operatorname{diag}(1,4)$. Calculate the conditional variance of $X_1$.
Why does changing $x_2$ change the conditional mean but not the conditional covariance?
If $Z=(1,-1)X_1$ for a two-dimensional target block, how do you obtain the conditional variance of $Z$ from the conditional covariance of $X_1$?
Name two checks that can reveal an error in a block conditional covariance calculation.