Deriving Gaussian Conditionals by Completing the Square

Why Completing the Square Gives a Conditional Gaussian

For a joint Gaussian vector, conditioning one block on another does not require guessing a new distribution. The joint density already contains the answer: hold the observed block fixed, collect the terms involving the block of interest, and complete the square. The resulting quadratic form is the kernel of a Gaussian density.

The calculation is useful because it shows exactly where the conditional mean and covariance come from. It also makes clear why nonsingularity matters: the square being completed must have a positive-definite quadratic coefficient.

Setup: Let \(X=(X_1^T,X_2^T)^T\) be jointly Gaussian, with \(X_1\) of dimension \(p\) and \(X_2\) of dimension \(q\). Write its mean and covariance as \[ \mu=\begin{pmatrix}\mu_1\\\mu_2\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}\Sigma_{11}&\Sigma_{12}\\\Sigma_{21}&\Sigma_{22}\end{pmatrix}, \] where \(\Sigma_{21}=\Sigma_{12}^T\). Assume \(\Sigma\) is positive definite. Define the precision matrix \(\Omega=\Sigma^{-1}\), partitioned conformably into blocks \(\Omega_{11},\Omega_{12},\Omega_{21},\Omega_{22}\).

Positive definiteness ensures that the joint density exists and that every principal block of \(\Omega\), including \(\Omega_{11}\), is positive definite. Thus \(\Omega_{11}^{-1}\) exists. For fixed \(x_2\), the centered values \(u=x_1-\mu_1\) and \(v=x_2-\mu_2\) turn the joint density’s exponent into a quadratic expression in \(u\).

Completing the Square in the Joint Density

The joint Gaussian density is proportional to the exponential of minus one half the centered quadratic form. Partitioning that form gives

\begin{pmatrix}u\\v\end{pmatrix}^{T} \Omega \begin{pmatrix}u\\v\end{pmatrix} = u^T\Omega_{11}u+2u^T\Omega_{12}v+v^T\Omega_{22}v.

The first two terms involve \(u\). Their square is found by shifting \(u\) by \(\Omega_{11}^{-1}\Omega_{12}v\):

u^T\Omega_{11}u+2u^T\Omega_{12}v = \bigl(u+\Omega_{11}^{-1}\Omega_{12}v\bigr)^T \Omega_{11} \bigl(u+\Omega_{11}^{-1}\Omega_{12}v\bigr) - v^T\Omega_{21}\Omega_{11}^{-1}\Omega_{12}v.

The last term here depends only on the fixed value \(x_2\). It therefore affects the density’s normalization as a function of \(x_1\), but not the location or shape of its Gaussian kernel. The center of the square is \(u=-\Omega_{11}^{-1}\Omega_{12}v\), and its quadratic coefficient is \(\Omega_{11}\).

Result: For a positive-definite joint covariance matrix, a version of the conditional density of \(X_1\) given \(X_2=x_2\) is \[ f_{X_1\mid X_2}(x_1\mid x_2) = \frac{|\Omega_{11}|^{1/2}}{(2\pi)^{p/2}} \exp\!\left[ -\frac12 \bigl(x_1-\mu_1+\Omega_{11}^{-1}\Omega_{12}(x_2-\mu_2)\bigr)^T \Omega_{11} \bigl(x_1-\mu_1+\Omega_{11}^{-1}\Omega_{12}(x_2-\mu_2)\bigr) \right]. \] Consequently, \[ X_1\mid X_2=x_2 \sim N\!\left( \mu_1-\Omega_{11}^{-1}\Omega_{12}(x_2-\mu_2), \ \Omega_{11}^{-1} \right). \]

To verify the normalizing constant, integrate the joint density over \(x_1\) with \(x_2\) fixed. Translating \(u\) by \(\Omega_{11}^{-1}\Omega_{12}v\) does not change the integral. The Gaussian integral for the positive-definite matrix \(\Omega_{11}\) is \((2\pi)^{p/2}|\Omega_{11}|^{-1/2}\). Dividing the joint density by this integral gives the displayed conditional density. This is the density ratio \(f_{X_1,X_2}(x_1,x_2)/f_{X_2}(x_2)\) wherever the marginal density is positive; here it is positive for every \(x_2\).

Reading the Parameters from Covariance Blocks

The completed square initially gives the answer in precision blocks. It can also be written using the covariance blocks. Set \[ B=\Sigma_{12}\Sigma_{22}^{-1}, \qquad C=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}. \] The inverses exist because \(\Sigma\) is positive definite, so its principal block \(\Sigma_{22}\) is positive definite.

The following factorization verifies how these blocks relate to the precision matrix:

\Sigma= \begin{pmatrix}I&B\\0&I\end{pmatrix} \begin{pmatrix}C&0\\0&\Sigma_{22}\end{pmatrix} \begin{pmatrix}I&0\\B^T&I\end{pmatrix}.

Indeed, the upper-right block of the product is \(B\Sigma_{22}=\Sigma_{12}\), and the upper-left block is \(C+B\Sigma_{22}B^T=\Sigma_{11}\); the lower blocks then agree by symmetry. The outer matrices are invertible. Since \(\Sigma\) is positive definite, this factorization also implies that \(C\) is positive definite. Inverting the factors gives

\Omega_{11}=C^{-1}, \qquad \Omega_{12}=-C^{-1}B.

Substitution into the completed-square result identifies the same conditional density with mean \(\mu_1+B(x_2-\mu_2)\) and covariance \(C\). This is an identification of the parameters in the square-completion calculation; the precision-block version remains the direct output of completing the square.

Worked Calculations

Worked Example: A Bivariate Conditional Density

Suppose \((X_1,X_2)^T\) is Gaussian with mean \((10,20)^T\) and covariance \[ \Sigma=\begin{pmatrix}4&3\\3&9\end{pmatrix}. \] Its determinant is \(4(9)-3^2=27>0\), and its first diagonal entry is positive, so this covariance matrix is positive definite. Find the conditional law of \(X_1\) when \(X_2=x_2\), then evaluate it at \(x_2=26\).

The inverse covariance is \[ \Omega=\frac1{27}\begin{pmatrix}9&-3\\-3&4\end{pmatrix} = \begin{pmatrix}1/3&-1/9\\-1/9&4/27\end{pmatrix}. \] Thus \(\Omega_{11}=1/3\), \(\Omega_{12}=-1/9\), and \(\Omega_{11}^{-1}=3\). Square completion gives conditional mean \[ 10-3(-1/9)(x_2-20)=10+\frac{x_2-20}{3} \] and conditional variance \(3\). At \(x_2=26\), the mean is \(10+6/3=12\). Therefore \(X_1\mid X_2=26\sim N(12,3)\).

As a covariance-block check, \(B=3/9=1/3\) and \(C=4-3(1/9)3=3\), giving the same mean and variance.

Worked Example: A Two-Dimensional Block Given a Scalar

Let \(X_1\) be two-dimensional and \(X_2\) scalar, with mean \(\mu_1=(1,-2)^T\), \(\mu_2=5\), and covariance \[ \Sigma=\begin{pmatrix}3&0&2\\0&5/4&-1\\2&-1&4\end{pmatrix}. \] To verify positive definiteness, take \(C=\begin{pmatrix}2&1/2\\1/2&1\end{pmatrix}\), \(\Sigma_{22}=4\), and \(B=(1/2,-1/4)^T\). The factorization above reconstructs this \(\Sigma\); \(C\) is positive definite since its first diagonal entry is positive and its determinant is \(2-1/4=7/4>0\).

From the factorization, \(\Omega_{11}=C^{-1}\). Direct inversion gives \[ \Omega_{11}=\frac{4}{7}\begin{pmatrix}1&-1/2\\-1/2&2\end{pmatrix} = \begin{pmatrix}4/7&-2/7\\-2/7&8/7\end{pmatrix}. \] Also \(C^{-1}B=(5/14,-3/7)^T\), so \(\Omega_{12}=-C^{-1}B=(-5/14,3/7)^T\). Completing the square therefore yields covariance \(\Omega_{11}^{-1}=C\) and mean \(\mu_1+B(x_2-5)\).

For \(x_2=7\), the deviation is \(2\), and the conditional mean is \[ \begin{pmatrix}1\\-2\end{pmatrix} +2\begin{pmatrix}1/2\\-1/4\end{pmatrix} = \begin{pmatrix}2\\-5/2\end{pmatrix}. \] Hence \(X_1\mid X_2=7\) has mean \((2,-5/2)^T\) and covariance \(\begin{pmatrix}2&1/2\\1/2&1\end{pmatrix}\).

Worked Example: A Scalar Given a Two-Dimensional Block

Suppose \(X_1\) is scalar and \(X_2\) is two-dimensional, with mean \(\mu_1=0\), \(\mu_2=(1,2)^T\), and covariance \[ \Sigma=\begin{pmatrix}4&2&-1\\2&4&0\\-1&0&1\end{pmatrix}. \] This matrix is positive definite: it has the factorization above with \(C=2\), \(\Sigma_{22}=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), and \(B=(1/2,-1)\). Each diagonal block is positive definite.

The precision-block coefficient for the scalar is \(\Omega_{11}=C^{-1}=1/2\), and \(\Omega_{12}=-C^{-1}B=(-1/4,1/2)\). For the observed value \(x_2=(3,1)^T\), the deviation is \(v=(2,-1)^T\). The completed-square center is \[ -\Omega_{11}^{-1}\Omega_{12}v =-2\left[(-1/4)(2)+(1/2)(-1)\right] =2. \] Thus \(X_1\mid X_2=(3,1)^T\) is Gaussian with mean \(2\) and variance \(\Omega_{11}^{-1}=2\). The covariance-block calculation agrees: \(Bv=(1/2)(2)+(-1)(-1)=2\), and \(C=2\).

Common Pitfall and Why the Derivation Matters

A common mistake is to read the coefficient of the quadratic term in the exponent as the covariance. In a Gaussian density, the exponent uses the inverse covariance: a kernel \(\exp[-(x-m)^TA(x-m)/2]\) has covariance \(A^{-1}\), not \(A\). In this derivation the coefficient is \(\Omega_{11}\), so the conditional covariance is \(\Omega_{11}^{-1}\).

Another tempting shortcut is to assume that the conditional mean is simply \(\mu_1\). The square shows the dependence on the observed block explicitly: its displacement \(x_2-\mu_2\) shifts the center by \(-\Omega_{11}^{-1}\Omega_{12}(x_2-\mu_2)\). When the cross-block contribution is zero, that shift vanishes; otherwise, it generally does not.

The method matters beyond providing a formula. It gives a direct way to derive and check conditional Gaussian laws from a joint density, including when either block has several coordinates. The positive-definiteness assumption guarantees that the completed square describes a proper density rather than a flat or unnormalizable kernel.

Key takeaway: Hold the observed block fixed, complete the quadratic form in the block of interest, and invert the resulting quadratic coefficient to obtain the conditional covariance. The center of that square is the conditional mean.

Check Your Understanding

Use the square-completion argument and state any assumptions needed.

In the precision-block expression, which matrix is the quadratic coefficient for \(x_1\), and what is the conditional covariance?
Why does completing the square shift \(x_1-\mu_1\) by a term involving \(x_2-\mu_2\)?
For a scalar pair with \(\Omega_{11}=2\) and \(\Omega_{12}=-1/2\), what are the conditional variance and the coefficient multiplying \(x_2-\mu_2\) in the conditional mean?
Why is \(\Omega_{11}\) invertible under the positive-definite joint covariance assumption?
Explain the error in claiming that a Gaussian kernel with quadratic coefficient \(A\) has covariance \(A\).