Conditioning a Gaussian Process at Finite Inputs

From a Gaussian Process to a Finite Conditional Law

A Gaussian process describes a collection of random variables, but a finite prediction question involves only finitely many of them. Choose observation inputs and test inputs, collect the corresponding process values into vectors, and their joint distribution is multivariate Gaussian. Conditioning then reduces to a block-matrix calculation. The essential bookkeeping is to keep the observation and test dimensions straight, and to distinguish an invertible observation covariance from a singular one.

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, let \(\mathcal{X}\) be an input set, and let \((X(x))_{x\in\mathcal{X}}\) be a real Gaussian process: every finite vector of its values has a multivariate Gaussian distribution, possibly singular. Write its mean function as \(m(x)=\mathbb{E}[X(x)]\) and covariance function as \(k(x,x')=\operatorname{Cov}(X(x),X(x'))\). Select \(n\) observation inputs \(x_1,\ldots,x_n\) and \(r\) test inputs \(z_1,\ldots,z_r\). Repeated inputs are allowed, though they can make the observation covariance singular.

Use column vectors for process values and their means. Define the observation vector \(Y=(X(x_1),\ldots,X(x_n))^{\mathsf T}\) and the test vector \(Z=(X(z_1),\ldots,X(z_r))^{\mathsf T}\). Their means are \(m_o\in\mathbb{R}^n\) and \(m_*\in\mathbb{R}^r\). Define the covariance blocks

Thus \(A_{ij}=k(x_i,x_j)\), \(B_{\ell i}=k(z_\ell,x_i)\), and \(C_{\ell q}=k(z_\ell,z_q)\). In particular, the block in the upper-right corner of the joint covariance is \(B^{\mathsf T}\), not \(B\). The full joint distribution is

Here \(\mathcal{N}\) denotes a multivariate Gaussian law, including a degenerate law when its covariance is singular. Conditioning on exact observed values means specifying \(Y=y\). If \(A\) is invertible, \(Y\) has a density and every \(y\in\mathbb{R}^n\) is in its support. If \(A\) is singular, only values in the support of \(Y\) are possible. In either case, the conditional law at a specified value is understood as a regular conditional distribution; for a nonsingular Gaussian vector the usual density-based conditional formula gives a version defined for every \(y\).

Proof. First suppose \(A\) is invertible. Define the residual vector \(R=Z-m_*-BA^{-1}(Y-m_o)\). Its mean is zero. By the covariance definitions and symmetry of covariance,

The vectors \(R\) and \(Y\) are jointly Gaussian because they are affine transformations of the jointly Gaussian vector \((Y,Z)\). For jointly Gaussian vectors, zero cross-covariance implies independence; this follows, for example, by factoring their joint Gaussian characteristic function when the cross-covariance block is zero. Hence \(R\) is independent of \(Y\). The identity \(Z=m_*+BA^{-1}(Y-m_o)+R\) now shows that, given \(Y=y\), the first two terms are fixed and \(R\) retains its unconditional Gaussian law. This gives the asserted conditional mean and covariance. The covariance is positive semidefinite because it is \(\operatorname{Cov}(R,R)\).

For singular \(A\), define \(A^+\) by diagonalizing the symmetric positive semidefinite matrix \(A\): invert each positive eigenvalue and leave each zero eigenvalue at zero. If \(v\) is in the null space of \(A\), then \(\operatorname{Var}(v^{\mathsf T}Y)=v^{\mathsf T}Av=0\), so \(v^{\mathsf T}(Y-m_o)=0\) almost surely. Its covariance with \(Z\) is \(Bv\), which must therefore be zero. Thus \(Bv=0\) for every \(v\) in the null space of \(A\), and consequently \(BA^+A=B\). Set \(R=Z-m_*-BA^+(Y-m_o)\). Direct calculation gives \(\operatorname{Cov}(R,Y)=B-BA^+A=0\). The same joint-Gaussian independence argument shows \(R\) is independent of \(Y\), while its covariance is \(C-BA^+B^{\mathsf T}\). For \(y-m_o\) in the range of \(A\), the identity \(AA^+(y-m_o)=y-m_o\) gives the conditional representation and formula. If \(y-m_o\) is outside that range, it has probability zero under a law supported on \(m_o+\operatorname{range}(A)\), and is not a possible observation. \(\square\)

Computing the Conditional Distribution

The block formula can be used directly once the inputs, means, and covariance blocks are specified. A useful way to understand its correction term is through the residual in the proof: \(BA^{-1}(Y-m_o)\) is the part of the test vector explained linearly by the observations, and the remaining residual is independent of them. The covariance subtracts the covariance explained by that linear term. The following examples show the calculation, including checks of dimensions and results.

Interpretation and a Common Error

The conditional distribution is itself Gaussian, but it is a distribution for the finite test vector, not a deterministic set of fitted values. The conditional mean gives the center of that distribution; the conditional covariance describes the remaining joint uncertainty at the test inputs. Its off-diagonal entries matter: they describe how uncertainty at different test inputs varies together after conditioning. The covariance may be singular, for example if test values are repeated or determined exactly by the observations; a Gaussian distribution need not have a density to be a valid conditional law.

A common mistake is to add the covariance correction instead of subtracting it. In the first example, this incorrect calculation gives \(3+2\cdot\tfrac14\cdot2=4\). But the residual variance is \(3+1-2=2\), and the correct formula gives \(3-2\cdot\tfrac14\cdot2=2\). Conditioning removes the part of test variation explained by the observations; it cannot increase the covariance in this fixed jointly Gaussian setup. Another frequent error is reversing the cross-covariance block: \(B\) is \(r\times n\), so it acts on the \(n\)-vector \(A^{-1}(y-m_o)\) to produce an \(r\)-vector. This is why the update is \(BA^{-1}(y-m_o)\).

Before applying the formula, check that the stated joint covariance is symmetric positive semidefinite, that \(A\) is invertible before using \(A^{-1}\), and that a singular-observation value lies in the support before using \(A^+\). For a singular \(A\), inconsistent repeated observations such as \(Y=(1,2)^{\mathsf T}\) in the last example have probability zero and cannot be treated as an ordinary conditioning event. The pseudoinverse does not make impossible data possible; it gives the conditional law only on the support.