A Two-Observation Gaussian Process Calculation

From Noisy Observations to Two Predictions

A Gaussian process calculation becomes concrete when the training and test inputs are fixed. The prior covariance between those inputs determines a joint Gaussian distribution; measurement noise changes the covariance of the observations, but it does not change the prior covariance of the latent test values. Conditioning the joint distribution then gives posterior means and a covariance matrix for the test values. Keeping these pieces separate is essential: the posterior covariance describes uncertainty about the latent process, not uncertainty about a future noisy measurement.

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, and let \((F(x))_{x\in\mathcal{X}}\) be a Gaussian process on an input set \(\mathcal{X}\). This means that every finite vector of process values has a multivariate Gaussian distribution. Write its mean function as \(m(x)=\mathbb{E}[F(x)]\), and its covariance kernel as \(k(x,x')=\operatorname{Cov}(F(x),F(x'))\). At two distinct training inputs \(x_1,x_2\), suppose the observations are \(Y_i=F(x_i)+\epsilon_i\), where the errors are independent \(N(0,\sigma^2)\) variables, independent of the process, and \(\sigma^2>0\). We want to predict the latent values \(F(z_1),F(z_2)\) at two test inputs.

The observation covariance is invertible under the stated positive-noise assumption. Indeed, \(K\) is a covariance matrix, so \(v^{\mathsf T}Kv\ge0\) for every vector \(v\). Thus for any nonzero \(v\), \(v^{\mathsf T}Cv=v^{\mathsf T}Kv+\sigma^2v^{\mathsf T}v>0\). The calculation below applies to a general Gaussian process, then uses a simple kernel that makes every entry exact.

Why the Conditioning Formula Works

Proof. Put \(X=(F(x_1),F(x_2))^{\mathsf T}\), \(Z=(F(z_1),F(z_2))^{\mathsf T}\), and \(Y=X+\epsilon\), where \(\epsilon=(\epsilon_1,\epsilon_2)^{\mathsf T}\). Finite-dimensional Gaussianity and independence of the Gaussian noise imply that \((Z,Y)\) is jointly Gaussian. Its means are \(m_*\) and \(m_X\). Independence of \(\epsilon\) from the process gives \(\operatorname{Cov}(Y)=K+\sigma^2 I=C\) and \(\operatorname{Cov}(Z,Y)=B\).

Define the residual \(R=Z-m_*-BC^{-1}(Y-m_X)\). It is a linear transformation of a jointly Gaussian vector, hence is jointly Gaussian with \(Y\). Directly expanding its covariance with \(Y\) gives \(\operatorname{Cov}(R,Y)=B-BC^{-1}C=0\). Jointly Gaussian vectors with zero cross-covariance are independent, the standard independence property of Gaussian vectors. Therefore \(R\) is independent of \(Y\), and its conditional mean given \(Y=y\) is its unconditional mean, zero. Rearranging the definition of \(R\) gives the stated conditional mean. Expanding the residual covariance gives \(\operatorname{Cov}(R)=K_*-BC^{-1}B^{\mathsf T}\): the two cross terms are each \(BC^{-1}B^{\mathsf T}\), while the covariance of \(BC^{-1}(Y-m_X)\) is \(BC^{-1}CC^{-1}B^{\mathsf T}=BC^{-1}B^{\mathsf T}\). Since \(R\) is independent of \(Y\), this residual covariance is also the conditional covariance. \(\square\)

Proof. The residual \(R\) in the theorem has covariance \(K_*-BC^{-1}B^{\mathsf T}\). For any fixed vector \(a\), \(a^{\mathsf T}(K_*-BC^{-1}B^{\mathsf T})a=\operatorname{Var}(a^{\mathsf T}R)\ge0\). This is exactly the definition of positive semidefiniteness. \(\square\)

A Kernel with Exact Covariances

For the examples, take \(\mathcal{X}=\mathbb{R}\), zero prior mean, and \(k(x,x')=1+xx'\). This is a valid covariance kernel: for any finite inputs \(u_1,\ldots,u_n\) and real coefficients \(a_1,\ldots,a_n\),

Thus every finite covariance matrix generated by this kernel is positive semidefinite. Equivalently, the process can be represented as \(F(x)=A+Bx\), where \(A\) and \(B\) are independent standard normal variables. The examples use training inputs \(-1,1\) and test inputs \(0,2\). With \(k(x,x')=1+xx'\), the training covariance entries are \(k(-1,-1)=2\), \(k(-1,1)=0\), \(k(1,-1)=0\), and \(k(1,1)=2\). The test covariance entries are \(k(0,0)=1\), \(k(0,2)=1\), \(k(2,0)=1\), and \(k(2,2)=5\). The cross-covariance entries, with test inputs in rows and training inputs in columns, are \(k(0,-1)=1\), \(k(0,1)=1\), \(k(2,-1)=-1\), and \(k(2,1)=3\).

A Common Error: Leaving Noise Out

The observation covariance is not just \(K\). It is \(K+\sigma^2I\), because the observed values include independent measurement errors. In the first worked example, omitting the noise would incorrectly use \(K^{-1}=\tfrac12I\) instead of \(C^{-1}=\tfrac25I\). The resulting incorrect mean at \(0\) would be \((1,1)\tfrac12(1,2)^{\mathsf T}=\tfrac32\), rather than the correct \(\tfrac65\). The incorrect variance at \(0\) would be \(1-(1,1)\tfrac12 I(1,1)^{\mathsf T}=1-\tfrac12(2)=0\), rather than \(\tfrac15\). This is not a harmless change in notation: it treats the noisy observations as if they were exact latent process values.

Another useful distinction is between a latent prediction and a future noisy observation. The variance calculated above is for \(F(z_r)\). If a new independent measurement \(F(z_r)+\epsilon_{\mathrm{new}}\) is predicted, its variance is the latent posterior variance plus \(\sigma^2\). Its covariance with a latent value at a different test input is unchanged, because the new measurement error is independent. Always identify which quantity is being predicted before reporting uncertainty.