Noiseless Gaussian Process Interpolation

Exact observations in a Gaussian process

A Gaussian process prediction is different when the data are exact rather than noisy: at an input where the process value has been observed, the conditional prediction must equal that value, with no remaining uncertainty. Away from the observed inputs, both the conditional mean and covariance depend on how the covariance kernel links those inputs to the rest of the process. The finite-dimensional Gaussian conditioning formula makes these effects explicit.

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, let \(T\) be an index set, and let \((f(x))_{x\in T}\) be a Gaussian process. This means every finite vector of process values is jointly Gaussian. Write \(m(x)=\mathbb{E}[f(x)]\) for its mean function and \(k(x,x')=\operatorname{Cov}(f(x),f(x'))\) for its covariance kernel. Choose distinct observed inputs \(x_1,\ldots,x_n\), and regard \(y_i\) as the exact observed value at \(x_i\). Define \(Y=(f(x_1),\ldots,f(x_n))^{\mathsf T}\), \(m_X=(m(x_1),\ldots,m(x_n))^{\mathsf T}\), and the covariance matrix \(K\) by \(K_{ij}=k(x_i,x_j)\).

For prediction at distinct inputs \(z_1,\ldots,z_r\), write \(Z=(f(z_1),\ldots,f(z_r))^{\mathsf T}\), \(m_Z=(m(z_1),\ldots,m(z_r))^{\mathsf T}\), \(K_{ZZ}=(k(z_a,z_b))_{a,b}\), and \(C=(k(z_a,x_i))_{a,i}\). If \(K\) is invertible, the conditional distribution given the exact data \(Y=y\) is

The theorem describes exact conditioning: the data enter through the residual \(y-m_X\), and the matrix \(C K^{-1}\) determines how that residual affects predictions. The covariance does not depend on the observed values \(y\). If an input is repeated in the prediction list, the same formula applies to that coordinate; it does not require all prediction inputs to be distinct from the observed ones.

Proof. Set \(R=Z-m_Z-C K^{-1}(Y-m_X)\). The vector \((R,Y)\) is jointly Gaussian because it is an affine transformation of a jointly Gaussian vector. Since \(\operatorname{Cov}(Y,Y)=K\) and \(\operatorname{Cov}(Z,Y)=C\),

For jointly Gaussian vectors, zero cross-covariance implies independence, so \(R\) and \(Y\) are independent. Also \(\mathbb{E}[R]=0\), and direct expansion gives \(\operatorname{Cov}(R)=K_{ZZ}-C K^{-1}C^{\mathsf T}\). Thus \(Z=m_Z+C K^{-1}(Y-m_X)+R\), where the residual \(R\) is independent of \(Y\), has mean zero and has the stated covariance. Conditioning on \(Y=y\) therefore gives the asserted Gaussian mean and covariance. \(\square\)

The covariance formula is positive semidefinite because it is the covariance matrix of \(R\). It may be singular: in particular, a prediction at an observed input has zero conditional variance. Invertibility of \(K\) matters. If distinct listed inputs nevertheless give a singular \(K\), the inverse formula cannot be used as written. One can remove redundant observations when they are consistent. More generally, the standard singular-Gaussian conditioning formula uses \(K^{+}\), the Moore–Penrose pseudoinverse, and requires \(y-m_X\) to lie in the range of \(K\); observations outside that range are incompatible with the model. Here the examples use invertible observed covariance matrices.

Why exact conditioning interpolates

Proof. For \(z_a=x_j\), the row of \(C\) corresponding to \(z_a\) is the \(j\)-th row of \(K\), namely \(e_j^{\mathsf T}K\), where \(e_j\) is the \(j\)-th coordinate vector. Its conditional mean is therefore \(m(x_j)+e_j^{\mathsf T}K K^{-1}(y-m_X)=m(x_j)+e_j^{\mathsf T}(y-m_X)=y_j\). The corresponding diagonal entry of \(K_{ZZ}\) is \(K_{jj}\). The conditional variance at this input is \(K_{jj}-e_j^{\mathsf T}K K^{-1}K e_j=K_{jj}-K_{jj}=0\). Applying the same calculation to each observed coordinate in the prediction list proves the final assertion. \(\square\)

This is interpolation in the precise sense that the conditional distribution assigns probability one to the observed value at each observed input. It does not say that the conditional mean equals the unknown process at every unobserved input, nor does it say that the conditional variance must be zero away from the data. Those depend on the covariance structure.

Common calculation errors and interpretation

A frequent error is to multiply by \(K\) where the formula requires \(K^{-1}\). In the finite-kernel example, \(Ky=(\tfrac12,-\tfrac12)^{\mathsf T}\). Using this incorrect vector would give the prediction mean \((\tfrac14,\tfrac12)(\tfrac12,-\tfrac12)^{\mathsf T}=\tfrac18-\tfrac14=-\tfrac18\), rather than the correct value \(-\tfrac12\). The correction is to solve \(K\alpha=y-m_X\), equivalently compute \(\alpha=K^{-1}(y-m_X)\), before multiplying by \(C\).

A second error is to report the prior variance \(K_{ZZ}\) as the conditional variance. Exact observations reduce uncertainty through the subtraction \(C K^{-1}C^{\mathsf T}\). At an observed input, that subtraction equals the prior variance exactly, leaving zero. The covariance reduction is not an arbitrary adjustment: it is the covariance of the residual after removing the part linearly predicted from the observations, as shown in the proof of the conditioning formula.

The interpolation property depends on conditioning on the process values themselves. If recorded values contain measurement error, then the observed quantities are not identical to \(f(x_i)\), and exact interpolation need not hold. For exact data, however, the formula gives a direct diagnostic: at every observed input, check that the predicted mean reproduces the data and the conditional variance is zero.