Noisy observations of a Gaussian process
A Gaussian process model describes uncertain latent values, but measurements of those values are rarely exact. A sensor, assay, or recording device can add error even when the underlying process value is fixed. Conditioning on the recorded data must account for that error: it changes how strongly the data inform the latent process, and it contributes additional uncertainty to a future noisy measurement.
Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Let \((f(x))_{x\in\mathcal{T}}\) be a Gaussian process indexed by an input set \(\mathcal{T}\), with mean function \(m(x)=\mathbb{E}[f(x)]\) and covariance kernel \(k(x,x')=\operatorname{Cov}(f(x),f(x'))\). “Gaussian process” means that every finite vector of process values has a multivariate Gaussian distribution, possibly degenerate. Choose training inputs \(x_1,\ldots,x_n\) and define the latent training vector \(F_X=(f(x_1),\ldots,f(x_n))^\mathsf{T}\).
Suppose the observed vector is \(Y=F_X+\varepsilon\), where \(\varepsilon\) is independent of the entire process and has a Gaussian distribution with mean zero and covariance \(R=\operatorname{diag}(r_1,\ldots,r_n)\), with each \(r_i>0\). Thus the observation errors are mutually independent and have strictly positive variances. Write \(m_X=(m(x_i))_{i=1}^n\), \(K_{XX}=(k(x_i,x_j))_{i,j=1}^n\). For query inputs \(x_1^*,\ldots,x_q^*\), define \(F_*=(f(x_a^*))_{a=1}^q\), \(m_*\), \(K_{**}\), and \(K_{*X}=(k(x_a^*,x_i))_{a,i}\); set \(K_{X*}=K_{*X}^\mathsf{T}\).
Proof. The vector \((F_*,Y)\) is jointly Gaussian because it is an affine transformation of the jointly Gaussian vector \((F_*,F_X,\varepsilon)\). Its mean is \((m_*,m_X)\), and its covariance blocks are \(K_{**}\), \(K_{*X}\), \(K_{X*}\), and \(C=K_{XX}+R\). In particular, independence of \(\varepsilon\) and the process makes the cross-covariance between \(F_*\) and \(Y\) equal to \(K_{*X}\). For any nonzero column vector \(a\in\mathbb{R}^n\), $$ a^\mathsf{T}Ca =\operatorname{Var}(a^\mathsf{T}F_X)+\sum_{i=1}^n r_i a_i^2>0, $$ because the variance term is nonnegative and at least one \(a_i\ne0\) has \(r_i>0\). Hence \(C\) is positive definite and invertible. Define \(A=K_{*X}C^{-1}\) and \(Z=F_*-m_*-A(Y-m_X)\). Then \(\mathbb{E}Z=0\), and $$ \operatorname{Cov}(Z,Y)=K_{*X}-AC=0. $$ The jointly Gaussian vectors \(Z\) and \(Y\) are therefore independent: their joint Gaussian characteristic function has no cross-covariance term and factors into the product of their marginal characteristic functions. Thus, conditional on \(Y=y\), \(F_*=m_*+A(y-m_X)+Z\), giving the stated conditional mean. Expanding the covariance of \(Z\), using \(AC=K_{*X}\), gives $$ \operatorname{Cov}(Z) =K_{**}-AK_{X*}-K_{*X}A^\mathsf{T}+ACA^\mathsf{T} =K_{**}-K_{*X}C^{-1}K_{X*}. $$ This is the conditional covariance, proving both formulas. \(\square\)
The matrix \(R\) belongs in the covariance of the observations, not in the covariance of the latent process at the query inputs. In the mean formula, the residual \(y-m_X\) is adjusted by \(C^{-1}\): more measurement noise generally means less trust in the recorded values. The conditional covariance does not depend on the particular observed value \(y\), although the conditional mean does.
Worked Example: One Noisy Observation at the Query Input
Let \(f(x)\) have prior mean \(0\) and variance \(1\) at one input \(x\). Observe \(Y=f(x)+\varepsilon\), where \(\varepsilon\) is independent with distribution \(N(0,1)\), and suppose the observed value is \(y=2\). Find the posterior distribution of the latent value and the predictive distribution of a fresh noisy measurement at \(x\).
Here \(K_{XX}=(1)\), \(R=(1)\), and \(K_{*X}=K_{**}=(1)\). The observation covariance is \(C=1+1=2\), which is positive. The conditional mean and variance of the latent value are $$ 0+1\cdot \frac12\cdot(2-0)=1, \qquad 1-1\cdot\frac12\cdot1=\frac12. $$ Therefore \(f(x)\mid Y=2\) is Gaussian with mean \(1\) and variance \(\tfrac12\).
Let a fresh measurement be \(Y_{\mathrm{new}}=f(x)+\varepsilon_{\mathrm{new}}\), with independent noise of variance \(1\), also independent of the training data and process. Its conditional mean remains \(1\), and its conditional variance is \(\tfrac12+1=\tfrac32\). Check: the posterior latent mean satisfies \(1=0+\tfrac12(2-0)\), and the fresh measurement variance is the latent posterior variance \(\tfrac12\) plus its own noise variance \(1\), giving \(\tfrac32\).
Latent uncertainty and predictive uncertainty
For a new measurement, the target is not just the latent vector \(F_*\). Suppose \(Y_*=F_*+\varepsilon_*\), where \(\varepsilon_*\) is independent of the process, the training errors, and the training data, and has mean zero and covariance \(R_*\). Conditional on \(Y=y\), the mean of \(Y_*\) is the latent posterior mean, while its covariance is the latent posterior covariance plus \(R_*\). The independence assumption matters: if future measurement error is correlated with the training errors, this simple addition does not apply.
Proof. The observation covariance \(C=K_{XX}+R\) is positive definite, so \(C^{-1}\) is positive definite. For any \(b\in\mathbb{R}^q\), let \(v=K_{X*}b\). Then $$ b^\mathsf{T}K_{*X}C^{-1}K_{X*}b=v^\mathsf{T}C^{-1}v\ge0. $$ Subtracting this nonnegative quantity from \(b^\mathsf{T}K_{**}b\) gives the quadratic form of the posterior covariance, proving the claimed order. For the predictive covariance, condition on the training data and use \(Y_*=F_*+\varepsilon_*\). The conditional cross-covariance between \(F_*\) and \(\varepsilon_*\) is zero by independence, so the covariance of the sum is the sum of the two covariances. \(\square\)
Worked Example: Correlated Latent Values and a Noisy Training Point
Let the latent vector at inputs \(a,b\) have mean \((0,0)^\mathsf{T}\) and covariance $$ K_{XX}= \begin{bmatrix} 1&\tfrac12\\ \tfrac12&1 \end{bmatrix}. $$ Observe only the value at \(a\), with independent measurement noise of variance \(1\), and observe \(y=2\). Find the posterior mean and covariance of \((f(a),f(b))^\mathsf{T}\).
The displayed covariance is valid: its diagonal entries are \(1\), and its determinant is \(1-\tfrac14=\tfrac34>0\). The single observation has variance \(1+1=2\), and its covariance with the latent query vector is \((1,\tfrac12)^\mathsf{T}\). The mean update is $$ \begin{bmatrix}0\\0\end{bmatrix} +\begin{bmatrix}1\\\tfrac12\end{bmatrix}\frac12(2-0) = \begin{bmatrix}1\\\tfrac12\end{bmatrix}. $$ For the covariance, subtract the outer product correction: $$ \begin{bmatrix}1&\tfrac12\\\tfrac12&1\end{bmatrix} -\frac12 \begin{bmatrix}1&\tfrac12\\\tfrac12&\tfrac14\end{bmatrix} = \begin{bmatrix}\tfrac12&\tfrac14\\\tfrac14&\tfrac78\end{bmatrix}. $$ Thus the observation updates both locations: the posterior mean at \(b\) is \(\tfrac12\), and its latent posterior variance is \(\tfrac78\). Check: the variance at the observed location is \(1-\tfrac12=\tfrac12\); the variance at \(b\) is \(1-(\tfrac12)^2/2=\tfrac78\). The posterior covariance matrix has determinant \(\tfrac12\cdot\tfrac78-(\tfrac14)^2=\tfrac38>0\), consistent with a valid covariance matrix.
Worked Example: Nonzero Prior Mean and a Noisy Future Reading
Suppose the latent values at \(a,b\) have mean \((1,-1)^\mathsf{T}\) and covariance $$ \begin{bmatrix}4&2\\2&9\end{bmatrix}. $$ Observe \(Y=f(a)+\varepsilon\), with noise variance \(1\), and \(y=3\). Find the posterior distribution of \(f(b)\) and the predictive variance of a fresh measurement at \(b\) with independent noise variance \(\tfrac12\).
The prior covariance is positive definite because its first diagonal entry is \(4>0\) and its determinant is \(4\cdot9-2\cdot2=32>0\). The observation mean is \(1\), its variance is \(4+1=5\), and the covariance between \(f(b)\) and \(Y\) is \(2\). Hence $$ \mathbb{E}[f(b)\mid Y=3] =-1+\frac25(3-1) =-1+\frac45 =-\frac15, \qquad \operatorname{Var}(f(b)\mid Y=3) =9-\frac{2\cdot2}{5} =\frac{41}{5}. $$ A fresh reading at \(b\) has the same conditional mean \(-\tfrac15\) and variance $$ \frac{41}{5}+\frac12=\frac{82}{10}+\frac5{10}=\frac{87}{10}. $$ Check: the mean update uses the centered observation \(3-1=2\), not \(3\). The posterior variance reduction is \(4/5\), exactly \(2^2/5\), while the extra \(\tfrac12\) in the predictive variance is the new measurement error.
What the noise term does—and a common error
In applications, \(K_{XX}+R\) is sometimes called the noisy observation covariance. It is the covariance of \(Y\), not of \(F_X\). Positive measurement variances also ensure invertibility even if some training inputs coincide or \(K_{XX}\) is singular. If the noise variances are allowed to be zero, the displayed inverse formula requires \(K_{XX}+R\) to be invertible; singular noiseless conditioning needs separate care and is not covered by this positive-noise assumption.
A common mistake is to use the noiseless interpolation formula when the data are noisy. In the first example, using \(K_{XX}^{-1}=1\) instead of \((K_{XX}+R)^{-1}=\tfrac12\) would give latent posterior mean \(2\) and variance \(1-1=0\). That calculation treats the recorded value as an exact latent value, contradicting the assumed measurement error. The correct calculation uses observation variance \(2\), giving mean \(1\) and latent variance \(\tfrac12\). A second frequent error is to report \(\tfrac12\) as the variance of a fresh noisy reading: \(\tfrac12\) describes uncertainty about the latent value, and the fresh reading has additional variance \(1\), for total variance \(\tfrac32\).
The distinction is operational. If the goal is to estimate the underlying process, report the posterior distribution of \(F_*\). If the goal is to predict what a measurement device will record, include the future measurement-error covariance. The posterior covariance reduction result applies to latent uncertainty; it does not say that a future noisy measurement is less variable than the latent process before conditioning.
Check Your Understanding
For each question, identify whether the target is latent or observed and keep the training noise separate from future measurement noise.
- For a scalar prior variance \(3\) and independent observation-noise variance \(1\), what is the variance of the observation?
- In the theorem, why is \(K_{XX}+R\) invertible when every training noise variance is strictly positive, even if \(K_{XX}\) is singular?
- If the latent posterior variance at a query is \(\tfrac34\) and a fresh independent measurement has noise variance \(\tfrac14\), what is the predictive variance?
- Why does the posterior covariance formula not depend on the observed value \(y\), while the posterior mean generally does?
- In the correlated two-location example, which covariance entry represents the posterior covariance between the two latent values?