Gaussian Conditioning With Measurement Noise

Why Measurement Noise Changes Conditioning

A linear observation of a Gaussian vector does not usually reveal the vector exactly. Instead, the recorded value combines a linear function of the unknown vector with measurement error. The conditional distribution must account for both the relationship between the vector and the observation and the amount of uncertainty contributed by that error.

The central calculation is a Gaussian conditioning problem for the pair consisting of the unknown vector and its noisy observation. The observation covariance includes a contribution from the vector itself and a contribution from the measurement noise. Consequently, changing the noise covariance changes the weight given to the observation.

Definition: Let $X\in\mathbb{R}^p$ have distribution $N(\mu,\Sigma)$, where $\Sigma$ is positive semidefinite. Let $\varepsilon\in\mathbb{R}^m$ be independent of $X$, with distribution $N(0,R)$, where $R$ is positive definite. For a fixed $m\times p$ matrix $H$, the noisy linear observation is $Y=HX+\varepsilon$. Its covariance matrix is $V=H\Sigma H^T+R$.

Because $R$ is positive definite, $V$ is positive definite and hence invertible. This lets us use the ordinary Gaussian conditional formula even when $\Sigma$ itself is singular. The realized observation will be denoted by $y$, and its discrepancy from the prior mean prediction $H\mu$ is $y-H\mu$.

The Conditional Distribution

The observation $Y$ has mean $H\mu$. Its covariance has two terms: $H\Sigma H^T$, the variability in $HX$, and $R$, the variability in the measurement error. The covariance between $X$ and $Y$ is $\Sigma H^T$, since the independent noise contributes no cross-covariance.

Theorem: Under the definition above, the conditional distribution of $X$ given $Y=y$ is Gaussian, with $$ E[X\mid Y=y]=\mu+\Sigma H^T(H\Sigma H^T+R)^{-1}(y-H\mu) $$ and $$ \operatorname{Var}(X\mid Y=y) =\Sigma-\Sigma H^T(H\Sigma H^T+R)^{-1}H\Sigma. $$ These formulas hold for every $y\in\mathbb{R}^m$, using the Gaussian conditional distribution specified by the joint law.

The matrix $\Sigma H^T V^{-1}$ maps the observation discrepancy into an adjustment to the prior mean. It depends on the prior covariance, the observation map, and the noise covariance. The conditional covariance does not depend on the realized value $y$: for this Gaussian model, different observation values change the conditional mean but not the conditional uncertainty.

Form the joint Gaussian vector.
$(X,Y)$ is an affine transformation of the independent Gaussian vectors $X$ and $\varepsilon$, so it is jointly Gaussian.

Calculate its covariance blocks.
The covariance of $Y$ is $V=H\Sigma H^T+R$, and the cross-covariance of $X$ and $Y$ is $\Sigma H^T$.

Apply Gaussian conditioning.
Use the block conditional mean and covariance formulas with observation mean $H\mu$ and invertible observation covariance $V$.

For completeness, write the joint mean and covariance as $$ E\begin{bmatrix}X\\Y\end{bmatrix} =\begin{bmatrix}\mu\\H\mu\end{bmatrix}, \qquad \operatorname{Var}\begin{bmatrix}X\\Y\end{bmatrix} = \begin{bmatrix} \Sigma & \Sigma H^T\\ H\Sigma & H\Sigma H^T+R \end{bmatrix}. $$ The block Gaussian conditional formula gives conditional mean equal to the first-block mean plus the cross-covariance times the inverse of the second-block covariance times the observation discrepancy. Substitution gives $$ \mu+\Sigma H^T V^{-1}(y-H\mu). $$ The conditional covariance is the first-block covariance minus the cross-covariance times $V^{-1}$ times its transpose: $$ \Sigma-\Sigma H^T V^{-1}H\Sigma. $$ Since $V$ is positive definite, all these products are defined. The joint Gaussian property and the block conditioning formula therefore establish both claimed expressions. $\square$

Worked Example: A Scalar Measurement With Substantial Noise

Suppose $X\sim N(10,4)$, and we observe $Y=X+\varepsilon$, where $\varepsilon\sim N(0,9)$ independently of $X$. If the recorded value is $y=13$, find the conditional distribution of $X$.

Here $H=1$, the prior mean is $10$, and the observation variance is $$ V=1(4)1+9=13. $$ The observation discrepancy is $13-10=3$, and the covariance between $X$ and $Y$ is $4$. Therefore the conditional mean is $$ 10+\frac{4}{13}(3)=10+\frac{12}{13}=\frac{142}{13}\approx 10.923. $$ The conditional variance is $$ 4-\frac{4(4)}{13}=4-\frac{16}{13}=\frac{36}{13}\approx 2.769. $$ Thus $X\mid Y=13$ has distribution $N(142/13,36/13)$. The mean moves from $10$ toward the observed value $13$, but does not reach it: the observation has variance $9$, so it is not treated as an exact measurement.

Worked Example: Two Components Observed Through Their Sum

Let $X=(X_1,X_2)^T\sim N((1,-1)^T,\Sigma)$, where $$ \Sigma=\begin{bmatrix}4&1\\1&2\end{bmatrix}. $$ We observe $Y=X_1+X_2+\varepsilon$, with independent $\varepsilon\sim N(0,1)$. Find the conditional mean and covariance when $Y=2$.

The observation matrix is $H=\begin{bmatrix}1&1\end{bmatrix}$. First calculate the variance of $Y$: $$ H\Sigma H^T =\begin{bmatrix}1&1\end{bmatrix} \begin{bmatrix}4&1\\1&2\end{bmatrix} \begin{bmatrix}1\\1\end{bmatrix} =8, \qquad V=8+1=9. $$ The prior predicted observation is $H\mu=1+(-1)=0$, so the discrepancy is $2$. Also, $$ \Sigma H^T =\begin{bmatrix}4&1\\1&2\end{bmatrix} \begin{bmatrix}1\\1\end{bmatrix} =\begin{bmatrix}5\\3\end{bmatrix}. $$ The conditional mean is therefore $$ \begin{bmatrix}1\\-1\end{bmatrix} +\begin{bmatrix}5\\3\end{bmatrix}\frac{2}{9} = \begin{bmatrix}14/9\\-1/3\end{bmatrix}. $$ For the conditional covariance, subtract the cross-covariance adjustment: $$ \begin{bmatrix}4&1\\1&2\end{bmatrix} -\frac{1}{9} \begin{bmatrix}5\\3\end{bmatrix} \begin{bmatrix}5&3\end{bmatrix} = \begin{bmatrix}11/9&-2/3\\-2/3&1\end{bmatrix}. $$ Hence the conditional distribution has mean $(14/9,-1/3)^T$ and the covariance matrix displayed above. The observation directly measures the sum, but both components shift because their prior covariance links them to that sum.

What the Noise Covariance Does

The effect of measurement noise is visible in the inverse $V^{-1}$ in both update formulas. When noise is large in a direction that the observation measures, variation in that direction is less informative about $X$. For multiple observations, off-diagonal entries of $R$ matter too: correlated errors mean that the observations do not supply as much independent information as their count alone might suggest.

Worked Example: Correlated Errors in Two Measurements

Suppose $X\sim N(0,1)$, and two measurements are $Y_1=X+\varepsilon_1$ and $Y_2=X+\varepsilon_2$. The noise is independent of $X$ and has covariance $$ R=\begin{bmatrix}1&1/2\\1/2&1\end{bmatrix}. $$ For the observed vector $y=(1,0)^T$, calculate the conditional distribution of $X$.

Here $H=(1,1)^T$, so $$ V=H H^T+R =\begin{bmatrix}2&3/2\\3/2&2\end{bmatrix}. $$ Its determinant is $4-9/4=7/4$, giving $$ V^{-1} =\frac{4}{7}\begin{bmatrix}2&-3/2\\-3/2&2\end{bmatrix} =\begin{bmatrix}8/7&-6/7\\-6/7&8/7\end{bmatrix}. $$ The cross-covariance between $X$ and $Y$ is $(1,1)$. Thus the mean adjustment is $$ (1,1)V^{-1}\begin{bmatrix}1\\0\end{bmatrix} =(2/7,2/7)\begin{bmatrix}1\\0\end{bmatrix} =2/7. $$ The conditional variance is $$ 1-(1,1)V^{-1}\begin{bmatrix}1\\1\end{bmatrix} =1-(1,1)\begin{bmatrix}2/7\\2/7\end{bmatrix} =3/7. $$ Therefore $X\mid Y=(1,0)^T\sim N(2/7,3/7)$. The positive correlation between the errors makes the two measurements partly redundant. In contrast, with independent unit-variance errors, the same observation vector would give mean $1/3$ and variance $1/3$. These are different updates because the noise covariance is different.

A precise comparison is possible when two noise covariance matrices are ordered. For symmetric matrices, write $R_1\preceq R_2$ when $R_2-R_1$ is positive semidefinite. This means that the second noise covariance is at least as large in every linear direction.

Theorem: Fix $\Sigma$ and $H$, and let $R_1$ and $R_2$ be positive definite with $R_1\preceq R_2$. Write $C(R)=\Sigma-\Sigma H^T(H\Sigma H^T+R)^{-1}H\Sigma$ for the conditional covariance. Then $C(R_1)\preceq C(R_2)$.

To prove the comparison, put $A=H\Sigma H^T+R_1$ and $D=R_2-R_1$. The matrix $A$ is positive definite and $D$ is positive semidefinite. Define $B=A^{-1/2}DA^{-1/2}$, which is positive semidefinite. Since $A+D=A^{1/2}(I+B)A^{1/2}$, $$ A^{-1}-(A+D)^{-1} =A^{-1/2}\{I-(I+B)^{-1}\}A^{-1/2}. $$ By the spectral theorem, the eigenvalues of $B$ are nonnegative. The eigenvalues of $I-(I+B)^{-1}$ are therefore $\lambda/(1+\lambda)\geq 0$, so $A^{-1}-(A+D)^{-1}$ is positive semidefinite. Subtracting the two conditional covariance formulas gives $$ C(R_2)-C(R_1) =\Sigma H^T\{A^{-1}-(A+D)^{-1}\}H\Sigma. $$ For any vector $u$, the quadratic form of this matrix at $u$ is the quadratic form of the positive semidefinite middle matrix at $H\Sigma u$, and hence is nonnegative. Thus $C(R_1)\preceq C(R_2)$. $\square$

Common Mistake and Practical Interpretation

A common mistake is to use $H\Sigma H^T$ as the observation covariance while forgetting $R$. That treats the recorded value as if it were the exact linear quantity $HX$. In the scalar example, omitting the noise would replace $13$ by $4$ in the denominator, producing a different and overly strong update. The correct observation covariance is always $H\Sigma H^T+R$ under this model.

The ordering result also needs careful interpretation. It compares conditional covariances, not the realized conditional means. A matrix comparison such as $C(R_1)\preceq C(R_2)$ does not say that one vector mean is larger or smaller than another. It says that the corresponding conditional variances of every linear contrast $a^TX$ are no greater under $R_1$, since $a^TC(R_1)a\leq a^TC(R_2)a$ for every vector $a$.

Key takeaway: For a noisy Gaussian linear observation, use the full observation covariance $H\Sigma H^T+R$. The resulting conditional mean adjusts the prior prediction according to the measurement's uncertainty, while the conditional covariance quantifies the remaining uncertainty.

Check Your Understanding

Use the update formulas and the assumptions stated in this tutorial.

In $Y=HX+\varepsilon$, why is the observation covariance $H\Sigma H^T+R$ rather than $H\Sigma H^T$ alone?
For a scalar prior with variance $5$, an exact linear observation coefficient $H=1$, and noise variance $5$, what is the coefficient multiplying the observation discrepancy in the conditional mean?
In a vector observation, what role does the cross-covariance $\Sigma H^T$ play in the conditional mean?
Why can positively correlated measurement errors make two observations partly redundant?
If $R_1\preceq R_2$, what does the conditional covariance comparison say about the variance of an arbitrary linear contrast $a^TX$?

Gaussian Conditioning With Measurement Noise

What You'll Learn