Conditioning on Linear Gaussian Measurements

From a Latent Vector to a Noisy Measurement

A measurement often reports only a linear summary of an unknown vector, and it rarely records that summary without error. Examples include several noisy sensors measuring different combinations of underlying signals, or repeated laboratory readings whose errors are correlated. In these settings, the central question is: after observing the measurement, what is the distribution of the latent vector?

The Gaussian model makes this question tractable. The measurement contributes information about the latent vector, shifting its conditional mean and reducing or reshaping its uncertainty. The observation matrix determines which directions are measured, while the error covariance determines how reliable and mutually related the measurements are.

Definition: Let $X\in\mathbb R^p$ be a latent random vector with $X\sim N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite. A noisy linear measurement of $X$ is $$ Y=AX+\varepsilon, $$ where $A$ is an $m\times p$ matrix and $\varepsilon\sim N_m(0,R)$ is independent of $X$, with $R$ positive definite. The observed value of $Y$ is denoted by $y$.

The positive definiteness assumptions ensure that the prior and measurement-error densities exist and that the measurement covariance $S=A\Sigma A^\mathsf T+R$ is invertible. The errors may be correlated: this is represented by nonzero off-diagonal entries of $R$. Independence is assumed between the entire error vector $\varepsilon$ and $X$, not between the separate components of $\varepsilon$.

The Conditional Distribution

The observed measurement has mean $A\mu$ and covariance $S=A\Sigma A^\mathsf T+R$. The cross-covariance between $X$ and $Y$ is $\Sigma A^\mathsf T$. These blocks determine the conditional law.

Theorem: Under the definition above, for every $y\in\mathbb R^m$, $$ X\mid Y=y\sim N_p\!\left( \mu+\Sigma A^\mathsf T S^{-1}(y-A\mu), \ \Sigma-\Sigma A^\mathsf T S^{-1}A\Sigma \right), \qquad S=A\Sigma A^\mathsf T+R. $$ The conditional mean adjusts the prior mean according to the measurement residual $y-A\mu$; the conditional covariance describes the remaining uncertainty.

Here is a direct derivation. Write $Z=Y-A\mu=A(X-\mu)+\varepsilon$, and define $K=\Sigma A^\mathsf T S^{-1}$ and $U=(X-\mu)-KZ$. The pair $(U,Z)$ is jointly Gaussian because it is an affine transformation of the independent Gaussian vectors $X$ and $\varepsilon$. Its cross-covariance is

\operatorname{Cov}(U,Z) =\operatorname{Cov}(X-\mu,Z)-K\operatorname{Var}(Z) =\Sigma A^\mathsf T-KS =0.

For jointly Gaussian vectors, zero cross-covariance implies independence. Thus $U$ and $Z$ are independent. Expanding the variance of $U$, using $\operatorname{Cov}(X-\mu,Z)=\Sigma A^\mathsf T$, gives

\operatorname{Var}(U) =\Sigma-\Sigma A^\mathsf T S^{-1}A\Sigma.

Since $X=\mu+KZ+U$, conditioning on $Y=y$ fixes $Z$ at $y-A\mu$, while the independent $U$ retains its unconditional distribution. Therefore the conditional mean is $\mu+K(y-A\mu)$, and the conditional covariance is $\operatorname{Var}(U)$, as claimed. $\square$

An Equivalent Precision Form

When both $\Sigma$ and $R$ are positive definite, the same conditional distribution can be derived by multiplying the prior density by the measurement likelihood. This gives a useful form in which the information from the prior and measurement adds through their precision matrices.

Result: Define $$ Q=\Sigma^{-1}+A^\mathsf T R^{-1}A, \qquad h=\Sigma^{-1}\mu+A^\mathsf T R^{-1}y. $$ Then $$ X\mid Y=y\sim N_p(Q^{-1}h,Q^{-1}). $$

To derive this expression, the prior density as a function of $x$ is proportional to $\exp[-\tfrac12(x-\mu)^\mathsf T\Sigma^{-1}(x-\mu)]$, and the likelihood of $y$ given $x$ is proportional to $\exp[-\tfrac12(y-Ax)^\mathsf T R^{-1}(y-Ax)]$. Their product is proportional to the conditional density of $x$ given $y$. The terms depending on $x$ in the combined exponent are

-\frac12\left[ x^\mathsf T Qx-2x^\mathsf T h \right] = -\frac12(x-Q^{-1}h)^\mathsf T Q(x-Q^{-1}h) +\frac12 h^\mathsf T Q^{-1}h.

The final term does not depend on $x$ and is absorbed into the normalizing constant. Since $Q$ is positive definite, the remaining expression is the kernel of a Gaussian density with mean $Q^{-1}h$ and covariance $Q^{-1}$. This establishes the precision form. It is algebraically equivalent to the covariance form above; the covariance form makes the role of the measurement residual especially visible, while the precision form emphasizes the addition of prior and measurement information. $\square$

Worked Updates

Worked Example: One Noisy Measurement of a Scalar

Suppose $X\sim N(10,4)$ and the observed measurement is $Y=X+\varepsilon$, where $\varepsilon\sim N(0,1)$ independently of $X$. If $y=12$, find the conditional distribution of $X$.

Here $A=1$, $\mu=10$, $\Sigma=4$, and $R=1$. Thus $S=A\Sigma A^\mathsf T+R=4+1=5$, and the measurement residual is $y-A\mu=12-10=2$. The conditional mean is

10+4\left(\frac{1}{5}\right)2 =10+\frac85 =11.6.

The conditional variance is

4-4\left(\frac{1}{5}\right)4 =4-\frac{16}{5} =\frac45.

Therefore $X\mid Y=12\sim N(11.6,0.8)$. As a check, the precision form gives variance $(1/4+1)^{-1}=4/5$ and mean $(4/5)(10/4+12)=11.6$.

Worked Example: Two Measurements with Correlated Errors

Let $X\sim N_2((0,0)^\mathsf T,I_2)$, let $A=I_2$, and suppose the measurement-error covariance is $$ R=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}. $$ For the observed vector $y=(2,0)^\mathsf T$, calculate the conditional distribution of $X$.

Because the measurement is $Y=X+\varepsilon$, its covariance is $$ S=I_2+R=\begin{pmatrix}2&1/2\\1/2&2\end{pmatrix}, \qquad S^{-1}=\frac1{15}\begin{pmatrix}8&-2\\-2&8\end{pmatrix}. $$ The determinant of $S$ is $4-1/4=15/4$, which verifies this inverse. The conditional mean is $S^{-1}y$, so

\begin{pmatrix}0\\0\end{pmatrix} +\frac1{15}\begin{pmatrix}8&-2\\-2&8\end{pmatrix} \begin{pmatrix}2\\0\end{pmatrix} = \begin{pmatrix}16/15\\-4/15\end{pmatrix}.

The conditional covariance is $I_2-S^{-1}$: $$ \begin{pmatrix}1&0\\0&1\end{pmatrix} -\frac1{15}\begin{pmatrix}8&-2\\-2&8\end{pmatrix} = \begin{pmatrix}7/15&2/15\\2/15&7/15\end{pmatrix}. $$ Thus $X\mid Y=y$ is bivariate normal with the displayed mean and covariance. In particular, the second component of the conditional mean is negative even though its observed measurement is zero. The positively correlated errors mean that a high first reading also makes a positive error in the second reading plausible; accounting for that association adjusts the estimate of the second latent component downward.

Worked Example: Measuring a Sum of Correlated Components

Suppose $$ X\sim N_2\!\left( \begin{pmatrix}1\\-1\end{pmatrix}, \begin{pmatrix}2&1\\1&2\end{pmatrix} \right), \qquad Y=X_1+X_2+\varepsilon, \qquad \varepsilon\sim N(0,1), $$ with $\varepsilon$ independent of $X$. Given $y=4$, find the conditional mean and covariance of $X$.

The measurement matrix is $A=(1\ \ 1)$. The prior mean predicts a measurement of $A\mu=1+(-1)=0$, so the residual is $4$. Also, $$ \Sigma A^\mathsf T =\begin{pmatrix}2&1\\1&2\end{pmatrix} \begin{pmatrix}1\\1\end{pmatrix} =\begin{pmatrix}3\\3\end{pmatrix}, \qquad S=A\Sigma A^\mathsf T+R=6+1=7. $$ Hence the conditional mean is $$ \begin{pmatrix}1\\-1\end{pmatrix} +\frac17\begin{pmatrix}3\\3\end{pmatrix}4 = \begin{pmatrix}19/7\\5/7\end{pmatrix}. $$ For the covariance, first calculate $$ \Sigma A^\mathsf T A\Sigma = \begin{pmatrix}3\\3\end{pmatrix} \begin{pmatrix}3&3\end{pmatrix} = \begin{pmatrix}9&9\\9&9\end{pmatrix}. $$ Therefore $$ \operatorname{Var}(X\mid Y=4) = \begin{pmatrix}2&1\\1&2\end{pmatrix} -\frac17\begin{pmatrix}9&9\\9&9\end{pmatrix} = \begin{pmatrix}5/7&-2/7\\-2/7&5/7\end{pmatrix}. $$ The measured sum is informative about the direction $(1,1)^\mathsf T$, but it does not directly measure the contrast between the two components. The negative conditional covariance reflects the remaining uncertainty in how a given sum is divided between them.

Common Pitfalls and Interpretation

Ignoring correlated measurement errors. Replacing a non-diagonal $R$ by a diagonal matrix changes $S$, the update to the mean, and the conditional covariance. The off-diagonal entries encode how errors move together and must be retained unless independence is justified.
Updating with the raw observation instead of its residual. The relevant quantity is $y-A\mu$, not simply $y$. A measurement equal to its prior-predicted value produces no change in the conditional mean.
Confusing measurement covariance with conditional covariance. The matrix $S$ is the covariance of the observed measurement before conditioning. The conditional covariance of $X$ is $\Sigma-\Sigma A^\mathsf T S^{-1}A\Sigma$; these matrices have different dimensions and describe different random vectors.
Treating a noisy measurement as an exact constraint. The formula assumes $R$ is positive definite. A zero-error or singular-noise model requires separate treatment and cannot be obtained by blindly using $R^{-1}$ in the precision form.

The update is useful whenever uncertainty about a Gaussian vector must be revised using linear sensor readings, noisy summaries, or correlated laboratory measurements. Its two forms serve different calculations: the covariance form highlights how the residual shifts the prior mean, while the precision form is convenient when prior and measurement information are naturally specified through inverse covariance matrices.

Check Your Understanding

Use the model and results above to answer the following questions.

In $Y=AX+\varepsilon$, what are the dimensions of $A$, $R$, and $S$ when $X\in\mathbb R^p$ and $Y\in\mathbb R^m$?
Why does a measurement equal to $A\mu$ leave the conditional mean at $\mu$?
For the scalar example, how would the conditional variance change if the measurement-error variance increased from $1$ to $4$?
In the two-measurement example, what feature of the error covariance explains the negative second component of the updated mean?
State the assumptions that make the precision form $Q^{-1}=(\Sigma^{-1}+A^\mathsf T R^{-1}A)^{-1}$ well-defined.