Sequential Conditioning and Gaussian Updating

Why Sequential Updating Needs a Conditional Observation

Suppose a Gaussian vector $X$ is observed indirectly through two blocks, $Y$ and $Z$. One can first condition on $Y$, then incorporate $Z$. For this to reproduce the law of $X$ given both blocks, the second update must account for what $Z$ says after $Y$ is already known. In particular, $Z$ generally cannot be treated as an observation independent of $Y$.

The key calculation is to remove from $Z$ its linear prediction from $Y$. The resulting residual is the part of the second block not explained by the first. Its covariance with the first-stage residual of $X$ determines how much the second block changes the conditional law.

Setup and the Sequential Update

Let $X\in\mathbb{R}^p$, $Y\in\mathbb{R}^q$, and $Z\in\mathbb{R}^r$ be jointly Gaussian, with mean vectors $\mu_X,\mu_Y,\mu_Z$. Assume their joint covariance matrix is positive definite, so the covariance matrices inverted below are nonsingular. Write $\Sigma_{AB}=\operatorname{Cov}(A,B)$, so $\Sigma_{BA}=\Sigma_{AB}^{T}$.

Conditioning first on $Y=y$, the Gaussian conditional mean and covariance of $X$ are

m_X(y)=\mu_X+\Sigma_{XY}\Sigma_{YY}^{-1}(y-\mu_Y), \qquad A=\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}.

The corresponding conditional mean of $Z$ given $Y=y$, and the conditional covariances needed for the second update, are

m_Z(y)=\mu_Z+\Sigma_{ZY}\Sigma_{YY}^{-1}(y-\mu_Y),

B=\Sigma_{XZ}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YZ}, \qquad D=\Sigma_{ZZ}-\Sigma_{ZY}\Sigma_{YY}^{-1}\Sigma_{YZ}.

Here $B$ is the covariance between $X$ and $Z$ after both have been linearly adjusted for $Y$, while $D$ is the covariance of the adjusted second observation. Under the positive-definiteness assumption, $D$ is positive definite.

Theorem: For any $y,z$, the conditional distribution of $X$ given $Y=y,Z=z$ is normal with mean and covariance $$ E[X\mid Y=y,Z=z]=m_X(y)+BD^{-1}\{z-m_Z(y)\}, \qquad \operatorname{Var}(X\mid Y=y,Z=z)=A-BD^{-1}B^T. $$ Thus successive conditioning agrees with conditioning on the two observed blocks jointly when the second update uses the residual $z-m_Z(y)$, the residual covariance $D$, and the residual cross-covariance $B$.

Why the Sequential Formula Works

Define the first-stage residuals

U=X-m_X(Y), \qquad V=Z-m_Z(Y).

These are jointly Gaussian. Their covariances are $\operatorname{Var}(U)=A$, $\operatorname{Cov}(U,V)=B$, and $\operatorname{Var}(V)=D$. Both residuals have zero covariance with $Y$; because the full vector is Gaussian, the pair $(U,V)$ is independent of $Y$.

To derive the update, subtract from $U$ its linear prediction from $V$:

R=U-BD^{-1}V.

Then $\operatorname{Cov}(R,V)=B-BD^{-1}D=0$, so $R$ and $V$ are independent by Gaussianity. Also,

\operatorname{Var}(R) =A-BD^{-1}B^T.

Since $U=BD^{-1}V+R$, conditioning on $Y=y,V=v$ fixes the second term in the linear part and leaves the distribution of $R$ unchanged. For the observed value $z$, $v=z-m_Z(y)$. This gives the stated conditional mean and covariance.

This derivation also explains why the update is genuinely sequential. First, $Y$ determines $m_X(y)$. Then the innovation $z-m_Z(y)$ shifts that mean by $BD^{-1}\{z-m_Z(y)\}$, while reducing the covariance from $A$ to $A-BD^{-1}B^T$.

Proof: The residuals $U,V$ are linear transformations of a jointly Gaussian vector. The conditional Gaussian formulas given above yield their stated covariance blocks, and their zero covariances with $Y$ imply that $(U,V)$ is independent of $Y$. The residual $R=U-BD^{-1}V$ is jointly Gaussian with $V$, has covariance zero with $V$, and therefore is independent of $V$. Its covariance is $$ \begin{aligned} \operatorname{Var}(R) &=A-BD^{-1}B^T-BD^{-1}B^T+BD^{-1}DD^{-1}B^T\\ &=A-BD^{-1}B^T. \end{aligned} $$ Consequently, conditional on $Y=y,V=v$, $U$ has mean $BD^{-1}v$ and covariance $A-BD^{-1}B^T$. Substituting $X=m_X(Y)+U$ and $v=z-m_Z(y)$ proves the theorem. $\square$

Agreement With Conditioning on Both Blocks at Once

The sequential expression is not a different approximation to the joint conditional law. It is exactly that law. To see why, define $V=Z-m_Z(Y)$. Since $m_Z(Y)$ is a function of $Y$, knowing $(Y,Z)$ is equivalent to knowing $(Y,V)$: from $Y,V$ one recovers $Z=V+m_Z(Y)$, and from $Y,Z$ one computes $V$. Thus the two pairs generate the same information.

The preceding derivation gives the conditional law of $X$ given $Y,V$. Since this is also the information in $Y,Z$, it is the conditional law given both original blocks. This proves the equivalence without needing to invert the entire covariance matrix of $(Y,Z)$.

Result: Under the stated joint Gaussian and positive-definite covariance assumptions, the staged update and direct conditioning on $(Y,Z)$ give identical conditional means and covariances. The residualization step is the matrix form of accounting for the first block before using the second.

Worked Matrix Updates

Worked Example: Two Noisy Measurements of a Scalar

Let $X\sim N(0,1)$, and observe $Y=X+\varepsilon_1$ and $Z=2X+\varepsilon_2$, where $\varepsilon_1,\varepsilon_2$ are independent $N(0,1)$ variables independent of $X$. Compute the conditional law for $Y=1,Z=0$.

For the first observation, $\operatorname{Var}(Y)=2$ and $\operatorname{Cov}(X,Y)=1$. Hence $m_X(y)=y/2$ and $A=1-1/2=1/2$.

Next, $\operatorname{Cov}(Z,Y)=2$ and $\operatorname{Var}(Z)=5$. Therefore

m_Z(y)=\frac{2}{2}y=y,\qquad B=\operatorname{Cov}(X,Z)-\frac{\operatorname{Cov}(X,Y)\operatorname{Cov}(Y,Z)}{\operatorname{Var}(Y)} =2-\frac{1\cdot2}{2}=1,

D=5-\frac{2^2}{2}=3.

For the observed values, the second-block residual is $z-m_Z(y)=0-1=-1$. The sequential update gives

E[X\mid Y=1,Z=0]=\frac12+\frac13(-1)=\frac16, \qquad \operatorname{Var}(X\mid Y,Z)=\frac12-\frac{1}{3}=\frac16.

As a direct check, the likelihood contributions have precision $1$, $1$, and $4$, from the prior, $Y$, and $Z$, respectively. The total precision is $6$, and the conditional mean is $(y+2z)/6=1/6$, with variance $1/6$. Both calculations agree.

Worked Example: A Two-Dimensional Target

Let $X=(X_1,X_2)^T\sim N_2(0,I_2)$, $Y=X_1+\varepsilon$, and $Z=X_1+X_2+\eta$, where $\varepsilon,\eta$ are independent $N(0,1)$ variables independent of $X$. Find the conditional law when $Y=2,Z=1$.

Write $e_1=(1,0)^T$ and $h=(1,1)^T$. The first observation has variance $2$ and covariance $\operatorname{Cov}(X,Y)=e_1$. Thus

m_X(y)=\begin{pmatrix}y/2\\0\end{pmatrix}, \qquad A=I_2-\frac{e_1e_1^T}{2} =\begin{pmatrix}1/2&0\\0&1\end{pmatrix}.

For the second block, $m_Z(y)=y/2$. Since its coefficient vector for $X$ is $h$, its residual cross-covariance and variance are

B=Ah=\begin{pmatrix}1/2\\1\end{pmatrix}, \qquad D=h^TAh+1=\frac12+1+1=\frac52.

The residual is $z-m_Z(y)=1-1=0$, so the mean remains $(1,0)^T$. The covariance is

A-\frac{BB^T}{D} = \begin{pmatrix}1/2&0\\0&1\end{pmatrix} -\frac25 \begin{pmatrix}1/4&1/2\\1/2&1\end{pmatrix} = \begin{pmatrix}2/5&-1/5\\-1/5&3/5\end{pmatrix}.

For a direct matrix check, the posterior precision is $I_2+e_1e_1^T+hh^T=\begin{pmatrix}3&1\\1&2\end{pmatrix}$. Its inverse is $\frac15\begin{pmatrix}2&-1\\-1&3\end{pmatrix}$, exactly the covariance above. Multiplying this inverse by $e_1y+hz=(3,1)^T$ gives mean $(1,0)^T$.

Worked Example: A Redundant Second Block

Let $X\sim N(0,1)$, $Y=X+\varepsilon$, and $Z=Y+\eta$, where $\varepsilon,\eta$ are mutually independent $N(0,1)$ variables independent of $X$. Show that $Z$ adds no information about $X$ once $Y$ is known.

As before, $m_X(y)=y/2$ and $A=1/2$. Now $\operatorname{Cov}(X,Z)=1$, $\operatorname{Cov}(Y,Z)=2$, and $\operatorname{Var}(Z)=3$. Therefore

B=1-\frac{1\cdot2}{2}=0, \qquad D=3-\frac{2^2}{2}=1, \qquad m_Z(y)=y.

The update term $BD^{-1}\{z-m_Z(y)\}$ is zero for every $y,z$, and the covariance remains $A=1/2$. Indeed, after $Y$ is observed, $Z-Y=\eta$ is independent noise and contains no additional information about $X$.

When the Second Block Changes Nothing

The examples suggest a general criterion. The covariance reduction is $BD^{-1}B^T$. Because $D$ is positive definite, this matrix is positive semidefinite. The second block leaves the conditional mean unchanged for every $z$ and leaves the covariance unchanged precisely when $B=0$. In the jointly Gaussian setting, $B=0$ also means that $X-m_X(Y)$ and $Z-m_Z(Y)$ are independent. Thus, once $Y$ is known, the remaining part of $Z$ is conditionally uninformative about $X$.

A zero update for one particular observed residual does not by itself imply that the block is uninformative. For example, the residual $z-m_Z(y)$ might happen to be zero even when $B\ne0$; the covariance can still decrease. The criterion for no information is $B=0$, not merely a zero realized shift in the mean.

Common Pitfall

Incorrect reasoning: “After finding the law of $X\mid Y=y$, condition that law directly on $Z=z$, treating $Z$ as if it were an observation from a model independent of $Y$.” This can discard the dependence between the observation blocks and produce an incorrect update.
Correction: Use the conditional observation model after the first block. Compute $m_Z(y)$, then the innovation $z-m_Z(y)$, its conditional covariance $D$, and its conditional cross-covariance $B$ with $X$. These quantities yield the valid second update.

The formulas here assume a positive-definite joint covariance, which guarantees the inverses used in the updates exist. Singular Gaussian blocks require restricting the conditioning values to their support and using an appropriate generalized-inverse formulation; the ordinary inverse formulas above should not be applied to singular blocks without that adjustment.

Check Your Understanding

Use the residualized second observation to answer each question.

In the update formula, what does $z-m_Z(y)$ represent, and why is it not generally equal to $z-\mu_Z$?
For the scalar noisy-measurement example, verify that the sequential conditional mean simplifies to $(y+2z)/6$ for arbitrary $y,z$.
What does $B=0$ imply about the second observation block after conditioning on the first?
Can the mean update be zero for a particular observed pair even if the second block reduces the conditional covariance? Explain.
Why do conditioning on $(Y,Z)$ and on $(Y,Z-m_Z(Y))$ use the same information?