Sequential Conditioning and Consistent Gaussian Updates

Why Update Sequentially?

In a Gaussian model, observations often arrive in stages: a new measurement is recorded, a prediction is revised, and another measurement follows. Updating after each observation is useful for computation and interpretation, but it should not change the answer relative to processing all observations at once. The key is to carry forward the full conditional mean and covariance, not just a point estimate.

We develop the update for a fixed latent vector observed through successive linear measurements. The observation errors will be independent across stages and independent of the latent vector. Under these assumptions, each stage has a simple update, and the final conditional distribution is the same as the joint Gaussian conditioning result.

Write \(m_t=E[X\mid Y_1=y_1,\ldots,Y_t=y_t]\) and let \(C_t\) be the corresponding conditional covariance matrix. At stage zero, \(m_0\) and \(C_0\) are the prior mean and covariance. These conditional distributions can be defined even if \(X\) has a degenerate Gaussian distribution. The positive-definite error covariance makes each new observation, given the previous observations, have a nonsingular Gaussian distribution.

The Sequential Gaussian Update

Given the current conditional distribution of \(X\), the next observation has a conditional mean \(H_tm_{t-1}\). Its difference from that mean is the innovation. Its covariance includes both uncertainty about \(X\) and the new measurement error.

To justify the update, condition first on the preceding observations. Under that conditional law, \(X\) has mean \(m_{t-1}\) and covariance \(C_{t-1}\); the new error is still independent with covariance \(R_t\). The conditional mean of \(Y_t\) is therefore \(H_tm_{t-1}\), and its conditional covariance is \(S_t\). For every nonzero vector \(a\in\mathbb{R}^{q_t}\), \(a^{\mathsf T}S_ta=(H_t^{\mathsf T}a)^{\mathsf T}C_{t-1}(H_t^{\mathsf T}a)+a^{\mathsf T}R_ta>0\), since \(C_{t-1}\) is positive semidefinite and \(R_t\) is positive definite.

The conditional cross-covariance of \(X\) with \(Y_t\) is \(C_{t-1}H_t^{\mathsf T}\). Applying the block Gaussian conditional formula, which requires the conditioned-on covariance \(S_t\) to be invertible but not the covariance of \(X\), gives the stated conditional mean and covariance. This proves the update at one stage. Starting from \(m_0,C_0\) and applying it successively proves the result for every \(t\). \(\square\)

The gain \(K_t\) describes how an innovation changes the estimate: directions in the observation that are strongly associated with uncertain components of \(X\) can shift the mean substantially. The covariance update depends on the model matrices and covariances, not on the observed value \(y_t\).

Sequential Conditioning Equals Joint Conditioning

Stack the observations into \(Y=(Y_1^{\mathsf T},\ldots,Y_T^{\mathsf T})^{\mathsf T}\), and likewise stack the matrices into \(H\). The joint error covariance is the block diagonal matrix \(R=\operatorname{diag}(R_1,\ldots,R_T)\), which is positive definite. A single joint update would use \(S=HC_0H^{\mathsf T}+R\). The next result states that this gives exactly the same conditional distribution as the recursion.

For a proof that does not assume \(X\) has a Lebesgue density, let \(\pi_0\) denote its probability distribution. For an observed value \(y_t\), the conditional density of \(Y_t\) given \(X=x\) is the Gaussian density \(\phi_{R_t}(y_t-H_tx)\). This density is strictly positive for every \(x\) and \(y_t\). Bayes' formula for a probability measure therefore gives the updated distribution

for every measurable set \(A\). The denominator is finite and strictly positive: the Gaussian density is bounded and positive everywhere, and \(\pi_{t-1}\) is a probability measure. Iterating this identity shows that \(\pi_T\) is proportional, relative to \(\pi_0\), to \(\prod_{t=1}^T\phi_{R_t}(y_t-H_tx)\). By independence of the errors, this product is the joint conditional density of the stacked observation given \(X=x\). Applying Bayes' formula once to the stacked observation gives the very same probability measure. The one-stage update theorem shows that each sequential \(\pi_t\) is the Gaussian distribution with the recursively calculated \(m_t,C_t\). Thus its final result equals the joint conditional law. No density for \(X\) is required. \(\square\)

Order, Assumptions, and a Common Pitfall

In this model, changing the order of the measurements does not change the final conditional distribution. The batch likelihood is a product, and multiplication is commutative; the consistency result then ensures that any valid sequential ordering reaches the same answer. Intermediate means and covariances can differ because they condition on different subsets of the observations.

Common mistake: treating a measurement as if its error were independent after conditioning when the errors are correlated. The recursion above uses independence of \(\varepsilon_t\) from earlier errors. If errors across stages are correlated, then after observing earlier measurements, the next error may have a nonzero conditional mean and may remain correlated with the latent-vector information from those measurements. Simply reusing \(Y_t=H_tX+\varepsilon_t\) with the original \(R_t\) can then give an incorrect innovation covariance and gain. One must first account for the conditional distribution of the new error given the earlier observations, or use the joint block covariance in a batch conditioning calculation.

A second practical check is that \(C_t\) should be symmetric positive semidefinite. The formula subtracts a positive semidefinite matrix from \(C_{t-1}\), but the result remains a valid covariance because it is obtained from the conditional Gaussian law. Numerical implementations should avoid rounding intermediate quantities prematurely, especially when observation covariances are close to singular. Here positive-definite \(R_t\) guarantees invertibility of each innovation covariance even if the current conditional covariance is singular.

Sequential updates are valuable when data arrive over time, when an analyst wants to inspect how each observation changes an estimate, or when processing a large collection of measurements in blocks. Their reliability depends on carrying forward both \(m_t\) and \(C_t\), using the correct conditional observation model at every step, and checking that the assumptions on measurement errors match the application.