Orthogonal Projection in Linear Filtering

Estimation as Orthogonal Projection

In filtering, an estimate at time $t$ can use the observations available by time $t$, but not the hidden state itself. Conditional expectation gives the least-squares estimate among all random variables measurable from those observations. Its defining feature is an orthogonality condition: the estimation error has zero expected product with every square-integrable observation-measurable quantity. In a linear Gaussian model, this condition reduces the estimation problem to a covariance calculation.

Work on a filtered probability space $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})$ carrying a Brownian motion $B_t$. Let $\mathcal{G}\subseteq\mathcal{F}_t$ be the sigma-algebra generated by the observations available at the time under consideration, with null sets included. No stochastic integral is needed for the projection arguments below. For a random vector $X$, square-integrability means $\mathbb{E}[|X|^2]<\infty$, where $|\cdot|$ is the Euclidean norm. The space $L^2(\mathcal{G})$ consists of square-integrable, $\mathcal{G}$-measurable random variables, identified when they agree almost surely.

Definition: A square-integrable estimate $\widehat X$ is orthogonal to the observation errors if $\mathbb{E}[(X-\widehat X)U]=0$ for every $U\in L^2(\mathcal{G})$. For vector $X$, this means $\mathbb{E}[(X-\widehat X)U]=0$ componentwise for every scalar $U\in L^2(\mathcal{G})$.

Conditional Expectation Characterizes the Projection

Theorem: Suppose $X\in L^2(\mathbb{P})$ is scalar-valued and $\mathcal{G}\subseteq\mathcal{F}$. A random variable $\widehat X\in L^2(\mathcal{G})$ equals $\mathbb{E}[X\mid\mathcal{G}]$ almost surely if and only if $$ \mathbb{E}[(X-\widehat X)U]=0\qquad\text{for every }U\in L^2(\mathcal{G}). $$ Moreover, for every $V\in L^2(\mathcal{G})$, $$ \mathbb{E}[(X-V)^2] =\mathbb{E}[(X-\widehat X)^2]+\mathbb{E}[(\widehat X-V)^2]. $$ Thus $\widehat X$ uniquely minimizes mean squared error over $L^2(\mathcal{G})$, up to almost-sure equality.

Proof. First set $\widehat X=\mathbb{E}[X\mid\mathcal{G}]$. The defining property of conditional expectation gives $\mathbb{E}[(X-\widehat X)U]=0$ when $U$ is bounded and $\mathcal{G}$-measurable. For arbitrary $U\in L^2(\mathcal{G})$, let $U_n=\max(-n,\min(U,n))$. Then $U_n$ is bounded and $\mathcal{G}$-measurable, and $\mathbb{E}[(X-\widehat X)U_n]=0$. Also $U_n\to U$ in $L^2$, so Cauchy–Schwarz gives $\mathbb{E}[(X-\widehat X)U_n]\to\mathbb{E}[(X-\widehat X)U]$. This proves orthogonality for every such $U$. Conversely, suppose $\widehat X\in L^2(\mathcal{G})$ satisfies the stated orthogonality. Let $M=\mathbb{E}[X\mid\mathcal{G}]$, which belongs to $L^2(\mathcal{G})$ because conditional expectation does not increase the $L^2$ norm. Both $X-\widehat X$ and $X-M$ are orthogonal to every member of $L^2(\mathcal{G})$; hence $\widehat X-M$ is also orthogonal to every such member. Taking $U=\widehat X-M$ yields $\mathbb{E}[(\widehat X-M)^2]=0$, so $\widehat X=M$ almost surely. Finally, write $X-V=(X-\widehat X)+(\widehat X-V)$, square, and take expectations. The cross term is zero by orthogonality, since $\widehat X-V\in L^2(\mathcal{G})$. This gives the identity. Its second term is nonnegative and is zero only when $V=\widehat X$ almost surely, proving the minimizing and uniqueness claims. $\square$

The result applies to any square-integrable state and any observation sigma-algebra; Gaussian assumptions are not needed. It does not say that the estimate is orthogonal only to the observation itself. It is orthogonal to every square-integrable function of the observations. That stronger condition is what characterizes conditional expectation among all observation-measurable estimates.

From Orthogonality to a Covariance-Weighted Update

Consider a finite-dimensional linear Gaussian observation model. Let $X\in\mathbb{R}^n$ have mean $m$ and covariance $P$, and let $$ Z=HX+\varepsilon, $$ where $H$ is a fixed $p\times n$ matrix and $\varepsilon$ is independent of $X$, Gaussian with mean zero and positive-definite covariance $R$. The observed data are represented by $Z$, so $\mathcal{G}=\sigma(Z)$. Define the innovation $Z-Hm$, its covariance $S=HPH^\mathsf{T}+R$, and the gain $K=PH^\mathsf{T}S^{-1}$. Since $R$ is positive definite and $HPH^\mathsf{T}$ is positive semidefinite, $S$ is positive definite and its inverse exists.

Theorem: In this model, the conditional mean given the observation is $$ \mathbb{E}[X\mid\sigma(Z)]=m+K(Z-Hm). $$ Its estimation error has covariance $$ P-PH^\mathsf{T}S^{-1}HP. $$

Proof. Set $\widehat X=m+K(Z-Hm)$ and $E=X-\widehat X$. The pair $(E,Z-Hm)$ is jointly Gaussian: both are linear combinations of the independent Gaussian vectors $X-m$ and $\varepsilon$. Its cross-covariance is $$ \operatorname{Cov}(E,Z-Hm) =\operatorname{Cov}(X-m,Z-Hm)-K\operatorname{Cov}(Z-Hm,Z-Hm) =PH^\mathsf{T}-KS=0. $$ For jointly Gaussian vectors, zero cross-covariance implies independence: their joint characteristic function is the exponential of a quadratic form, and a zero cross-covariance makes that quadratic form the sum of the separate forms, so the characteristic function factors. Thus $E$ is independent of $Z-Hm$, and hence of $Z$. It has mean zero, so $\mathbb{E}[E\mid\sigma(Z)]=0$. Therefore $\mathbb{E}[X\mid\sigma(Z)]=\widehat X$. To compute the error covariance, use $E=(X-m)-K(Z-Hm)$, $\operatorname{Cov}(X-m,Z-Hm)=PH^\mathsf{T}$, and $\operatorname{Cov}(Z-Hm)=S$. Expanding gives $$ \operatorname{Cov}(E)=P-PH^\mathsf{T}K^\mathsf{T}-KHP+KSK^\mathsf{T} =P-KHP =P-PH^\mathsf{T}S^{-1}HP, $$ where $K=PH^\mathsf{T}S^{-1}$ and $S$ is symmetric. $\square$

The covariance weighting is essential: $K$ scales each innovation by how strongly the state is correlated with the observation, relative to the total observation covariance. More generally, if one searches only among affine estimates $m+A(Z-Hm)$, orthogonality to the innovation requires $$ \operatorname{Cov}\big(X-m-A(Z-Hm),Z-Hm\big) =PH^\mathsf{T}-AS=0. $$ Since $S$ is invertible, this equation has the unique solution $A=PH^\mathsf{T}S^{-1}$. For non-Gaussian variables this gives the best affine estimate, but it need not be the conditional expectation: orthogonality to the observation alone does not imply orthogonality to every measurable function of it. In the Gaussian case, the independence argument above supplies that stronger property.

Worked Covariance Updates

Worked Example: A Scalar Observation

Let $X\sim N(2,4)$, and observe $Z=X+\varepsilon$, where $\varepsilon\sim N(0,1)$ is independent of $X$. Find the conditional mean and error variance when the observed value is $Z=5$.

Here $m=2$, $P=4$, $H=1$, and $R=1$. The innovation covariance is $S=HPH^\mathsf{T}+R=4+1=5$, so $K=PH^\mathsf{T}S^{-1}=4/5$. The observed innovation is $5-1\cdot2=3$, giving $$ \widehat X=2+\frac45\cdot3=2+\frac{12}{5}=\frac{22}{5}. $$ The error variance is $P-PH^\mathsf{T}S^{-1}HP=4-4\cdot\frac15\cdot4=\frac45$.

Check: Completing the square in the two Gaussian densities gives posterior precision $1/4+1=5/4$, hence posterior variance $4/5$. The posterior mean is $(2/4+5)/(5/4)=(11/2)(4/5)=22/5$, matching the projection calculation.

Worked Example: Two State Components and One Observation

Let $X=(X_1,X_2)^\mathsf{T}$ have mean $(0,0)^\mathsf{T}$ and covariance $P=\begin{pmatrix}1&0\\0&4\end{pmatrix}$. Observe $Z=X_1+X_2+\varepsilon$, with independent $\varepsilon\sim N(0,1)$. Compute the conditional mean and error covariance when $Z=3$.

Here $H=(1\quad1)$. Thus $HPH^\mathsf{T}=1+4=5$, $S=5+1=6$, and $$ K=PH^\mathsf{T}S^{-1} =\begin{pmatrix}1\\4\end{pmatrix}\frac16 =\begin{pmatrix}1/6\\2/3\end{pmatrix}. $$ The innovation is $3-Hm=3$. Therefore $$ \widehat X=\begin{pmatrix}0\\0\end{pmatrix} +\begin{pmatrix}1/6\\2/3\end{pmatrix}3 =\begin{pmatrix}1/2\\2\end{pmatrix}. $$ The error covariance is $$ P-PH^\mathsf{T}S^{-1}HP =\begin{pmatrix}1&0\\0&4\end{pmatrix} -\frac16\begin{pmatrix}1&4\\4&16\end{pmatrix} =\begin{pmatrix}5/6&-2/3\\-2/3&4/3\end{pmatrix}. $$

Check: the error’s covariance with the observation is $PH^\mathsf{T}-KS=(1,4)^\mathsf{T}-(1/6,2/3)^\mathsf{T}6=(0,0)^\mathsf{T}$, as orthogonality requires. The diagonal entries $5/6$ and $4/3$ are nonnegative, and the covariance matrix has determinant $(5/6)(4/3)-(2/3)^2=10/9-4/9=2/3$, consistent with a positive-definite error covariance.

Worked Example: Correlated Observation Noise

Let $X\in\mathbb{R}^2$ have mean zero and covariance $I$, and observe $Z=X+\varepsilon$, where $\varepsilon$ is independent of $X$ with covariance $R=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}$. Given $Z=(2,0)^\mathsf{T}$, compute the estimate and error covariance.

Here $H=I$, so $S=P+R=\begin{pmatrix}2&1/2\\1/2&2\end{pmatrix}$. Its determinant is $4-1/4=15/4$, and direct multiplication verifies $$ S^{-1}=\frac{4}{15}\begin{pmatrix}2&-1/2\\-1/2&2\end{pmatrix} =\begin{pmatrix}8/15&-2/15\\-2/15&8/15\end{pmatrix}. $$ Since $P=I$, the gain is $K=S^{-1}$. The conditional mean is $$ KZ=\begin{pmatrix}8/15&-2/15\\-2/15&8/15\end{pmatrix} \begin{pmatrix}2\\0\end{pmatrix} =\begin{pmatrix}16/15\\-4/15\end{pmatrix}. $$ The error covariance is $I-K=\begin{pmatrix}7/15&2/15\\2/15&7/15\end{pmatrix}$.

Check: $KS=I$, so $\operatorname{Cov}(X-KZ,Z)=I-KS=0$. The off-diagonal entries in the gain and error covariance matter: correlated observation noise means the two observed components cannot be treated as independent measurements.

Why the Projection View Matters

The orthogonality condition converts a qualitative requirement—use the observations efficiently—into an equation that can be checked and solved. In a Gaussian filtering problem, the current estimate is a conditional expectation, while its covariance update quantifies what uncertainty remains after incorporating the observation. The same projection logic also clarifies the boundary of a covariance-only calculation: for a non-Gaussian model, a covariance-based gain generally gives only the best affine estimate, not necessarily the best estimate among all measurable functions of the data.

Common pitfall: using covariance without scaling by observation uncertainty. In the scalar example, a mistaken calculation might set the gain equal to $P=4$, producing the estimate $2+4(5-2)=14$. This ignores the observation variance and is not orthogonal to the observation error. Indeed, if that affine rule is applied to the random observation, its error-observation covariance is $P-4S=4-4\cdot5=-16$, not zero. The correct gain is $P/S=4/5$, which accounts for both signal and noise uncertainty and gives zero error-observation covariance.

Check Your Understanding

Use the orthogonality condition and covariance formulas to answer these questions.

Why must the test variable $U$ range over all of $L^2(\mathcal{G})$ to characterize conditional expectation, rather than only over the raw observation $Z$?
For a scalar prior variance $P>0$, observation coefficient $H=1$, and noise variance $R>0$, derive the gain and explain how it changes as $R$ increases.
In the two-component example, verify directly that the reported error covariance equals $P-KHP$.
Explain why zero covariance between the estimation error and the observation is sufficient for the conditional-mean conclusion in the jointly Gaussian model.
For a non-Gaussian state and observation, what estimation property does the covariance equation $AS=\operatorname{Cov}(X,Z)$ establish, and what does it not establish by itself?