Linear Filters and Mean-Square Convergence

Infinite Linear Filters

A finite weighted sum of observations is straightforward to define. An infinite weighted sum requires more care: its partial sums must approach a random variable in a specified sense. For stationary inputs, mean-square convergence is especially useful because it controls both expectations and covariances. The coefficient condition that guarantees convergence depends on the covariance structure of the input, so it is important to distinguish a general sufficient condition from sharper conditions available in special cases.

Work on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Let $(X_n)_{n\in\mathbb{Z}}$ be a real-valued, weakly stationary sequence with $\mathbb{E}[X_n]=\mu$ and $\operatorname{Cov}(X_{n+h},X_n)=\gamma(h)$ for every $n,h\in\mathbb{Z}$. Weak stationarity means these expectations and covariances exist and have the stated time-invariant form. In particular, $\mathbb{E}[X_n^2]=\mu^2+\gamma(0)<\infty$. The two-sided index set allows the same causal filter to be defined at every integer time.

Definition: Given real coefficients $a_0,a_1,\ldots$, the order-$m$ truncation of the causal linear filter at time $n$ is $Y_n^{(m)}=\sum_{j=0}^{m}a_jX_{n-j}$. The infinite filter exists in mean square if, for each $n$, there is a random variable $Y_n$ with $\mathbb{E}[(Y_n^{(m)}-Y_n)^2]\to0$ as $m\to\infty$. This convergence is also called convergence in $L^2$.

Mean-square convergence is stronger than convergence in probability, and it is not the same as almost-sure convergence. Here the definition concerns the expected squared error. The exact test is the Cauchy criterion: the truncations must become arbitrarily close in $L^2$ to one another. Since $L^2$ is complete, an $L^2$-Cauchy sequence has an $L^2$ limit; completeness of $L^2$ is a standard result.

An Exact Cauchy Criterion

The covariance sequence gives an exact expression for the squared $L^2$ distance between two truncations. For integers $0\le r\le s$, the difference between the truncation ending at $s$ and the one ending at $r-1$ is $\sum_{j=r}^{s}a_jX_{n-j}$. Its mean is $\mu\sum_{j=r}^{s}a_j$. Centering each $X$ and using $\operatorname{Cov}(X_{n-j},X_{n-k})=\gamma(k-j)$ gives the criterion below.

Proposition (Exact mean-square criterion): The truncations $Y_n^{(m)}$ converge in mean square if and only if $$ \mu^2\left(\sum_{j=r}^{s}a_j\right)^2+ \sum_{j=r}^{s}\sum_{k=r}^{s}a_ja_k\gamma(k-j)\longrightarrow0 $$ as $r,s\to\infty$, uniformly over $s\ge r$. The expression is the squared $L^2$ norm of the tail from $r$ through $s$, and it does not depend on $n$.

Proof. For any $n$ and $r\le s$, write $D=\sum_{j=r}^{s}a_jX_{n-j}$. Its centered version is $D-\mathbb{E}D=\sum_{j=r}^{s}a_j(X_{n-j}-\mu)$. The identity $\mathbb{E}[D^2]=(\mathbb{E}D)^2+\operatorname{Var}(D)$ yields $$ \mathbb{E}[D^2]=\mu^2\left(\sum_{j=r}^{s}a_j\right)^2+ \sum_{j=r}^{s}\sum_{k=r}^{s}a_ja_k\operatorname{Cov}(X_{n-j},X_{n-k}). $$ By weak stationarity, $\operatorname{Cov}(X_{n-j},X_{n-k})=\gamma(k-j)$, so this is exactly the displayed expression and is independent of $n$. The truncations are $L^2$-Cauchy precisely when $\mathbb{E}[(Y_n^{(s)}-Y_n^{(r-1)})^2]\to0$ for all such tails, which is precisely the stated condition. Completeness of $L^2$ makes this Cauchy condition equivalent to existence of a mean-square limit. $\square$

This proposition is exact, but it may be inconvenient to apply directly. A simple condition that works for every weakly stationary input is absolute summability of the coefficients. Let $\|Z\|_2=(\mathbb{E}[Z^2])^{1/2}$. The triangle inequality in $L^2$ gives $$ \left\|\sum_{j=r}^{s}a_jX_{n-j}\right\|_2 \le \sum_{j=r}^{s}|a_j|\,\|X_{n-j}\|_2 =\sqrt{\mu^2+\gamma(0)}\sum_{j=r}^{s}|a_j|. $$ Thus a summable coefficient tail makes the filter tail small in mean square.

Theorem (Absolute-summability condition and stationarity): Suppose $(X_n)_{n\in\mathbb{Z}}$ is weakly stationary with finite second moment and $\sum_{j=0}^{\infty}|a_j|<\infty$. Then the truncations $Y_n^{(m)}$ converge in mean square for every $n$. Their limits $(Y_n)_{n\in\mathbb{Z}}$ form a weakly stationary sequence, with mean $\mathbb{E}[Y_n]=\mu\sum_{j=0}^{\infty}a_j$.

Proof. Put $C=\sqrt{\mu^2+\gamma(0)}=\|X_n\|_2$, which is finite and independent of $n$. For $s\ge r$, the $L^2$ triangle inequality gives $$ \|Y_n^{(s)}-Y_n^{(r-1)}\|_2 \le C\sum_{j=r}^{s}|a_j|. $$ The right side is bounded by $C\sum_{j=r}^{\infty}|a_j|$, which tends to zero as $r\to\infty$. Therefore the truncations are $L^2$-Cauchy and converge in $L^2$, for each $n$. Since $|\mathbb{E}[Y_n^{(m)}-Y_n]|\le\|Y_n^{(m)}-Y_n\|_2$, their means converge, and $\mathbb{E}[Y_n^{(m)}]=\mu\sum_{j=0}^{m}a_j$ tends to $\mu\sum_{j=0}^{\infty}a_j$. Each finite truncation has a mean independent of $n$. Its covariance at times $n$ and $n+h$ is also independent of $n$: expanding the finite sums expresses it using only coefficients and values of $\gamma$ at differences of indices. To pass to the limit, center the truncations. For fixed $n,h$, let $U_m=Y_n^{(m)}-\mathbb{E}[Y_n^{(m)}]$, $V_m=Y_{n+h}^{(m)}-\mathbb{E}[Y_{n+h}^{(m)}]$, and let $U,V$ denote the corresponding centered limits. Mean-square convergence implies $U_m\to U$ and $V_m\to V$ in $L^2$. By Cauchy–Schwarz, $$ |\mathbb{E}[U_mV_m]-\mathbb{E}[UV]| \le \|U_m-U\|_2\|V_m\|_2+\|U\|_2\|V_m-V\|_2\longrightarrow0. $$ The norms $\|V_m\|_2$ are bounded because $V_m$ converges in $L^2$. For every $m$, the covariance of the finite truncation at lag $h$ is independent of $n$; taking its limit shows the same for the output covariance. The output has a constant mean and finite second moment, so it is weakly stationary. $\square$

Worked Applications

Worked Example: A Geometric Filter of a General Stationary Input

Suppose a weakly stationary input has mean $\mu=2$ and variance $\gamma(0)=3$. Filter it with $a_j=3^{-j}$, $j\ge0$. Find the output mean and give a mean-square error bound for truncating after $m$.

The coefficients are absolutely summable. Their partial sums and total sum are $$ \sum_{j=0}^{m}3^{-j}=\frac{1-3^{-(m+1)}}{1-\frac13} =\frac32\left(1-3^{-(m+1)}\right), \qquad \sum_{j=0}^{\infty}3^{-j}=\frac32. $$ The theorem gives output mean $2\cdot\frac32=3$. Also $\|X_n\|_2=\sqrt{2^2+3}=\sqrt7$. The omitted coefficient tail is $$ \sum_{j=m+1}^{\infty}3^{-j} =\frac{3^{-(m+1)}}{1-\frac13} =\frac{1}{2\cdot3^m}. $$ Consequently, for any finite $s>m$, $$ \|Y_n^{(s)}-Y_n^{(m)}\|_2 \le\sqrt7\sum_{j=m+1}^{s}3^{-j} \le\frac{\sqrt7}{2\cdot3^m}. $$ The bound tends to zero as $m\to\infty$. In particular, at $m=0$ the bound is $\sqrt7/2$, from $\sqrt7\sum_{j=1}^{\infty}3^{-j}=\sqrt7/2$. The limiting mean also checks directly: the finite-filter means $2\cdot\frac32(1-3^{-(m+1)})$ tend to $3$.

Worked Example: A Square-Summable Filter That Is Not Absolutely Summable

Let $(X_n)$ be mean-zero white noise with variance $1$, so $\operatorname{Cov}(X_{n+h},X_n)=1$ if $h=0$ and $0$ otherwise. Take $a_j=1/(j+1)$. Show directly that the filter converges in mean square even though its coefficients are not absolutely summable.

For a tail from $r$ through $s$, the exact criterion reduces to $$ \mathbb{E}\left[\left(\sum_{j=r}^{s}\frac{X_{n-j}}{j+1}\right)^2\right] =\sum_{j=r}^{s}\frac{1}{(j+1)^2}. $$ For $r\ge1$, put $k=j+1$. Since $1/k^2\le1/[k(k-1)]=1/(k-1)-1/k$ for every $k\ge2$, $$ \sum_{j=r}^{s}\frac{1}{(j+1)^2} \le\sum_{k=r+1}^{s+1}\left(\frac{1}{k-1}-\frac{1}{k}\right) =\frac1r-\frac{1}{s+1} \le\frac1r. $$ This bound tends to zero as $r\to\infty$, so the truncations are Cauchy in mean square. But the coefficients are not absolutely summable: for each integer $q\ge1$, the terms with $2^q\le k<2^{q+1}$ contribute at least $2^q/2^{q+1}=1/2$ to the harmonic sum $\sum_{k\ge1}1/k$. Infinitely many such blocks force that sum to diverge. Thus absolute summability is sufficient, not necessary. As a check on the convergence calculation, the variance of every finite tail is exactly its displayed sum because distinct white-noise terms have zero covariance.

Worked Example: Zero-Variance White Noise

Let $(X_n)$ be mean-zero white noise with variance $\sigma^2=0$, and take $a_j=1$ for every $j$. Determine whether the filter truncations converge in mean square.

For each $n$, $\mathbb{E}[X_n^2]=0$, so $X_n=0$ almost surely. For any finite $r\le s$, the tail has squared $L^2$ norm $$ \mathbb{E}\left[\left(\sum_{j=r}^{s}X_{n-j}\right)^2\right] =\sigma^2(s-r+1)=0\cdot(s-r+1)=0. $$ Every truncation is therefore zero almost surely, and the truncations converge in mean square to zero, although $\sum_{j=0}^{\infty}|a_j|=\infty$. For comparison, with the same coefficients and mean-zero white noise of variance $\sigma^2=1$, a tail of length $s-r+1$ has squared $L^2$ norm $1\cdot(s-r+1)$. Taking $s=r$ gives a tail norm squared of $1$ at every $r$, so the truncations are not Cauchy. These computations verify that the variance-zero and positive-variance cases have different conclusions.

White Noise and a Common Pitfall

For mean-zero white noise with variance $\sigma^2>0$, the exact tail expression is $$ \mathbb{E}\left[\left(\sum_{j=r}^{s}a_jX_{n-j}\right)^2\right] =\sigma^2\sum_{j=r}^{s}a_j^2. $$ Hence the causal filter converges in mean square if and only if $\sum_{j=0}^{\infty}a_j^2<\infty$. Indeed, square summability makes every tail sum tend to zero; conversely, if the truncations are Cauchy, the nonnegative tail sums must tend to zero, which implies convergence of the series of squares. The strict assumption $\sigma^2>0$ is needed for the converse. When $\sigma^2=0$, every finite tail has squared norm zero, so every coefficient sequence gives mean-square convergence.

Common mistake. It is incorrect to say without qualification that a white-noise filter converges exactly when its coefficients are square-summable. For the zero-variance example, that claim would reject $a_j=1$, even though every truncation is identically zero. The corrected statement requires $\sigma^2>0$ for the “if and only if” square-summability criterion; at $\sigma^2=0$, convergence holds for every coefficient sequence. Another common overstatement is to treat absolute summability as necessary. The harmonic-coefficient example shows why that is false: the coefficients are not absolutely summable, but their squares are summable and the white-noise filter converges.

The practical distinction is useful. Absolute summability supplies a simple guarantee for any weakly stationary input with finite second moment. When the input has additional covariance structure, the exact Cauchy criterion can allow weaker coefficient conditions. Once mean-square convergence is established, the output’s weak stationarity follows from the finite truncations and passage to the limit; no assumption of independence or Gaussianity is required for that conclusion.

Check Your Understanding

In each question, specify the hypotheses you use and distinguish a sufficient condition from a necessary one.

For a weakly stationary input with mean $\mu$ and variance $\gamma(0)$, explain why absolute summability of $(a_j)$ makes the filter truncations Cauchy in $L^2$.
State the exact squared $L^2$ norm of a finite filter tail in terms of $\mu$, $a_j$, and $\gamma$.
For mean-zero white noise of variance $4$, what condition on the coefficients is necessary and sufficient for mean-square convergence?
Why does the square-summability equivalence in the white-noise case require positive variance?
What two properties must be verified to conclude that a mean-square limit is weakly stationary?

Linear Filters and Mean-Square Convergence

What You'll Learn