Laws of Large Numbers Under Time-Series Dependence

Why a Law of Large Numbers Needs More Than Stationarity

A time average is often used to estimate a population mean. For independent observations, familiar laws of large numbers justify this step under suitable moment conditions. In a time series, observations may remain dependent across time, so the key question is whether that dependence still permits the average to settle near its expectation. Stationarity alone does not guarantee that it does.

This tutorial develops two useful routes to convergence. The ergodic theorem gives almost-sure convergence when the stationary process is ergodic and has a finite first moment. Mixing conditions provide a practical sufficient condition for ergodicity, and they also let us bound the variability of sample averages. These results concern convergence, not the distribution of a normalized estimation error.

Stationarity, Ergodicity, and the Time Average

Definition: A process $\{X_t\}$ is strictly stationary if the joint distribution of any finite collection $(X_{t_1},\ldots,X_{t_m})$ is unchanged when the same integer is added to every time index. A strictly stationary process is ergodic if every event unchanged by a shift of the entire time series has probability zero or one. Informally, there are no nontrivial, shift-invariant features of the process that differ across realizations.

For a stationary process with $E|X_0|<\infty$, the expectation $E[X_t]$ is the same at every time. Write this common value as $\mu=E[X_0]$. The time average of the first $n$ observations is $\bar X_n=n^{-1}\sum_{t=1}^n X_t$. The central result is that ergodicity makes this average reveal the population mean.

Theorem (Birkhoff ergodic theorem for a stationary sequence): If $\{X_t\}$ is strictly stationary and ergodic and $E|X_0|<\infty$, then $$ \bar X_n=\frac{1}{n}\sum_{t=1}^n X_t \longrightarrow E[X_0]=\mu \quad\text{almost surely as } n\to\infty. $$

Almost-sure convergence means that, with probability one, the sequence of realized time averages approaches $\mu$. It is stronger than convergence in probability. The finite first absolute moment is the relevant integrability condition; the theorem does not require independent observations or a finite variance.

Mixing as a Sufficient Condition

A mixing condition formalizes the idea that events far apart in time become nearly independent. For a stationary process, let $\mathcal F_{-\infty}^t$ contain events determined by observations up to time $t$, and let $\mathcal F_{t+k}^{\infty}$ contain events determined by observations from time $t+k$ onward. The alpha-mixing coefficient at lag $k$ is

\alpha(k)=\sup_t\ \sup_{\substack{A\in\mathcal F_{-\infty}^t\\B\in\mathcal F_{t+k}^{\infty}}} \left|P(A\cap B)-P(A)P(B)\right|.

The process is alpha-mixing (or strongly mixing) if $\alpha(k)\to 0$ as $k\to\infty$. This definition concerns dependence between whole past and future sigma-fields, not just the correlation between two observations. Vanishing correlations alone do not establish mixing.

Useful implication: A strictly stationary alpha-mixing process with $\alpha(k)\to0$ is ergodic. Therefore, if $E|X_0|<\infty$, its sample average converges almost surely to $E[X_0]$ by the ergodic theorem. Mixing is a sufficient condition for ergodicity, not a necessary one.

This route is often convenient because the mixing assumption describes how dependence fades. It is also important to keep the conclusion in proportion: the ergodic theorem guarantees convergence, but does not by itself give a finite-sample error bound or a convergence rate.

A Variance Argument for Convergence in Mean Square

A second route makes the role of declining dependence visible in a variance calculation. Suppose $\{X_t\}$ is strictly stationary, has finite second moment, and has mean $\mu$. Define $Y_t=X_t-\mu$, $\gamma_k=\operatorname{Cov}(X_t,X_{t-k})$, and $\bar Y_n=\bar X_n-\mu$. Stationarity gives $\operatorname{Cov}(Y_t,Y_s)=\gamma_{|t-s|}$, so counting the pairs of observations at each lag yields

\operatorname{Var}(\bar X_n) =\operatorname{Var}(\bar Y_n) =\frac{1}{n^2}\left(n\gamma_0+2\sum_{k=1}^{n-1}(n-k)\gamma_k\right) =\frac{1}{n}\left(\gamma_0+2\sum_{k=1}^{n-1}\left(1-\frac{k}{n}\right)\gamma_k\right).

If $\gamma_k\to0$, the weighted averages of the lag covariances in this expression tend to zero. To see why, fix any $\varepsilon>0$. Choose $K$ so that $|\gamma_k|<\varepsilon$ for every $k>K$. The finitely many terms through $K$, divided by $n$, tend to zero as $n\to\infty$; the absolute contribution from the remaining terms is at most $2\varepsilon$. Since $\varepsilon$ can be made arbitrarily small, $\operatorname{Var}(\bar X_n)\to0$. Thus $\bar X_n\to\mu$ in mean square, and hence in probability.

For bounded observations, alpha-mixing gives a direct way to verify covariance decay. If $|X_t|\le M$, then $|X_t-\mu|\le 2M$. The standard covariance inequality for bounded variables measurable with respect to separated past and future gives

|\gamma_k|\le 4\|X_t-\mu\|_\infty^2\alpha(k) \le 16M^2\alpha(k).

Consequently, $\alpha(k)\to0$ implies $\gamma_k\to0$, and the variance argument applies. This establishes convergence in mean square for bounded stationary alpha-mixing observations. Summability of the mixing coefficients is not needed for this convergence result.

Worked Example: A Variance Bound for Bounded Mixing Data

Suppose hypothetical daily observations $X_t$, measured in units, are strictly stationary, $0\le X_t\le10$, and have mean $\mu=5$. Assume their mixing coefficients satisfy $\alpha(k)\le0.01(0.5)^k$ for every $k\ge1$. Bound the variance of the average of $n=500$ days.

Since $X_t$ lies between 0 and 10, its variance is at most $25$: for a variable in an interval of width 10, the maximum variance is $10^2/4=25$. Also, $|X_t-\mu|\le10$, so the covariance inequality gives $|\gamma_k|\le400\alpha(k)$. The geometric bound sums to

\sum_{k=1}^{\infty}\alpha(k) \le 0.01\sum_{k=1}^{\infty}(0.5)^k =0.01.

Using the variance formula and replacing each weight $1-k/n$ by its upper bound 1,

\operatorname{Var}(\bar X_{500}) \le \frac{1}{500}\left(25+2(400)(0.01)\right) =\frac{33}{500}=0.066.

Thus the variance of the 500-day average is at most $0.066$ squared units under these assumptions. For example, Chebyshev's inequality gives $P(|\bar X_{500}-5|\ge1)\le0.066$. The bound is conservative; it is not a claim that the actual variance equals $0.066$. As the sample size increases, the same argument gives a variance bound tending to zero.

Finite-Moment Observations

Boundedness is convenient, but many economic variables are not bounded. A finite higher moment can replace boundedness in a covariance argument. One standard form of Davydov's inequality states that for variables separated by a lag with alpha-mixing coefficient $\alpha(k)$, and $p,q>1$ satisfying $1/p+1/q<1$,

|\operatorname{Cov}(U,V)| \le C\,\|U\|_p\|V\|_q\, \alpha(k)^{\,1-1/p-1/q},

where $C$ is a finite constant and $\|U\|_p=(E|U|^p)^{1/p}$. In particular, if $E|X_0|^{2+\delta}<\infty$ for some $\delta>0$, this inequality gives, for a finite constant $C_\delta$,

|\gamma_k| \le C_\delta\|X_0-\mu\|_{2+\delta}^{\,2} \alpha(k)^{\delta/(2+\delta)}.

If $\alpha(k)\to0$, the right-hand side tends to zero. The variance argument then establishes convergence in mean square, and thus in probability. In addition, alpha-mixing implies ergodicity, so the ergodic theorem gives almost-sure convergence whenever $E|X_0|<\infty$. The higher moment in this argument is useful for controlling covariances; it is not needed for the ergodic theorem itself.

Worked Example: A Finite Fourth Moment and Slowly Decaying Mixing

Let $Y_t=X_t-\mu$ be a hypothetical centered, strictly stationary series with $\|Y_0\|_4\le1$, and suppose $\alpha(k)\le0.000625/k^2$ for $k\ge1$. Use Davydov's inequality with $\delta=2$, and take its constant to be 8 for this bound. Show that its sample average converges in mean square and bound the variance for $n=1000$.

Here $\delta/(2+\delta)=1/2$, so

|\gamma_k| \le 8\|Y_0\|_4^2\alpha(k)^{1/2} \le 8\left(\frac{0.000625}{k^2}\right)^{1/2} =\frac{0.2}{k}.

Also, $\gamma_0=E[Y_0^2]\le\|Y_0\|_4^2\le1$. Substituting these bounds into the variance formula gives

\operatorname{Var}(\bar Y_{1000}) \le\frac{1}{1000}\left(1+0.4\sum_{k=1}^{999}\frac{1}{k}\right) \le\frac{1+0.4(1+\log 999)}{1000} \approx0.004163.

The harmonic sum bound uses $\sum_{k=1}^{m}1/k\le1+\log m$. More generally, the resulting bound is at most $\{1+0.4(1+\log(n-1))\}/n$, which tends to zero. Thus the average converges in mean square. The coefficients in this example are not summable at the displayed $1/k$ covariance bound, but summability is not required for convergence of the average. Since the series is also alpha-mixing with coefficients tending to zero and has a finite first moment, its average converges almost surely as well.

Why Stationarity Alone Is Not Enough

Worked Example: A Stationary Average That Does Not Reveal the Mean

Let $Z$ equal $1$ or $-1$, each with probability $1/2$, and set $X_t=Z$ for every time $t$. This hypothetical series is strictly stationary and bounded, with $E[X_t]=0$. Its sample average is

\bar X_n=\frac{1}{n}\sum_{t=1}^{n}Z=Z \quad\text{for every }n.

Therefore $P(|\bar X_n-0|>1/2)=1$ for every $n$: the average does not converge in probability to the population mean. Each realization stays at either 1 or $-1$, so the time average converges to the realized value $Z$, not to $E[Z]=0$. The process is stationary but not ergodic; the event that the whole series equals 1 is unchanged by a time shift and has probability $1/2$. This example shows why stationarity by itself cannot justify replacing a population mean by a single series' time average.

Common Mistakes and Practical Interpretation

Treating stationarity as an LLN. Stable means and covariances do not rule out a non-ergodic process. Check an ergodicity or suitable mixing condition before concluding that a time average converges to the population mean.
Using only a declining autocorrelation function. Declining correlations may be useful, but zero correlation at every nonzero lag does not establish independence, mixing, or ergodicity. Dependence in nonlinear features can remain.
Confusing the modes of convergence. The variance argument establishes convergence in mean square, and therefore in probability. The ergodic theorem establishes almost-sure convergence under its own assumptions. State which conclusion the assumptions support.
Reading an LLN as an inference result. Convergence says that averages approach a target as the sample grows; it does not by itself provide a confidence interval, a finite-sample guarantee, or a distribution for estimation error.

The practical lesson is to match the claim to the assumptions. For a stationary ergodic series with an integrable observation, the time average converges almost surely to its population mean. Alpha-mixing is one usable sufficient condition for ergodicity. Moment and covariance bounds can additionally quantify convergence in probability or mean square, but a rate or sampling distribution requires further analysis.

Check Your Understanding

Use the assumptions stated in each question; distinguish almost-sure convergence from convergence in probability.

State the moment and dependence conditions in the ergodic theorem that imply almost-sure convergence of a stationary sample average.
Why does strict stationarity alone fail to guarantee convergence of $\bar X_n$ to $E[X_0]$? Describe the role of the constant random series example.
For bounded stationary observations with $\alpha(k)\to0$, what covariance property follows, and what mode of convergence can the variance argument establish?
If $E|X_0|^4<\infty$ and $\alpha(k)\to0$, explain why Davydov's inequality can imply that lag covariances vanish.
Does an LLN alone give a confidence interval or a convergence rate? Explain briefly.