Why Stationarity Does Not Guarantee a Representative Time Average
In time-series analysis, it is common to estimate a population feature by averaging observations from one long run of a process. That strategy needs more than stationarity. Stationarity says that the probabilistic behavior does not change when time is shifted; it does not by itself ensure that a single realized path reflects the population distribution. Ergodicity is the additional property that connects long-run averages along a path to population expectations.
The distinction matters whenever an analyst interprets a long sample mean or variance as an estimate of a population moment. An average can settle down as the sample grows but settle at a path-specific value rather than the unconditional population value. The key question is therefore not simply whether a process is stationary, but whether it is stationary and ergodic for the quantity being averaged.
Definition and the Ergodic Theorem
Consider a strictly stationary process \(\{X_t\}_{t\in\mathbb{Z}}\). A shift moves every time index forward by one while leaving the process’s joint probability law unchanged. An event is shift-invariant if shifting the entire path does not change whether the event occurs, apart from events of probability zero.
Ergodicity does not mean that observations are independent. It concerns what happens to averages over a long path, not whether nearby observations are related. The result that makes this property useful is Birkhoff’s ergodic theorem.
The expectation in this statement is the population expectation under the stationary distribution. “Almost surely” means that the convergence holds for all paths except a set with probability zero. It does not say that a finite sample average equals the population expectation, or that every possible realized path must converge.
For \(g(x)=x\), the condition is \(E|X_0|<\infty\), and an ergodic process satisfies \[ \frac{1}{n}\sum_{t=1}^{n}X_t \longrightarrow E[X_0] \quad\text{almost surely.} \] For a raw moment of order \(r\), take \(g(x)=x^r\); the required condition is \(E|X_0|^r<\infty\). Thus, ergodicity and the relevant moment condition together justify using a long-run average to recover that population moment.
From Raw Moments to the Sample Variance
The same result explains why the usual time-series sample variance can reveal the population variance under appropriate conditions. Suppose the process is strictly stationary and ergodic, and \(E[X_0^2]<\infty\). Write the time average as \(\bar{X}_n=n^{-1}\sum_{t=1}^{n}X_t\), and define the variance with denominator \(n\) as \(v_n=n^{-1}\sum_{t=1}^{n}(X_t-\bar{X}_n)^2\). Expanding the square gives the identity
The ergodic theorem applied to \(g(x)=x\) gives \(\bar{X}_n\to E[X_0]\) almost surely. Applied to \(g(x)=x^2\), it gives \(n^{-1}\sum X_t^2\to E[X_0^2]\) almost surely. Subtracting the limits yields
Using \(n-1\) instead of \(n\) in the sample-variance denominator does not change this limit: the resulting variance is \(n v_n/(n-1)\), and \(n/(n-1)\to 1\). This argument establishes convergence of the sample variance; it does not provide a rate of convergence or an inferential procedure.
Worked Example: A Time Average in an IID Bernoulli Process
Suppose, hypothetically, that \(X_t\) records whether a machine produces a defective item at time \(t\): \(X_t=1\) for a defect and \(X_t=0\) otherwise. Assume the observations are independent and identically distributed Bernoulli random variables with defect probability \(p=0.06\). The process is strictly stationary and ergodic, and \(E|X_t|=0.06<\infty\). The ergodic theorem therefore says that the fraction of defects in a long run converges almost surely to \(0.06\).
For a hypothetical run of 10 observations, suppose the recorded values are \(1,0,0,0,1,0,0,0,0,0\). The time average is
This finite-sample average is 0.20, or 20% defective items, while the model’s population mean is 0.06. There is no contradiction: ergodicity describes the limit as the number of observations grows, not the value that every short run must produce. The example also shows why a convergence result should not be treated as a guarantee of close agreement in a small sample.
A Stationary Mixture Can Have a Random Limit
A useful counterexample is a mixture in which a hidden state is drawn once and then affects the distribution of every observation. Conditional on that state, observations may behave quite regularly. Unconditionally, however, the process can be stationary without being ergodic.
Worked Example: A Non-Ergodic Mixture of Bernoulli Processes
Suppose a hidden state \(S\) is selected once, with \(P(S=L)=P(S=H)=1/2\). Conditional on \(S=L\), let the \(X_t\) be IID Bernoulli with probability \(0.2\); conditional on \(S=H\), let them be IID Bernoulli with probability \(0.8\). This is a hypothetical model for a production line that operates in one of two persistent regimes.
The process is strictly stationary: shifting time does not change either regime’s IID law, or the probabilities assigned to the two regimes. Its unconditional mean is
But the time average converges to \(0.2\) if \(S=L\) and to \(0.8\) if \(S=H\). For instance, a hypothetical 10-observation path in the low regime with two defects has average \(2/10=0.20\). Extending the run does not make its average converge to the unconditional mean of \(0.5\); it makes the average converge to the mean of the regime that was drawn.
This process is not ergodic. The event that the long-run average exceeds \(0.5\) is invariant under shifting the path, and it has probability \(1/2\), not zero or one. The hidden regime persists in the path, so an observer of one long realization learns its regime-specific mean rather than the mixture’s unconditional mean.
The same issue affects the variance. Since \(X_t\) is binary, \(E[X_t^2]=E[X_t]=0.5\), so the unconditional variance is \(0.5-(0.5)^2=0.25\). Within either regime, the variance is \(0.2(0.8)=0.16\). A long sample variance within one realized regime converges to \(0.16\), not the mixture variance \(0.25\). The difference reflects variation between regime means as well as variation within a regime.
Ergodicity Is Not Independence
A process can be ergodic even when its observations have strong dependence. For example, consider a process that alternates between 0 and 2, with the starting phase chosen at random: its path is either \(0,2,0,2,\ldots\) or \(2,0,2,0,\ldots\), each with probability \(1/2\). Shifting time swaps the two equally likely phases, so the process is stationary. The shift also moves between the two phases; the only invariant events are the empty event and the event containing both phases. Thus, the process is ergodic.
Each path spends half its time at 0 and half at 2 in the long run. Its time average therefore converges to the population mean \(1\), and its average of squared observations converges to the population second moment \(2\). Yet the centered observations alternate between \(-1\) and \(1\), so their lag-\(h\) correlation is \((-1)^h\); dependence persists at every lag. The example illustrates that ergodicity is about long-run averages, not independence or the disappearance of correlation.
When Time Averages Reveal Population Moments
For a time average to identify a population moment, check both the process property and the moment condition. Stationarity identifies a stable population distribution; ergodicity connects a typical single path to that distribution; integrability makes the expectation being averaged finite.
| Target quantity | Function to average | Moment condition |
|---|---|---|
| Population mean \(E[X_0]\) | \(g(x)=x\) | \(E|X_0|<\infty\) |
| Raw moment \(E[X_0^r]\) | \(g(x)=x^r\) | \(E|X_0|^r<\infty\) |
| Population variance | First and second raw moments | \(E[X_0^2]<\infty\) |
For a stationary process that is not ergodic, the time average still has a limit under the theorem’s integrability condition, but that limit can depend on the path’s invariant component. This is why a large number of observations does not, by itself, establish that a sample moment estimates the unconditional population moment.
Common Mistake
- Wrong reasoning: “The process is stationary, so its sample mean converges to its population mean.”
- Correction: Stationarity alone is insufficient. For the ergodic theorem to identify the time-average limit with \(E[X_0]\), the process must also be ergodic, and \(E|X_0|\) must be finite. In a stationary non-ergodic mixture, a path’s limit can be a regime-specific conditional mean.
Check Your Understanding
Use the definitions and conditions in this tutorial to answer the following questions.
- State the ergodicity condition in terms of shift-invariant events.
- What integrability condition is needed to apply the ergodic theorem to a time average of \(X_t^2\)?
- In the Bernoulli mixture, why does a long-run average not converge to the unconditional mean of \(0.5\)?
- Why can the alternating 0-and-2 process be ergodic even though its observations are strongly dependent?
- Under what assumptions does the time-series sample variance converge to the population variance?