The Continuous-Time Ergodic Theorem

Long-Run Time Spent in Each State

A stationary distribution describes how a continuous-time Markov chain behaves when its distribution does not change with time. The ergodic theorem gives a different, pathwise interpretation: along almost every trajectory, the fraction of a long observation interval spent in a state approaches that state's stationary probability. The result concerns time spent in states, not merely the fraction of jumps that land there.

Let $(X_t)_{t\ge0}$ be a nonexplosive, irreducible continuous-time Markov chain on a finite or countable state space $S$, defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Use its natural filtration $(\mathcal{F}_t)_{t\ge0}$. Its conservative generator is $Q=(q_{ij})$, with $q_{ij}\ge0$ for $i\ne j$, finite exit rates $q_i=-q_{ii}$, and rows summing to zero. Nonexplosion means that almost surely only finitely many jumps occur in every bounded time interval. For a bounded function $f:S\to\mathbb{R}$, define the time average up to time $t>0$ by $A_t(f)=t^{-1}\int_0^t f(X_s)\,ds$. For a state $i$, choosing $f(j)=\mathbf{1}_{\{i\}}(j)$ makes this the fraction of time spent in $i$.

Theorem (Continuous-time ergodic theorem): Suppose the chain is irreducible and positive recurrent, and let $\pi$ be its stationary probability distribution. Then, for every bounded $f:S\to\mathbb{R}$ and every initial state $x\in S$, $$ \frac{1}{t}\int_0^t f(X_s)\,ds \longrightarrow \sum_{j\in S}\pi_j f(j) \qquad\text{almost surely as }t\to\infty. $$ In particular, $t^{-1}\int_0^t\mathbf{1}_{\{i\}}(X_s)\,ds\to\pi_i$ almost surely for every $i\in S$. Here positive recurrence means that, for a state $r$, the cycle from entering $r$ until the next entry into $r$ after leaving it has finite expected duration. In an irreducible chain, this property does not depend on the choice of $r$.

Irreducibility ensures that the chain communicates between states; positive recurrence supplies return cycles with finite mean duration. We use two standard facts: an irreducible recurrent chain hits any fixed state almost surely from every starting state, and the iid strong law of large numbers holds for integrable random variables. The return-cycle argument below shows why these facts yield the stated time average. The existence and uniqueness of $\pi$ for an irreducible positive recurrent chain are standard results from the earlier material on stationary distributions and positive recurrence.

Proof by Regeneration at a State

First suppose the chain starts at a state $r$ with $q_r>0$. Let $C$ be the time from starting in $r$ to the next entry into $r$ after the chain has left $r$. The initial holding time at $r$ is exponential with mean $1/q_r>0$; positive recurrence gives $0<\mathbb{E}_r C<\infty$. At the return time, the strong Markov property says that the chain starts afresh from $r$. Thus successive cycle lengths and the path segments within those cycles are independent and identically distributed.

For each state $j$, define the expected time spent in $j$ during one cycle by $a_j=\mathbb{E}_r\int_0^C\mathbf{1}_{\{j\}}(X_s)\,ds$. These quantities are nonnegative, and summing over states gives $\sum_j a_j=\mathbb{E}_r C$, by summing the state indicators and using monotone convergence. Therefore $\nu_j=a_j/\mathbb{E}_r C$ defines a probability distribution. We first identify it as stationary.

Proof. Let $g:S\to\mathbb{R}$ be bounded, and fix $s\ge0$. Because the event $\{t<C\}$ is known by time $t$, the Markov property gives $\mathbb{E}_r[\mathbf{1}_{\{t<C\}}P(s)g(X_t)]=\mathbb{E}_r[\mathbf{1}_{\{t<C\}}g(X_{t+s})]$, where $P(s)g(i)=\mathbb{E}_i[g(X_s)]$. Integrating in $t$ and using boundedness and $\mathbb{E}_r C<\infty$, we obtain \[ \mathbb{E}_r\int_0^C P(s)g(X_t)\,dt = \mathbb{E}_r\int_s^{C+s}g(X_u)\,du. \] The right-hand integral minus $\int_0^C g(X_u)\,du$ equals $\int_C^{C+s}g(X_u)\,du-\int_0^s g(X_u)\,du$. By the strong Markov property at $C$, $X_C=r$, so the two boundary integrals have equal expectations. Hence $\sum_i\nu_iP(s)g(i)=\sum_i\nu_i g(i)$. This holds for every bounded $g$ and every $s\ge0$, so $\nu$ is stationary. Uniqueness of the stationary probability for an irreducible positive recurrent chain gives $\nu=\pi$. $\square$

Now let $L_k$ be the duration of cycle $k$, and let $R_k=\int_{\text{cycle }k}f(X_s)\,ds$. The pairs $(L_k,R_k)$ are iid; $\mathbb{E}L_k=\mathbb{E}_rC<\infty$, and $|R_k|\le\|f\|_\infty L_k$, so $R_k$ is integrable. The iid strong law gives \[ \frac{1}{n}\sum_{k=1}^n L_k\longrightarrow\mathbb{E}_r C,\qquad \frac{1}{n}\sum_{k=1}^n R_k\longrightarrow\mathbb{E}_r\int_0^C f(X_s)\,ds \quad\text{almost surely.} \] The ratio of these limits is $\sum_j\pi_jf(j)$, by the definition of $\nu$ and the identification $\nu=\pi$.

To pass from completed cycles to an arbitrary time $t$, let $N(t)$ count the completed cycles by time $t$. The first strong-law limit implies $N(t)/t\to1/\mathbb{E}_r C$ almost surely. The unfinished cycle contributes at most $\|f\|_\infty L_{N(t)+1}$ in absolute value. For iid nonnegative integrable $L_k$, $L_k/k\to0$ almost surely: for each $\varepsilon>0$, $\sum_k\mathbb{P}(L_k>\varepsilon k)<\infty$ by integrability, and the Borel–Cantelli lemma applies; taking a countable sequence of $\varepsilon$'s gives the limit. Consequently $L_{N(t)+1}/t\to0$ almost surely. The unfinished cycle therefore has no effect on the long-run average. From any initial state $x$, the chain hits $r$ in finite time almost surely by irreducibility and recurrence; the bounded contribution before that hit, divided by $t$, tends to zero. This proves the theorem for every $x$. $\square$

Worked Occupation-Time Calculations

Worked Example: A Two-State Chain

Consider a chain on $S=\{0,1\}$ with rates $q_{01}=2$ and $q_{10}=1$. Its generator is $Q=\begin{bmatrix}-2&2\\1&-1\end{bmatrix}$. The row sums are $-2+2=0$ and $1-1=0$. Both states communicate, and this finite irreducible chain is positive recurrent.

Write $\pi=(\pi_0,\pi_1)$. Solving $\pi Q=0$ gives $-2\pi_0+\pi_1=0$, so $\pi_1=2\pi_0$. Together with $\pi_0+\pi_1=1$, this gives $3\pi_0=1$, hence $\pi=(\tfrac13,\tfrac23)$. The ergodic theorem says that the fraction of time spent in state $0$ converges almost surely to $\tfrac13$, and the fraction spent in state $1$ converges to $\tfrac23$, from either initial state.

Check: $\pi Q=(\tfrac13(-2)+\tfrac23(1),\ \tfrac13(2)+\tfrac23(-1))=(0,0)$, and $\tfrac13+\tfrac23=1$.

Worked Example: A Three-State Birth–Death Chain

Let $S=\{0,1,2\}$, with transitions $0\to1$ at rate $2$, $1\to0$ at rate $1$, $1\to2$ at rate $3$, and $2\to1$ at rate $2$. The generator is $Q=\begin{bmatrix}-2&2&0\\1&-4&3\\0&2&-2\end{bmatrix}$. The row sums are $-2+2+0=0$, $1-4+3=0$, and $0+2-2=0$. The chain is finite, irreducible, and positive recurrent.

From $\pi Q=0$, the first coordinate gives $-2\pi_0+\pi_1=0$, so $\pi_1=2\pi_0$. The third gives $3\pi_1-2\pi_2=0$, so $\pi_2=\tfrac32\pi_1=3\pi_0$. Normalization gives $6\pi_0=1$, and therefore $\pi=(\tfrac16,\tfrac13,\tfrac12)$. For $A=\{0,2\}$, add the state probabilities: the long-run fraction of time in $A$ is $\pi_0+\pi_2=\tfrac16+\tfrac12=\tfrac23$, almost surely.

Check: The coordinates of $\pi Q$ are $-\tfrac{2}{6}+\tfrac{1}{3}=0$, $\tfrac{2}{6}-\tfrac{4}{3}+1=0$, and $1-\tfrac{2}{2}=0$. Also $\tfrac16+\tfrac13+\tfrac12=1$, and $\pi_0+\pi_2=\tfrac16+\tfrac36=\tfrac23$.

Worked Example: A Directed Three-State Cycle

On $S=\{0,1,2\}$, let the only jumps be $0\to1$ at rate $1$, $1\to2$ at rate $2$, and $2\to0$ at rate $3$. Then $Q=\begin{bmatrix}-1&1&0\\0&-2&2\\3&0&-3\end{bmatrix}$. Its row sums are $-1+1+0=0$, $0-2+2=0$, and $3+0-3=0$; the chain is finite and irreducible.

The stationarity equations give $-\pi_0+3\pi_2=0$, $\pi_0-2\pi_1=0$, and $2\pi_1-3\pi_2=0$. Thus $\pi_0=3\pi_2$ and $\pi_1=\tfrac12\pi_0$. Taking $\pi_2=2\pi_0/6$ is not the normalization; instead, substitute $\pi_1=\pi_0/2$ and $\pi_2=\pi_0/3$ into the sum. Then $\pi_0+\pi_0/2+\pi_0/3=11\pi_0/6=1$, so $\pi=(\tfrac6{11},\tfrac3{11},\tfrac2{11})$. The long-run fraction in state $1$ is $\tfrac3{11}$ almost surely.

Check: The coordinates of $\pi Q$ are $-\tfrac6{11}+\tfrac6{11}=0$, $\tfrac6{11}-\tfrac6{11}=0$, and $\tfrac6{11}-\tfrac6{11}=0$. The entries sum to $\tfrac{6+3+2}{11}=1$.

Why Jump Frequencies Are Not Time Fractions

A common mistake is to count visits to states and assume those counts give the long-run occupation fractions. In the two-state example, the embedded jump chain alternates between $0$ and $1$, so its jump-visit frequencies are $\tfrac12$ and $\tfrac12$. But the holding time in state $0$ has mean $1/2$, while the holding time in state $1$ has mean $1$. The actual time fractions are $\tfrac13$ and $\tfrac23$, as the calculation of $\pi$ shows. The reason is that time occupation weights visits by the duration of the holding intervals.

The theorem also requires irreducibility and positive recurrence. For a simple example of why irreducibility matters, take two states with $Q=0$. Each state is absorbing, so the long-run occupation fractions are $(1,0)$ if the chain starts at $0$, and $(0,1)$ if it starts at $1$. There is no single stationary probability that gives the occupation fractions for both initial states. Positive recurrence matters as well: the irreducible rate-$\tfrac12$ nearest-neighbour chain on the integers is null recurrent, by the recurrence classification of the embedded simple symmetric random walk. It has no stationary probability distribution, so the theorem's conclusion in terms of stationary probabilities does not apply.

Finally, the conclusion is a time-average statement: it concerns $t^{-1}\int_0^t f(X_s)\,ds$. It does not, by itself, assert that the distribution of $X_t$ at one fixed time converges to $\pi$; convergence to equilibrium is a distinct question.

Check Your Understanding

For each question, specify the assumptions you use and verify any proposed stationary probabilities.

For a two-state chain with rates $q_{01}=3$ and $q_{10}=2$, solve for $\pi$ and state the long-run fraction of time in state $0$.
In the directed-cycle example, what is the long-run fraction of time spent in $\{0,2\}$? Show the sum you use.
Why does the regenerative proof require finite expected cycle length, not just almost-sure return to the reference state?
For the two-state chain with rates $2$ and $1$, compare the jump-visit frequencies with the time-occupation fractions. Explain the difference using the mean holding times.
What does the continuous-time ergodic theorem assert about a bounded function $f$, and what separate kind of convergence does it not assert?

The Continuous-Time Ergodic Theorem

What You'll Learn