Equilibrium and the Time Between Returns
A chain may visit every state repeatedly and still spend very different amounts of calendar time in those states. The distinction matters: a state with a high jump rate can be visited often but occupied only briefly. For an irreducible continuous-time Markov chain, the precise criterion for a probability distribution to describe equilibrium is positive recurrence: the expected time between returns to a state must be finite.
Let \((X_t)_{t\ge0}\) be a non-explosive continuous-time Markov chain on a countable state space \(S\), defined on a probability space \((\Omega,\mathcal F,\mathbb P)\), with natural filtration \((\mathcal F_t)_{t\ge0}\). Write \(Q=(q_{ij})\) for its generator, \(q_i=-q_{ii}\) for its exit rate, and \(P(t)\) for its transition matrix at time \(t\). Non-explosion means that almost surely only finitely many jumps occur in every bounded time interval. Irreducibility means every state can be reached from every other state with positive probability. We use row distributions, so an invariant probability \(\pi\) satisfies \(\pi P(t)=\pi\) for every \(t\ge0\).
The time \(R_i\) includes the initial holding time in \(i\) and the subsequent excursion away from \(i\). This convention makes it the length of a full cycle from an entry into \(i\) to the next entry. In a chain with more than one state, irreducibility gives \(q_i>0\), and the initial holding time has mean \(1/q_i\).
The basic accounting behind the result is the continuous-time return-cycle formula. During a cycle from \(i\) to its next entry, define the occupation time of state \(j\) by \(Z_j=\int_0^{R_i}\mathbf{1}_{\{X_t=j\}}\,dt\). When the mean cycle length is finite, expected occupation per unit expected cycle length gives the invariant probability: \(\pi_j=\mathbb E_i[Z_j]/\mathbb E_i[R_i]\). In particular, the time spent in \(i\) during this cycle is just its initial holding time, whose mean is \(1/q_i\). Thus \(\pi_i=1/(q_i\mathbb E_i[R_i])\).
Proof. We use the standard regenerative return-cycle theorem: for an irreducible, non-explosive CTMC, a finite-mean return cycle to \(i\) defines an invariant probability by expected occupation per cycle, and any invariant probability obeys the return-cycle identity \(\pi_i=1/(q_i\mathbb E_i[R_i])\). This return-cycle theorem is a standard renewal-reward result; its full measure-theoretic proof is beyond this tutorial and is treated with the continuous-time ergodic theory. Now suppose first that the chain is positive recurrent. Its mean cycle length is finite, so the regenerative theorem constructs an invariant probability. If \(\widetilde\pi\) is any invariant probability, irreducibility implies \(\widetilde\pi_i>0\) for every state \(i\): if \(\widetilde\pi_i=0\), invariance and a positive-probability path into \(i\) would force the mass of its starting state to be zero, and repeating this argument contradicts total mass one. The return-cycle identity then gives \(\mathbb E_i[R_i]=1/(q_i\widetilde\pi_i)\), so the chain is positive recurrent. Finally, if \(\pi\) and \(\widetilde\pi\) are two invariant probabilities, both have the same return-cycle occupation formula and the same cycle mean; hence their coordinates agree at every state. Therefore the invariant probability is unique. For a one-state chain, \(P(t)=(1)\), so its only probability distribution is invariant. \(\square\)
The theorem is about equilibrium probabilities, not simply about whether a path returns. Recurrence concerns eventual returns; positive recurrence adds a finite-mean requirement in calendar time. The formula for \(\pi_i\) makes the role of holding rates explicit: even if two states are visited equally often at jump times, the one with longer holding times receives more stationary mass.
Worked Example: A Two-State Chain
Consider a CTMC on \(S=\{0,1\}\), defined on a probability space with its natural filtration, whose rates are \(q_{01}=2\) and \(q_{10}=3\). The generator is \(Q=\begin{bmatrix}-2&2\\3&-3\end{bmatrix}\). Its row sums are \(-2+2=0\) and \(3-3=0\). The chain is irreducible and non-explosive: its exit rates are bounded by \(3\).
Starting at \(0\), the chain holds there for mean \(1/2\), moves to \(1\), holds there for mean \(1/3\), and then returns to \(0\). Therefore \(\mathbb E_0[R_0]=\tfrac12+\tfrac13=\tfrac56\). The return-cycle formula gives \(\pi_0=1/(2\cdot\tfrac56)=\tfrac35\). The cycle spends mean \(1/3\) in state \(1\), so \(\pi_1=(\tfrac13)/(\tfrac56)=\tfrac25\). These masses sum to \(\tfrac35+\tfrac25=1\).
Check by the generator equations. The row vector \(\pi=(\tfrac35,\tfrac25)\) gives \(\pi Q=( -\tfrac65+\tfrac65,\ \tfrac65-\tfrac65)=(0,0)\). For a finite-state chain, \(\pi Q=0\) implies \(\pi e^{tQ}=\pi\) for all \(t\ge0\), since \(\pi Q^k=0\) for every integer \(k\ge1\) and the matrix exponential is its power series. Thus the computed probability is invariant.
Worked Example: A Three-State Cycle with Unequal Rates
On \(S=\{1,2,3\}\), let the chain move only \(1\to2\), \(2\to3\), and \(3\to1\), at respective rates \(1,2,3\). Its generator is \(Q=\begin{bmatrix}-1&1&0\\0&-2&2\\3&0&-3\end{bmatrix}\). The row sums are \(-1+1+0=0\), \(0-2+2=0\), and \(3+0-3=0\). All exit rates are bounded by \(3\), so the chain is non-explosive, and the directed cycle makes it irreducible.
Each return cycle to state \(1\) includes one holding time in each state. Their mean durations are \(1\), \(1/2\), and \(1/3\), so \(\mathbb E_1[R_1]=1+\tfrac12+\tfrac13=\tfrac{11}{6}\). The mean occupation amounts in states \(1,2,3\) are respectively \(1,\tfrac12,\tfrac13\). Dividing each by \(\tfrac{11}{6}\) gives \(\pi=(\tfrac6{11},\tfrac3{11},\tfrac2{11})\), whose entries sum to \(\tfrac{11}{11}=1\).
Check. Multiplying by the generator gives first coordinate \(-\tfrac6{11}+3\cdot\tfrac2{11}=0\), second coordinate \(\tfrac6{11}-2\cdot\tfrac3{11}=0\), and third coordinate \(2\cdot\tfrac3{11}-3\cdot\tfrac2{11}=0\). Thus \(\pi Q=0\), and the finite-state matrix-exponential argument verifies invariance. The unequal stationary masses reflect the unequal holding times: the chain spends the longest average time in state \(1\), where its exit rate is smallest.
Worked Example: Immigration and Death
Consider the immigration-death process on \(S=\{0,1,2,\ldots\}\) with immigration rate \(1\) and per-individual death rate \(1\). Thus \(q_{n,n+1}=1\), \(q_{n,n-1}=n\) for \(n\ge1\), and \(q_{nn}=-(1+n)\) for \(n\ge1\), while \(q_{00}=-1\). At state \(0\), the generator row sums to \(-1+1=0\). At \(n\ge1\), the row sum is \(n+1-(n+1)=0\). The chain is irreducible and non-explosive; its rates grow at most linearly, and the linear-rate non-explosion criterion applies.
Set \(\pi_n=e^{-1}/n!\) for \(n\ge0\). These entries sum to one because \(\sum_{n=0}^{\infty}1/n!=e\). At \(n=0\), the incoming rate from state \(1\) is \(\pi_1\cdot1=e^{-1}\), equal to the outgoing rate \(\pi_0\cdot1=e^{-1}\). For \(n\ge1\), incoming flow is \(\pi_{n-1}\cdot1+\pi_{n+1}(n+1)=e^{-1}/(n-1)!+e^{-1}/n!\). Since \(1/(n-1)!=n/n!\), this equals \(e^{-1}(n+1)/n!=\pi_n(n+1)\), the outgoing flow. These balances, together with non-explosion, give invariance.
The mean cycle length at \(0\) follows from the return-cycle identity: \(q_0=1\) and \(\pi_0=e^{-1}\), hence \(\mathbb E_0[R_0]=1/(\pi_0q_0)=e\), which is finite. This agrees with the theorem: the chain is positive recurrent and its invariant probability is unique.
Jump Frequencies Are Not Time Probabilities
A useful way to check the distinction is to consider the embedded jump chain, which records the states visited at jump times and ignores how long the chain waits in each state. In the three-state example, that chain follows the deterministic cycle, so each state has long-run visit frequency \(1/3\). But its CTMC invariant probability is \((\tfrac6{11},\tfrac3{11},\tfrac2{11})\), not \((\tfrac13,\tfrac13,\tfrac13)\). The jump chain counts visits; the CTMC equilibrium weights the duration of those visits.
Common mistake. Assigning equal stationary probability to the three states because the jump chain visits them equally often gives the wrong answer. For the generator in the worked example, this proposed vector is \((\tfrac13,\tfrac13,\tfrac13)\), and its product with \(Q\) is \((\tfrac23,-\tfrac13,-\tfrac13)\), not \((0,0,0)\). The correction is to account for holding times: the mean holds are \(1,\tfrac12,\tfrac13\), which normalize to \((\tfrac6{11},\tfrac3{11},\tfrac2{11})\). Always check the proposed probability against invariance, rather than treating jump counts as time occupation.
Non-explosion is also part of the theorem’s setting. It ensures the chain is defined through every finite time and that return-cycle occupation is meaningful without missing mass at an explosion time. Irreducibility rules out separate closed classes with different invariant probabilities. The conclusion is consequently precise: in this setting, finite expected calendar-time return cycles are necessary and sufficient for a unique equilibrium probability.
Check Your Understanding
Use the return-cycle interpretation and verify numerical answers by substitution.
- For a two-state chain with rates \(q_{01}=4\) and \(q_{10}=2\), compute the invariant probabilities and the mean return-cycle length to state \(0\).
- For the three-state directed cycle with rates \(1,2,3\), why are the jump-chain frequencies \(1/3\) each, while the CTMC invariant probabilities are unequal?
- State the return-cycle identity relating \(\pi_i\), \(q_i\), and \(\mathbb E_i[R_i]\). What does it imply if the expected cycle length is infinite?
- Check whether \((\tfrac12,\tfrac12)\) is invariant for the two-state generator \(\begin{bmatrix}-2&2\\3&-3\end{bmatrix}\) by multiplying the row vector by the generator.
- Which two hypotheses in the theorem prevent separate equilibrium classes and loss of probability through infinitely many jumps in finite time?