Equilibrium Probability and Recurrence
For a finite irreducible continuous-time Markov chain, long-run equilibrium can be described by a probability distribution that is unchanged by running the chain for any amount of time. On an infinite state space, recurrence alone does not guarantee such a probability: a recurrent chain can return to each state eventually while spending too much time spreading its visits across infinitely many states for those visits to have a probability distribution. A transient chain has an even more basic obstruction: it does not return to a state with probability one.
Throughout, let \((X_t)_{t\ge0}\) be a conservative, nonexplosive, irreducible continuous-time Markov chain on a countable state space \(S\), defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Assume its exit rates \(q_i=-q_{ii}\) are finite. Write \(P(t)=(p_{ij}(t))\) for its transition matrix at time \(t\). A probability row vector \(\pi\) is an invariant probability if \(\pi P(t)=\pi\) for every \(t\ge0\). For an irreducible chain, a state is recurrent if, after leaving it, the chain returns with probability one; it is transient if the return probability is less than one. A recurrent state is positive recurrent if its mean return time after leaving is finite, and null recurrent if that mean is infinite. These classifications agree across states in an irreducible chain.
The key idea in the proof is to sample the continuous-time chain at equally spaced times. Invariance then gives an invariant probability for a discrete-time chain. A discrete-time return-time identity forces a finite expected sampled return, and a careful first-step decomposition converts that fact into a finite expected continuous-time return. The mixture over possible exit states must be handled explicitly; finiteness for each exit state separately would not by itself establish finiteness of the mixture.
Sampling and the Return-Time Argument
Fix \(h>0\) and define \(Y_n=X_{nh}\) for \(n\ge0\). This is a discrete-time Markov chain with transition matrix \(P(h)\). It is irreducible: for any \(i,j\in S\), irreducibility of the continuous-time chain supplies a finite path of positive-rate transitions from \(i\) to \(j\). There is positive probability of making those transitions in the time interval \((0,h)\), in the prescribed order, and then making no further jump before time \(h\). Thus \(p_{ij}(h)>0\). If \(\pi\) is invariant for the continuous-time chain, then \(\pi P(h)=\pi\), so it is also invariant for the sampled chain.
We use the standard discrete-time Kac return-time formula: for an irreducible discrete-time chain with invariant probability \(\pi\), the first return time \(\tau_i^+=\inf\{n\ge1:Y_n=i\}\), when \(Y_0=i\), satisfies \(\mathbb{E}_i[\tau_i^+]=1/\pi_i\). In particular, \(\pi_i>0\) and this expectation is finite. This standard result is the discrete-time return-time identity; the argument below uses it only for the sampled chain.
Proof of the theorem's necessary direction. Suppose that \(\pi\) is an invariant probability. Fix a state \(i\), and use the sampled chain just defined. For \(j\ne i\), let \(H_i=\inf\{n\ge0:Y_n=i\}\) when the sampled chain starts at \(j\). On the first sampled step from \(i\), either \(Y_1=i\), in which case \(\tau_i^+=1\), or \(Y_1=j\ne i\), in which case \(\tau_i^+=1+H_i\). Conditioning on \(Y_1\) and using the Markov property gives
The left side is finite by Kac's formula. All terms on the right are nonnegative, so in particular \(\sum_{j\ne i}p_{ij}(h)\mathbb{E}_j[H_i]<\infty\).
Set \(r=1-p_{ii}(h)\). In a nontrivial irreducible chain, \(r>0\): there is positive probability of leaving \(i\) before time \(h\) and then making no further jump before \(h\). Let \(G=\inf\{n\ge1:Y_n\ne i\}\), the first sampled exit from \(i\). Each sampled step taken while the chain is at \(i\) has exit probability \(r\), so \(\mathbb{P}_i(G=m)=p_{ii}(h)^{m-1}r\) for \(m\ge1\), and \(\mathbb{E}_i[G]=1/r<\infty\). For \(j\ne i\), summing over the possible values of \(G\) gives \(\mathbb{P}_i(Y_G=j)=\sum_{m\ge1}p_{ii}(h)^{m-1}p_{ij}(h)=p_{ij}(h)/r\). Consequently, the expected sampled time to hit \(i\) after this sampled exit is
This calculation verifies that the particular random mixture of exit states has finite expected hitting time. By time \((G+H_i)h\), the continuous-time chain has sampled a state different from \(i\) at time \(Gh\) and then sampled \(i\) again. It must therefore have left \(i\) by time \(Gh\), and its first return after leaving is no later than \((G+H_i)h\). The expected value of this bound is finite, since both \(\mathbb{E}_i[G]\) and the expected sampled hitting time after \(G\) are finite. Thus the mean continuous-time return time after leaving \(i\) is finite. Since \(i\) was arbitrary, the irreducible chain is positive recurrent. The one-state chain is absorbing and has its point mass as an invariant probability, so it does not create an exception to the stated necessary direction. \(\square\)
Worked Examples: The Three Recurrence Types
Worked Example: A Null Recurrent Symmetric Walk
Consider the CTMC on \(S=\mathbb{Z}\) that jumps from \(k\) to \(k+1\) at rate \(1\) and from \(k\) to \(k-1\) at rate \(1\). Its generator has \(q_{k,k+1}=q_{k,k-1}=1\), \(q_{kk}=-2\), and all other entries zero. Each generator row sums to \(1+1-2=0\). The chain is nonexplosive because its total jump rate is always \(2\). Its embedded jump chain is the simple symmetric random walk on \(\mathbb{Z}\).
That embedded walk is recurrent: starting at \(1\), the probability of hitting \(0\) before \(N\) is \((N-1)/N\), which tends to \(1\) as \(N\) tends to infinity. The same argument applies on the negative side, so after each departure from \(0\) the walk returns to \(0\) with probability one. The chain is null recurrent rather than positive recurrent: if it were positive recurrent, the positive-recurrence theorem would give an invariant probability, which the calculation below rules out.
For this bounded-rate chain, invariance implies the generator balance equations \(\pi Q=0\). At state \(k\), that equation is \(\pi_{k-1}+\pi_{k+1}-2\pi_k=0\), or equivalently \(\pi_{k+1}-\pi_k=\pi_k-\pi_{k-1}\). The successive differences are therefore constant, so \(\pi_k=A+Bk\) for constants \(A,B\). A nonnegative summable sequence on all of \(\mathbb{Z}\) of this form must have \(B=0\), since otherwise it is negative on one tail; it must then have \(A=0\), since a positive constant is not summable. Thus there is no invariant probability. This agrees with the theorem for null recurrent chains.
Worked Example: A Transient Biased Walk
Now let the state space again be \(\mathbb{Z}\), with rate \(2\) from \(k\) to \(k+1\) and rate \(1\) from \(k\) to \(k-1\). The generator row at \(k\) has entries \(q_{k,k+1}=2\), \(q_{k,k-1}=1\), \(q_{kk}=-3\), and row sum \(2+1-3=0\). The total jump rate is \(3\), so the chain is nonexplosive. Its embedded jump chain moves right with probability \(2/3\) and left with probability \(1/3\).
To check transience, consider the probability, starting at \(1\), of hitting \(0\) before \(N\). The gambler's-ruin calculation for right-step probability \(p=2/3\), left-step probability \(q=1/3\), and \(a=q/p=1/2\) gives \(\mathbb{P}_1(T_0<T_N)=(a-a^N)/(1-a^N)\). As \(N\) tends to infinity this tends to \(1/2\). The events of hitting \(0\) before \(N\) increase to the event of ever hitting \(0\): any path that hits \(0\) in finite time has a finite maximum before that hit. Hence the probability of ever returning from \(1\) to \(0\) is \(1/2\), strictly less than \(1\). The irreducible chain is transient.
There is also no invariant probability, as can be checked directly. The balance equation at \(k\) would require \(2\pi_{k-1}+\pi_{k+1}-3\pi_k=0\). Its characteristic equation is \(x^2-3x+2=0=(x-1)(x-2)\), so every solution has the form \(\pi_k=A+B2^k\). If this sequence is summable over \(\mathbb{Z}\), its terms must tend to zero as \(k\) tends to \(-\infty\), which forces \(A=0\). Then summability as \(k\) tends to \(+\infty\) forces \(B=0\). No nonzero probability solution exists, in agreement with the theorem.
Worked Example: Positive Recurrence in an Immigration-Death Chain
On \(S=\{0,1,2,\ldots\}\), let individuals arrive at rate \(2\), and let each individual die at rate \(1\). Thus \(q_{n,n+1}=2\), \(q_{n,n-1}=n\) for \(n\ge1\), and \(q_{nn}=-(n+2)\); at \(n=0\), \(q_{00}=-2\). For \(n\ge1\), the row sum is \(2+n-(n+2)=0\), and at \(n=0\) it is \(2-2=0\). The chain is irreducible. It is nonexplosive: on any bounded time interval there are finitely many arrivals almost surely, and the number of deaths cannot exceed the initial population plus the number of arrivals.
The invariant candidate is the Poisson distribution with mean \(2\): \(\pi_n=e^{-2}2^n/n!\) for \(n\ge0\). Its entries are nonnegative and \(\sum_{n\ge0}\pi_n=e^{-2}\sum_{n\ge0}2^n/n!=e^{-2}e^2=1\). For \(n\ge1\), the incoming generator balance is \(2\pi_{n-1}+(n+1)\pi_{n+1}\). Substitution gives \(2\pi_{n-1}=n\pi_n\) and \((n+1)\pi_{n+1}=2\pi_n\), so the incoming total is \((n+2)\pi_n\), exactly the outgoing rate times \(\pi_n\). At \(n=0\), the incoming flow is \(\pi_1=2\pi_0\), equal to the outgoing flow \(2\pi_0\).
For completeness, invariance also follows from the construction of the process. If the initial population is Poisson with mean \(2\), each initial individual survives to time \(t\) with probability \(e^{-t}\), so the surviving initial population is Poisson with mean \(2e^{-t}\). Arrivals form a rate-\(2\) Poisson process; an arrival at time \(s\le t\) survives to \(t\) with probability \(e^{-(t-s)}\), so the number of surviving arrivals at \(t\) is Poisson with mean \(\int_0^t 2e^{-(t-s)}\,ds=2(1-e^{-t})\). These independent populations sum to a Poisson variable with mean \(2e^{-t}+2(1-e^{-t})=2\). Thus \(\pi P(t)=\pi\) for every \(t\), and the chain is positive recurrent by the irreducible positive-recurrence theorem.
A Common Normalization Error
A formal solution of balance equations is not automatically a probability distribution. For the symmetric walk above, the balance equations admit the constant sequence \(\pi_k=C\). A tempting but wrong calculation is to set \(C=1/\sum_{k\in\mathbb{Z}}1\) and treat the result as a positive normalizing constant. The sum is infinite, so no positive \(C\) makes the sequence sum to \(1\); setting \(C=0\) gives total mass \(0\), not \(1\). The correction is to check summability before normalization. More generally, an invariant measure of infinite total mass is not an invariant probability.
Invariant probability and positive recurrence describe equilibrium in the sense of a time-invariant probability law. They should not be conflated with a claim about convergence from every initial distribution: convergence questions require additional hypotheses and are treated separately. The distinction needed here is simpler: recurrence says whether return occurs; positive recurrence says the mean return time is finite; and an invariant probability requires enough concentration across states to have total mass one.
Check Your Understanding
Use the definitions and return-time argument above to answer each question.
- For a fixed \(h>0\), why does an invariant probability \(\pi\) for \(X_t\) also satisfy \(\pi P(h)=\pi\)?
- In the sampled-chain proof, why is it necessary to show that \(\sum_{j\ne i}p_{ij}(h)\mathbb{E}_j[H_i]\) is finite, rather than only noting that each \(\mathbb{E}_j[H_i]\) is finite?
- For the symmetric walk with rates \(1\) in each direction, solve the balance equation and explain why no nonzero solution of the resulting form is a probability on \(\mathbb{Z}\).
- For the biased walk with right rate \(2\) and left rate \(1\), compute the limiting probability of hitting \(0\) before \(N\), starting from \(1\), as \(N\) tends to infinity.
- For the immigration-death chain, verify directly that \(2\pi_{n-1}=n\pi_n\) and \((n+1)\pi_{n+1}=2\pi_n\) for \(n\ge1\).