Convergence to Equilibrium for CTMCs

From Stationarity to Convergence

A stationary distribution describes an equilibrium that remains unchanged in time. It does not, by itself, say that a chain started from a particular state approaches that equilibrium. For continuous-time Markov chains, bounded rates allow a useful bridge to discrete time: observe a discrete chain at the event times of an independent Poisson process. The resulting representation averages over different numbers of discrete steps, and that averaging removes periodic oscillations.

Let \((X_t)_{t\ge0}\) be a time-homogeneous continuous-time Markov chain on a countable state space \(S\), defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Write \(P(t)=(p_{ij}(t))\), where \(p_{ij}(t)=\mathbb{P}_i(X_t=j)\), and let its conservative generator be \(Q=(q_{ij})\). Thus \(q_{ij}\ge0\) for \(i\ne j\), and each row sums to zero. Set \(q_i=-q_{ii}=\sum_{j\ne i}q_{ij}\), the total rate of leaving state \(i\). Irreducibility means that every state can be reached from every other state with positive probability at some time. Positive recurrence means that the expected return time, measured in continuous time, is finite for each state.

The conclusion concerns the distribution at a fixed large time, not merely the fraction of time spent in each state. The earlier continuous-time ergodic theorem concerns long-run time averages; convergence to equilibrium is a stronger, different assertion. The proof below reduces the assertion to the discrete-time convergence theorem: an irreducible, aperiodic, positive recurrent countable-state Markov chain with transition matrix \(R\) and stationary distribution \(\pi\) satisfies \(R^n_{ij}\to\pi_j\). We use that standard discrete-time result from Markov-chain theory.

Uniformization and the Convergence Proof

Choose a constant \(c>0\) strictly larger than \(\sup_i q_i\), and define \(R=I+Q/c\), where \(I\) is the identity matrix. For distinct states, \(r_{ij}=q_{ij}/c\); on the diagonal, \(r_{ii}=1-q_i/c\). These entries are nonnegative, and each row sums to \(1\), since \(\sum_j r_{ij}=1+\sum_jq_{ij}/c=1\). In particular, \(r_{ii}>0\) at every state. This strict inequality will ensure aperiodicity.

To see the representation behind this matrix, let \((Y_n)_{n\ge0}\) be a discrete-time Markov chain with transition matrix \(R\), and independently let \(N_t\) be a Poisson process of rate \(c\). The process \(Y_{N_t}\) has the same transition law as the CTMC. In a time interval of length \(h\), the probability of no Poisson event is \(1-ch+o(h)\), the probability of one event is \(ch+o(h)\), and the probability of two or more events is \(o(h)\). Therefore its transition matrix over that interval is \(I+ch(R-I)+o(h)=I+hQ+o(h)\), which has generator \(Q\). Equivalently, conditioning on the Poisson count gives the uniformization formula

The weights in this sum are the probabilities that a rate-\(c\) Poisson process has \(n\) events by time \(t\). They sum to \(1\). Bounded rates ensure that the CTMC is nonexplosive, so this construction describes its transition probabilities for all finite times.

Proof. Fix \(i,j\in S\). Since \(Q=c(R-I)\), the off-diagonal entries of \(R\) are positive exactly where the corresponding off-diagonal rates of \(Q\) are positive. Thus any communicating path for the CTMC gives a path for \(R\), and \(R\) is irreducible. Also \(r_{ii}>0\), so the chain with transition matrix \(R\) can return to each state in one step with positive probability; its period is \(1\). A stationary distribution \(\pi\) for the CTMC satisfies \(\pi Q=0\), and hence \(\pi R=\pi+\pi Q/c=\pi\). An irreducible discrete-time chain with an invariant probability distribution is positive recurrent. The discrete-time convergence theorem therefore gives \(R^n_{ij}\to\pi_j\).

Let \(a_n=R^n_{ij}\), so \(0\le a_n\le1\). By uniformization, \(p_{ij}(t)\) is the average of \(a_n\) when \(n\) has a Poisson distribution with mean \(ct\). Given \(\varepsilon>0\), choose \(N\) so that \(|a_n-\pi_j|<\varepsilon\) for every \(n\ge N\). Splitting the sum at \(N\) gives

Here the bound \(1\) applies because both \(a_n\) and \(\pi_j\) lie in \([0,1]\). For fixed \(N\), \(\mathbb{P}(N_t<N)=e^{-ct}\sum_{n=0}^{N-1}(ct)^n/n!\to0\) as \(t\to\infty\): this is a finite sum of terms that each tend to zero. Thus \(\limsup_{t\to\infty}|p_{ij}(t)-\pi_j|\le\varepsilon\). Since this holds for every \(\varepsilon>0\), \(p_{ij}(t)\to\pi_j\).

For completeness, this coordinate-wise convergence also gives total variation convergence for a fixed starting state \(i\). Choose a finite set \(F\subseteq S\) with \(\pi(F)>1-\varepsilon\). On \(F\), the sum \(\sum_{j\in F}|p_{ij}(t)-\pi_j|\) tends to zero because it has finitely many terms. The mass outside \(F\) under \(P_i(X_t\in\cdot)\) tends to \(1-\pi(F)<\varepsilon\), since the mass on \(F\) converges to \(\pi(F)\). The sum of absolute differences outside \(F\) is at most the sum of the two outside masses, hence is eventually at most \(3\varepsilon\). Since \(\varepsilon\) is arbitrary, the full sum tends to zero. For a fixed initial distribution, first restrict its mass to a finite set of starting states with arbitrarily small remaining mass, and apply the same argument to the finite mixture. \(\square\)

Worked Consequences

What the Assumptions Protect

Positive recurrence is essential for convergence to a stationary probability. For example, consider the continuous-time simple symmetric random walk on the integers \(\mathbb{Z}\), jumping to each neighbor at rate \(1\). Its rates are bounded and the chain is irreducible, but it is null recurrent and has no stationary probability distribution. Indeed, a stationary distribution would have to satisfy \(\pi_{n-1}+\pi_{n+1}-2\pi_n=0\) at every \(n\). This implies that the successive differences \(\pi_{n+1}-\pi_n\) are constant, so \(\pi_n\) is an affine function of \(n\). No nonzero nonnegative affine sequence on all of \(\mathbb{Z}\) can have finite sum. There is therefore no \(\pi\) to which the transition probabilities could converge as a probability distribution.

Irreducibility matters as well: if two states are both absorbing, a chain started at either state stays there forever, so there is no single limiting distribution shared by both starting states. Bounded rates supply the simple uniformization argument used here and ensure nonexplosion; they are a sufficient hypothesis for this result, not a claim that every unbounded-rate chain fails to converge.

Common pitfall. It is wrong to infer that a periodic jump chain prevents a CTMC from converging. In the two-state example, the jump-chain probabilities from state \(1\) alternate between \(1\) and \(0\) at successive steps. Treating those step probabilities as if they were \(p_{11}(t)\) would incorrectly predict oscillation forever. At time \(t\), however, the number of jumps is random, and the correct calculation is \(p_{11}(t)=(1+e^{-2t})/2\to1/2\). Stationarity means \(\pi P(t)=\pi\) at every time; convergence says that transition probabilities from each fixed starting state approach that stationary distribution.