Recurrence and Transience of CTMC States

Returns After a Departure

In continuous time, a state is occupied for a positive holding time before the chain jumps. Counting that initial holding period as a “return” would blur the distinction between remaining in a state and leaving it and coming back. The natural recurrence question is therefore: after the chain departs from a state, does it ever visit that state again? For a nonabsorbing state, this question can be answered using only the sequence of states visited at jump times.

Let $(X_t)_{t\ge0}$ be a nonexplosive, time-homogeneous CTMC on a finite or countable state space $S$, defined on a probability space $(\Omega,\mathcal F,\mathbb P)$, with natural filtration $(\mathcal F_t)_{t\ge0}$ and generator $Q=(q_{ij})$. Assume the exit rate $q_i=-q_{ii}$ is finite for every $i\in S$. For $q_i>0$, the holding time in state $i$ is exponential with rate $q_i$. At the next jump, the destination is $j\ne i$ with probability $q_{ij}/q_i$. The generator rows sum to zero, so $\sum_{j\ne i}q_{ij}=q_i$; thus these jump probabilities sum to one.

Definition: For $q_i>0$, define the embedded jump chain $(Y_n)_{n\ge0}$ by recording the states visited at successive jumps, with $Y_0$ the starting state. Its transition probabilities are $r_{ij}=q_{ij}/q_i$ for $j\ne i$, and $r_{ii}=0$. The post-departure return probability is $f_i=\mathbb P_i(\inf\{n\ge1:Y_n=i\}<\infty)$. The state $i$ is recurrent if $f_i=1$, and transient if $f_i<1$. Its expected number of post-departure visits is $V_i=\mathbb E_i[\sum_{n\ge1}\mathbf 1_{\{Y_n=i\}}]$, which may be infinite.

Here $\mathbb P_i$ denotes the law of the chain started at $i$. Because the CTMC is nonexplosive, its jump times do not accumulate in finite time, and the jump chain records its successive actual state changes. Recurrence concerns whether the state is visited again, not how long the chain waits between jumps.

A state with $q_i=0$ is different: since its generator row has no off-diagonal rates, it is absorbing. It never departs, so the post-departure return probability above is not defined for it. Under the common occupancy convention, an absorbing state is called recurrent because a chain started there remains there for all time. We keep that case separate rather than treating “no departure” as a return after departure.

Return Probability and Expected Visits

Theorem: For any nonabsorbing state $i$, the embedded jump chain satisfies $$ V_i=\sum_{n=1}^{\infty}\mathbb P_i(Y_n=i) =\begin{cases} \dfrac{f_i}{1-f_i},& f_i<1,\\[6pt] \infty,& f_i=1. \end{cases} $$ Consequently, $i$ is recurrent exactly when its expected number of post-departure visits is infinite; it is transient exactly when that expectation is finite.

Proof. Let $T_1=\inf\{n\ge1:Y_n=i\}$, and, after each return, define $T_{k+1}=\inf\{n>T_k:Y_n=i\}$, with an infinite value if the indicated return never occurs. The number of post-departure visits is $N_i=\sum_{n\ge1}\mathbf 1_{\{Y_n=i\}}$. For each integer $k\ge1$, the event $\{N_i\ge k\}$ is the event that $T_k<\infty$. At each return index, the future transition probabilities are those of the jump chain started at $i$: this follows by conditioning on each possible finite value of that index and applying the Markov property at that deterministic index. Therefore, after any return, the conditional probability of a further return is $f_i$. Repeated conditioning gives $\mathbb P_i(N_i\ge k)=f_i^k$. For a nonnegative integer-valued random variable, $\mathbb E_i[N_i]=\sum_{k\ge1}\mathbb P_i(N_i\ge k)$, by summing its indicator representation. Hence $V_i=\sum_{k\ge1}f_i^k$, which equals $f_i/(1-f_i)$ when $f_i<1$ and diverges when $f_i=1$. Since $f_i$ is a probability, these cases exhaust all possibilities and give the stated classification. $\square$

The theorem also gives the equivalent series test $V_i=\sum_{n\ge1}r_{ii}^{(n)}$, where $r_{ii}^{(n)}=\mathbb P_i(Y_n=i)$. In applications, one may find the return probability directly or sum the $n$-step return probabilities. The theorem concerns the number of visits in the jump chain; it does not say that recurrent states have finite mean return time in calendar time. That distinction is important, and the classification here does not include positive recurrence.

Worked Examples Using the Jump Chain

Worked Example: A Two-State CTMC Is Recurrent

Let $S=\{0,1\}$, and let the generator be $$ Q=\begin{bmatrix}-2&2\\3&-3\end{bmatrix}. $$ Each row sums to zero: $-2+2=0$ and $3-3=0$, so the exit rates are $q_0=2$ and $q_1=3$. Find the recurrence classification of state $0$.

The embedded jump probabilities are $r_{01}=2/2=1$ and $r_{10}=3/3=1$. Each row of the embedded transition matrix therefore sums to one, and $$ R=\begin{bmatrix}0&1\\1&0\end{bmatrix}. $$ Starting from $0$, the chain jumps to $1$ at its first jump and back to $0$ at its second. Thus $\mathbb P_0(T_1=2)=1$, where $T_1=\inf\{n\ge1:Y_n=0\}$, so $f_0=1$. Subsequent visits occur at every even jump index. Therefore $V_0=\sum_{m\ge1}1=\infty$, and state $0$ is recurrent.

Check: The first-return probability is one, and the return-probability theorem gives $V_0=\infty$. The different exit rates affect the holding times, but not this jump-chain calculation.

Worked Example: A State That Jumps to an Absorbing State

Let $S=\{0,1\}$ and $$ Q=\begin{bmatrix}0&0\\1&-1\end{bmatrix}. $$ The row sums are $0+0=0$ and $1-1=0$. State $0$ is absorbing, while the exit rate from state $1$ is $q_1=1$. Classify state $1$.

From state $1$, the embedded jump probabilities are $r_{10}=1/1=1$ and $r_{11}=0$. The jump-chain row for state $1$ sums to $1$. Once the chain jumps from $1$ to $0$, it stays there, so there is no later jump-chain visit to $1$. Thus $f_1=0$, and $$ V_1=\sum_{n\ge1}\mathbb P_1(Y_n=1)=0. $$ State $1$ is transient.

Check: Substitution in the theorem gives $V_1=f_1/(1-f_1)=0/(1-0)=0$. State $0$ is not assigned a post-departure return probability by this definition because it has no departure. Under the occupancy convention, it is absorbing and recurrent.

Worked Example: Symmetric Random Walk on the Integers

Consider a CTMC on $S=\mathbb Z$ with $q_{k,k+1}=q_{k,k-1}=\tfrac12$ and all other off-diagonal rates zero. For every $k$, the generator row sums to $\tfrac12+\tfrac12-1=0$, so $q_k=1$. Its embedded jump chain moves one step right or left with probability $\tfrac12$ each; each row sums to $\tfrac12+\tfrac12=1$. The CTMC is nonexplosive: its jump rate is constantly $1$, so the number of jumps in each finite time interval is finite almost surely. We show that state $0$ is recurrent.

Starting from $0$, the first jump goes to either $1$ or $-1$. By symmetry it is enough to show that the embedded walk started at $1$ hits $0$ with probability one. Fix an integer $N\ge2$, and let $h_k$ be the probability that the walk started at $k\in\{0,\ldots,N\}$ hits $0$ before $N$. The boundary values are $h_0=1$ and $h_N=0$. For $1\le k\le N-1$, conditioning on the first jump gives $h_k=\tfrac12h_{k-1}+\tfrac12h_{k+1}$. This equation says the successive differences $h_k-h_{k-1}$ are constant. The boundary values then yield $h_k=1-k/N$: its endpoints are $1$ and $0$, and $\tfrac12(1-(k-1)/N)+\tfrac12(1-(k+1)/N)=1-k/N$, verifying the equation.

In particular, the probability of hitting $0$ before $N$ from $1$ is $h_1=1-1/N=(N-1)/N$. As $N$ increases, these events increase: hitting $0$ before $N$ implies hitting $0$ before $N+1$. Their union is the event of ever hitting $0$ from $1$, since any finite path that hits $0$ has a finite maximum before that hit. Thus the probability of ever hitting $0$ is $\lim_{N\to\infty}(N-1)/N=1$. The same holds from $-1$ by symmetry, so $f_0=1$, and the theorem gives $V_0=\infty$.

Check: The finite-interval formula gives $h_1=1/2$ at $N=2$, and $h_1=2/3$ at $N=3$; both equal $(N-1)/N$. These probabilities increase to $1$, as required for the recurrence conclusion.

Rates, Visits, and a Common Pitfall

For a state with $q_i>0$, each visit to $i$ has a mean holding time $1/q_i$. In the usual CTMC construction, holding times at successive visits have this mean, and the destination probabilities are given by the embedded jump chain. Consequently, if $U_i=\int_0^\infty \mathbf 1_{\{X_t=i\}}\,dt$ is total occupation time, then $$ \mathbb E_i[U_i]=\frac{1}{q_i}(1+V_i). $$ The $1$ counts the initial visit, while $V_i$ counts visits after departure. This identity follows by adding the holding times over visits and taking expectations; for a possibly infinite number of visits, nonnegativity permits the sum and expectation to be interchanged. Thus changing the exit rate changes the time spent per visit, but does not change the embedded return probability. In particular, recurrence is not a statement that the chain spends a specified fraction of time in the state.

Common pitfall. Off-diagonal entries of a generator are rates, not jump probabilities. For the two-state example, treating $q_{01}=2$ and $q_{10}=3$ as probabilities would produce a purported two-jump return probability $2\cdot3=6$, which is impossible. The correction is to divide each rate by the exit rate in its row: $r_{01}=2/q_0=2/2=1$ and $r_{10}=3/q_1=3/3=1$. The correct return probability is $1$.

Check Your Understanding

Use post-departure visits, and distinguish jump probabilities from rates.

For a nonabsorbing state $i$, what is the embedded jump probability $r_{ij}$ in terms of $q_{ij}$ and $q_i$?
If $f_i=\tfrac12$, compute $V_i$ using the return-probability theorem.
A state has $f_i=1$. What does this imply about its expected number of post-departure visits?
Why is an absorbing state treated separately under a definition based on returning after departure?
In the symmetric random-walk example, why does the probability of hitting $0$ before $N$ from $1$ increase to the probability of ever hitting $0$?

Recurrence and Transience of CTMC States

What You'll Learn