The Embedded Jump Chain

From Continuous Time to a Sequence of Jumps

A continuous-time Markov chain records both which states are visited and when they are visited. Sometimes the order of visited states is the main object of interest. Recording the state at each successive jump removes the clock times and leaves a discrete-time chain. Its transition probabilities are determined by the off-diagonal rates of the generator, normalized by the total rate of leaving the current state.

Let \((X_t)_{t\ge0}\) be a time-homogeneous continuous-time Markov chain on a finite or countable state space \(S\), defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), with filtration \((\mathcal{F}_t)_{t\ge0}\). Write its generator as \(Q=(q_{ij})\). For distinct states \(i\) and \(j\), \(q_{ij}\ge0\) is the rate of a jump from \(i\) to \(j\). The generator’s rows sum to zero, so the total exit rate from \(i\) is \(q_i=-q_{ii}=\sum_{j\ne i}q_{ij}\). We assume these rates are finite. The holding time at \(i\), when \(q_i>0\), is exponential with rate \(q_i\).

Set \(\sigma_0=0\). For each successive jump, define \(\sigma_{n+1}=\inf\{t>\sigma_n:X_t\ne X_{\sigma_n}\}\), whenever this time is finite, and let \(Y_n=X_{\sigma_n}\). Thus \(Y_n\) records the state at the \(n\)th jump epoch, with \(Y_0=X_0\). The sequence is considered only for jump epochs that occur before any accumulation of jump times. If the process reaches a state with no possible departure, there is no further genuine jump epoch.

The zero diagonal in the embedded chain is deliberate: a jump means that the state changes. The diagonal entry \(q_{ii}\) of the generator is not a rate of jumping from \(i\) back to \(i\); it is the negative total exit rate. The embedded chain describes jump destinations, not elapsed time.

Why the Rates Normalize to Jump Probabilities

Proof. Fix a current state \(i\). For each \(j\ne i\) with \(q_{ij}>0\), let the clock for a jump to \(j\) be exponential with rate \(q_{ij}\), with the clocks independent. Clocks with rate zero can be omitted. The next jump is to \(j\) exactly when the clock for \(j\) rings before all the other clocks. For \(t\ge0\), the probability density that the clock for \(j\) rings at time \(t\) and every other clock has not rung by then is \(q_{ij}e^{-q_{ij}t}\prod_{k\ne i,j}e^{-q_{ik}t}=q_{ij}e^{-q_it}\). Integrating over \(t\ge0\), the probability that \(j\) is the destination is \(\int_0^\infty q_{ij}e^{-q_it}\,dt=q_{ij}/q_i\), since \(q_i>0\). Independent exponential clocks have probability zero of tying, so there is almost surely a unique first clock. Summing the destination probabilities gives \(\sum_{j\ne i}q_{ij}/q_i=q_i/q_i=1\), using \(q_i=\sum_{j\ne i}q_{ij}\). After a jump, the time-homogeneous Markov jump mechanism applies the same clock rule at the new state; consequently, conditional on the jump history and the current state, the next-state probabilities depend only on that current state. This is the Markov property for the embedded sequence. \(\square\)

The theorem requires \(q_i>0\). If \(q_i=0\), nonnegativity of the off-diagonal rates and their sum \(q_i\) imply \(q_{ij}=0\) for every \(j\ne i\). The state has no possible departure, and dividing by \(q_i\) is not valid. This case needs a separate convention if a stochastic matrix on all of \(S\) is desired.

Computing an Embedded Transition Matrix

More generally, when every state has positive exit rate, the embedded transition matrix on \(S\) is row-stochastic: its entries are nonnegative and each row sums to one. If the state space is countable, the row sum is understood as a sum over all distinct destination states; the finite exit-rate assumption ensures that this sum is \(q_i/q_i=1\).

States with No Possible Departure

When \(q_i=0\), the process remains at \(i\) forever once it reaches \(i\). The sequence of genuine jump destinations ends there: there is no next jump to record. In some calculations, it is convenient to extend the sequence by declaring that it stays at \(i\) thereafter. This produces an ordinary discrete-time Markov chain with an absorbing row at \(i\), but the added repetitions are a bookkeeping convention, not actual jumps.

Common Mistake: Treating the Generator Diagonal as a Jump

A frequent error is to divide every entry of a generator row by the exit rate and treat the result as a transition row. In the first example, this would assign the row \((-5/5,2/5,3/5)=(-1,2/5,3/5)\) to state \(1\). Its entries sum to \(-1+2/5+3/5=0\), and the negative entry cannot be a probability. The correction is to use only off-diagonal rates as destination rates, set the embedded diagonal to zero, and divide those rates by \(q_1\). This gives \((0,2/5,3/5)\), a valid probability row.

The embedded chain is useful when a question concerns the order of visited states—for example, which state is visited next, or which sequence of states occurs before absorption. It is not enough for questions about the time spent in each state. The generator contains both kinds of information: its off-diagonal rates determine the jump destinations after normalization, while the total exit rate determines the holding-time distribution. Keeping those roles separate prevents a jump probability from being mistaken for a rate.