From a Sequence of States to a Random Path
A discrete-time Markov chain records a state at each integer time. A continuous-time Markov chain records a state at every real time \(t\ge0\). Its sample path is therefore a time-indexed function: it shows not only which states occur, but also when changes happen. For countable state spaces with the usual discrete topology, the paths considered here are right-continuous and piecewise constant. This makes the timing of state changes part of the model.
Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, let \(S\) be a finite or countable state space, and let \((X_t)_{t\ge0}\) be an \(S\)-valued process. Its natural filtration is \(\mathcal{F}_t=\sigma(X_u:0\le u\le t)\), the information in the process up to time \(t\). Write \(\mu_i=\mathbb{P}(X_0=i)\); as a distribution, \(\mu\) is a row vector.
The conditional-probability identity is an almost-sure statement. The transition probabilities specify what happens from each state, including states that may have probability zero under a particular initial distribution. A transition kernel is row-stochastic: for each starting state \(i\), its probabilities over all destination states sum to one. Unlike a discrete-time chain, a CTMC has a state at every real time, and the intervals between changes are part of its path.
What the Sample Paths Look Like
For a path \(t\mapsto X_t(\omega)\), right-continuity means its value at a time is the value approached immediately from later times. Having left limits means that at each \(t>0\), the values immediately before \(t\) approach a limit in \(S\). Equip a countable state space with the discrete topology: distinct states are separated, so convergence to a state means eventual equality to that state. Consequently, a path that is right-continuous and has left limits cannot move through infinitely many changes on a bounded time interval.
Proof. Suppose instead there are infinitely many jump times in \([0,T]\). A sequence of distinct jump times then has a subsequence converging to some \(t\in[0,T]\), by compactness of the interval. A further subsequence approaches \(t\) from below or from above. If it approaches from below, the left limit at \(t\) implies that \(x(u)\) is constant for all \(u\) sufficiently close to \(t\) with \(u<t\). There can be no jumps in that interval, contradicting the subsequence. If it approaches from above, right-continuity at \(t\) implies that \(x(u)=x(t)\) for all \(u\) sufficiently close to \(t\) with \(u\ge t\), again contradicting the subsequence. At \(t=0\) only the from-above case is possible, and at \(t=T\) only the from-below case is needed. Thus infinitely many jumps are impossible. \(\square\)
A jump time is a time at which the path changes state. Between consecutive jump times the state stays fixed. This path description is different from saying that a process is “constant between observations”: the observations in discrete time occur only at integers, whereas a CTMC has a defined state at each time between them. The paths below illustrate several ways those two descriptions can differ.
Worked Example: The Poisson Counting Process as a CTMC
Let \((N_t)_{t\ge0}\) be a Poisson counting process with rate \(\lambda>0\), on a probability space with its natural filtration, and take \(S=\{0,1,2,\ldots\}\). The path starts at \(0\), increases by one at each arrival, and is constant between arrivals. Independent Poisson increments give, for \(0\le s\le t\),
For a fixed \(i\), the row sum is \(\sum_{j\in S}p(s,t;i,j)=\sum_{k=0}^{\infty}e^{-\lambda(t-s)}(\lambda(t-s))^k/k!=1\), by the exponential series. The independent-increments property implies that, given \(\mathcal{F}_s\), the increment \(N_t-N_s\) has this same Poisson distribution regardless of the earlier history. Hence \(N_t\) satisfies the Markov conditional-probability identity.
For example, with \(\lambda=1\), \(s=0\), \(t=1\), and \(i=0\), the probability of exactly two arrivals is \(e^{-1}1^2/2!=e^{-1}/2\). On that event, the path makes two unit jumps at the two arrival times; it does not move continuously from \(0\) to \(2\). Also, the probability of at least two arrivals in an interval of length \(h>0\) is \(1-e^{-\lambda h}(1+\lambda h)\), so observations at the endpoints can miss intermediate jumps.
Transition Probabilities Determine Joint Distributions
The initial distribution and the transition probabilities describe not only one-time probabilities but also joint distributions at finitely many ordered times. The result follows by repeatedly conditioning on the information available at the preceding observation time.
Proof. For \(m=0\), the statement is the probability of the single event \(X_{t_0}=i_0\), so it holds. Suppose the formula holds for \(m-1\). The event \(A=\{X_{t_0}=i_0,\ldots,X_{t_{m-1}}=i_{m-1}\}\) belongs to \(\mathcal{F}_{t_{m-1}}\). By the defining conditional-probability identity, \(\mathbb{P}(A,X_{t_m}=i_m)=\mathbb{E}[\mathbf{1}_A\mathbb{P}(X_{t_m}=i_m\mid\mathcal{F}_{t_{m-1}})]\). On \(A\), the conditional probability inside the expectation is \(p(t_{m-1},t_m;i_{m-1},i_m)\), a fixed number. Factoring it out gives \(\mathbb{P}(A,X_{t_m}=i_m)=p(t_{m-1},t_m;i_{m-1},i_m)\mathbb{P}(A)\). Substituting the induction formula for \(\mathbb{P}(A)\) proves the claim for \(m\). \(\square\)
This factorization is useful when a question concerns states at selected times. It does not say that those observations reveal the entire path: the path also contains the times and destinations of changes between observations.
Worked Example: A Discrete-Time Chain Held Between Integers
Let \(Y_n\) be a discrete-time chain on \(S=\{0,1\}\) with transition matrix \(P=\begin{bmatrix}0&1\\1&0\end{bmatrix}\). Each row sums to \(1\), and the chain switches state at every integer step. Define a continuous-time process by \(X_t=Y_{\lfloor t\rfloor}\), where \(\lfloor t\rfloor\) is the greatest integer not exceeding \(t\). This process is right-continuous and constant between integer times.
For \(s\le t\), the number of discrete steps between \(X_s\) and \(X_t\) is \(r=\lfloor t\rfloor-\lfloor s\rfloor\). Given \(X_s=i\), \(X_t=i\) if \(r\) is even and \(X_t=1-i\) if \(r\) is odd. Thus each transition row has one entry equal to \(1\) and the other equal to \(0\); its row sum is \(1\). Since the future of \(Y\) given its history depends only on its current state, the conditional law of \(X_t\) given \(\mathcal{F}_s\) depends only on \(X_s\) and the times \(s,t\). Therefore \(X\) is a CTMC in the stated, possibly time-inhomogeneous, sense.
For \(s=\tfrac14\) and \(t=\tfrac34\), the floor difference is \(0\), so the state is unchanged. For \(s=\tfrac14\) and \(t=\tfrac54\), the floor difference is \(1\), so the state flips. This construction is one particular way to turn a discrete-time sequence into a continuous-time path; it does not make the original sequence itself a path with states defined at every real time.
Worked Example: A Markov Path That Is Not Time-Homogeneous
Let \(U\) be uniform on \((0,1)\), and define \(X_t=0\) for \(t<U\) and \(X_t=1\) for \(t\ge U\), on \(S=\{0,1\}\). The natural filtration records whether the switch has occurred. The path has one jump, at \(U\), and is right-continuous because its value at the jump time is \(1\).
If the chain is in state \(1\) at time \(s\), it remains there, so \(p(s,t;1,1)=1\) and \(p(s,t;1,0)=0\). If it is in state \(0\) at time \(s<1\), the observed history says exactly that \(U>s\). Conditional on \(U>s\), \(U\) is uniform on \((s,1)\). Therefore, for \(s\le t\le1\), \(p(s,t;0,1)=(t-s)/(1-s)\) and \(p(s,t;0,0)=(1-t)/(1-s)\); for \(t\ge1\), these probabilities are \(1\) and \(0\). For \(s=t<1\), the formulas give \(0\) and \(1\). In every case the row sums to \(1\), and the state-\(1\) row also sums to \(1\).
At \(s=\tfrac14\) and \(t=\tfrac12\), the transition probability from \(0\) to \(1\) is \((\tfrac12-\tfrac14)/(1-\tfrac14)=1/3\), and the probability of remaining in \(0\) is \((1-\tfrac12)/(1-\tfrac14)=2/3\). These probabilities depend on the absolute times: the available interval for \(U\), given that the process is still in state \(0\), shrinks as \(s\) increases. This is a Markov chain, but not a time-homogeneous one.
Common Pitfalls and Why the Path View Matters
Common mistake: treating endpoint values as a continuous interpolation. For the Poisson process with \(\lambda=1\), the probability of exactly two arrivals by time \(1\) is \(e^{-1}/2\). A wrong reconstruction from \(N_0=0\) and \(N_1=2\) is the straight line \(x(t)=2t\). That is not a valid path in the state space \(\{0,1,2,\ldots\}\), and it fails to show either arrival time. The correction is to use the actual counting path: it stays at \(0\), jumps to \(1\) at the first arrival, stays there until the second, and then jumps to \(2\).
Two further distinctions are important. First, a CTMC need not be time-homogeneous: the conditional transition probabilities may depend on both \(s\) and \(t\), as in the uniform-switch example. Second, a discrete-time chain specifies only its integer-time sequence. A continuous-time path requires additional information about the states between those times and when changes occur. Different path constructions can agree at integer times while differing between them.
The examples here have paths with finitely many jumps on every bounded interval. More general constructions may raise questions about infinitely many jumps in finite time; that issue requires separate treatment. The path definition and conditional transition identity are the starting point for studying those questions, as well as the special case of time-homogeneous chains.
Check Your Understanding
Use the definitions and verify each probability calculation from its transition rule.
- For a CTMC, what does the conditional-probability identity assert about \(X_t\) given \(\mathcal{F}_s\)?
- Why does a right-continuous path with left limits in a countable discrete state space have only finitely many jumps on a bounded interval?
- For a Poisson process with rate \(\lambda\), write the probability of exactly \(k\) arrivals between \(s\) and \(t\), and check that the probabilities over all \(k\ge0\) sum to \(1\).
- In the integer-held alternating chain, compare the state at \(s=\tfrac14\) with the state at \(t=\tfrac34\), and then with the state at \(t=\tfrac54\).
- For the uniform-switch process, compute the transition probabilities from state \(0\) at \(s=\tfrac12\) to time \(t=\tfrac34\). Do they depend only on \(t-s\)?