The Time-Homogeneous Markov Property

Conditioning on the Present and the Elapsed Time

A continuous-time process records states at every time, so its history up to time $t$ contains much more information than the single state $X_t$. The Markov property says that, when predicting the future, this additional history is irrelevant once $X_t$ is known. Time homogeneity adds a second requirement: the prediction depends on the length of time ahead, not on the absolute time at which the prediction is made.

Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space, let $S$ be a finite state space, and let $(X_t)_{t\ge0}$ be an $S$-valued process. Use its natural filtration $\mathcal{F}_t=\sigma(X_s:0\le s\le t)$, which represents the observed history through time $t$. A conditional probability given $X_t=i$ is directly defined by an event ratio only when $\mathbb{P}(X_t=i)>0$. To state a property for every state, it is convenient to give a transition probability kernel, including rows for states not reached under a particular initial law.

Definition: The process has the time-homogeneous Markov property if there are probabilities $p_{ij}(h)$, for $i,j\in S$ and $h\ge0$, such that for every $t,h\ge0$ and $j\in S$, $$ \mathbb{P}(X_{t+h}=j\mid\mathcal{F}_t)=p_{X_t,j}(h)\quad\text{almost surely}. $$ For each $i$ and $h$, the numbers $p_{ij}(h)$ are nonnegative and sum to $1$ over $j\in S$. The same $p_{ij}(h)$ must work at every $t$: the future conditional law depends on the current state $i$ and elapsed time $h$, not on the absolute time $t$. At $h=0$, $p_{ij}(0)=1$ if $i=j$, and $0$ otherwise.

The displayed conditional probability is a random variable: its value is determined by the observed past. The definition says that this random variable is, almost surely, a function only of $X_t$. It does not say that $X_t$ has the same distribution at different times. In particular, time homogeneity is a property of conditional transition probabilities, not a claim that the process is stationary.

Proposition (Testing with history events): Fix $t,h\ge0$, $j\in S$, and proposed values $p_{ij}(h)$. The identity $\mathbb{P}(X_{t+h}=j\mid\mathcal{F}_t)=p_{X_t,j}(h)$ almost surely holds if and only if, for every event $A\in\mathcal{F}_t$, $$ \mathbb{P}(A\cap\{X_{t+h}=j\})=\mathbb{E}\!\left[\mathbf{1}_A p_{X_t,j}(h)\right]. $$

Proof. If the conditional-probability identity holds, multiply both sides by $\mathbf{1}_A$ and take expectations. The defining property of conditional expectation gives the stated equality. Conversely, suppose the equality holds for every $A\in\mathcal{F}_t$. The random variable $p_{X_t,j}(h)$ is $\mathcal{F}_t$-measurable because $X_t$ is observed by time $t$. The defining characterization of conditional expectation says that an $\mathcal{F}_t$-measurable random variable $Z$ is $\mathbb{P}(X_{t+h}=j\mid\mathcal{F}_t)$ exactly when its integral over every $A\in\mathcal{F}_t$ equals $\mathbb{P}(A\cap\{X_{t+h}=j\})$. Taking $Z=p_{X_t,j}(h)$ proves the identity almost surely. $\square$

This criterion makes the role of “the whole past” precise: one may test the conditional statement by checking it against every event determined by that past. In the examples below, the verification will use a larger information set that includes the event mechanism. Conditioning first on that larger set and then on the observed history proves the required statement for $\mathcal{F}_t$.

A Useful Finite-State Verification Method

Consider a process driven by events of a Poisson process $(N_t)_{t\ge0}$ of rate $\lambda>0$. At each event, the state changes according to a fixed row-stochastic matrix $K=(K_{ij})$ on the finite state space $S$, using fresh randomness independent of the Poisson process and of the earlier event choices. Row-stochastic means $K_{ij}\ge0$ and $\sum_jK_{ij}=1$ for each $i$. Between events the state remains constant. The Poisson process has independent increments, and the number of events in an interval of length $h$ has probabilities $e^{-\lambda h}(\lambda h)^m/m!$, for $m=0,1,\ldots$.

Theorem: In this event-driven construction, the process has the time-homogeneous Markov property. Its conditional transition probabilities are $$ p_{ij}(h)=\sum_{m=0}^{\infty}e^{-\lambda h}\frac{(\lambda h)^m}{m!}(K^m)_{ij}, $$ where $K^0$ is the identity matrix.

Proof. Let $\mathcal{G}_t$ contain the Poisson events through time $t$ and all the choices used at those events. Then $\mathcal{F}_t\subseteq\mathcal{G}_t$, and $X_t$ is known given $\mathcal{G}_t$. Conditional on $\mathcal{G}_t$, the number $M=N_{t+h}-N_t$ of future events is independent of the past, with $\mathbb{P}(M=m)=e^{-\lambda h}(\lambda h)^m/m!$. If $X_t=i$ and $M=m$, the transition choices at those $m$ events give probability $(K^m)_{ij}$ of ending at $j$: for $m=0$ this is the identity matrix, and each additional event multiplies the state probabilities by $K$. Summing over the disjoint possibilities $M=m$ gives $\mathbb{P}(X_{t+h}=j\mid\mathcal{G}_t)=p_{X_t,j}(h)$. This expression depends on the observed past only through $X_t$, and its formula does not involve $t$. Since it is $\mathcal{F}_t$-measurable, conditioning it on $\mathcal{F}_t$ leaves it unchanged; the tower property of conditional expectation therefore gives $\mathbb{P}(X_{t+h}=j\mid\mathcal{F}_t)=p_{X_t,j}(h)$. Finally, each row of $K^m$ sums to $1$, so the nonnegative weighted sum defining each row of $p(h)$ also sums to $\sum_{m\ge0}e^{-\lambda h}(\lambda h)^m/m!=1$. Thus these are transition probabilities and the stated property holds. $\square$

Worked Examples with Event-Driven Processes

Worked Example: A Two-State Process That Flips

Let $S=\{0,1\}$. Events occur according to a Poisson process of rate $1$, and at every event the state flips: $0$ becomes $1$, and $1$ becomes $0$. The event transition matrix is $$ K=\begin{bmatrix}0&1\\1&0\end{bmatrix}. $$

Both rows sum to $1$. After $m$ events the state is unchanged if $m$ is even and flipped if $m$ is odd. Write $a=h$, since the rate is $1$. Starting from either state, the probability of an even number of events is $$ \sum_{r=0}^{\infty}e^{-a}\frac{a^{2r}}{(2r)!} =\frac{e^{-a}}{2}(e^a+e^{-a}) =\frac{1+e^{-2a}}{2}. $$ The odd-event probability is $1$ minus this quantity, namely $(1-e^{-2a})/2$. Thus, from state $0$, the probabilities of being in $0$ and $1$ after $h$ time units are $(1+e^{-2h})/2$ and $(1-e^{-2h})/2$, respectively; from state $1$, these probabilities are reversed. By the theorem, these probabilities apply conditional on the entire history through any time $t$, not just conditional on the state at time $t$.

Check: For either starting state the two probabilities sum to $(1+e^{-2h}+1-e^{-2h})/2=1$. At $h=0$, they are $1$ for staying in the initial state and $0$ for flipping, as required.

Worked Example: A Three-State Process That Resets Uniformly

Let $S=\{1,2,3\}$. Events occur at rate $1$. At every event, choose the new state uniformly from $S$, independently of the state before the event. The event transition matrix is $$ K=\begin{bmatrix} \tfrac13&\tfrac13&\tfrac13\\ \tfrac13&\tfrac13&\tfrac13\\ \tfrac13&\tfrac13&\tfrac13 \end{bmatrix}. $$ Each row sums to $\tfrac13+\tfrac13+\tfrac13=1$. Also, $K^2=K$: after one reset the state is uniform, and another independent uniform reset leaves it uniform.

Take an elapsed time $h=\ln 2$. The probability of no event is $e^{-\ln 2}=\tfrac12$, and the probability of at least one event is $1-\tfrac12=\tfrac12$. Starting from state $i$, no event leaves the process at $i$; one or more events leave its state uniform. Therefore the conditional probability of state $i$ at the end is $$ \tfrac12+\tfrac12\cdot\tfrac13=\tfrac23, $$ and the conditional probability of either particular state $j\ne i$ is $$ \tfrac12\cdot\tfrac13=\tfrac16. $$ These probabilities hold given every history through $t$, with the starting state in these calculations replaced by the observed $X_t$. In particular, the resulting probabilities do not depend on the absolute time $t$.

Check: For each starting state, the three destination probabilities sum to $\tfrac23+\tfrac16+\tfrac16=1$. The probability of remaining in the starting state includes both the no-event case and the chance of resetting back to that state; omitting the latter would incorrectly give $\tfrac12$.

Worked Example: A Finite Process That Moves Up and Then Stays

Let $S=\{0,1,2\}$. Events occur at rate $1$. At an event, state $0$ moves to $1$, state $1$ moves to $2$, and state $2$ stays at $2$. The event transition matrix is $$ K=\begin{bmatrix} 0&1&0\\ 0&0&1\\ 0&0&1 \end{bmatrix}. $$ The row sums are $0+1+0=1$, $0+0+1=1$, and $0+0+1=1$, so $K$ is a valid transition matrix.

Set $h=\ln 2$, so the mean number of events over the interval is $\ln 2$, and the probability of no event is $\tfrac12$. Starting in state $0$, zero events leave the process in $0$, exactly one event puts it in $1$, and two or more events put it in $2$. The three probabilities are therefore $$ p_{00}(h)=\tfrac12,\qquad p_{01}(h)=e^{-\ln 2}\frac{(\ln 2)^1}{1!}=\frac{\ln 2}{2}, $$ and $$ p_{02}(h)=1-\tfrac12-\frac{\ln 2}{2}=\frac{1-\ln 2}{2}. $$ Starting in state $1$, no event leaves the process in $1$, and one or more events put it in $2$, so $p_{11}(h)=\tfrac12$ and $p_{12}(h)=\tfrac12$. Starting in state $2$, it stays there with probability $p_{22}(h)=1$. In every case these probabilities describe the future conditional on the full history, because the theorem applies.

Check: The row from state $0$ sums to $\tfrac12+\tfrac{\ln 2}{2}+\tfrac{1-\ln 2}{2}=1$. The row from state $1$ sums to $\tfrac12+\tfrac12=1$, and the row from state $2$ sums to $1$. The probability $(1-\ln 2)/2$ is the probability of at least two events; it is nonnegative because $\ln 2<1$.

Markov Does Not Automatically Mean Time-Homogeneous

A process can satisfy the Markov property and still have transition probabilities that change with absolute time. For example, define a deterministic process with $S=\{0,1\}$ by $X_t=1$ for $1\le t<2$ and $X_t=0$ otherwise. At each fixed time, the process and its future are deterministic, so the conditional law of its future given its history is the same as the conditional law given its current state: both are concentrated on the deterministic future value. Thus it has the Markov property. But its transition probabilities are not time-homogeneous.

Indeed, at $t=0$, the process is in state $0$ and $X_{t+1}=X_1=1$ with probability $1$. At $t=3$, it is again in state $0$, but $X_{t+1}=X_4=0$ with probability $1$. Both comparisons start in state $0$ and use elapsed time $h=1$. A time-homogeneous kernel would require the same value $p_{01}(1)$ in both cases, but the two conditional probabilities are $1$ and $0$. Hence no such time-independent kernel exists.

Common pitfall. It is not enough to verify that a future probability can be calculated from the current state at one particular time. That checks dependence on history at that time, but not homogeneity. For homogeneity, the same elapsed-time probability must work at every absolute time and for every state being tested. Conversely, a process whose marginal probabilities change with time may still be time-homogeneous: the event-driven examples above have homogeneous conditional probabilities even though their distributions can change as events accumulate.

Check Your Understanding

For each question, distinguish conditioning on the full history from dependence on absolute time.

State the time-homogeneous Markov property for a finite-state process using $\mathcal{F}_t$.
In the two-state flip example, what are the probabilities of an even and an odd number of events during an interval of length $h$ at rate $1$?
For the uniform reset example at elapsed time $h=\ln 2$, why is the probability of ending in the starting state $\tfrac23$, rather than $\tfrac12$?
In the process that moves from $0$ to $1$ to $2$, what event-count cases lead from state $0$ to state $2$ over an interval?
Give the two equal-current-state, equal-elapsed-time comparisons that show the deterministic pulse process is not time-homogeneous.

The Time-Homogeneous Markov Property

What You'll Learn