The Markov Property for State Processes

What the Markov Property Says

A multi-state history records which states a subject occupies and when transitions occur. At a given time, the subject’s current state may summarize everything in the earlier history that matters for predicting what happens next. The Markov property is the mathematical condition expressing that claim: conditional on the present state, the future state process is independent of the past.

This is a property of a process relative to its chosen state definition. It is not a claim that the past has no relationship to the future at all: the past can matter through the current state. The question is whether the earlier history provides any additional information once the current state is known.

Definition: Let $X(t)$ be a state process, with state space $\mathcal{S}$, and let $\mathcal{H}_t=\sigma\{X(u):0\leq u\leq t\}$ denote the information in the process history up to time $t$. The process is Markov if, for every $t$, every $k\geq1$, every $0<s_1<\cdots<s_k$, and every $j_1,\ldots,j_k\in\mathcal{S}$, $$ P\!\left(X(t+s_1)=j_1,\ldots,X(t+s_k)=j_k\mid\mathcal{H}_t\right) = P\!\left(X(t+s_1)=j_1,\ldots,X(t+s_k)=j_k\mid X(t)\right) \quad\text{almost surely.} $$ Equivalently, given $X(t)$, the future process is conditionally independent of the history before and including $t$. The one-time version, for a single future time $t+s$, is a useful special case.

The condition is about the conditional distribution of the future, not just its average or its most likely state. It applies to all future times and joint patterns of future states. In particular, matching probabilities of being in one state at one horizon does not, by itself, establish the Markov property.

The equality is understood almost surely: it must hold except on histories that have probability zero. In a finite-state model, the practical comparison is between histories that lead to the same current state and have positive probability. Their conditional future distributions must agree.

Reading the Condition in Continuous Time

For a continuous-time process, $t$ is the time at which the history is being summarized, while $s>0$ is time elapsed after $t$. In the one-time form, the condition is

P(X(t+s)=j\mid\mathcal{H}_t)=P(X(t+s)=j\mid X(t)) \quad\text{for each }j\in\mathcal{S}.

The right-hand side may depend on both $t$ and $s$. The Markov property alone does not require that probabilities depend only on the elapsed time $s$. That additional restriction is time-homogeneity, a separate assumption. Nor does the Markov property alone specify a particular transition model or guarantee that its assumptions fit a given application.

Continuous time also makes it important to be precise about what counts as the past. The history $\mathcal{H}_t$ includes the sequence and timing of states up to $t$, not merely the state occupied immediately before $t$. If time spent in the current state, previous transitions, or another feature of the path changes the future distribution after conditioning on $X(t)$, then the process is not Markov with respect to that state definition.

Worked Example: Detecting Dependence on Time Already Spent Ill

Consider a hypothetical model with states 0 (disease-free), 1 (ill), and 2 (dead). At time $t$, two possible histories both end in state 1. In history A, the subject entered state 1 one month ago; in history B, the subject entered it twelve months ago. Suppose the model specifies that, conditional on each full history, the probability of death in the next six months is 0.10 after A and 0.30 after B.

For the event $D=\{X(t+6\text{ months})=2\}$, the two conditional probabilities are

P(D\mid\text{history A})=0.10,\qquad P(D\mid\text{history B})=0.30.

Both histories have the same current state, $X(t)=1$, but the future probabilities differ by $0.30-0.10=0.20$, or 20 percentage points. Therefore the future distribution is not determined by the current state alone, and the process is not Markov for this three-state definition. This conclusion concerns the stipulated model; it does not establish that duration in illness causes death.

The difference could reflect a genuine duration dependence, or it could indicate that the state definition omits relevant information. The calculation identifies a failure of the stated Markov condition; it does not by itself decide how the model should be revised.

Justification: Conditional Independence

The conditional-independence wording and the conditional-distribution equation express the same restriction. To see why, take a past event $A\in\mathcal{H}_t$ and a future event $B$, and suppose the Markov condition gives $P(B\mid\mathcal{H}_t)=q(X(t))$, where $q(X(t))=P(B\mid X(t))$. By iterated conditioning,

\begin{aligned} P(A\cap B\mid X(t)) &=E\!\left[1_A1_B\mid X(t)\right]\\ &=E\!\left[1_A E(1_B\mid\mathcal{H}_t)\mid X(t)\right]\\ &=E\!\left[1_A q(X(t))\mid X(t)\right]\\ &=P(A\mid X(t))P(B\mid X(t)). \end{aligned}

This is the factorization that defines conditional independence of the past event and the future event given the current state. The argument applies to events describing several future times as well as to a single future state. It explains why the property is stronger than finding one future summary that happens not to vary across histories.

Worked Example: An Exponential Waiting Time

Suppose, hypothetically, that a subject starts in state 0 (alive and event-free) and moves once to absorbing state 1 (event occurred). Let $T$ be the time of the transition and assume $T\sim\mathrm{Exp}(\lambda)$ with constant rate $\lambda=0.04$ per month. For this model, the subject is in state 0 at time $t$ exactly when $T>t$. For any $t\geq0$ and $s>0$,

\begin{aligned} P(X(t+s)=1\mid X(t)=0) &=P(T\leq t+s\mid T>t)\\ &=\frac{P(t<T\leq t+s)}{P(T>t)}\\ &=\frac{e^{-\lambda t}-e^{-\lambda(t+s)}}{e^{-\lambda t}}\\ &=1-e^{-\lambda s}. \end{aligned}

For the next three months, this probability is $1-e^{-0.04(3)}=1-e^{-0.12}\approx0.1131$. Thus, among subjects still in state 0, the probability of entering state 1 within the next three months is about 11.31%, regardless of how long they have already remained in state 0. In this simple two-state model, histories available at time $t$ differ in how long the subject has waited, but the displayed conditional probability does not depend on that duration.

The calculation uses the exponential survival probability $P(T>v)=e^{-\lambda v}$ and the stated model assumption that $T$ has that distribution. It illustrates a Markov process; it does not show that every Markov state process must have an exponential waiting time. The calculation concerns this one-transition model and does not supply a general test for Markov behavior in richer state processes.

Worked Example: Checking a Future-State Distribution Across Histories

Consider a hypothetical model with states 1 (stable), 2 (ill), and 3 (dead). At time $t$, among subjects in state 1, suppose history A has conditional probability 0.60 and history B has probability 0.40. Assume the model specifies the following distribution of $X(t+2\text{ months})$ after each history:

History up to $t$	$P(X(t+2)=1\mid\text{history})$	$P(X(t+2)=2\mid\text{history})$	$P(X(t+2)=3\mid\text{history})$
A, with $X(t)=1$	0.70	0.20	0.10
B, with $X(t)=1$	0.70	0.20	0.10

The probabilities in each row sum to $0.70+0.20+0.10=1.00$. For example, the marginal probability of being in state 2 at $t+2$ given current state 1 is

P(X(t+2)=2\mid X(t)=1) =(0.60)(0.20)+(0.40)(0.20)=0.20.

The joint conditional probability of history A and state 2 at $t+2$, given the current state, is $(0.60)(0.20)=0.12$. This equals $P(\text{A}\mid X(t)=1)P(X(t+2)=2\mid X(t)=1)=(0.60)(0.20)=0.12$, as conditional independence requires. The same calculation for history B gives $0.40(0.20)=0.08$, also equal to $0.40(0.20)$.

These calculations verify the factorization for the displayed future time and state under the specified model. To conclude that the process is Markov, the model must specify that the full future distribution, including joint distributions at multiple future times, is likewise the same after histories with the same current state. Agreement in this single table is not enough on its own.

Assumptions, Comparisons, and Failure Modes

The Markov property is a modeling assumption, not a consequence of drawing a state-transition diagram. It is meaningful only after the states and the time origin have been defined. The histories being compared must refer to the same current state under that definition, and the conditional future distributions must be well-defined for those histories. A state process can be Markov under one state definition and non-Markov under another.

A common alternative is a history-dependent model, in which future transition behavior can depend on features of the earlier path beyond the current state. For example, future risk may vary with time already spent in the current state. A semi-Markov model is one way to represent dependence of transitions on sojourn time. As discussed in the earlier material on refining states to restore the Markov property, including relevant history in a richer state can sometimes make an expanded process Markov. These are modeling choices, not automatic fixes: the richer state must be scientifically interpretable and supported by the available information.

When assessing whether the assumption is plausible, ask whether histories ending in the same state could differ in a scientifically relevant way, such as their duration in that state or their previous transitions. If the model implies different future distributions for such histories, the Markov assumption fails for the current state definition. In observed data, estimated differences can also reflect sampling variability; an observed discrepancy is not, by itself, proof of a population-level violation.

Confusing Markov with time-homogeneous. Markov means the future is conditionally independent of the past given the present state. It does not require the conditional distribution to be unchanged across calendar times.
Confusing Markov with exponential waiting times. The exponential example shows one setting where the property holds. It is incorrect to conclude that every Markov process has exponential sojourn times.
Checking only the current state at one future time. The definition concerns the future distribution, including joint behavior across future times. One matching probability is not sufficient evidence for the full condition.
Treating a state diagram as proof. A diagram describes allowed transitions; it does not show that the current state contains all relevant predictive information from the history.

The Markov assumption can simplify the description of a multi-state process, but it may misrepresent the process when relevant history is omitted. Its adequacy depends on the scientific question and the state definition, not just on mathematical convenience.

Check Your Understanding

Answer each question using the definition of the Markov property. Short solutions follow the questions.

In the continuous-time condition, what does $\mathcal{H}_t$ contain?
Two histories end in the same state at time $t$, but their probabilities of reaching state 2 within the next month are 0.08 and 0.19. What does this imply for the Markov assumption under that state definition?
Does the Markov property alone require conditional future probabilities to depend only on elapsed time $s$, rather than on $t$ and $s$?
Why does agreement in the probability of one future state at one time not establish the full Markov property?

Solutions. (1) $\mathcal{H}_t$ contains the process history $\{X(u):0\leq u\leq t\}$, including the states occupied and their timing. (2) It violates the condition: the future probability differs across histories with the same present state, by $0.19-0.08=0.11$, or 11 percentage points. (3) No. The Markov property allows dependence on both $t$ and $s$; requiring dependence only on elapsed time is the separate time-homogeneity assumption. (4) The property requires the conditional distribution of the future, including joint patterns at multiple future times, to depend on the history only through the current state. One probability checks only one part of that requirement.

History up to \(t\)	\(P(X(t+2)=1\mid\text{history})\)	\(P(X(t+2)=2\mid\text{history})\)	\(P(X(t+2)=3\mid\text{history})\)
A, with \(X(t)=1\)	0.70	0.20	0.10
B, with \(X(t)=1\)	0.70	0.20	0.10

The Markov Property for State Processes

What You'll Learn