Markovization By State Augmentation

Why Enlarge the State?

A process can have history-dependent transition rates even when its observed state is known. For example, two people who are both currently ill may have different future mortality rates because they accumulated different amounts of a relevant exposure before becoming ill. If the model records only “ill,” it merges situations with different predicted futures.

State augmentation addresses this problem by adding a summary of the past to the current state. The goal is not to record every detail of the trajectory. It is to find a history variable that retains the information needed to describe future transitions. The augmented process is Markov if, conditional on its current augmented state and current time, its future law no longer depends on earlier history.

This is a modeling condition, not an automatic consequence of adding a variable. It requires that the added summary capture all the past information relevant to the future dynamics. The earlier material on the Markov property and state sufficiency gives the underlying criterion; here the focus is how to construct and check an augmented representation.

An Augmented Illness-Death Model

Consider an illness-death process with states 0 (healthy), 1 (ill), and 2 (dead), where death is absorbing and the allowed transitions are \(0\to1\), \(0\to2\), and \(1\to2\). Define \(E(t)\) as cumulative time spent healthy up to time \(t\), measured in years. It records a feature of the past that remains available after illness begins.

While a person is healthy, \(E(t)\) increases at one year per year. Once the person is ill or dead, it remains constant. Thus \(E(t)\) is continuous and satisfies \(\dot E(t)=1\) in state 0 and \(\dot E(t)=0\) in states 1 and 2. Suppose the transition intensities depend on the current time, illness-death state, and \(E(t)\), but not on other details of the path.

Write the three possible intensities as \(q_{01}(t,e)\), \(q_{02}(t,e)\), and \(q_{12}(t,e)\). The augmented state is \(Z(t)=(X(t),E(t))\). A transition changes \(X(t)\) but leaves \(E(t)\) unchanged at the instant of transition; in state 0, the exposure coordinate then continues to increase.

The key is that the augmented state keeps the value \(e\) after the transition from healthy to ill. If the post-illness death rate depends on accumulated healthy time, the current illness state alone does not preserve that information. With \(e\) included, the intensity at the current time is specified by the current augmented state.

Generator and Markov Property

The transition rates describe jumps, while the exposure coordinate also evolves continuously. Both parts can be represented in the generator. For a differentiable test function \(f(i,e)\), the generator \(\mathcal{A}_t\) gives the instantaneous rate of change in the conditional expectation of \(f(Z(t))\). The continuous-motion term is the exposure drift; the remaining terms account for jumps.

The derivative appears only in state 0 because \(E(t)\) has unit speed there and is fixed in states 1 and 2. The jump terms have the standard form “rate times change in \(f\).” This representation also makes clear why simply writing a transition-rate matrix for the three illness-death labels is insufficient when rates vary with \(e\): the process includes a continuously changing coordinate as well as discrete jumps.

A short justification follows from a first-order expansion over an interval of length \(h\). In state 0, if no jump occurs, the exposure changes from \(e\) to \(e+h\), so \(f(0,e+h)-f(0,e)=h\,\partial f(0,e)/\partial e+o(h)\). A jump to state \(j\) occurs with probability \(q_{0j}(t,e)h+o(h)\), contributing \(q_{0j}(t,e)h\{f(j,e)-f(0,e)\}+o(h)\). The probability of multiple jumps is \(o(h)\). Summing, subtracting \(f(0,e)\), and dividing by \(h\) gives the first generator expression as \(h\) tends to zero. The same argument in states 1 and 2 gives the other expressions. Because the rates and exposure evolution depend on the past only through the current \((X(t),E(t))\) and \(t\), the future conditional law is determined by that augmented state and time: this is the Markovization.

Numerical Rates in the Augmented State

A proposed augmented model should specify all allowed transitions at each augmented state. The following example illustrates how the state-dependent rates are evaluated. The figures are hypothetical, and rates are measured per year.

When Augmentation Works—and When It Does Not

The construction is valid when the added variable is sufficient for the relevant history dependence. For example, if future intensities depend on cumulative healthy time but not on the sequence of healthy and ill episodes, \(E(t)\) can be enough. In the illness-death model above, there is no return from illness to health, so cumulative healthy time stops changing after illness onset. In a process with repeated visits to health, the definition and dynamics of the exposure must still match the scientific mechanism; total healthy time and time in the current healthy spell are different variables.

Useful diagnostics are conceptual and empirical. First, list the historical features that the proposed rates depend on, and verify that each can be calculated from \((X(t),E(t))\) and \(t\). Second, compare histories that lead to the same augmented state: if the model still assigns them different future transition laws, the augmentation is not sufficient. Third, check that the exposure updates correctly along every possible path, including after each transition. A mismatch in the update rule can create an apparently Markov representation that does not correspond to the intended model.

State augmentation is one alternative to retaining an explicit history-dependent intensity or using a semi-Markov description based on time spent in a state. The appropriate representation depends on what history matters. A duration-dependent rate may be described using the current sojourn duration; a rate depending on a cumulative exposure may require that exposure as a state coordinate. The earlier material on semi-Markov transition kernels provides a framework for transition laws that depend on sojourn history. Augmentation instead aims to make the chosen enlarged process Markov.

The approaches are not interchangeable by definition. A single duration variable may omit earlier exposure, while cumulative exposure may omit the duration of the current illness spell. If both features affect the future, both may need to be included. Conversely, adding unnecessary variables can make the state space harder to estimate and interpret. This tutorial’s construction does not establish that an arbitrary history-dependent model has a finite-dimensional Markov representation; some histories cannot be summarized adequately by the proposed coordinates.

Common Mistakes and Limitations

• Keeping only the clinical label. Treating all currently ill individuals as having the same future risk is wrong if the modeled intensity depends on their recorded exposure histories. Include the relevant history summary in the state.
• Assuming any added variable restores Markov behavior. It does so only if the augmented current state and time determine the future conditional law. Check the dependence of every transition rate and the evolution rule for the added variable.
• Confusing a cumulative total with a current duration. Total time spent healthy and time since the most recent transition into health can differ. Define the history variable precisely and update it according to that definition.
• Calling a short-interval approximation an exact probability. The relation \(P(\text{transition }i\to j\text{ in }[t,t+h)\mid\mathcal{F}_t)=q_{ij}(t,E(t))h+o(h)\) is local. A finite-interval probability requires accounting for changes in state and rates during the interval.