State-Transition Models for Changing Conditions

Representing a System That Changes

Many systems do not produce a single outcome and stop. Weather changes from day to day, a machine moves between operating conditions, and a patient’s health may be classified into different states at successive visits. To model such a system, we need to describe both its possible states and the probabilities of moving from one state to another.

A state-transition model records the current state and uses transition probabilities to describe the next state. In this tutorial, each transition represents one fixed time step, such as one day. The model can generate a sequence of states, called a path, that represents one possible history of the system.

Definition: A finite-state transition model has a finite set of states \(1,\ldots,k\) and transition probabilities \(P_{ij}=P(X_{t+1}=j\mid X_t=i)\). Here, \(X_t\) is the state at time \(t\), \(i\) is the current state, and \(j\) is the next state. In a first-order model, the distribution of the next state depends on the current state, not on the earlier path. If the transition probabilities do not change over time, the model is time-homogeneous.

The probabilities are usually displayed in a transition matrix. Rows correspond to the current state, and columns correspond to the next state. Each entry must be between 0 and 1, and each row must sum to 1: from any current state, the next step must lead to one of the listed states.

\sum_{j=1}^{k} P_{ij}=1 \quad\text{for each current state }i

The row convention is important. An entry \(P_{ij}\) does not describe the probability of having state \(i\) now given state \(j\) tomorrow; it describes the probability of moving from \(i\) now to \(j\) next. A transition matrix is a probability model, not by itself evidence that the system really follows those transitions.

Building a Weather Transition Model

Consider a hypothetical model with three daily weather states: Sunny (S), Cloudy (C), and Rainy (R). Suppose the model assigns the following probabilities to tomorrow’s weather, conditional on today’s weather.

Today	Tomorrow: Sunny	Tomorrow: Cloudy	Tomorrow: Rainy
Sunny	0.70	0.20	0.10
Cloudy	0.30	0.40	0.30
Rainy	0.20	0.30	0.50

For example, the model says that if today is Cloudy, the probability of Rainy tomorrow is 0.30. The Cloudy row sums to \(0.30+0.40+0.30=1.00\), as required. The other two rows also sum to 1.00.

Before using this model, check its assumptions. The three states must cover the outcomes the model intends to represent, and each day must be assigned to one state. The transition probabilities are treated as fixed from day to day. Finally, tomorrow’s state is modeled using today’s state alone. That last assumption may be unrealistic if, for example, a multi-day weather pattern matters even after today’s weather is known. These are modeling choices, not consequences of the matrix.

Worked Example: Interpreting and Checking a Transition Matrix

Use the hypothetical weather matrix to find the probability that tomorrow is Rainy if today is Cloudy. Check whether the Rainy row gives a valid probability distribution.

Transition probability. Read the entry in the Cloudy row and Rainy column. It is \(P_{CR}=0.30\), so the model assigns a 30% chance of Rainy tomorrow given Cloudy today.

Row check. The Rainy row contains probabilities 0.20, 0.30, and 0.50. Their sum is \(0.20+0.30+0.50=1.00\). Each value is between 0 and 1, and the row sums to 1, so it is a valid distribution for the next state.

This calculation assumes that the listed model applies to the day in question and that the weather state has been classified as Rainy today. The result is conditional on today’s state; it is not the overall probability of rain on an unspecified day.

Simulating a Path

To simulate a path, begin with a specified initial state. At each step, use the row for the current state to select the next state according to its transition probabilities. One way to do this is to generate a uniform random number \(U\) between 0 and 1 and divide the interval from 0 to 1 into consecutive pieces whose lengths are the row probabilities. The random-number mapping method is the same basic idea used to sample categorical outcomes with unequal probabilities.

For the Cloudy row, assign \(0\leq U<0.30\) to Sunny, \(0.30\leq U<0.70\) to Cloudy, and \(0.70\leq U\leq1\) to Rainy. The exact handling of interval endpoints has probability zero for an ideal continuous uniform draw, so it does not affect the model.

After selecting tomorrow’s state, use its row for the following transition. Do not keep using the initial-state row: the current state updates after every step. The sequence of states is dependent because each step uses the state reached on the previous step.

Worked Example: Simulating Six Daily Transitions

Suppose today is Cloudy. For six successive transitions, use the hypothetical random numbers \(0.62,\ 0.12,\ 0.83,\ 0.95,\ 0.41,\ 0.71\). Generate the path.

Transition 1: Starting from Cloudy, \(0.62\) falls in the Cloudy interval \([0.30,0.70)\). Tomorrow is Cloudy.

Transition 2: From Cloudy, \(0.12\) falls in the Sunny interval \([0,0.30)\). The next day is Sunny.

Transition 3: From Sunny, the intervals are \([0,0.70)\) for Sunny, \([0.70,0.90)\) for Cloudy, and \([0.90,1]\) for Rainy. The number \(0.83\) selects Cloudy.

Transition 4: From Cloudy, \(0.95\) selects Rainy because it is in \([0.70,1]\).

Transition 5: From Rainy, the intervals are \([0,0.20)\) for Sunny, \([0.20,0.50)\) for Cloudy, and \([0.50,1]\) for Rainy. The number \(0.41\) selects Cloudy.

Transition 6: From Cloudy, \(0.71\) selects Rainy. Including the starting day, the path is Cloudy, Cloudy, Sunny, Cloudy, Rainy, Cloudy, Rainy. Each random number was interpreted using the row for the state immediately before that transition.

Estimating Long-Run State Frequencies

A single path shows one possible sequence, but it can also provide estimates of how frequently the system occupies each state. If a simulated path records \(T\) days, count the days in each state and divide by \(T\). The resulting proportions are empirical state frequencies. They estimate the model’s long-run state frequencies, but a finite simulation will vary from run to run.

Definition: For a path \(X_1,\ldots,X_T\), the simulated frequency of state \(j\) is the number of recorded days in state \(j\), divided by \(T\). If the initial state is included among the \(T\) recorded days, it contributes to the count; if recording begins after the first transition, it does not. State clearly which convention is used.

A related calculation gives a useful benchmark. A stationary distribution is a set of state probabilities that remains unchanged after one transition. Write its probabilities as \(\pi_1,\ldots,\pi_k\). The defining equations are

\pi_j=\sum_{i=1}^{k}\pi_iP_{ij} \quad\text{for each }j, \qquad \sum_{j=1}^{k}\pi_j=1.

The first equation says that the probability of being in state \(j\) after a transition equals the total probability of arriving there from every possible current state. The second equation requires the state probabilities to sum to 1. When a model has a unique stationary distribution and its conditions support long-run stabilization, its entries provide the long-run state proportions to compare with simulation. A simulated frequency is an estimate, not a guarantee that a particular run will match the stationary value exactly.

Worked Example: Solving for Stationary Weather Frequencies

Find a stationary distribution for the hypothetical weather matrix. Let \(\pi_S,\pi_C,\pi_R\) denote the stationary probabilities of Sunny, Cloudy, and Rainy.

The balance equation for Sunny is \(\pi_S=0.70\pi_S+0.30\pi_C+0.20\pi_R\). Rearranging gives \(0.30\pi_S=0.30\pi_C+0.20\pi_R\). The equation for Cloudy is \(\pi_C=0.20\pi_S+0.40\pi_C+0.30\pi_R\), or \(0.60\pi_C=0.20\pi_S+0.30\pi_R\).

From the first rearranged equation, \(\pi_S=\pi_C+\frac{2}{3}\pi_R\). Substitute this into the second: \(0.60\pi_C=0.20(\pi_C+\frac{2}{3}\pi_R)+0.30\pi_R\). Thus \(0.60\pi_C=0.20\pi_C+\frac{13}{30}\pi_R\), so \(0.40\pi_C=\frac{13}{30}\pi_R\), and therefore \(\pi_C=\frac{13}{12}\pi_R\). It follows that \(\pi_S=\frac{13}{12}\pi_R+\frac{2}{3}\pi_R=\frac{7}{4}\pi_R\).

Use the requirement that the probabilities sum to 1: \((\frac{7}{4}+\frac{13}{12}+1)\pi_R=1\). The coefficient is \(\frac{23}{6}\), so \(\pi_R=\frac{6}{23}\). Therefore, \(\pi_C=\frac{13}{46}\) and \(\pi_S=\frac{21}{46}\). In decimal form, the stationary probabilities are approximately 0.4565 Sunny, 0.2826 Cloudy, and 0.2609 Rainy. They sum to 1.

As a check, the Rainy balance equation is \(\pi_R=0.10\pi_S+0.30\pi_C+0.50\pi_R\). Substituting the results gives \(0.10(21/46)+0.30(13/46)+0.50(6/23)=12/46=6/23\), as required. Under this model, a sufficiently long path is expected to spend proportions of time near these values, although a finite path will not generally equal them exactly.

Putting the Simulation and Benchmark Together

Suppose a hypothetical simulation records 1,000 days and observes 460 Sunny days, 281 Cloudy days, and 259 Rainy days. The simulated frequencies are

\hat f_S=\frac{460}{1000}=0.460,\qquad \hat f_C=\frac{281}{1000}=0.281,\qquad \hat f_R=\frac{259}{1000}=0.259.

These frequencies sum to 1.000. They are close to the stationary values \(0.4565,\ 0.2826,\ 0.2609\), respectively. The difference is plausible in a finite random run. If the frequencies were far from the benchmark, useful checks would include whether the transition intervals were coded correctly, whether the current state was updated at every step, and whether the run was long enough for its initial condition to have little influence. A longer run may provide a more stable estimate, but any one finite run still has simulation variability.

Define states and a time step.
List the distinct states the model will use and specify what one transition means, such as one day.

Specify and check transitions.
For each current state, give probabilities for every possible next state. Confirm that each entry is between 0 and 1 and every row sums to 1.

Simulate successive states.
Start from a stated initial state, use its row to select the next state, then use the new state’s row for the next transition.

Summarize the path.
Count the recorded days in each state and divide by the number of recorded days. Compare those frequencies with a stationary distribution only when the model supports that benchmark.

Common Mistakes and Why the Model Matters

State-transition models make it possible to represent changing conditions without treating each time point as an unrelated draw. They are useful for exploring how a system evolves under stated assumptions and for estimating long-run occupancy of its states. Their outputs are conditional on the model: changing the states, transition probabilities, initial state, or time step can change the simulated path and its summaries.

Using the wrong row. The next transition must use the row for the current state, not the state at the start of the whole simulation.
Treating steps as independent. The next state is sampled conditionally on the current state. Successive states in a path are generally dependent.
Reversing the matrix. With the row convention used here, row \(i\), column \(j\) means current state \(i\) followed by next state \(j\).
Calling a simulated frequency an exact probability. A finite path gives an estimate that varies by chance; it does not establish an exact long-run value.
Assuming a matrix makes the model realistic. Probabilities must reflect a defensible description of the system. A matrix can be internally valid and still be a poor model of real changes.

For example, the wrong reasoning “the Sunny row says 70% Sunny, so every day has a 70% chance of being Sunny” ignores the current state. The correction is that 0.70 is the probability of Sunny tomorrow given Sunny today. If today is Cloudy, the relevant probability is 0.30; if today is Rainy, it is 0.20.

Check Your Understanding

Use the definitions and procedures above to answer these questions.

In a transition matrix, what does the entry in the Cloudy row and Rainy column represent?
Why must every row of a transition matrix sum to 1?
If a simulation reaches Rainy, which row should be used to choose the following state, and why?
For a path of 200 recorded days with 58 days Sunny, calculate the simulated Sunny frequency.
In your own words, explain why a simulated long-run frequency may differ from the stationary probability in a finite run.