Markov State Transitions in Choice Models

Why State Transitions Matter

In a dynamic choice problem, an action can affect more than the outcome in the current period: it may also change the state in which the decision-maker makes the next choice. A firm that maintains equipment, for example, may influence its future condition. By contrast, the next period’s market demand might evolve independently of that maintenance decision. Representing these processes correctly is essential: a model that assigns the wrong transition probabilities describes a different decision problem.

This tutorial focuses on the probability law for the next observed state, conditional on the current state and the action taken. It does not require solving for an optimal action. Instead, the aim is to specify the state-transition structure clearly enough that it can later be used in a dynamic choice model.

The state must contain the information the model needs to predict the next state, given the action. If some variable that affects transitions is omitted, the stated Markov property may fail. A transition law is time-homogeneous when these probabilities do not change with \(t\). That assumption is useful for a single transition matrix per action, but it is not automatic; a model with calendar-time effects may instead have time-varying matrices.

For any current state and action, transition probabilities must be nonnegative and sum to one across all possible next states. These are basic probability requirements, not optional modelling choices. A matrix convention used here places current states in rows and next states in columns, so each row sums to one.

Exogenous and Choice-Induced State Changes

A state component is exogenous to the choice in a model when its transition distribution does not depend on the chosen action, conditional on the current state. For example, a demand category may move from low to high demand at the same rate whether or not a firm maintains its equipment. A state component has choice-induced transitions when its conditional distribution changes with the action; equipment condition may improve more often under maintenance than without it.

These labels describe the transition structure specified by the model. They do not, by themselves, establish a causal effect from observational data. If an action is selected in response to information not included in the observed state, differences in subsequent states across actions may partly reflect that selection. A causal interpretation requires appropriate structural assumptions or research design; the transition matrix alone does not supply them.

When a state has several components, one can specify the joint transition probability directly. Alternatively, one may factor it into component probabilities if conditional independence is justified. For example, conditional independence of next period’s demand and equipment condition, given the current state and action, means their joint probability is the product of their component probabilities. This is an assumption about the joint transition law, not a consequence of merely recording two state variables.

Building Action-Specific Matrices

When transitions depend on the action, a separate matrix is generally needed for each action. Let the finite state space be \(x_1,\ldots,x_K\). For action \(a\), define \(P_a\) by setting its \((i,j)\) entry to \(P(x_{t+1}=x_j\mid x_t=x_i,a_t=a)\). Thus, row \(i\) records the distribution of next states from current state \(x_i\), under action \(a\). Each entry is between zero and one, and each row sums to one.

The following example combines the exogenous demand process above with an equipment-condition component that responds to maintenance. The joint state is \(x=(D,C)\), where \(D\in\{L,H\}\) is demand and \(C\in\{G,W\}\) is equipment condition (good or worn). The ordering for both rows and columns will be \((L,G),(L,W),(H,G),(H,W)\). Fixing and stating this ordering prevents a common source of matrix errors.

Now suppose that without maintenance, next condition is \(G\) with probability \(0.85\) from current \(G\), and \(0.20\) from current \(W\). The complementary probabilities are for next condition \(W\). Keep the same conditional-independence assumption and demand process.

Why the Product Formula Works

The matrix calculations rely on the conditional-independence assumption. For a current state \(x=(D,C)\) and action \(a\), that assumption states that next demand \(D'\) and next condition \(C'\) are independent conditional on \(x,a\). The definition of conditional independence then gives the joint transition probability as a product.

To verify the row-sum claim, fix current state and action, and write \(q(d')\) for the demand transition probability and \(r(c')\) for the condition transition probability. Summing the product over all possible next-state pairs gives

This is a justification for the row sums when the factorization applies. If the components are dependent, multiplying their separate transition probabilities is not valid; the joint probabilities must be specified directly (or constructed using a model for their dependence).

From a Matrix to a Distribution Over States

A transition matrix can also update a probability distribution over current states. Represent the current distribution as a row vector \(\pi_t\), whose \(i\)th entry is the probability of state \(x_i\). If the same action \(a\) is applied in every current state, then the next-state distribution is \(\pi_{t+1}=\pi_tP_a\). If the action varies by current state, use in each row the transition probabilities for the action prescribed in that state.

State-Dependent Actions and Common Pitfalls

In a decision model, an action may depend on the current state. A rule \(\pi(x)\) that specifies an action for each state implies a transition matrix whose row for \(x_i\) is taken from \(P_{\pi(x_i)}\). For example, if the firm maintains equipment whenever it is worn and takes no maintenance action when it is in good condition, use the \((L,G)\) and \((H,G)\) rows of \(P_N\), and the \((L,W)\) and \((H,W)\) rows of \(P_M\). This row-by-row construction preserves the stated state ordering. It describes transitions under a specified rule, not the probabilities of choosing each action.

A frequent mistake is to multiply marginal transition probabilities without checking conditional independence. Another is to put current states in columns in one matrix and rows in another, then compare entries as if the ordering were unchanged. The correction is to state the conditioning variables and matrix ordering explicitly, verify the factorization assumption when using products, and check that every row sums to one.

• Do not label every changing state exogenous. Ask whether its transition distribution depends on the action, conditional on the current state.
• Do not confuse action-specific transitions with an action rule. \(P_a\) describes transitions under a fixed action; a state-dependent rule selects a row from the appropriate action-specific matrix.
• Do not infer causation from observed transition differences alone. Action selection may depend on information that is not represented in the state.