Serially Correlated Choice Shocks and State Augmentation

Why Serial Correlation Changes the State

A dynamic choice model is memoryless when its state contains enough information to determine the distribution of future payoffs and states, conditional on the current state and action. If payoff shocks persist over time, the observed state alone may not contain that information. A shock realized today can both affect today's decision and predict tomorrow's payoff shock. The choices observed in the past can therefore matter for predicting future choices, even after conditioning on the currently observed state.

The modeling response is to include the relevant current shock in the state. The resulting state may be partly latent to the econometrician, but it can still be the correct state for the decision-maker's dynamic problem. This is a structural modeling step, not merely a correction to a standard error or a different choice-probability formula.

A Choice Model With Persistent Payoff Shocks

Let \(x_t\) denote the observed state at time \(t\), \(a_t\) the chosen action, and \(z_t\) a vector of payoff shocks. The vector can have one component for each action, so that \(z_{a,t}\) is the persistent shock associated with action \(a\). A simple utility specification is

The key feature is that \(z_t\) evolves over time rather than being redrawn independently each period. For example, it may follow a vector autoregression of the form

where \(\eta_{t+1}\) is a new innovation. In a scalar version, a common specification is \(|\rho|<1\) and \(\eta_{t+1}\sim N(0,\sigma_\eta^2)\), independent across time. A positive \(\rho\) means a high current shock tends to predict a high next-period shock. The model must also specify how the observed state evolves, for example through a transition probability \(P(x_{t+1}\mid x_t,a_t)\).

For the formulation below, suppose the decision-maker observes \(z_t\) when choosing. The econometrician may not observe it. Suppose also that the observed-state transition and shock transition together have a specified conditional distribution given the current state and action. If they are conditionally independent, their joint transition distribution factors into \(P(x_{t+1}\mid x_t,a_t)\) and \(Q(z_{t+1}\mid z_t,x_t,a_t,x_{t+1})\). If they are not independent, the model must specify the joint distribution directly.

The decision-maker's state is then \(s_t=(x_t,z_t)\). The observed component alone may still follow a Markov transition, but it is not generally a sufficient state for the choice problem: knowing \(x_t\) does not reveal the current persistent shock or the distribution of the next one.

Augmenting the State and Writing the Bellman Equation

Let \(V(x,z)\) be the value to a decision-maker who begins a period with observed state \(x\) and persistent shock \(z\). For a finite action set \(\mathcal A\), let \(u(x,a,z)\) be current deterministic utility, and let \(\beta\in(0,1)\) be the discount factor. Write \(Q(dx',dz'\mid x,z,a)\) for the joint transition distribution of next period's observed state and shock. With no additional transitory choice shocks, the Bellman equation is

The integral is an expectation over both next-period components. If they are conditionally independent under the model, this expectation can instead be written as a sum or integral over \(x'\), followed by one over \(z'\), using their separate transition distributions. If there are also independent, transitory choice shocks \(\varepsilon_t\), they enter the current payoff but not the persistent state; the Bellman equation then takes the expectation over \(\varepsilon_t\) of the maximum over actions.

The important change is not the discounting or the maximization. It is the conditioning information in the continuation value: the distribution of \(z'\) depends on \(z\), so the value is a function of both \(x\) and \(z\). In a model with choice-specific persistent shocks, choices compare both current shock realizations and continuation values that depend on those realizations.

The result follows directly from the definition of a Markov-sufficient state. Current utility is determined by \((x_t,z_t,a_t)\), so earlier history provides no further information about the current payoff once those variables are known. By assumption, the distribution of the next state \((x_{t+1},z_{t+1})\) is also determined by \((x_t,z_t,a_t)\). Applying the same argument at each future date, the conditional distribution of future payoffs and states is determined by the current augmented state and future actions, not by the earlier path. Thus the Bellman problem can be written using \((x_t,z_t)\) rather than the full history. This justification requires the stated transition condition; adding a shock variable is not sufficient if the shock transition still depends on omitted history.

What the Econometrician Observes

State augmentation describes the decision-maker's problem; it does not assert that the econometrician observes the added state. If \(z_t\) is latent, the econometrician must account for it when relating the model to observed choices. In general, the probability of a future choice conditional on the observed state alone averages over the possible current shocks. The distribution used for that average can depend on the observed history, because past choices provide information about persistent shocks.

This distinction matters for likelihood construction and for conditional choice probabilities. A probability conditional on \((x_t,z_t)\) is not generally the same object as a probability conditional only on \(x_t\). When \(z_t\) is unobserved, the latter is a mixture over the shock's conditional distribution given the information available to the econometrician. Treating that mixture as a fixed probability depending only on \(x_t\) can be wrong if the mixture weights change with history.

Scope and Modeling Conditions

Several distinctions prevent common specification errors. A persistent payoff shock is not the same as a fixed latent type. A type is typically constant over the modeled horizon, whereas a serially correlated shock changes according to a transition law. Nor is it the same as a transitory shock that is independently redrawn each period: a transitory shock does not, by itself, predict future payoffs. A model can contain both a persistent component \(z_t\) and a transitory component \(\varepsilon_t\), but their roles and distributions should be specified separately.

The Markov formulation also relies on an information assumption. The Bellman equation above treats the decision-maker as observing \(z_t\). If the decision-maker does not observe the shock and instead learns about it from signals, then the decision-maker's information or beliefs may need to be represented in the state. Similarly, if the shock transition depends on omitted history, adding \(z_t\) alone will not restore the Markov property. The model must include enough information to make current payoffs and transition distributions history-independent conditional on the state and action.

Common Mistake: Averaging Away Persistence

A tempting but generally incorrect argument is: “The shock is unobserved, so average it out and solve a model with state \(x_t\) only.” Averaging is not automatically invalid, but the averaging distribution must be correct for the information being conditioned on. If the conditional distribution of \(z_t\) changes with past choices and the shock predicts future shocks, then a single average based only on \(x_t\) loses relevant information. The resulting continuation values and choice probabilities can be misspecified.

The correction is to specify the persistent shock's transition law, include its current realization in the decision-maker's state, and derive the Bellman equation over the augmented state. If the shock is latent to the econometrician, the model's observed-data probabilities must then account for the distribution of that state given the observed history. The shock can be integrated out only with the appropriate conditional distribution, not by assuming that history no longer matters.