Primitives and Timing in Dynamic Discrete Choice

Why Primitives and Timing Matter

A dynamic discrete-choice model describes an agent who repeatedly chooses among a finite or countable set of actions while circumstances can change over time. A firm may continue operating, invest, or exit; a consumer may retain or replace a durable good. In either case, the choice today affects both the current payoff and the opportunities available later.

Before solving or estimating such a model, specify its primitives: who chooses, which actions are feasible, what the agent observes, how choices produce payoffs, and how the relevant circumstances evolve. The timing of information and choice is part of that specification. If a payoff shock is observed before choosing, for example, it can influence the action; if it is observed afterward, it cannot.

Definition: A stationary infinite-horizon dynamic discrete-choice model has periods \(t=0,1,2,\ldots\); a time-invariant discount factor \(0\leq\beta<1\); a payoff-relevant state \(x_t\); a feasible action set \(A(x_t)\); and time-invariant payoff and information-evolution rules. The agent chooses an action in each period to maximize expected discounted lifetime payoff. The state and the agent’s information must contain what is needed to describe current payoffs, feasible choices, and future relevant uncertainty.

Agents, States, Actions, and Shocks

An agent is the decision-maker whose sequence of choices is being modeled. In a firm model, the agent might be one firm; in a consumer model, it might be one consumer. A model may describe one agent or a population of agents facing common or different primitives. The unit of decision-making should be explicit because it determines what a state and an action refer to.

The state \(x_t\) summarizes payoff-relevant circumstances at the start of period \(t\). It can include observed characteristics such as a firm’s productivity, a consumer’s accumulated experience, or a machine’s age. The feasible action set \(A(x_t)\) may depend on the state: for example, a machine that has already been scrapped cannot be repaired. An action \(a_t\in A(x_t)\) is the discrete decision made by the agent in period \(t\).

A common payoff specification separates a systematic component from choice-specific shocks. Write the period-\(t\) payoff from action \(a\) as \(u(x_t,a;\theta)+\varepsilon_t(a)\), where \(u\) is the systematic payoff, \(\theta\) collects parameters, and \(\varepsilon_t(a)\) is the shock associated with that action. The vector of shocks may contain a different component for every feasible action. This notation does not itself impose a distribution or independence condition; those are additional assumptions to state when needed.

For the timing convention used here, the agent observes \(x_t\) and the current vector of choice-specific shocks before choosing. The agent then selects \(a_t\), receives the corresponding payoff, and faces uncertainty about next period. Thus current shocks can affect the current decision. A model in which the agent chooses before observing those shocks has a different information structure and generally different choice behavior.

Worked Example: Choosing Between Repair and Replacement

A hypothetical firm has a machine in a particular condition. It must either repair or replace it. Both actions are feasible, and the firm observes the current shocks before choosing. Let \(\beta=0.9\). Suppose the current systematic payoffs are \(-3\) utility units for repair and \(-8\) for replacement. The observed shocks are \(1\) for repair and \(0\) for replacement. Assume the expected discounted continuation utilities, conditional on each action, are respectively \(10.8\) and \(16.2\) utility units.

The total current-plus-continuation comparison for repair is \( -3+1+10.8=8.8 \) utility units. For replacement it is \( -8+0+16.2=8.2 \) utility units. Given the stated continuation utilities and no other feasible actions, the firm chooses repair. Repair has the higher total, even though replacement has the higher continuation utility: the current payoff and the observed current shock also matter.

This calculation uses continuation utilities as inputs; it does not explain how to solve for them. The example’s timing assumption is essential: because the firm observes the shocks before choosing, the values \(1\) and \(0\) enter the comparison for the two actions.

Timing and the Lifetime Objective

A precise timeline prevents current information, action, payoff, and future uncertainty from being conflated. Under the convention above, the sequence is:

Observe current information.
The agent observes the current state \(x_t\) and the current choice-specific shock vector \(\varepsilon_t\).

Choose a feasible action.
The agent chooses \(a_t\in A(x_t)\), using the information available at the decision point.

Receive the current payoff.
The realized payoff is \(u(x_t,a_t;\theta)+\varepsilon_t(a_t)\).

Move to the next period.
The next state and next-period information are drawn according to the model’s specified conditional law.

A policy is a rule for choosing an action from the information available to the agent. Given a policy, the agent’s lifetime objective, conditional on current information, is the expected discounted sum of payoffs:

\mathbb{E}\left[\sum_{j=0}^{\infty}\beta^j \left\{u(x_{t+j},a_{t+j};\theta)+\varepsilon_{t+j}(a_{t+j})\right\} \;\middle|\; \text{current information}\right].

The agent seeks a feasible policy that maximizes this objective. Here \(\beta\) is the common per-period discount factor: a payoff \(j\) periods ahead receives weight \(\beta^j\). The restriction \(\beta<1\), together with suitable bounds or integrability conditions on payoffs, supports a finite expected discounted objective. These primitives define the decision problem; they do not by themselves specify an estimation method or how to compute an optimal policy.

Stationarity and State Sufficiency

Stationary means that the payoff and information-evolution rules do not depend directly on calendar time. For example, the same state-action pair has the same systematic payoff schedule in every period, and the same state-action pair faces the same conditional law for next period’s relevant information. The agent’s state, shocks, and actions can still change over time. Stationarity is a property of the rules, not a claim that circumstances or choices remain constant.

A state is sufficient when it summarizes the history well enough for predicting the future consequences of decisions. In particular, histories that give the same state must have the same feasible actions and current payoff specification. They must also imply the same conditional law of future payoff-relevant information, given actions. That includes the joint law of future states and shocks when both can matter. Matching the marginal law of each separately is not enough.

This condition can be expressed recursively: conditional on the agent’s full current information, including the current state and observed current shocks, and the chosen action, the law of next period’s state and shocks must be specified without needing additional omitted history; the same requirement must continue to hold in later periods. If the next-state and next-shock relationship depends on history even when their separate marginal distributions do not, the state has omitted information. A model can sometimes correct this by expanding the state to include a relevant history variable.

Worked Example: Why the Joint Law Can Matter

Consider two histories that would both be recorded as the same state \(x_t\). After a fixed action, let \(S\) indicate whether the next state is favorable, and let \(E\) be a binary next-period payoff shock. Under either history, both \(S\) and \(E\) equal 1 with probability \(1/2\). Suppose the next-period payoff component of interest is \(SE\) utility units.

Under history 1, suppose \(S=E\) with probability 1: the pair is \((1,1)\) or \((0,0)\), each with probability \(1/2\). Then \( \mathbb{E}[SE\mid\text{history 1}]=(1)(1)(1/2)+(0)(0)(1/2)=0.5 \). Under history 2, suppose \(S\) and \(E\) are independent. Then \( \mathbb{E}[SE\mid\text{history 2}]=P(S=1,E=1)=0.5\times0.5=0.25 \).

The separate marginal distributions agree, but the expected future payoff differs by \(0.25\) utility units. If this payoff component is discounted one period with \(\beta=0.9\), the difference in its discounted expectation is \(0.9\times0.25=0.225\) utility units. Therefore, the two histories cannot be treated as equivalent for this decision problem merely because they imply the same marginal laws. The state must retain information needed to capture the relevant joint law, or the model must otherwise specify that joint law conditional on the state and action.

Stationarity Does Not Mean Constant Circumstances

A common modeling challenge is a payoff or choice opportunity that seems to depend on elapsed time. The solution may be to include the relevant duration or remaining time in the state. Then the rules can remain stationary even though the state changes predictably from one period to the next.

Worked Example: A Contract With Three Periods Remaining

Suppose a hypothetical firm earns \(10\) dollars per period while a contract is active, and there are three periods remaining. Let \(\beta=0.9\), and assume the firm receives no contract payoff after the final period. The discounted value of the remaining contract payoffs is \(10+0.9(10)+0.9^2(10)=10+9+8.1=27.1\) dollars.

If the state records only “contract active,” it treats a firm with one period remaining like one with three, even though their future payoff opportunities differ. Instead let \(r\) be the number of periods remaining and include \(r\) in the state. The payoff rule is \(10\) dollars when \(r>0\), and the next-period state has one fewer period remaining until the contract ends. The state changes, but this rule does not depend on calendar time. The model can represent the finite contract within an infinite-horizon framework by including a post-contract state with no contract payoff.

The calculation does not say that all duration dependence can be ignored. It shows how to represent it in the state so that the payoff rule and evolution rule can be stated consistently over time.

Common Mistakes

Leaving the information timing implicit. Saying that a shock affects choice without saying when it is observed is incomplete. State whether the agent observes it before choosing, and specify the decision information accordingly.
Calling a state sufficient because it predicts each future variable separately. Separate marginal distributions need not determine the joint distribution. If future payoffs depend on state-shock combinations, specify their conditional joint law or retain the omitted information in the state.
Equating stationarity with constant states or payoffs realized. Stationarity means time-invariant rules. States and realized payoffs can vary; duration can be included in the state when it affects future opportunities.
Ignoring continuation consequences. Comparing only current payoffs can select the wrong action when actions lead to different future prospects. A valid comparison must account for the expected discounted consequences of each action.

Key takeaway: A dynamic discrete-choice model is defined not just by a list of actions and payoffs, but by who observes what and when, how actions affect current rewards and future information, and whether the state captures the full relevant history. Stationary primitives can govern changing states over an infinite horizon.

Check Your Understanding

Answer each question using the model definitions and timing in this tutorial.

Under the timing convention in this tutorial, which information does the agent observe before choosing an action?
Why is it not generally enough for two histories to imply the same marginal distributions of next-period state and shock?
A choice-specific shock is realized only after the action is selected. Can it directly affect that period’s choice under the stated timing? Explain.
What does stationarity require, and what does it not require about the state or realized payoff?
A state records that a business is open but omits how long its permit remains valid. When could that omission make the state insufficient?