Two-Step Conditional Choice Probability Estimation

Why Use a Two-Step Estimator?

In a dynamic discrete choice model, an agent compares current payoffs with the discounted value of future states. A nested fixed-point estimator repeatedly solves the model’s value equations while searching over structural parameters. The conditional choice probability (CCP) approach offers an alternative: estimate choice probabilities and state transitions first, then use them to construct value functions and recover payoff parameters. This avoids repeatedly solving the dynamic program inside the parameter search.

The method is not assumption-free. It relies on a correctly specified state that captures the relevant information for future choices, a transition law that can be estimated, and choice shocks with a known distribution. In the standard version here, shocks are independently and identically distributed Type I extreme value across actions and periods, with scale normalized to one. The discount factor \(\beta\) is known and satisfies \(0\leq\beta<1\). As in Hotz–Miller inversion, the resulting CCP odds identify differences in choice-specific values; the two-step estimator uses those differences together with value equations to recover structural payoffs.

Model and Two-Step Logic

Let \(x\) denote the observed state, \(a\) an action from a finite choice set, and \(u(a,x;\theta)\) the deterministic flow payoff, indexed by structural parameter vector \(\theta\). After choosing \(a\), the next state \(x'\) follows \(P(x'\mid x,a)\). The agent also receives an unobserved choice shock \(\varepsilon_a\). Define the choice-specific value before that shock as

Here \(V(x)\) is the expected value before the current period’s choice shocks are realized. Under the normalized Type I extreme-value assumption, the CCP is

where \(\gamma\) is the Euler–Mascheroni constant. Choose a reference action \(a_0\). Taking the log ratio of two CCPs gives

Thus, once the value function and transition probabilities are available, the CCP log odds reveal the difference in deterministic flow payoffs. For a known or normalized reference payoff, the value function can itself be recovered from the reference-action equation. Since \(V(x)=v(a_0,x)-\log p(a_0\mid x)+\gamma\), it satisfies

Step 1: Estimate Choice Probabilities and Transitions

With a finite state space and adequate observations in each state, estimate \(p(a\mid x)\) by the within-state action frequency. Estimate \(P(x'\mid x,a)\) by the frequency of next state \(x'\) among observations with current state \(x\) and action \(a\). For continuous or high-dimensional states, these conditional probabilities generally require a parametric or flexible statistical model rather than raw cell frequencies.

The data and model must support these estimates. The state must contain the payoff-relevant information that makes it appropriate to condition choices and transitions on \(x\); the observed transition must follow the specified Markov law conditional on current state and action; and there must be observations supporting the relevant actions and next states. In particular, the inversion requires positive CCPs for the actions being compared. A zero estimated probability makes the log odds undefined, while a probability near zero can make the inversion unstable.

Step 2: Solve Values and Recover Payoffs

If the reference payoff \(u(a_0,x;\theta)\) is known, the reference-action equations form a linear system in the state values. In vector notation, with \(P_0\) the transition matrix for the reference action and \(c(x)=u(a_0,x;\theta)-\log p(a_0\mid x)+\gamma\), the system is

The inverse exists when \(0\leq\beta<1\) and \(P_0\) is a stochastic matrix. Its rows sum to one, so its spectral radius is at most one; consequently the series \(\sum_{k=0}^{\infty}\beta^kP_0^k\) converges and equals \((I-\beta P_0)^{-1}\). If the reference payoff contains unknown parameters, the value system depends on those parameters and must be solved accordingly; it cannot simply be treated as known.

After solving the value system, recover each flow-payoff difference by rearranging the CCP log-odds equation:

The right side is an inverted payoff difference. To obtain structural parameters, specify how the payoff difference depends on \(\theta\), then match the inverted differences to that structure across states and actions. Exactly identified models can yield a system of equations; with more state-action comparisons than parameters, a common approach is to choose parameters that minimize a weighted discrepancy between the structural payoff differences and their inverted counterparts.

What the Two Steps Require

The procedure is compact, but each step has substantive requirements. In the first step, estimated CCPs and transitions must represent the same underlying dynamic process. Sparse state-action cells can yield noisy probabilities; lack of action support means the counterfactual transition for that action cannot be learned from those data without additional modeling restrictions. In the second step, the payoff specification and its normalizations must distinguish the parameters of interest. CCPs identify relative choice-specific values, not an unrestricted common level of utility across all actions.

The approach matters because it separates flexible estimation of behavior and state evolution from structural payoff recovery. It can be computationally attractive when the state space is manageable and the first-step probabilities can be estimated reliably. Its two-step form also means that the recovered values and payoffs depend on estimated inputs. Treating those inputs as exact can understate uncertainty; inference with estimated choice probabilities requires accounting for first-step estimation error.

Common Mistakes

• Reversing the transition difference. The inversion subtracts \(\beta(P_a-P_{a_0})V\) from the log odds. Switching the order changes the sign of the continuation adjustment.
• Calling log odds a flow-payoff difference. Log odds equal a difference in choice-specific values. The discounted difference in expected future values must be removed to obtain the current flow-payoff difference.
• Using zero CCPs in a logarithm. If an estimated probability is zero, the log-odds inversion is undefined. This signals a support or estimation problem, not evidence of an infinite payoff that can be taken literally.
• Ignoring payoff normalization. Choice probabilities alone do not identify a common utility level. State the reference action and the payoff restrictions used to recover structural parameters.