Nested Fixed-Point Estimation

Why Estimation Is Nested

In a static choice model, a parameter vector can often be mapped directly to choice probabilities and then to a likelihood. Dynamic choice models add a step: current choices depend on the expected value of future opportunities. That value is itself defined by a Bellman equation whose solution depends on the model parameters. Consequently, evaluating the likelihood at a candidate parameter vector requires solving the dynamic program first.

This repeated calculation is the defining feature of nested fixed-point estimation. The inner problem computes the value function and implied choice probabilities at a fixed parameter vector. The outer problem compares the resulting likelihood across parameter vectors. If the parameter vector changes, the Bellman equation generally changes, so its solution and the choice probabilities must be updated as well.

From Primitives to Choice Probabilities

Consider a finite-horizon-free, infinite-horizon model with discount factor \(\beta\), where \(0\leq\beta<1\). At each date, an individual observes a state \(s\) and chooses an action \(j\) from a finite set. The deterministic flow payoff is \(u_j(s;\theta)\), and the action-specific shock is \(\varepsilon_j\). After choosing \(j\), the next state follows the transition probabilities \(P(s'\mid s,j)\). For clarity, suppose transitions are known and do not depend on \(\theta\); the likelihood can also include transition terms when transition parameters are estimated.

Assume the choice shocks are independent Type I extreme-value draws with scale one, independently over actions and dates. Let \(V(s;\theta)\) denote the expected value at state \(s\), before the current choice shocks are realized. The choice-specific value, excluding the current shock, is \(v_j(s;\theta)=u_j(s;\theta)+\beta\sum_{s'}P(s'\mid s,j)V(s';\theta)\). The Bellman equation integrates the maximum over current shocks.

Here \(\gamma\) is Euler’s constant, arising from the mean of the normalized Type I extreme-value distribution. It is common across actions at a given state; it affects the level of the value function but not the relative choice probabilities. Integrating the same shocks gives the conditional choice probability \(p_j(s;\theta)=\exp(v_j(s;\theta))/\sum_k\exp(v_k(s;\theta))\). Thus a probability cannot generally be calculated from current flow payoffs alone: the choice-specific values include the discounted value of future states.

Why the Inner Solution Is Well Defined

The Bellman operator maps a candidate value function \(W\) into a new value function by substituting \(W\) for \(V\) in the choice-specific values and then applying the log-sum expression. For any two candidate functions \(W\) and \(\widetilde W\), each action’s continuation value differs by at most \(\beta\lVert W-\widetilde W\rVert_\infty\), where \(\lVert\cdot\rVert_\infty\) is the largest absolute difference across states. The log-sum-exp function changes by no more than the largest absolute change in its inputs. Therefore the Bellman operator \(T_\theta\) satisfies \(\lVert T_\theta W-T_\theta\widetilde W\rVert_\infty\leq\beta\lVert W-\widetilde W\rVert_\infty\).

Because \(\beta<1\), this is a contraction. On a finite state space, the contraction mapping result implies a unique fixed point \(V(\cdot;\theta)\). Iterating \(V^{(m+1)}=T_\theta V^{(m)}\) converges to it from any finite starting vector. This justifies the inner solution and gives a practical way to compute it. It does not remove the need to solve again when \(\theta\) changes.

Constructing the Nested Likelihood

Suppose the data contain \(N\) individuals observed over periods \(t=1,\ldots,T_i\), with state \(s_{it}\) and chosen action \(a_{it}\). Conditional on the observed states, and under the shock assumptions above, the choice likelihood is the product of the probabilities assigned to the actions actually observed. Its log form is

For every trial value of \(\theta\), the analyst solves the Bellman equation and recomputes the probabilities before evaluating this expression. Maximum likelihood selects a parameter vector that maximizes \(\ell(\theta)\). If the model also specifies estimated transition probabilities and the state transitions are included in the joint likelihood, their log probabilities are added. If transitions are known or the objective is the conditional choice likelihood, no such transition terms are added.

Numerical Accuracy and Interpretation

The outer objective is only as reliable as the inner calculation. If the value iteration is stopped too early, the resulting choice probabilities may be inaccurate, and their log probabilities may shift the likelihood comparison. This is especially consequential when observed choices receive very small predicted probabilities: taking a logarithm makes such contributions strongly negative. A practical implementation should use a stated convergence tolerance, check that the Bellman residual is small, and ensure that reported likelihood values are stable to a tighter inner tolerance.

Warm-starting the inner solver at a new parameter vector with the previous value function can save computation, but it does not change the target: the iteration must still converge to the fixed point for the current parameter vector. Nor should a previously computed probability table be reused after changing parameters unless it is known to remain valid under the model.

Common Mistake

• Wrong reasoning: “I computed the value function and probabilities once, so I can hold them fixed while changing \(\theta\) in the likelihood.”
• Correction: The value function is defined by a Bellman equation containing \(\theta\). Changing \(\theta\) generally changes flow payoffs, values, and choice probabilities. Re-solve the Bellman equation at each candidate parameter vector before evaluating its likelihood.

A second error is to treat the likelihood as a measure of whether the model is true. It is instead the probability, under the specified model, of the observed choices (or joint observations if other terms are included), evaluated at a parameter vector. Maximum likelihood selects the vector that makes the observed data most likely within that model specification; it does not by itself establish a causal interpretation or validate the specification.