Counterfactual Policy Changes to Payoffs and Transitions

Why a Policy Change Requires a New Dynamic Solution

A subsidy can make one action more attractive immediately; a change in eligibility, service access, or operating conditions can instead alter which states follow an action. Either change can affect choices today and the states encountered tomorrow. In a dynamic choice model, the second effect feeds back into today’s decision through its continuation value.

A counterfactual policy calculation therefore does more than add a subsidy to an estimated choice probability. It keeps the estimated preference parameters fixed, specifies the policy’s effects on payoffs and transitions, and solves the resulting choice problem under those new conditions. The choice probabilities are calculated from that new solution.

Definition: A policy counterfactual evaluates choices and values under a specified alternative policy while holding the estimated preference parameters fixed. Policy-dependent payoffs and state transitions may change; the preference parameters do not. The counterfactual value function and choice probabilities must be computed for the alternative policy.

The Dynamic Choice Model and Policy Inputs

Consider a stationary infinite-horizon model with a finite set of states \(x\), a finite set of available actions \(a\), and discount factor \(0<\beta<1\). In state \(x\), the agent receives deterministic payoff \(u(x,a;\theta,\pi)\) and an action-specific choice shock \(\varepsilon_a\). Here \(\theta\) denotes the estimated preference parameters and \(\pi\) denotes the policy. After choosing \(a\), the next state is drawn from \(F_\pi(x' \mid x,a)\).

Assume choice shocks are independent across actions and periods, independent of the state transition conditional on the state and action, and Type I extreme value with scale normalized to one. These are the assumptions under which the logit formulas below apply. The value function \(V_\pi(x)\) is the expected discounted value before the current period’s shocks are observed. Define the choice-specific value, excluding the current shock, by

q_\pi(x,a)=u(x,a;\theta,\pi)+\beta\sum_{x'}F_\pi(x'\mid x,a)V_\pi(x').

The first term is the current deterministic payoff. The second is the discounted expected value of the next state. With the stated shock normalization, integrating out the current shocks gives the stationary Bellman equation and conditional choice probabilities:

V_\pi(x)=\gamma+\log\left(\sum_a \exp\{q_\pi(x,a)\}\right), \qquad P_\pi(a\mid x)= \frac{\exp\{q_\pi(x,a)\}}{\sum_j\exp\{q_\pi(x,j)\}},

where \(\gamma\) is the Euler–Mascheroni constant, approximately \(0.5772\). The common \(\gamma\) affects the level of the value function under this shock normalization, but it cancels from the choice probabilities. In a policy counterfactual, both \(u\) and \(F\) may change. The parameter \(\theta\) and the discount factor \(\beta\) remain fixed.

Key distinction: A policy change to \(u\) changes the immediate payoff; a policy change to \(F\) changes expected future values. Either can alter \(q_\pi(x,a)\), and an altered transition law can change the value function even when current payoffs are unchanged.

Solving the Counterfactual Value Function

For an infinite-horizon stationary counterfactual, guess a value for every state, calculate each action’s expected continuation value using the counterfactual transition law, and update the values with the Bellman equation. Repeat until the updates are sufficiently small. Then calculate choice probabilities using the converged choice-specific values.

Specify the policy.
Write the counterfactual payoff \(u_\pi(x,a;\theta)\) and transition probabilities \(F_\pi(x'\mid x,a)\). Keep \(\theta\) fixed.

Initialize state values.
Choose starting values \(V^{(0)}(x)\), such as zero for every state.

Update values and choices.
Calculate \(q^{(k)}_\pi(x,a)\) using \(V^{(k)}\), then set \(V^{(k+1)}(x)=\gamma+\log\sum_a\exp\{q^{(k)}_\pi(x,a)\}\).

Check convergence and report probabilities.
Continue until the largest change in state values is below a chosen numerical tolerance. Use the converged choice-specific values to compute \(P_\pi(a\mid x)\).

Why Value Iteration Converges

For this finite-state model, the Bellman update is a contraction when \(0<\beta<1\). This fact justifies iterating the counterfactual Bellman equation rather than relying on an arbitrary stopping point. To see why, let two candidate value functions \(V\) and \(W\) differ by at most \(d=\max_x|V(x)-W(x)|\). For any state and action, their expected continuation values differ by at most \(d\), because transition probabilities are nonnegative and sum to one. Thus the corresponding choice-specific values differ by at most \(\beta d\).

The log-sum-exp function changes by no more than the largest change in its inputs. In particular, if every input changes by at most \(c\), then the log-sum-exp changes by at most \(c\). Applying this with \(c=\beta d\) shows that the updated value functions differ by at most \(\beta d\). Since \(\beta<1\), successive Bellman updates contract distances between candidate value functions. The Bellman equation consequently has a unique fixed point, and value iteration converges to it.

Worked Example: A Subsidy in a One-State Infinite-Horizon Model

Suppose there is one state, two actions (work and school), and the state returns to itself after either action. The discount factor is \(\beta=0.90\). Estimated preferences imply deterministic payoffs of 1.00 payoff unit for work and 1.40 payoff units for school. A policy adds a school subsidy of 0.50 payoff unit. Hold these preferences and the transition law fixed.

Because both actions lead back to the same state, the continuation value \(\beta V\) is common to both actions. Before the subsidy, the school choice probability is therefore

P(\text{school})= \frac{\exp(1.40)}{\exp(1.00)+\exp(1.40)} =\frac{1}{1+\exp(-0.40)} \approx 0.5987.

The inclusive value equation reduces to \(V=\gamma+\log\{\exp(1.00)+\exp(1.40)\}+\beta V\). Solving this equation gives

V= \frac{0.5772+\log\{\exp(1.00)+\exp(1.40)\}}{1-0.90} \approx \frac{0.5772+1.913}{0.10} =24.90.

Under the subsidy, the school payoff becomes \(1.40+0.50=1.90\), while the work payoff remains 1.00. Thus

P_\pi(\text{school})= \frac{1}{1+\exp\{-(1.90-1.00)\}} \approx 0.7109, \qquad V_\pi= \frac{0.5772+\log\{\exp(1.00)+\exp(1.90)\}}{0.10} \approx 28.18.

The counterfactual raises the probability of choosing school from about 0.5987 to 0.7109 and raises the value from 24.90 to 28.18 payoff units. In this particular model, the common continuation value cancels from the choice probability. That cancellation need not occur when actions lead to different future states.

When Policy Changes State Transitions

A transition change affects an action’s choice-specific value through the expected value of the next state. It can therefore change current choices even if the current payoff schedule is unchanged. In a finite-horizon model, the calculation is especially direct: use the specified terminal values as continuation values, calculate current choice-specific values under the policy transitions, and then integrate over the current shocks. This is a counterfactual calculation, not an estimate of new preferences.

Worked Example: Improving the Transition to a Valuable State

Consider one current state and two actions, A and B, with discount factor \(\beta=0.90\). At the end of this period, the agent reaches either a good state with value 2.00 or a poor state with value 0.00. Current deterministic payoffs are 0.30 for A and 0.00 for B. Initially, A leads to the good state with probability 0.40, while B does so with probability 0.50. A policy changes only A’s transition probability, raising it to 0.80. The payoffs, terminal values, and preference parameters stay fixed.

Before the policy change, expected continuation values are \(0.40(2.00)+0.60(0.00)=0.80\) after A and \(0.50(2.00)+0.50(0.00)=1.00\) after B. The choice-specific values are

q(A)=0.30+0.90(0.80)=1.02, \qquad q(B)=0.00+0.90(1.00)=0.90.

Consequently, the probability of A is \(1/[1+\exp\{-(1.02-0.90)\}]\approx0.5300\). After the policy change, A’s expected continuation value is \(0.80(2.00)+0.20(0.00)=1.60\), so its new choice-specific value is \(0.30+0.90(1.60)=1.74\). B’s value remains 0.90. The counterfactual probability of A is therefore

P_\pi(A)=\frac{1}{1+\exp\{-(1.74-0.90)\}} =\frac{1}{1+\exp(-0.84)} \approx0.6985.

The transition policy increases the probability of A from about 0.5300 to 0.6985 without changing either action’s current payoff. Its effect operates through the expected value of the next state.

Combining Payoff and Transition Changes Across States

When actions have different transition laws, policy can affect choices differently across current states. The following finite-horizon example computes those choices state by state. Its terminal values are treated as given, so the calculations do not require an infinite-horizon fixed point; the same separation between current payoffs and continuation values applies in a stationary solution.

Worked Example: A Subsidy and State-Dependent Transition Reform

There are two current states, L and H, and actions A and B. The discount factor is 0.90. At the end of the period, the terminal value is 0 in L and 2 in H. Initially, action A pays 0.20 in either current state; action B pays 0.60. From L, the probability of reaching H is 0.30 after A and 0.50 after B. From H, it is 0.70 after A and 0.50 after B. A reform adds 0.40 to A’s current payoff and changes A’s probability of reaching H to 0.70 from L and 0.90 from H. B’s payoff and transitions remain unchanged.

Before the reform, in state L the choice-specific values are \(q(A)=0.20+0.90[0.30(2)+0.70(0)]=0.74\) and \(q(B)=0.60+0.90[0.50(2)+0.50(0)]=1.50\). Thus \(P(A\mid L)=1/[1+\exp(0.76)]\approx0.3186\). In state H, they are \(q(A)=0.20+0.90[0.70(2)]=1.46\) and \(q(B)=1.50\), giving \(P(A\mid H)=1/[1+\exp(0.04)]\approx0.4900\).

Under the reform, A’s payoff is 0.60. In L, \(q_\pi(A)=0.60+0.90[0.70(2)]=1.86\), while \(q_\pi(B)=1.50\). Hence \(P_\pi(A\mid L)=1/[1+\exp(-0.36)]\approx0.5890\). In H, \(q_\pi(A)=0.60+0.90[0.90(2)]=2.22\), while \(q_\pi(B)=1.50\), so \(P_\pi(A\mid H)=1/[1+\exp(-0.72)]\approx0.6726\). Both the subsidy and the transition change enter the new choice-specific values.

For example, suppose the initial population has probability 0.60 of being in L and 0.40 of being in H. Its initial probability of choosing A is \(0.60(0.3186)+0.40(0.4900)=0.3872\) under the original policy and \(0.60(0.5890)+0.40(0.6726)=0.6224\) under the reform. The same starting state distribution is used for this comparison; subsequent state distributions can be simulated using the corresponding choice probabilities and transition laws.

From Conditional Choices to State Distributions

A state-specific choice probability is not the same as the population share choosing an action. To obtain an aggregate choice share, average the conditional probabilities over the relevant state distribution. To project the distribution forward, combine those probabilities with the action-specific transition laws. For example, if the probability of being in state \(x\) is \(\mu_\pi(x)\), then the next-period probability of state \(x'\) is

\mu'_\pi(x')= \sum_x \mu_\pi(x)\sum_a P_\pi(a\mid x)F_\pi(x'\mid x,a).

This equation makes clear why a policy can affect future aggregate choices through two channels: it changes the conditional action probabilities, and it can change the transition probabilities that determine which states agents enter. For a multi-period projection, repeat the update using the new state distribution at each period. The initial distribution should be stated; it is not automatically the same as a long-run distribution under the counterfactual policy.

Common Mistakes

Changing probabilities without re-solving values. Adding a subsidy to the baseline choice-specific payoff and reporting a new logit probability ignores any policy effects on continuation values. Recompute the Bellman solution whenever the relevant payoff or transition changes.
Changing current payoffs but leaving policy transitions in the baseline. A transition reform changes expected continuation values even if preferences and current payoffs are fixed. Use the counterfactual \(F_\pi\) inside the expectation.
Re-estimating preferences automatically. This exercise is a fixed-parameter counterfactual: retain \(\theta\) and change only the specified policy inputs. It is not a claim that the reform changes preferences.
Confusing conditional and aggregate choice probabilities. \(P_\pi(a\mid x)\) describes choices among agents in state \(x\). An overall share also depends on how many agents occupy each state.

Key takeaway: A disciplined dynamic policy counterfactual specifies new payoffs and transition laws, holds estimated preferences fixed, recomputes values, and derives new conditional choice probabilities from those values. State-distribution projections then combine the new choices with the new transitions.

Check Your Understanding

Answer each question using the distinction between current payoffs, continuation values, and state distributions.

In a stationary model, a subsidy changes one action’s payoff but not any transition probability. Which quantities must be recomputed for a fixed-parameter counterfactual?
Why can a transition reform change a choice probability when all current deterministic payoffs remain unchanged?
Suppose two actions have identical transitions from every state. Explain why their common continuation value does not affect their relative logit probability in that state.
What additional information is needed to turn state-specific choice probabilities into an aggregate choice share?
In value iteration, why does the condition \(0<\beta<1\) support convergence to a unique stationary value function under the stated finite-state model?