Choice-Specific Values and Conditional Choice Probabilities

From Dynamic Values to Choices

In a dynamic discrete choice model, an agent chooses among actions while anticipating how each action affects future states and payoffs. The agent also observes a current payoff shock that the econometrician does not observe. These two ingredients create an important distinction: a choice-specific value summarizes the expected payoff from an action before its current shock is known, while the utility that determines the actual choice includes that shock.

When the shocks are independent Type I extreme value, this distinction gives a convenient link between values and choices. A set of choice-specific values determines conditional choice probabilities through a softmax formula. The expected value before the shock is realized is the expected maximum of the shocked utilities, not the maximum of the choice-specific values alone.

Definition: Let $s$ denote the state observed before a choice, $j\in\mathcal{J}$ an available action, and $\epsilon_j$ its current payoff shock. The choice-specific value $v_j(s)$ is the expected payoff from taking action $j$, including its continuation value, but excluding the current shock $\epsilon_j$. The realized choice utility is $U_j(s)=v_j(s)+\epsilon_j$.

Assume a finite action set, a discount factor $0\leq\beta<1$, and state transitions that follow a Markov rule conditional on the current state and chosen action. Write $u_j(s)$ for the current payoff excluding the shock, and $P(s'\mid s,j)$ for the probability of next state $s'$. If $V(s')$ is the expected value at that next state before its current choice shocks are drawn, then

v_j(s)=u_j(s)+\beta\sum_{s'}P(s'\mid s,j)V(s').

The transition term averages over possible next states and discounts their values. The agent then adds the current shock to each action's value and chooses an action with the largest realized utility. Thus, conditional on the state, action $j$ is chosen when $v_j(s)+\epsilon_j\geq v_k(s)+\epsilon_k$ for every available action $k$.

Extreme-Value Shocks and Conditional Choice Probabilities

Use the location-zero Type I extreme-value distribution with scale $\sigma>0$, independently across actions conditional on the state and the information entering the model. Its cumulative distribution function is

F(\epsilon)=\exp\left[-\exp\left(-\frac{\epsilon}{\sigma}\right)\right].

The scale governs the dispersion of shocks relative to differences in choice-specific values. A common location shift in all shocks does not affect which action has the largest utility; the location-zero convention fixes that otherwise arbitrary level. With this convention, the shocks have mean $\gamma\sigma$, where $\gamma\approx0.5772$ is the Euler–Mascheroni constant.

Result: Under independent Type I extreme-value shocks with common scale $\sigma$, the conditional choice probability for action $j$ in state $s$ is $$ P(j\mid s)=\frac{\exp(v_j(s)/\sigma)}{\sum_{k\in\mathcal{J}}\exp(v_k(s)/\sigma)}. $$ The expected maximum realized utility, or value before the current shocks are observed, is $$ V(s)=\mathbb{E}_{\epsilon}\left[\max_{j\in\mathcal{J}}\{v_j(s)+\epsilon_j\}\right] =\gamma\sigma+\sigma\log\left(\sum_{j\in\mathcal{J}}\exp(v_j(s)/\sigma)\right). $$

These probabilities are conditional on the state and the model's parameters, with the current shocks integrated out. They sum to one, and every available action has a strictly positive probability for finite values. Increasing $v_j(s)$ relative to other values raises action $j$'s probability. Increasing $\sigma$ makes probabilities less concentrated: shocks are more dispersed relative to the value differences.

Why the Softmax Formula Holds

For completeness, the probability formula follows directly from the shock distribution and the choice rule. Suppress the state argument and condition on the shock $\epsilon_j=e$. Action $j$ is selected if $\epsilon_k\leq e+v_j-v_k$ for each $k\ne j$. Independence therefore gives

P(j\text{ chosen}\mid \epsilon_j=e) =\prod_{k\ne j}F(e+v_j-v_k).

The density of $\epsilon_j$ is $f(e)=\sigma^{-1}\exp(-e/\sigma)\exp[-\exp(-e/\sigma)]$. Integrating the conditional probability against this density and substituting $z=\exp(-e/\sigma)$ gives

\begin{aligned} P(j\text{ chosen}) &=\int_{-\infty}^{\infty} f(e)\prod_{k\ne j}F(e+v_j-v_k)\,de\\ &=\int_0^\infty \exp\left[-z\sum_{k\in\mathcal{J}}\exp((v_k-v_j)/\sigma)\right]\,dz\\ &=\frac{1}{\sum_{k\in\mathcal{J}}\exp((v_k-v_j)/\sigma)} =\frac{\exp(v_j/\sigma)}{\sum_{k\in\mathcal{J}}\exp(v_k/\sigma)}. \end{aligned}

The expected maximum follows from its distribution. For $M=\max_j\{v_j+\epsilon_j\}$, independence implies

P(M\leq m) =\prod_j F(m-v_j) =\exp\left[-\exp\left(-\frac{m-\sigma\log\sum_j\exp(v_j/\sigma)}{\sigma}\right)\right].

This is a Type I extreme-value distribution with location $\sigma\log\sum_j\exp(v_j/\sigma)$ and scale $\sigma$. Its mean is its location plus $\gamma\sigma$, yielding the stated expression for $V(s)$. The log-sum term is often called the inclusive value. It accounts for the possibility that a favorable shock makes an action with a lower deterministic value the best realized option.

Worked Examples

Worked Example: Two Actions and a Half-Unit Shock Scale

Suppose a state has two actions with choice-specific values $v_1=1.2$ and $v_2=0.4$, measured in utility units. Let the independent shock scale be $\sigma=0.5$. Compute the conditional probabilities and the value before the shocks are observed.

Choice probabilities. The relative value difference is $1.2-0.4=0.8$, which is $0.8/0.5=1.6$ scale units. Therefore

P(1)=\frac{1}{1+\exp(-1.6)}\approx0.8320,\qquad P(2)=\frac{1}{1+\exp(1.6)}\approx0.1680.

The probabilities sum to $1.0000$. Action 1 is more likely because its choice-specific value is higher, but action 2 still has positive probability because its shock can be sufficiently favorable.

Expected maximum. Factoring out the larger value is numerically convenient:

\begin{aligned} V &=0.5772(0.5)+0.5\log\left[\exp(1.2/0.5)+\exp(0.4/0.5)\right]\\ &=1.2+0.2886+0.5\log\left(1+\exp(-1.6)\right)\\ &\approx1.2+0.2886+0.0920 =1.5806. \end{aligned}

This is an expected value in utility units before the current shocks are observed. It exceeds the larger choice-specific value, $1.2$, because the selected action can receive a favorable shock.

Worked Example: Continuation Values Determine Current Probabilities

At state $s$, an agent chooses between actions A and B. The discount factor is $\beta=0.9$. After A, the next state is good with probability $0.7$ and bad with probability $0.3$; after B, those probabilities are $0.2$ and $0.8$. Suppose $V(\text{good})=4$, $V(\text{bad})=1$, and current payoffs excluding shocks are $u_A(s)=2$ and $u_B(s)=3.2$. Let $\sigma=0.5$.

Calculate the choice-specific values. The expected next-state value under A is $0.7(4)+0.3(1)=3.1$, while under B it is $0.2(4)+0.8(1)=1.6$. Hence

v_A(s)=2+0.9(3.1)=4.79,\qquad v_B(s)=3.2+0.9(1.6)=4.64.

Although B has the larger current payoff, A has the larger total choice-specific value because its continuation value is higher.

Calculate the probabilities. The value difference is $4.79-4.64=0.15$, or $0.15/0.5=0.3$ scale units:

P(A\mid s)=\frac{1}{1+\exp(-0.3)}\approx0.5744,\qquad P(B\mid s)=\frac{1}{1+\exp(0.3)}\approx0.4256.

The probabilities sum to $1.0000$. The model does not imply that A is chosen with certainty; the current shocks leave B a substantial conditional probability.

Calculate the pre-shock value. Using the larger value to factor the log-sum,

\begin{aligned} V(s) &=0.5772(0.5)+0.5\log\left[\exp(4.79/0.5)+\exp(4.64/0.5)\right]\\ &=4.79+0.2886+0.5\log\left(1+\exp(-0.3)\right)\\ &\approx4.79+0.2886+0.2772=5.3558. \end{aligned}

The transition probabilities enter the calculation of each action's continuation value. The extreme-value formula is applied only after those choice-specific values have been formed.

Worked Example: A Realized Choice Is Not Its Ex Ante Probability

Return to two actions with $v_1=1.2$, $v_2=0.4$, and $\sigma=0.5$. Suppose the shocks realized in one particular decision are $\epsilon_1=-0.3$ and $\epsilon_2=0.1$. Determine the chosen action and distinguish this realization from its conditional probability.

The realized utilities are

U_1=1.2-0.3=0.9,\qquad U_2=0.4+0.1=0.5.

Thus action 1 is chosen in this realization because $0.9>0.5$. Before seeing these shocks, however, its conditional choice probability was approximately $0.8320$, not $1$. The probability describes the frequency implied by the shock distribution across repeated draws at the same state and values; it is not the realized utility and does not determine every individual choice.

Assumptions, Normalizations, and Interpretation

The formulas rely on specific assumptions. The shocks must have the stated Type I extreme-value distribution, be independent across actions, and share a common scale. The agent must observe the current shocks before choosing, while the conditional probabilities integrate over shocks that the econometrician does not observe. The continuation-value calculation also requires the transition probabilities and the relevant future values to be defined conditional on state and action. If shocks are correlated across actions or have another distribution, the softmax and log-sum formulas generally do not apply in this form.

The location convention matters for the level of $V(s)$, not for choice probabilities. If the same constant is added to every choice-specific value, the softmax probabilities are unchanged. Likewise, shifting all shocks by a common constant shifts the expected maximum but leaves the selected action unchanged. Some formulations center the shocks to have mean zero; under that alternative convention the $\gamma\sigma$ term is removed from the expected-maximum formula. Use one convention consistently when writing the Bellman equation.

The scale has both a probabilistic and a normalization role. A larger $\sigma$ spreads probability more evenly across actions for fixed value differences. In many applications, the scale of utility is not separately informative without an additional normalization, because multiplying all payoffs and the shock scale by the same positive constant leaves the choice probabilities unchanged.

Common Pitfalls

Confusing value with realized utility. The choice-specific value $v_j(s)$ excludes the current shock. The agent chooses using $v_j(s)+\epsilon_j$.
Taking the maximum too early. In general, $V(s)$ is not $\max_j v_j(s)$. It is the expected maximum after shocks are added, given by the log-sum expression under the stated assumptions.
Dropping the scale. The exponent is $v_j(s)/\sigma$, not just $v_j(s)$, and the inclusive value is multiplied by $\sigma$.
Forgetting the shock-location convention. With location-zero shocks, the expected maximum contains $\gamma\sigma$. A centered-shock convention changes that level term, so the two conventions should not be mixed.
Treating a conditional probability as a deterministic choice. A probability such as $0.5744$ describes the model's choice probability conditional on the state; one realized decision still selects a single action.

Key takeaway: First form each action's value from its current payoff and expected continuation value. Then add its current shock to obtain realized utility. With independent Type I extreme-value shocks, the value differences determine softmax conditional choice probabilities, and the pre-shock value is the expected maximum, including the location term specified by the shock normalization.

Check Your Understanding

Answer each question using the definitions and assumptions stated above.

In one sentence, distinguish a choice-specific value from a realized choice utility.
Two actions have values $v_1=2$ and $v_2=1$, with scale $\sigma=1$. Write the conditional choice probability for action 1.
Why can an action with a lower choice-specific value still be chosen?
In a dynamic model, how do transition probabilities enter the choice-specific value?
What part of the expected-maximum formula changes if shocks are centered to have mean zero rather than given the location-zero convention?