Why Evaluate an Action by Its Future Consequences?
At a treatment decision, the outcome of interest is usually not an immediate response to that single treatment. The current action can change later covariates, which affect both later treatment choices and the eventual outcome. A decision rule therefore needs to evaluate a current action together with the best future decisions it makes possible. A stage-specific Q-function records that evaluation as a function of the current history and action.
The finite-horizon Bellman recursion expresses optimal value through successive conditional expectations and maximizations. The Q-function makes one part of that recursion especially explicit: for each current history, it compares the expected final outcome under each available current action, with optimal decisions used thereafter. The action attaining the largest value is the optimal action at that history.
Definition and Identification
Consider \(K\) treatment stages. Immediately before treatment at stage \(t\), let \(H_t\) denote the observed history, including baseline covariates and the treatments and covariates observed so far. Let \(A_t\) be the current treatment, and let \(Y\) be the final outcome, with larger values defined to be better. A policy \(d=(d_1,\ldots,d_K)\) assigns an available treatment \(d_t(h_t)\) to each history at each stage.
For a precise recursive form, write \(H_{t+1}\) for the next pre-treatment history, which incorporates the current treatment and the covariates that follow it. At the terminal stage, the Q-function is the conditional expected final outcome under the current action. Earlier Q-functions average the next-stage optimal value over the next-history distribution induced by the current action:
These conditional expectations have a causal interpretation when the usual identification conditions hold. Assume consistency, so observed outcomes and covariates agree with their potential counterparts under the treatments actually received. Assume sequential exchangeability: at each stage, conditional on \(H_t\), treatment assignment is independent of the future potential covariates and outcomes under the treatment sequences being considered. Also assume sequential positivity: each action needed for the target decisions has positive probability at histories where those decisions are to be evaluated. Under these conditions, the observed conditional distributions of next histories and terminal outcomes identify the corresponding intervention-specific expectations. Without exchangeability, for example, conditioning on observed treatment and history need not recover the outcome distribution under an intervention.
For clarity, suppose each stage has a finite set of available actions, histories and outcomes are measurable, and \(E[|Y|]\) is finite. The conditional expectations then exist, and the maximum over a finite action set is attained. A fixed tie-breaking rule, if needed, selects one action when multiple actions maximize \(Q_t\). For more general action spaces, existence and measurability of a maximizing rule require additional conditions; a supremum alone does not guarantee an available maximizing action.
Worked Example: Choosing at the Terminal Stage
At the final treatment stage, suppose the conditional mean outcome for each history and action is as follows. Larger outcomes are better.
| History | Mean if \(A_K=0\) | Mean if \(A_K=1\) | Maximizing action |
|---|---|---|---|
| \(h_K=\text{low risk}\) | 7 | 9 | 1 |
| \(h_K=\text{high risk}\) | 5 | 3 | 0 |
At the low-risk history, \(Q_K(h_K,0)=7\) and \(Q_K(h_K,1)=9\), so \(V_K(h_K)=\max(7,9)=9\), attained by action 1. At the high-risk history, \(Q_K(h_K,0)=5\) and \(Q_K(h_K,1)=3\), so \(V_K(h_K)=\max(5,3)=5\), attained by action 0. Thus the maximizing action depends on the history; selecting one treatment for everyone would not implement both stage-specific choices.
Why the Maximizing Action Is the Decision
At a fixed history, every candidate current action is compared on the same scale: expected final outcome under that action followed by optimal future decisions. This makes maximization direct. The formal argument also shows why a current action should not be selected using its immediate effect alone.
To justify the result, use the recursion defining \(Q_t\). For any available action \(a_t\), \(Q_t(h_t,a_t)\) is the conditional expected value of \(V_{t+1}(H_{t+1})\), or the expected terminal outcome when \(t=K\). By definition, \(V_t(h_t)=\max_{a_t}Q_t(h_t,a_t)\). Therefore,
The equality follows because \(d_t^*(h_t)\) is a maximizing action. At stage \(K\), this equality says precisely that the selected terminal action attains the largest conditional expected outcome. At each earlier stage, the recursion averages the next-stage value, so choosing an action that maximizes the current Q-function also maximizes the current conditional expected value when optimal later choices are used. Applying this argument successively from the terminal stage back to stage 1 establishes the claim. If more than one action ties, each tied action attains the same value. \(\square\)
Worked Example: A Current Action With Better Future Prospects
There are two treatment stages. At stage 2, the next history is either \(G\) or \(B\). The stage-2 conditional mean outcomes are:
| Stage-2 history | \(Q_2(h_2,0)\) | \(Q_2(h_2,1)\) | \(V_2(h_2)\) |
|---|---|---|---|
| \(G\) | 4 | 8 | 8 |
| \(B\) | 6 | 3 | 6 |
Thus the optimal stage-2 action is 1 at \(G\), yielding 8, and 0 at \(B\), yielding 6. Suppose the distribution of the stage-2 history depends on the stage-1 action:
| Stage-1 action | \(P(H_2=G\mid A_1)\) | \(P(H_2=B\mid A_1)\) |
|---|---|---|
| 0 | 0.8 | 0.2 |
| 1 | 0.3 | 0.7 |
For stage-1 action 0, the Q-function averages the optimal stage-2 values over the resulting history distribution: \(Q_1(h_1,0)=0.8(8)+0.2(6)=6.4+1.2=7.6\). For stage-1 action 1, it is \(Q_1(h_1,1)=0.3(8)+0.7(6)=2.4+4.2=6.6\). Since \(7.6>6.6\), the maximizing stage-1 action is 0. The calculation does not compare stage-1 actions by an immediate outcome; it compares their expected outcomes after the best available stage-2 decisions.
History-Specific Choices and Comparisons
A Q-function can produce a different action for different histories. This is not a contradiction: the conditional expected outcomes being compared can differ across histories, even though the decision objective is the same. For instance, a treatment may be preferable for one risk profile and inferior for another.
Worked Example: Personalized Versus Static Treatment
Suppose two histories, \(h_0\) and \(h_1\), each occur with probability \(0.5\). At the decision stage, the conditional mean outcomes are:
| History | \(Q(h,0)\) | \(Q(h,1)\) | Best action |
|---|---|---|---|
| \(h_0\) | 9 | 7 | 0 |
| \(h_1\) | 3 | 8 | 1 |
The history-specific rule chooses action 0 at \(h_0\) and action 1 at \(h_1\). Its expected outcome is \(0.5(9)+0.5(8)=4.5+4=8.5\). If a static rule must choose the same action for everyone, its value under action 0 is \(0.5(9)+0.5(3)=6\), while its value under action 1 is \(0.5(7)+0.5(8)=3.5+4=7.5\). The better static choice is action 1, with value 7.5, which is less than 8.5. This comparison assumes the history proportions are the stated target-population proportions and that the conditional means are causally identified.
These comparisons clarify the difference between a Q-based decision and two alternatives. A myopic decision maximizes a current or immediate outcome, ignoring subsequent consequences; the two-stage example shows why that can select the wrong first action. A static decision assigns the same action regardless of history; the personalized example shows when that restriction can sacrifice expected outcome. An alternative is to evaluate every possible complete policy and select the best one. The Q-function recursion instead organizes the comparison stage by stage, using each later-stage optimal value to assess earlier actions.
Diagnostics, Failure Modes, and Limits
The maximizing rule is only as credible as the Q-functions used to construct it. In data applications, the conditional expectations are estimated, and errors can change which action appears to maximize them. The following checks help identify problems with the decision itself or its interpretation:
- Check outcome direction. Confirm that larger values really are preferable. If the outcome is a cost or an adverse event, the decision objective may instead require minimizing the expected outcome or redefining the outcome as a benefit.
- Check support at relevant histories. If an action is never, or almost never, observed for histories where the rule would choose it, its action-specific Q-function is not reliably identified from those data. This is a positivity problem, not evidence that the fitted maximum is trustworthy.
- Check history construction. A history must contain the variables needed to define the decision and the relevant treatment and covariate sequence. Omitting prognostic information can undermine sequential exchangeability; indexing or timing errors can condition on information unavailable at the decision.
- Check the action comparison, not just fitted values. If estimated Q-values for two actions are nearly tied, small estimation changes can switch the selected action. A stable estimated value does not necessarily imply a stable decision.
- Check that future decisions are optimized inside the Q-function. Averaging the outcome after a fixed, nonoptimal future policy answers a different question from the Q-function defined here.
An optimal action is defined relative to the outcome, the available actions, the history, and the population in which the expectation is taken. It does not establish that the chosen action is best for every individual, nor does it guarantee good performance if the conditional outcome models are inaccurate or the target histories lack treatment support. Further, if a maximum is not attained in a more general action space, writing “choose the maximizing action” is not justified without extra assumptions. These qualifications do not change the finite-action definition; they mark where its direct application needs care.
Check Your Understanding
Answer each question using the definitions and calculations above. Solutions are included below the questions.
- At a terminal-stage history, the expected outcomes for actions 0 and 1 are 6 and 4. Which action maximizes the Q-function, and what is the value function?
- At a two-stage problem, the optimal next-stage values are 10 and 2. Under current action 0, their probabilities are 0.25 and 0.75; under current action 1, they are 0.5 and 0.5. Calculate both current-stage Q-values and identify the maximizing action.
- What does positivity contribute to interpreting an observed conditional mean as an intervention-specific Q-function?
- If two actions have equal Q-values at a history, must one of them have a larger expected outcome under the stated criterion?
Solutions. (1) Action 0 maximizes because \(6>4\), and the value is \(V_K(h_K)=6\). (2) \(Q(h,0)=0.25(10)+0.75(2)=2.5+1.5=4\); \(Q(h,1)=0.5(10)+0.5(2)=5+1=6\). Thus action 1 maximizes. (3) Positivity ensures that the action-specific conditional distributions needed for the intervention are represented in the observed data at the histories being evaluated; without support, the intervention-specific mean cannot be recovered from those observations without further assumptions. (4) No. The actions tie under the stated expected-outcome criterion, so either is a maximizing action.