Why a Three-Stage Calculation Requires Backward Steps
With three treatment decisions, the best choice at the first decision depends on the value of the later decisions that follow it. Q-learning handles this dependence by starting with the final decision, replacing the final outcome with the best estimated continuation value, and repeating that operation backward. The calculations are straightforward only if the fitted quantities have a clear interpretation: observational conditional means are not automatically causal values.
This example uses binary treatments and a binary baseline characteristic. There are no intermediate covariates apart from the treatment history. The example is deliberately small enough to check every number, but it still shows how the final rule can depend on the history available at each decision.
When the Fitted Regressions Have a Causal Meaning
Write \(Y^{a_1,a_2,a_3}\) for the terminal outcome that would be observed under the treatment sequence \((a_1,a_2,a_3)\). The example and the causal interpretation of backward Q-learning rely on the following assumptions.
- Consistency: if a person receives \((A_1,A_2,A_3)=(a_1,a_2,a_3)\), then \(Y=Y^{a_1,a_2,a_3}\).
- Sequential exchangeability: at each time \(t\), conditional on \(H_t\), the treatment \(A_t\) is independent of the collection of potential outcomes under all treatment sequences from that point onward.
- Positivity: for every history \(h_t\) with positive probability under the target population, each treatment that may be selected has positive conditional probability: \(P(A_t=a\mid H_t=h_t)>0\).
- Integrability: the terminal outcome has a finite expectation under the treatment sequences considered.
The causal continuation values are defined backward. At the last decision, \(Q_3(h_3,a_3)\) is the expected terminal outcome under the specified history and final treatment. At an earlier decision, \(Q_t(h_t,a_t)\) is the expected best continuation value after taking \(a_t\). The maximum is over the finite treatment set \(\{0,1\}\).
Under consistency, sequential exchangeability, and positivity, these observed-data conditional expectations identify the corresponding intervention-based continuation values. Without those assumptions, the same regressions may describe treatment selection rather than the consequences of intervening. In an application, the conditional expectations are estimated from data; the calculations below use a small constructed data set to make the fitted values exact and transparent.
Worked Example: Fitting the Final-Stage Q-Function
Worked Example: Fitting the Final-Stage Q-Function
Suppose the terminal outcome in a balanced teaching data set is generated by
The data contain all 16 combinations of \(X,A_1,A_2,A_3\), with one record for each combination. Thus a saturated cell-mean fit reproduces the displayed deterministic outcome in every cell. This is a numerical illustration, not a claim that one record per cell would generally estimate conditional means precisely.
For a fixed \(X,A_1,A_2\), write the outcome as \(B+cA_3\), where
Consequently, \(Q_3(H_3,0)=B\) and \(Q_3(H_3,1)=B+c\). For example, when \(X=0,A_1=0,A_2=1\), \(B=-1\) and \(c=-0.6\), so the fitted values are \(-1\) for \(A_3=0\) and \(-1.6\) for \(A_3=1\). Treatment 0 is preferred in that history.
Repeating the same substitution in every history gives the complete final-stage fit.
| \(X\) | \(A_1\) | \(A_2\) | \(Q_3(H_3,0)\) | \(Q_3(H_3,1)\) | Best \(A_3\) |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 1.5 | 1 |
| 0 | 0 | 1 | -1 | -1.6 | 0 |
| 0 | 1 | 0 | 1 | 4.5 | 1 |
| 0 | 1 | 1 | 0 | 1.4 | 1 |
| 1 | 0 | 0 | 2 | 0.3 | 0 |
| 1 | 0 | 1 | 3 | -0.8 | 0 |
| 1 | 1 | 0 | 0 | 0.3 | 1 |
| 1 | 1 | 1 | 1 | -0.8 | 0 |
The final-stage decision is history-dependent: the sign of \(c\) determines whether \(A_3=1\) improves the fitted continuation value. For instance, at \((X,A_1,A_2)=(1,1,0)\), \(c=0.3\), so treatment 1 is preferred; at \((1,1,1)\), \(c=-1.8\), so treatment 0 is preferred.
Worked Example: Forming the Second-Stage Pseudo-Outcome
Worked Example: Forming the Second-Stage Pseudo-Outcome
For each observed history through \(A_2\), the second-stage pseudo-outcome is the larger of the two fitted final-stage values. In notation, it is \(\widetilde Y_2=\max_{a_3}Q_3(H_3,a_3)\). The row \((X,A_1,A_2)=(0,1,0)\), for example, has \(Q_3\) values 1 and 4.5, so its pseudo-outcome is 4.5.
Applying that calculation to all histories yields the saturated second-stage regression. Each pair in the middle column is ordered as the fitted value for \(A_2=0\), then for \(A_2=1\).
| \(X\) | \(A_1\) | \(Q_2(H_2,0), Q_2(H_2,1)\) | Best \(A_2\) | Best continuation value |
|---|---|---|---|---|
| 0 | 0 | 1.5, -1 | 0 | 1.5 |
| 0 | 1 | 4.5, 1.4 | 0 | 4.5 |
| 1 | 0 | 2, 3 | 1 | 3 |
| 1 | 1 | 0.3, 1 | 1 | 1 |
For the last two rows, the arithmetic follows directly from the final-stage table. At \(X=1,A_1=0,A_2=0\), the best final-stage value is \(\max(2,0.3)=2\); at \(A_2=1\), it is \(\max(3,-0.8)=3\). Thus the stage-two comparison is \(2\) versus \(3\), and \(A_2=1\) is selected. At \(X=1,A_1=1\), the corresponding maxima are \(\max(0,0.3)=0.3\) and \(\max(1,-0.8)=1\), again selecting \(A_2=1\).
In an ordinary sample, \(\widetilde Y_2\) is calculated for each participant using that participant’s observed \(H_3\). A regression of this pseudo-outcome on \(H_2,A_2\) estimates \(Q_2\). Here the saturated cell means give the displayed values exactly.
Worked Example: Forming the First-Stage Rule
Worked Example: Forming the First-Stage Rule
The first-stage pseudo-outcome is \(\widetilde Y_1=\max_{a_2}Q_2(H_2,a_2)\). Taking the maximum within each \((X,A_1)\) history gives
Therefore the first decision is \(A_1=1\) when \(X=0\), because \(4.5>1.5\), and \(A_1=0\) when \(X=1\), because \(3>1\). Following those first decisions, the second-stage table selects \(A_2=0\) for \(X=0,A_1=1\), and \(A_2=1\) for \(X=1,A_1=0\). The final-stage table then selects \(A_3=1\) in the first history and \(A_3=0\) in the second.
The resulting rule on histories reached by its own first two decisions is
The displayed \(d_2\) and \(d_3\) give the decisions on histories reached under \(d_1\) and \(d_2\). A complete rule can also specify actions at histories that the rule itself never reaches; the tables above provide those actions.
Why Backward Maximization Works
To justify the recursion, at stage 3 consistency identifies the observed outcome with the potential outcome for the received treatment, while exchangeability makes the received treatment independent of the relevant potential outcomes given \(H_3\). Positivity ensures that each candidate treatment is represented at histories where its value is required. Thus the conditional mean defining \(Q_3\) equals the conditional mean under intervention on \(A_3\).
Now fix a history \(H_2=h_2\) and treatment \(A_2=a_2\). Under the same assumptions, the conditional distribution of subsequent outcomes under each final treatment is identified. Since there are only two final actions, the best attainable continuation value is the maximum of their two conditional values. Taking its conditional expectation given \(H_2=h_2,A_2=a_2\) gives \(Q_2(h_2,a_2)\). Repeating this argument at stage 1 gives \(Q_1(h_1,a_1)\). Integrability ensures these conditional expectations and comparisons are well-defined.
For optimality, take any competing rule \(d'\). At stage 3, for each history, the maximizing rule has a continuation value at least as large as the value under \(d'_3\), by the definition of a maximum. Conditional expectation preserves this inequality, so the same comparison holds for the continuation values at stage 2 after any fixed \(A_2\). Maximizing over \(A_2\) therefore gives a value at least as large as the value from \(d'_2\) followed by \(d'_3\). Applying the same argument to the first-stage choice shows that the recursively maximizing rule has expected outcome at least as large as the full value of \(d'\). This is the finite-horizon backward-induction argument. \(\square\)
This result concerns causal continuation values, not arbitrary observational regressions. If sequential exchangeability fails, for example because an unmeasured prognostic factor influences both treatment and outcome, the observed conditional means may rank treatments differently from their intervention values. The algebraic maximization would still run, but its causal conclusion would not follow.
Diagnostics, Alternatives, and Limitations
In real data, each backward step adds estimation uncertainty: errors in the final-stage fit affect its maximum, which becomes the target in the second-stage fit, and then the first-stage fit. Useful checks include examining treatment counts and estimated treatment probabilities within relevant histories, checking whether fitted values extrapolate beyond observed support, and comparing the selected actions with the underlying treatment-specific fitted values. A rule that changes sharply across nearby histories can signal sparse data, model instability, or a near tie rather than a robust clinical distinction.
Positivity deserves particular attention. If a treatment is never observed at a history, the corresponding conditional mean cannot be learned from that history without extrapolation. A fitted model may still return a number, but that does not repair the missing information. Likewise, a saturated fit is transparent here because every cell is populated; with many covariates and stages, it may be too sparse to use. Parsimonious regression, regularization, or other modeling choices can improve stability but introduce assumptions about functional form and treatment interactions.
Q-learning is a regression-based alternative to approaches such as g-computation or inverse-probability weighting. Its backward regressions directly target conditional continuation values, while weighting approaches reweight observed treatment paths. These strategies have different modeling and stability properties; none removes the need to define a target population and make assumptions that connect observed data to interventions. This tutorial’s calculation does not address uncertainty quantification or methods for choosing among competing model specifications.
Common Pitfall
A second practical error is to maximize the observed terminal outcome once at the end and then use that same outcome as the target at every earlier stage. The correct targets are constructed backward: maximize the stage-three Q-function to form the stage-two pseudo-outcome, fit the stage-two Q-function, then maximize it to form the first-stage pseudo-outcome.
Check Your Understanding
Answer each question, then compare your reasoning with the short solutions below.
- At history \((X,A_1,A_2)=(1,0,1)\), what are the two fitted final-stage values, and which treatment is selected?
- For \(X=0,A_1=1\), calculate both second-stage values and identify the selected \(A_2\).
- What first-stage action is selected when \(X=1\), and what two values determine that choice?
- Why is positivity needed if the analyst can still compute a regression prediction for an unobserved treatment-history cell?
Solutions. (1) The final-stage values are \(3\) for \(A_3=0\) and \(-0.8\) for \(A_3=1\), so select \(A_3=0\). (2) The values are \(4.5\) for \(A_2=0\) and \(1.4\) for \(A_2=1\), so select \(A_2=0\). (3) At \(X=1\), the first-stage values are \(Q_1(1,0)=3\) and \(Q_1(1,1)=1\); select \(A_1=0\). (4) A model can extrapolate a prediction where no treatment is observed, but the data then provide no empirical comparison for that action at that history. Positivity is what supports identification of the treatment-specific conditional value from observed data rather than unsupported extrapolation.