Q-Learning With Three Treatment Stages

From Terminal Predictions to Earlier Decisions

With three treatment decisions, the outcome of an early action depends not only on what happens immediately afterward, but also on the treatments that can be chosen later. Backward Q-learning handles this dependence by starting at the final treatment stage. It predicts the terminal contribution under each final action, chooses the best predicted continuation, and uses that prediction as part of the target for the preceding stage.

The essential operation is repeated twice: a fitted later-stage Q-function supplies a pseudo-outcome for an earlier-stage regression. This tutorial works through that sequence, including the intermediate rewards, the conditioning histories, and the maximization at each stage. The finite-horizon Bellman recursion and the general meaning of Q-functions are assumed from the earlier material on those subjects.

The histories must be timed correctly: \(H_t\) contains information observed before \(A_t\), while \(H_{t+1}\) may include new covariates observed after \(A_t\). In a longitudinal study, these histories ordinarily include prior treatments and covariates. A regression at stage 2 therefore uses stage-2 histories and treatments, not just the baseline variables.

For causal interpretation, assume consistency, sequential exchangeability of each treatment given its recorded history, and positivity for the actions being evaluated. The conditional means must be estimable from the observed data, and the action sets must permit a well-defined maximum (for example, finite action sets). These conditions do not make a misspecified regression correct; they describe the setting in which the observed conditional distributions can identify the relevant causal quantities.

The Three Backward Regression Targets

In a sample, the procedure first fits a model for \(Q_3\), using \(R_3+Y\) as the response and \(H_3,A_3\) as predictors. For each individual, the fitted terminal prediction is then maximized over feasible stage-3 actions. Adding the observed stage-2 reward gives the stage-2 pseudo-outcome. Regressing that target on \(H_2,A_2\) estimates \(Q_2\).

The same operation is repeated once more. Maximize the fitted stage-2 Q-function at each observed \(H_2\), add \(R_1\), and regress the resulting target on \(H_1,A_1\). The final fitted model is \(\widehat Q_1\).

The pseudo-outcome is not a newly observed clinical outcome. It is a constructed regression target: an observed reward plus a model-based prediction of future reward under the best available later action. The maximization happens separately for each person’s observed history, because later treatment can adapt to information that becomes available after the earlier decision.

Why the Pseudo-Outcome Targets the Earlier Q-Function

The regression construction follows directly from the recursive conditional-mean definitions. This is a statement about the target of each regression, not a claim that any particular fitted model is correctly specified.

Derivation. By the definition of \(Z_2\), conditioning on \(H_2,A_2\) gives $$ E[Z_2\mid H_2,A_2] =E\!\left[R_2+\max_{a_3\in\mathcal{A}_3(H_3)}Q_3(H_3,a_3)\mid H_2,A_2\right]. $$ The expression on the right is exactly the defining conditional mean for \(Q_2(H_2,A_2)\). The same substitution at stage 1 gives $$ E[Z_1\mid H_1,A_1] =E\!\left[R_1+\max_{a_2\in\mathcal{A}_2(H_2)}Q_2(H_2,a_2)\mid H_1,A_1\right] =Q_1(H_1,A_1). $$ These equalities require the displayed conditional expectations to exist; finite expected rewards are a sufficient condition. In estimation, fitted later-stage functions replace the population Q-functions, so the sample pseudo-outcomes are estimated targets rather than known conditional means. \(\square\)

A second useful fact explains why regression is the natural fitting step when squared-error loss is used.

Proof. Write \(Z-g=(Z-m)+(m-g)\). Squaring and taking expectations yields $$ E[(Z-g)^2]=E[(Z-m)^2]+E[(m-g)^2]+2E[(Z-m)(m-g)]. $$ The cross-term is zero: conditioning on \(H,A\), the factor \(m-g\) is fixed and \(E[Z-m\mid H,A]=0\). Thus $$ E[(Z-g)^2]=E[(Z-m)^2]+E[(m-g)^2]\geq E[(Z-m)^2]. $$ Equality holds exactly when \(g=m\) almost surely. In practice, the regression model restricts the available functions \(g\), so the fitted model is the best approximation within that model class under the chosen loss, not automatically the true conditional mean. \(\square\)

Diagnostics, Alternatives, and Limitations

Several checks help identify a faulty backward fit. Confirm that every target contains the reward from the current stage exactly once, and that the continuation term comes from the next stage’s fitted Q-function. Check that maximization uses the feasible action set for the person’s history. Inspect the number of observations supporting each relevant history-action combination; sparse support makes conditional means and their maxima unstable. Finally, examine fitted values and pseudo-outcomes for implausible magnitudes, unexpected ranges, and sensitivity to a small number of observations.

• Reward timing error: omitting \(R_2\) or adding it twice changes every affected stage-2 target and carries that error into stage 1.
• Wrong maximization order: maximizing an average over future histories instead of maximizing separately at each observed history can understate the value of adaptive treatment.
• Unsupported actions: a model may return predictions for actions rarely or never observed in a history subgroup. Those predictions rely on extrapolation, not direct information in the data.
• Unstable maxima: when two fitted actions have similar values, small prediction changes can switch the selected action. A high maximized prediction alone is not evidence that the decision is reliable.

Backward Q-learning is a regression-based alternative to inverse-probability weighting approaches, including marginal structural models: it models outcome means and propagates estimated continuation values rather than constructing treatment weights. It also differs from direct policy-learning approaches, which target a decision rule or its value more directly instead of fitting a sequence of conditional outcome regressions. These approaches involve different modeling and stability trade-offs; none removes the need for adequate treatment support and credible assumptions.

The procedure’s main limitation is that errors in later-stage predictions enter earlier-stage targets. Misspecification, sparse data, or noisy maximization at stage 3 can therefore affect both \(\widehat Q_2\) and \(\widehat Q_1\). The construction defines the sequence of targets, but does not by itself guarantee good estimation, reliable decisions, or valid uncertainty quantification.