Common Errors In Backward Q-Learning

Why Small Backward-Step Errors Matter

Backward Q-learning constructs a treatment rule one decision at a time. At the final stage, it estimates the expected outcome under each available treatment and selects the better option. At the earlier stage, it must use the value of following that optimized final-stage rule—not simply the outcome that happened under the final treatment a person happened to receive. Confusing these quantities can reverse an earlier treatment decision even when the final-stage model is correct.

Three errors are especially easy to make: using observed future outcomes instead of the optimized pseudo-outcome; reversing the sign of a treatment contrast; and maximizing a prediction that belongs to the wrong decision stage. The calculations below make each distinction explicit.

The Two-Stage Quantities to Keep Separate

Consider two treatment stages. Let \(H_1\) be the information available before the first treatment \(A_1\), and let \(H_2\) be the information available before the second treatment \(A_2\). The final outcome is \(Y\). For clarity, suppose each action set is finite. The stage-two Q-function and its optimized value are

The stage-one Q-function evaluates each possible first treatment while assuming that the best available second-stage treatment will be used afterward:

In practice, backward Q-learning estimates the stage-two Q-function, calculates a fitted maximum for each observed \(H_2\), and uses those fitted maxima as responses in the stage-one regression. Identification from observational data requires consistency, sequential exchangeability at each treatment stage, and positivity for the relevant histories and actions. The conditional-mean models must also represent the quantities being estimated if the fitted rules are to have the intended interpretation. These conditions do not excuse bookkeeping errors: even a correctly specified model can be used incorrectly.

To see why, fix a history \(h_2\). Among all second-stage actions, \(Q_2(h_2,a_2)\) is the conditional expected outcome under that action, so its largest value is \(V_2(h_2)\). Now fix \(h_1\) and a candidate \(a_1\). Averaging the best achievable continuation value over the resulting \(H_2\) distribution gives \(Q_1(h_1,a_1)\). Thus the best first-stage action maximizes the expected outcome achievable after that action. More formally, for any alternative two-stage rule \((g_1,g_2)\), conditional on \(H_2\),

Taking conditional expectations given \(H_1,A_1\) shows that the expected outcome under any continuation rule after \(A_1=a_1\) is no greater than \(Q_1(H_1,a_1)\). Maximizing over \(a_1\) therefore gives a value at least as large as that of any rule. For finite action sets, a maximizing action can be selected at each history; ties may be resolved by any fixed rule. \(\square\)

Error 1: Regressing on the Outcome That Happened

The observed outcome reflects the actual second-stage treatment, \(A_2\). The stage-one pseudo-outcome instead represents the best predicted continuation available at that person’s \(H_2\). These differ whenever the treatment actually received does not maximize the stage-two Q-function. Regressing \(Y\) directly on \(H_1,A_1\) can consequently compare first-stage treatments under the observed distribution of later treatments, rather than under optimized continuation.

The algebra pinpoints what the direct-outcome substitution changes. Given \(H_2\), the observed outcome has conditional mean \(Q_2(H_2,A_2)\), whereas the optimized pseudo-outcome is \(V_2(H_2)\). Since \(V_2(H_2)\geq Q_2(H_2,A_2)\) pointwise, the values agree only when the received action is maximizing (or its Q-value ties for the maximum). Therefore, replacing \(V_2(H_2)\) with \(Y\) does not generally estimate \(Q_1\).

Error 2: Reversing the Treatment Contrast

For a binary treatment coded \(0\) and \(1\), define the treatment contrast at a fixed history as

If \(\Delta(h)>0\), action \(1\) has the larger predicted outcome; if \(\Delta(h)<0\), action \(0\) does. A coefficient is interpretable as this contrast only when the model’s coding and parameterization support that interpretation. In particular, changing the order of subtraction changes the sign, not the preferred action.

The decision rule itself follows from a simple identity: for two actions, \(\max\{Q(h,0),Q(h,1)\}=Q(h,1)\) when and only when \(Q(h,1)-Q(h,0)\geq 0\). If the contrast is defined in the opposite order, the corresponding inequality must also reverse. Treating a negative reverse contrast as evidence for action \(0\) while retaining the original contrast convention mixes the two definitions.

Error 3: Maximizing a Prediction From the Wrong Stage

A prediction must be maximized over the action that is being chosen at that stage. At stage two, compare \(Q_2(H_2,a_2)\) over possible \(a_2\). At stage one, compare \(Q_1(H_1,a_1)\) over possible \(a_1\), after averaging the optimized stage-two value over the \(H_2\) distribution induced by each first-stage action. A large stage-two prediction at one possible future history is not itself the value of a first-stage choice.

The general justification for the stage-one averaging step is the definition of conditional expectation: given \(H_1=h_1,A_1=a_1\), the continuation value \(V_2(H_2)\) varies with the future history. Its conditional mean, not its largest possible value, is the expected value associated with that first-stage choice. Maximizing a stage-two model over \(A_2\) is appropriate; maximizing over future histories or over an action that is not being chosen at the current stage is not.

Practical Checks, Alternatives, and Limits

A short audit of the computation can catch these errors before interpreting a rule:

Backward Q-learning is a plug-in, regression-based approach. Other approaches, including g-computation and inverse-probability-weighted methods, organize estimation differently, but they target a specified treatment strategy or value; they do not make an incorrectly defined contrast or decision stage meaningful. Q-learning’s attraction is that it builds a rule directly from conditional outcome models. Its limitations include dependence on the conditional-mean models and the availability of treatment choices across relevant histories. The diagnostics above address implementation logic; they do not establish model adequacy, exchangeability, or positivity.

A common mistake is to say, “The observed outcome is the best available response, so it should be used at the earlier stage.” That reasoning confuses a realized outcome under the received treatment with the expected value under an optimized continuation. The correction is to construct the stage-two fitted maximum first, then use that quantity as the stage-one pseudo-outcome. Similarly, a positive or negative coefficient should not be translated into a rule until its contrast direction and action coding have been checked.