Why a Pseudo-Outcome Is Needed
In backward Q-learning, the earlier-stage model needs an outcome that includes both reward already accrued and the value of the best available future treatment. At the later stage, that future value is not observed as a single outcome: it must be estimated from the later-stage Q-function and optimized over the available actions. The resulting quantity is called a pseudo-outcome. It is a constructed response for estimating an earlier-stage conditional mean, not a newly observed clinical outcome.
A detail matters when constructing it. An earlier reward may not be part of the history recorded before the later treatment. In that case, it is incorrect to replace its conditional mean by the reward itself inside an expectation conditional on the later history. The pseudo-outcome can still use the observed earlier reward; the derivation of its conditional mean must simply retain the appropriate conditional expectation.
Two-Stage Setup and Definition
Consider two treatment stages. Let \(H_1\) be the observed history before the first treatment \(A_1\), and let \(R_1\) be the reward accrued after the first treatment. Let \(H_2\) denote the observed history available before the second treatment \(A_2\). It contains the information needed to model the second-stage outcome and treatment assignment. It may include \(R_1\), but it need not do so. Let \(R_2\) be the second-stage reward. The total reward is \(R_1+R_2\).
For a history \(h_2\) and an available second-stage action \(a_2\), define the second-stage Q-function as the conditional mean of the second-stage reward under that action. Write \(V_2(h_2)\) for its largest value over the allowed actions. The action set may be finite, or it may be a set over which the maximum is attained.
In practice, \(Q_2\) is estimated, and the estimated continuation value is substituted into the pseudo-outcome. The definition above first describes the population target using the true \(Q_2\). The first-stage model is then fit to the constructed responses, using the first-stage history and treatment as predictors.
Why the Conditional Mean Is the Earlier-Stage Target
The key calculation is the law of iterated expectations. Since \(V_2(H_2)\) is a function of \(H_2\), it can be taken outside a conditional expectation given \(H_2\). The earlier reward \(R_1\), however, can be taken outside only if it is measurable given that history. In the general case its conditional mean must remain in the expression.
Proof. By definition, \(Z_1=R_1+V_2(H_2)\). Condition first on \(H_2,H_1,A_1\). Linearity of conditional expectation gives \(E[Z_1\mid H_2,H_1,A_1]=E[R_1\mid H_2,H_1,A_1]+E[V_2(H_2)\mid H_2,H_1,A_1]\). Because \(V_2(H_2)\) is measurable with respect to \(H_2\), the second term equals \(V_2(H_2)\). Taking the conditional expectation of both sides given \(H_1,A_1\), and applying the tower property to the left side, yields the stated identity. The first term on the right is a conditional mean, not necessarily \(R_1\) itself. \(\square\)
This statistical identity connects to a causal value under standard identification assumptions. Suppose consistency holds; treatment assignment at each stage is sequentially exchangeable given the history available at that stage; and each action used by the target regime has positive probability at histories where it is prescribed. Assume also that \(H_2\) is sufficient for identifying the second-stage conditional outcome under treatment. These conditions allow the observed conditional distributions to represent the corresponding intervention distributions.
For a fixed first action \(a_1\) and a second-stage rule \(d_2(H_2)\), the expected total reward under those interventions can be written by conditioning on the later history. If \(R_1\) is not determined by that history, its conditional mean must appear explicitly:
The inner conditional mean of \(R_1\) is essential unless \(R_1\) is known from \(H_2\). Under consistency and sequential exchangeability, the intervention distribution of the history and earlier reward agrees with the corresponding observed conditional distribution given \(H_1=h_1,A_1=a_1\); the second-stage conditional outcome under \(d_2\) is represented by \(Q_2(H_2,d_2(H_2))\). Choosing an action that maximizes \(Q_2\) at each \(H_2\) therefore replaces that term by \(V_2(H_2)\). The resulting value is the conditional mean of the pseudo-outcome.
Worked Examples
Worked Example: An Earlier Reward Not Recorded in the Later History
Suppose \(H_1\) and \(A_1\) are fixed, and the later history \(H_2\) has two possible values. Conditional on the chosen first treatment, \(P(H_2=0)=0.4\) and \(P(H_2=1)=0.6\). The earlier reward is not determined by \(H_2\), but its conditional means are \(E[R_1\mid H_2=0,H_1,A_1]=2\) and \(E[R_1\mid H_2=1,H_1,A_1]=1\).
At \(H_2=0\), suppose the two second-stage Q-values are 3 and 5. Thus \(V_2(0)=\max(3,5)=5\). At \(H_2=1\), suppose they are 4 and 2, so \(V_2(1)=4\). The conditional mean pseudo-outcome is therefore
The same result follows by separating the two contributions: \(E[R_1\mid H_1,A_1]=0.4(2)+0.6(1)=1.4\), while \(E[V_2(H_2)\mid H_1,A_1]=0.4(5)+0.6(4)=4.4\). Their sum is \(1.4+4.4=5.8\). It would be unjustified to write \(E[R_1\mid H_2]=R_1\): the reward varies within a later-history stratum. The observed pseudo-outcome for a particular person uses that person’s \(R_1\), whereas this calculation of its conditional mean uses the appropriate conditional means.
Worked Example: Constructing Individual Pseudo-Outcomes
Consider two people with observed earlier rewards \(R_1=1\) and \(R_1=4\). Their later histories are \(H_2=x\) and \(H_2=y\), respectively. An estimated second-stage model gives \( \hat Q_2(x,0)=6\), \( \hat Q_2(x,1)=4\), \( \hat Q_2(y,0)=2\), and \( \hat Q_2(y,1)=5\).
For the first person, the estimated optimized continuation value is \(\max(6,4)=6\), so the pseudo-outcome is \(1+6=7\). For the second person, it is \(\max(2,5)=5\), so the pseudo-outcome is \(4+5=9\). The responses supplied to the first-stage regression are thus 7 and 9.
The observed second-stage treatment does not determine which Q-value is used in this construction. For example, if the second person happened to receive action 0, the pseudo-outcome still uses 5, the maximum of the two fitted values, rather than 2, the fitted value for the action actually received. The target is the value under an optimized continuation, not the person’s observed second-stage reward.
Worked Example: Comparing First-Stage Choices
Suppose a baseline history \(h_1\) has two possible first-stage actions. The conditional distribution of \(H_2\) is the same for both actions: \(P(H_2=u)=0.5\) and \(P(H_2=v)=0.5\). For \(A_1=0\), the conditional means of \(R_1\) at \(u,v\) are 1 and 2; for \(A_1=1\), they are 0 and 1. The optimized continuation values are \(V_2(u)=6\) and \(V_2(v)=4\).
For \(A_1=0\), the conditional mean pseudo-outcome is \(0.5(1+6)+0.5(2+4)=3.5+3=6.5\). For \(A_1=1\), it is \(0.5(0+6)+0.5(1+4)=3+2.5=5.5\). Thus the first-stage conditional mean is larger for action 0 by \(6.5-5.5=1\). The comparison includes both immediate reward and future optimized value; comparing only the conditional means of \(R_1\), which are 1.5 and 0.5, would happen to give the same ranking here but would not generally give the correct value difference if the later-history distributions or continuation values differed by first action.
Constructing and Checking the Response
Use the observed later-stage outcome and the pre-treatment history and action to estimate \(Q_2(H_2,A_2)\), under the assumptions and support required for that model.
For each observed \(H_2\), calculate \(\hat V_2(H_2)=\max_{a_2\in\mathcal A_2(H_2)}\hat Q_2(H_2,a_2)\). Respect any action constraints in \(\mathcal A_2(H_2)\).
Set \(\hat Z_1=R_1+\hat V_2(H_2)\). Do not replace \(R_1\) by a predicted mean in each individual response unless that is an explicitly different estimation procedure.
Regress \(\hat Z_1\) on the first-stage history and treatment to estimate \(Q_1(H_1,A_1)\). The conditional mean, rather than each individual pseudo-outcome, is the Q-function target.
Several diagnostics follow directly from this construction. Check that the maximization uses the correct action set and that each fitted value is evaluated at the same person’s later history. Compare the distribution of \(\hat V_2(H_2)\), \(R_1\), and \(\hat Z_1\) for implausible values or coding errors. Inspect whether some actions have little or no support in relevant histories; a fitted maximum can otherwise select an action whose value is mostly extrapolated. Also verify that the history used in the later-stage model contains the variables needed to support its conditional outcome interpretation. A pseudo-outcome calculation cannot repair a deficient history or unsupported action.
Common Mistakes, Alternatives, and Limitations
- Replacing an unmeasured-by-history reward with itself inside a conditional expectation. The incorrect step is \(E[R_1+V_2(H_2)\mid H_2]=R_1+V_2(H_2)\) when \(R_1\) is not measurable given \(H_2\). The correct expression is \(E[R_1\mid H_2]+V_2(H_2)\). The individual pseudo-outcome still contains observed \(R_1\); the conditional-mean derivation must not treat it as known from \(H_2\).
- Using the observed later treatment rather than optimizing. Substituting \(Q_2(H_2,A_2)\) targets the value of the observed later treatment, not the best continuation. The optimized value uses \(\max_{a_2}Q_2(H_2,a_2)\).
- Confusing a pseudo-outcome with a directly observed total outcome. \(R_1+\hat V_2(H_2)\) combines an observed reward with a model-based estimate. Its uncertainty and possible model error are not visible by treating it as an ordinary measured response.
An alternative is direct g-computation: estimate conditional outcome distributions or means and average them under specified treatment choices, carrying the calculation through the histories. Backward pseudo-outcome regression packages the later optimized value into a response and estimates the earlier conditional mean with a regression model. These are related ways to implement sequential conditional-mean calculations, but they organize modeling differently. Either approach relies on valid histories, treatment support, and adequate estimation of the relevant conditional means. In finite samples, taking a maximum of noisy fitted values can also make the constructed future value unstable, especially where treatment data are sparse. The pseudo-outcome provides a clear target; it does not guarantee that its estimated version is accurate.
Check Your Understanding
Answer each question, then compare your work with the solutions below.
- If \(R_1\) is not measurable with respect to \(H_2\), what term replaces \(R_1\) when deriving the pseudo-outcome’s conditional mean given \(H_2\)?
- At a history \(h_2\), the two second-stage Q-values are 3 and 8. What continuation value enters the pseudo-outcome, and why?
- Suppose \(P(H_2=u)=0.25\), \(P(H_2=v)=0.75\), the conditional means of \(R_1\) are 4 and 2, and the optimized continuation values are 6 and 3. Calculate the conditional mean pseudo-outcome.
- Explain why an observed second-stage action with fitted Q-value 2 need not contribute 2 to that person’s pseudo-outcome if another available action has fitted Q-value 5.
Solutions. (1) Use \(E[R_1\mid H_2,H_1,A_1]\), with conditioning variables included as appropriate; do not substitute \(R_1\) itself unless it is determined by the conditioning history. (2) The continuation value is \(\max(3,8)=8\), because the pseudo-outcome represents optimized future value. (3) The answer is \(0.25(4+6)+0.75(2+3)=0.25(10)+0.75(5)=2.5+3.75=6.25\). (4) The pseudo-outcome adds the maximum fitted value, 5, rather than the value for the action actually received; it targets optimized continuation rather than observed-treatment continuation.