Worked Example With Two Binary Decisions

From Terminal Predictions to Two Decisions

In a two-stage treatment problem, the best second treatment can depend on what happened between decisions. The best first treatment must account for that later opportunity: its value is the expected outcome after choosing the best available second treatment at each subsequent history. This tutorial carries out that calculation for binary treatments and covariates, beginning with fitted terminal Q-values and ending with both treatment rules.

Let \(X\) be a baseline covariate, \(A_1\in\{0,1\}\) the first treatment, \(L\) an intermediate covariate measured before the second decision, \(A_2\in\{0,1\}\) the second treatment, and \(Y\) the final outcome, with larger values preferred. The stage-two history is \(H_2=(X,A_1,L)\). The data must contain support for the treatment choices and histories to which the fitted rules will be applied.

These conditional means have a causal interpretation under consistency, sequential exchangeability at both treatment decisions, and positivity for treatment options in the histories being evaluated. In particular, consistency links observed outcomes to the potential outcomes under received treatments; sequential exchangeability rules out unmeasured confounding of each treatment decision given its measured history; and positivity requires that the relevant actions have positive probability at those histories. For estimation by Q-learning, the conditional-mean models must also be correctly specified, or sufficiently flexible to estimate the relevant means. With binary covariates, saturated cell means provide a transparent illustration.

Worked Two-Stage Calculation

Suppose \(X\) and \(L\) are binary. A saturated terminal model is fitted by allowing a separate intercept and treatment contrast for every \((X,A_1,L)\) history. Equivalently, it gives a separate predicted mean for each value of \(A_2\) in each history. Assume the fitted predictions are as follows.

To fit the stage-one Q-function, average the pseudo-outcomes over the distribution of \(L\) conditional on each \((X,A_1)\). Suppose that distribution is estimated, or known for this illustration, to satisfy \(P(L=1\mid X=0,A_1=0)=0.25\), \(P(L=1\mid X=0,A_1=1)=0.75\), and \(P(L=1\mid X=1,A_1=a_1)=0.5\) for either \(a_1\).

Why the Backward Calculation Works

Proof. Fix a baseline value \(x\) and any first-stage choice \(a_1\). For each possible \(l\), every second-stage action has conditional expected outcome \(Q_2(x,a_1,l,a_2)\). Hence no second-stage rule can have conditional expected outcome greater than \(V_2(x,a_1,l)=\max_{a_2}Q_2(x,a_1,l,a_2)\) at that history. Averaging over \(L\) gives an upper bound of \(E[V_2(x,a_1,L)\mid X=x,A_1=a_1]=Q_1(x,a_1)\) for the expected outcome after choosing \(a_1\). A rule selecting a maximizing second-stage action at every \(l\) attains this bound, since the history sets are finite and a maximum exists. Finally, among the two first-stage choices, selecting one that maximizes \(Q_1(x,a_1)\) attains the greatest of these upper bounds. This establishes optimality at every \(x\), and averaging over \(X\) establishes optimality in the population. The identification assumptions ensure that these conditional means correspond to the expected potential outcomes under the specified treatment choices. \(\square\)

In a fitted analysis, the same steps produce a plug-in rule: replace the population Q-functions by their estimates and maximize those estimates. The theorem explains the target of this calculation; it does not by itself guarantee that a fitted rule is accurate in a finite sample.

The Second-Stage Rule Uses the Full History

This distinction is important when describing a regime. A rule defined for every possible history is not the same as its behavior on histories that occur after the rule’s own first-stage choice. Reporting only the behavior along the regime can be useful, but it must not be mistaken for a rule that ignores \(X\) and \(A_1\).

Verification, Alternatives, and Failure Modes

One verification is to compute the value of the derived rule from the stage-one fitted values. If \(P(X=0)=0.4\) and \(P(X=1)=0.6\), its fitted value is \(0.4(14.75)+0.6(15.5)=5.9+9.3=15.2\). For comparison, a rule that always selects \(A_1=1\), while still using the maximizing second-stage action for each history, has fitted value \(0.4(14.75)+0.6(13)=5.9+7.8=13.7\). The difference is \(15.2-13.7=1.5\) fitted outcome units. This is a model-based comparison, not evidence of a causal improvement unless the identification and model assumptions are credible.

For binary histories, direct enumeration of complete regimes is a useful check: calculate the value of each candidate combination of first- and second-stage decisions and verify that none exceeds the backward maximum. Enumeration becomes cumbersome as the number of histories grows; backward Q-learning exploits the sequential structure. Other approaches, such as inverse-probability-weighted comparisons, target regime values differently and can provide a useful contrast, but do not remove the need for adequate treatment support and careful causal assumptions.

• Check the maximization row by row. For every history, compare both terminal predictions. A verbal rule based only on \(L\) can contradict rows where \(X\) or \(A_1\) changes the treatment contrast.
• Check the distribution used for averaging. The stage-one calculation weights each pseudo-outcome by \(P(L\mid X,A_1)\), not by an overall marginal distribution of \(L\). Using the wrong weights changes the Q-values.
• Check support and model form. Sparse histories can make cell means unstable or undefined. A terminal model that omits important treatment interactions can also conceal history-dependent contrasts. Inspect cell counts, fitted contrasts, and sensitivity to plausible model specifications.
• Do not treat estimated ties as decisive evidence. If two fitted predictions are equal, either action maximizes the fitted value at that history. If they are merely close, sampling variability can easily reverse their ordering; the earlier material on ties and nonregular inference explains why such decisions warrant caution.

The example uses finite binary variables and fitted means to make every step visible. In practice, continuous covariates require modeling conditional means over a larger history space, and estimated treatment rules can be sensitive to extrapolation, model misspecification, and overfitting. Correct backward arithmetic cannot repair a lack of support or a poorly estimated terminal model.