Terminal-Stage Q-Model Construction

Why the Terminal Q-Model Is the Starting Point

In a finite treatment sequence, the terminal decision is the last opportunity to choose an action. There is no later treatment decision whose value must be folded into the outcome. The key task at this stage is therefore direct: estimate the expected final outcome for each available terminal action at a given pre-treatment history, then choose the action with the highest expected outcome. The terminal-stage Q-model supplies those conditional means.

This is the terminal component of the backward reasoning used in two-stage Q-learning. Here the emphasis is narrower: defining the terminal Q-function correctly, constructing a model for it, and interpreting its fitted values as action comparisons. Let higher values of the outcome represent better outcomes; if the scientific goal is to minimize a loss, the corresponding decision is a minimum instead.

Definition: Let $H_K$ be the observed history immediately before the terminal treatment $A_K$, and let $Y$ be the final outcome. For an available action $a$, the terminal-stage Q-function is $$ Q_K(h,a)=E[Y\mid H_K=h,A_K=a]. $$ A terminal Q-model is a statistical model $m(h,a;\beta)$ for this conditional mean. Its fitted version, $m(h,a;\hat{\beta})$, estimates the expected final outcome at history $h$ if action $a$ is taken.

The history $H_K$ contains the measured covariates and earlier treatment information needed to describe the decision point, but only information observed before $A_K$. Including post-treatment variables in $H_K$ changes the conditioning target and can introduce bias. The outcome $Y$ is the final outcome relevant to the decision; it may be a terminal measurement or a prespecified summary of outcomes accumulated over the treatment sequence.

When the Conditional Mean Has a Causal Meaning

The conditional mean in the definition is an observed-data quantity. To interpret it as the expected outcome under an intervention that sets the terminal action to $a$, additional assumptions are needed. Write $Y^a$ for the potential final outcome under terminal action $a$, among individuals with their observed pre-terminal histories.

Identification conditions: For each action being compared and each target history $h$: (1) consistency, so $Y=Y^a$ when $A_K=a$; (2) terminal-stage conditional exchangeability, $Y^a\perp A_K\mid H_K$; and (3) positivity, so action $a$ has positive probability (or positive density at the relevant action) given $H_K=h$. Under these conditions, $$ E[Y^a\mid H_K=h]=E[Y\mid H_K=h,A_K=a]=Q_K(h,a). $$

Consistency requires the observed treatment to correspond to the intervention being described; it also presumes that the action is defined sufficiently precisely. Exchangeability says that, after conditioning on recorded history, treatment assignment does not retain information about the potential outcome. Positivity ensures that the data contain treated individuals at the histories where the comparison is to be made. These assumptions are not guaranteed by fitting a regression. They are requirements for interpreting its conditional means causally.

Constructing and Using the Model

Choose a response model appropriate for the outcome and for the intended conditional mean. For a continuous outcome, a linear regression can model $E[Y\mid H_K,A_K]$. For a binary outcome, a logistic regression can model the conditional probability $P(Y=1\mid H_K,A_K)$; the Q-function is then the predicted probability, not the linear predictor on the log-odds scale. More flexible regression methods are possible, but they still need to estimate the same conditional mean.

After fitting the model, evaluate it at the same history $h$ under each admissible terminal action. The comparison is among predicted outcomes, not among treatment coefficients in isolation. With a finite action set $\mathcal A$, the model-based decision is

\hat d_K(h)=\arg\max_{a\in\mathcal A}m(h,a;\hat{\beta}).

If the fitted means tie, a prespecified tie-breaking rule is needed to return one action. A tie does not establish that the actions are clinically identical; it may reflect limited precision, model restrictions, or genuinely equal conditional means.

Worked Example: Evaluating a Linear Terminal Q-Model

Suppose the fitted conditional mean for a continuous final outcome is $m(h,a;\hat{\beta})=10+0.4h+2a$, where $h$ is a measured history score and the available actions are $a=0$ and $a=1$. At history $h=3$, calculate both fitted means and select the action.

For action $a=0$, $m(3,0;\hat{\beta})=10+0.4(3)+2(0)=10+1.2=11.2$.

For action $a=1$, $m(3,1;\hat{\beta})=10+0.4(3)+2(1)=10+1.2+2=13.2$.

Since $13.2>11.2$, the model selects $a=1$ at $h=3$. This calculation uses fitted conditional means at the specified history. The treatment coefficient alone is not the decision rule in general; the comparison must be made using the model's predictions for the actions under consideration.

Two Results That Explain the Construction

The first result says why selecting the largest terminal Q-value is the right decision under the stated assumptions. Suppose $\mathcal A$ is a finite set and the conditional means $Q_K(h,a)$ are defined for every $a\in\mathcal A$ at the target history. Then a maximizing action exists. A fixed tie-breaking rule makes the choice well-defined.

Theorem: Under consistency, terminal-stage conditional exchangeability, and positivity, a rule $d_K^*(h)$ that maximizes $Q_K(h,a)$ over $a\in\mathcal A$ maximizes the conditional expected potential outcome among the available terminal actions: $$ E[Y^{d_K^*(h)}\mid H_K=h]\geq E[Y^a\mid H_K=h]\quad\text{for every }a\in\mathcal A. $$

Proof. By the identification conditions, for every $a\in\mathcal A$, $E[Y^a\mid H_K=h]=Q_K(h,a)$. By definition, $d_K^*(h)$ is a maximizing action, so $Q_K(h,d_K^*(h))\geq Q_K(h,a)$ for every $a$. Substituting the identified conditional means gives the claimed inequality. $\square$

The next result explains why conditional-mean regression is the appropriate prediction target when squared error is used. It applies to any random variable $Y$ with finite second moment and any predictor $g(H_K,A_K)$ with finite second moment.

Result: The conditional mean $Q_K(H_K,A_K)=E[Y\mid H_K,A_K]$ minimizes expected squared prediction error. For any such $g$, $$ E[(Y-g(H_K,A_K))^2] = E[(Y-Q_K(H_K,A_K))^2] + E[(Q_K(H_K,A_K)-g(H_K,A_K))^2]. $$

Derivation. Write $Q=Q_K(H_K,A_K)$. Expanding the square gives $E[(Y-g)^2]=E[(Y-Q+Q-g)^2]$. The cross term is zero because $E[(Y-Q)(Q-g)]=E[E[(Y-Q)(Q-g)\mid H_K,A_K]]$, and $Q-g$ is a function of $(H_K,A_K)$. Therefore $E[(Y-Q)(Q-g)\mid H_K,A_K]=(Q-g)E[Y-Q\mid H_K,A_K]=0$. The remaining terms are $E[(Y-Q)^2]$ and $E[(Q-g)^2]$, proving the identity. The second term is nonnegative, so no predictor has smaller squared-error risk than the conditional mean. $\square$

This result identifies the population target; it does not prove that a particular fitted regression is accurate. Estimation error, model restrictions, limited data, and lack of treatment support can all make fitted conditional means poor approximations to $Q_K$.

Worked Example: History-Stratified Conditional Means

For illustration, suppose history is grouped into low and high strata, and observed outcomes are available for both actions in each stratum. Estimate each conditional mean by its within-cell sample average.

In the low-history stratum, the outcomes under $a=0$ are $4,6,5$, so $\hat Q_K(\text{low},0)=(4+6+5)/3=15/3=5$. Under $a=1$, the outcomes are $7,8,6$, so $\hat Q_K(\text{low},1)=(7+8+6)/3=21/3=7$. The fitted comparison selects $a=1$ for the low stratum.

In the high-history stratum, the outcomes under $a=0$ are $8,10,9$, so $\hat Q_K(\text{high},0)=(8+10+9)/3=27/3=9$. Under $a=1$, the outcomes are $7,9,8$, so $\hat Q_K(\text{high},1)=(7+9+8)/3=24/3=8$. The fitted comparison selects $a=0$ for the high stratum.

Thus the cell-mean model yields different estimated actions across the two history strata. These are estimates, not proof of causal benefit: interpreting them causally still requires exchangeability, consistency, and positivity, and small cell counts make the estimates unstable.

Worked Example: A Binary Outcome and Logistic Predictions

Suppose a fitted logistic model for a binary outcome is $\operatorname{logit}\{m(h,a;\hat{\beta})\}=-2+0.4h+0.9a$, where $m$ denotes the predicted probability that $Y=1$. At history $h=2$, calculate the predicted probability for each action and decide which the model favors.

For $a=0$, the linear predictor is $-2+0.4(2)+0.9(0)=-1.2$. Hence $m(2,0;\hat{\beta})=1/(1+\exp(1.2))\approx0.2315$.

For $a=1$, the linear predictor is $-2+0.4(2)+0.9(1)=-0.3$. Hence $m(2,1;\hat{\beta})=1/(1+\exp(0.3))\approx0.4256$.

The model favors $a=1$, because $0.4256>0.2315$. The comparison is on the probability scale, since that is the conditional mean of the binary outcome. The logistic link is strictly increasing, so in this particular model the ordering also agrees with the ordering of the two linear predictors.

Diagnostics, Alternatives, and Limitations

A fitted Q-model can produce a clear action recommendation even when the underlying evidence is weak. Check its predictions and data support before interpreting that recommendation.

Check action support at relevant histories. If one action is rarely or never observed in a history region, its fitted mean may depend on extrapolation. With no support, positivity fails and the observed data do not identify that action's conditional mean there.
Assess conditional-mean fit. Compare predicted and observed outcomes across treatment and history groups, using residual patterns or calibration summaries appropriate to the outcome. Systematic residual patterns indicate that the model may not capture important features of the conditional mean.
Check stability of the selected action. If small changes to the fitted model or sample change the maximizing action, the recommendation is uncertain even if the fitted values look plausible. Report uncertainty rather than treating the selected action as known.
Check the outcome direction and scale. Confirm that larger modeled values really mean better outcomes. For binary outcomes, compare predicted probabilities rather than interpreting coefficients as changes in probability.

History-stratified means make few modeling assumptions within the observed cells, but can be very imprecise when cells are small and cannot directly provide predictions for histories not represented in the table. Regression pools information across histories and can provide smoother predictions, but relies on its functional form. The earlier material on inverse-probability weighting provides a different route to regime comparisons: weighting emphasizes treatment assignment probabilities, whereas a terminal Q-model directly predicts outcomes conditional on treatment and history. Neither approach repairs a failure of the assumptions needed for causal identification.

The terminal-stage construction also has limits. It does not by itself account for uncertainty in earlier-stage value estimates, solve confounding from unmeasured variables, or guarantee that a fitted action rule has good performance in a new population. Its decision is optimal only relative to the available actions, the specified outcome, the target histories, and the validity of the conditional means used in the comparison.

Common pitfall: Choosing an action from the sign or size of a treatment coefficient without evaluating the fitted conditional means at the history of interest. The correction is to compute $m(h,a;\hat{\beta})$ for each candidate action at the same $h$, then compare those values. A coefficient can summarize a model parameterization; the predictions are the quantities that define the model-based action.

Check Your Understanding

Answer each question, then compare with the solutions below.

State the terminal Q-function and name the three identification conditions that permit its causal interpretation.
A fitted model gives $m(h,0)=6.4$ and $m(h,1)=5.9$. Which action does the model select at $h$, assuming larger outcomes are better?
For a binary outcome, a logistic model predicts linear predictors $-0.8$ for action 0 and $-0.2$ for action 1 at the same history. Which action has the larger predicted probability?
What does it mean if action 1 is never observed among people with a particular history $h$?

Solutions. (1) $Q_K(h,a)=E[Y\mid H_K=h,A_K=a]$. Its causal interpretation requires consistency, terminal-stage conditional exchangeability, and positivity. (2) Select action 0, since $6.4>5.9$. (3) Select action 1: the logistic function is strictly increasing, and $-0.2>-0.8$, so its predicted probability is larger. (4) Positivity fails at that history for action 1; its conditional mean is not identified from the observed data there, and a model prediction would rely on extrapolation.