Deriving Sequential Outcome Regressions Under a Fixed Regime

Why the Regressions Run Backward

A treatment regime specifies which treatment to assign at each decision, possibly as a function of the history available at that decision. To find the mean outcome under that regime, it is not enough to predict the outcome conditional on the treatments and covariates actually observed. The future covariates must also be averaged over the distributions they would have under the regime. The backward sequence of conditional outcome regressions performs these operations one decision at a time.

The introductory material on identifying a counterfactual mean by backward recursion establishes the basic g-computation idea. Here the focus is the precise construction of the regressions: which treatment value enters each conditional mean, which covariate distribution is used, and what conditions make the calculations meaningful.

Definition: Suppose there are $K$ treatment decisions, at times $t=0,\ldots,K-1$. Let $L_0$ be baseline covariates, $A_t$ the treatment at time $t$, $L_{t+1}$ the covariates measured after that treatment, and $Y$ the terminal outcome measured after $L_K$. The history just before treatment $A_t$ is $H_t=(L_0,A_0,L_1,\ldots,A_{t-1},L_t)$, with $H_0=L_0$. A regime $d=(d_0,\ldots,d_{K-1})$ assigns $A_t=d_t(H_t)$. Define $Q_K(h_K)=E[Y\mid H_K=h_K]$, and, recursively for $t=K-1,\ldots,0$, define $$ Q_t(h_t)=\int Q_{t+1}\bigl(h_t,d_t(h_t),l_{t+1}\bigr)\, P\bigl(dl_{t+1}\mid H_t=h_t,A_t=d_t(h_t)\bigr). $$ The functions $Q_t$ are the sequential outcome regressions under regime $d$. Each recursion assigns treatment its regime value and averages over the next covariate distribution under that assigned treatment.

The notation $Q_{t+1}(h_t,d_t(h_t),l_{t+1})$ means the next regression evaluated at the history extended by the assigned treatment and the next covariate. If $L_{t+1}$ is discrete, the integral is a sum. If it is continuous, the integral uses its conditional density when one exists. In either case, the averaging distribution is conditional on the treatment value specified by the regime, not on an unspecified or future observed treatment.

Conditions and the Regime Recursion

The observed-data regressions identify the regime-specific recursion under assumptions about how treatments are assigned and how observed data relate to counterfactual data. Write $Y^d$ for the outcome that would be observed under regime $d$, and $H_t^d$ for the history that would arise before decision $t$ under that regime.

Assumptions for the recursion:

Consistency and no interference: for a person whose observed treatment history follows the regime through the relevant time, the observed covariates and outcome equal their corresponding counterfactual values under that history; one person's assigned treatment does not change another person's outcome.
Sequential exchangeability: at each decision, conditional on the observed history $H_t$, treatment assignment is independent of the counterfactual next covariate and subsequent counterfactual covariates and outcome under the treatment continuations being considered.
Regime-relevant positivity: for histories occurring with positive probability under the regime, the conditional probability of the regime-assigned treatment is positive: $P(A_t=d_t(H_t)\mid H_t)>0$.
Integrability: $Y^d$ is integrable, and the conditional expectations and integrals used in the recursion are well-defined. Bounded outcomes are a sufficient integrability condition. Integrability of the observed $Y$ alone is not enough.

Theorem: Under consistency, sequential exchangeability, regime-relevant positivity, and the stated integrability conditions, the counterfactual mean is $$ E[Y^d]=E[Q_0(L_0)]. $$ Moreover, at each time $t$, $Q_t(H_t^d)$ is a version of the conditional mean of $Y^d$ given the regime history $H_t^d$.

Derivation. Start at the end of follow-up. Given the history $H_K^d$, sequential exchangeability and consistency identify the conditional mean of the counterfactual outcome with the observed conditional mean $E[Y\mid H_K]$. This is $Q_K(H_K^d)$. Now suppose that at time $t+1$, $Q_{t+1}(H_{t+1}^d)$ represents the conditional mean of $Y^d$ given $H_{t+1}^d$. By the law of iterated expectations, $$ E[Y^d\mid H_t^d] = E\!\left[E[Y^d\mid H_{t+1}^d]\mid H_t^d\right] = E[Q_{t+1}(H_{t+1}^d)\mid H_t^d]. $$ Under regime $d$, the next treatment is $d_t(H_t^d)$. Sequential exchangeability and consistency identify the conditional distribution of $L_{t+1}^d$, given $H_t^d=h_t$, with the observed conditional distribution of $L_{t+1}$ given $H_t=h_t,A_t=d_t(h_t)$. Substituting that distribution into the last conditional expectation gives exactly the defining integral for $Q_t(h_t)$. This proves the conditional-mean claim backward from $K$ to $0$. Finally, taking expectations of the claim at $t=0$ gives $E[Y^d]=E[Q_0(L_0)]$. Integrability permits these conditional expectations and the required iterated expectations. $\square$

This derivation also shows why future covariates are integrated out rather than held fixed at their observed values. At each step, the next regression depends on the next covariate; the current regression averages that dependence over the covariate distribution produced by the assigned treatment.

Worked Calculations

Worked Example: One Decision and a Binary Follow-Up Covariate

Suppose there is a baseline covariate $X$, one treatment $A_0$, a binary follow-up covariate $Z=L_1$, and an outcome $Y$. Consider the regime $d_0(X)=1$. In a population with $X=x$, suppose $P(Z=1\mid X=x,A_0=1)=0.75$, and the fitted terminal conditional means are $E[Y\mid X=x,A_0=1,Z=0]=2$ and $E[Y\mid X=x,A_0=1,Z=1]=10$.

The terminal regression is $Q_1(x,z)=E[Y\mid X=x,A_0=1,Z=z]$. The regime-specific regression at baseline sums over both possible values of $Z$:

\begin{aligned} Q_0(x) &=Q_1(x,0)P(Z=0\mid X=x,A_0=1) +Q_1(x,1)P(Z=1\mid X=x,A_0=1)\\ &=2(1-0.75)+10(0.75)\\ &=0.5+7.5=8. \end{aligned}

Thus the regime-specific conditional mean for this baseline stratum is 8. The calculation uses the distribution of $Z$ under $A_0=1$; using the distribution of $Z$ under another treatment value would answer a different question.

Worked Example: Two Decisions with a Dynamic Second Treatment

Let $A_0$ be the first treatment, $L_1$ a binary intermediate covariate, $A_1$ the second treatment, $L_2$ a binary later covariate, and $Y$ the outcome. Use the regime $A_0=1$, $A_1=1-L_1$. Assume the conditional probabilities and terminal means below apply to the histories generated by this regime:

$P(L_1=1\mid A_0=1)=0.4$.
When $L_1=0$, the assigned $A_1=1$, and $P(L_2=1\mid A_0=1,L_1=0,A_1=1)=0.2$. The means of $Y$ at $L_2=0,1$ are 4 and 14.
When $L_1=1$, the assigned $A_1=0$, and $P(L_2=1\mid A_0=1,L_1=1,A_1=0)=0.7$. The means of $Y$ at $L_2=0,1$ are 3 and 8.

First integrate out $L_2$. For $L_1=0$, the regression after the first decision is $Q_1(0)=4(0.8)+14(0.2)=3.2+2.8=6$. For $L_1=1$, it is $Q_1(1)=3(0.3)+8(0.7)=0.9+5.6=6.5$.

Next average over $L_1$ under the first assigned treatment: $Q_0=Q_1(0)P(L_1=0\mid A_0=1)+Q_1(1)P(L_1=1\mid A_0=1)$. Therefore,

$$ Q_0=6(0.6)+6.5(0.4)=3.6+2.6=6.2. $$

The second treatment is not fixed at one value for everyone: it is assigned according to $1-L_1$. Each branch uses the outcome means and the $L_2$ distribution corresponding to that branch's assigned treatment.

Worked Example: Integrating a Continuous Covariate

Suppose there is one treatment decision, the regime assigns $A_0=1$, and the follow-up covariate $L_1$ is uniformly distributed on $[0,1]$ conditional on that assignment. Let the terminal regression be $Q_1(l)=1+l+2l^2$. The baseline regression is the integral of this function over the regime-specific distribution:

\begin{aligned} Q_0 &=\int_0^1(1+l+2l^2)\,dl\\ &=\left[l+\frac{l^2}{2}+\frac{2l^3}{3}\right]_0^1\\ &=1+\frac12+\frac23=\frac{13}{6}\approx2.167. \end{aligned}

Substituting the mean of $L_1$ into the regression would give $1+0.5+2(0.5)^2=2$, not $13/6$. The regression is nonlinear, so evaluating it at the mean covariate does not generally equal its average over the covariate distribution.

Diagnostics, Alternatives, and Limitations

A sequential regression calculation can be numerically correct but scientifically unreliable if its conditional models or assumptions are poor. Check that each regression is evaluated at the treatment value prescribed by the regime, and that each integration distribution conditions on that same value. For continuous covariates, inspect whether the fitted regression behaves plausibly over the covariate values receiving appreciable probability under the regime. For discrete covariates, confirm that all relevant categories have been included and that conditional probabilities sum to one. Sparse histories, extreme fitted probabilities, or a need to extrapolate into unsupported treatment-covariate combinations signal potential positivity problems.

The most common conceptual error is to leave a future treatment at its observed value. For example, predicting $Y$ conditional on observed $A_1$ and then averaging over the observed $A_1$ distribution does not implement a regime that sets $A_1=d_1(H_1)$. The correction is to evaluate the next regression at $A_1=d_1(H_1)$, then average over the next covariate distribution conditional on that assigned value. A related error is to plug a mean covariate into a nonlinear regression; the continuous example shows why the required operation is integration, not substitution.

These regressions are the outcome-regression or g-computation route to a regime-specific mean. Inverse-probability weighting instead reweights observed outcomes by treatment probabilities; it does not construct the same sequence of conditional means. Ordinary regression that adjusts for observed treatment history can describe conditional associations, but without the regime assignments and sequential integration it is not, by itself, the regime-specific mean. The regression approach depends on adequate outcome and covariate-distribution estimation, while weighting depends strongly on treatment probabilities. Neither method removes the need for consistency, exchangeability, and positivity.

A further limitation is that observed-data integrability does not guarantee a finite regime-specific mean. The recursion and theorem require $Y^d$ to be integrable, or another condition strong enough to ensure that the target and the conditional expectations exist. For a bounded outcome this issue is avoided. For an unbounded outcome, the analyst must not infer finiteness merely from a finite observed-data mean.

Check Your Understanding

Answer each question, then check the short solutions below.

In the recursion for $Q_t$, which treatment value is used inside $Q_{t+1}$, and which distribution is used to average over $L_{t+1}$?
Suppose $L_{t+1}$ has three possible values with probabilities $0.2,0.5,0.3$ under the assigned treatment, and the corresponding next-regression values are $1,4,7$. Calculate $Q_t$.
Why can a nonlinear terminal regression not generally be evaluated at the conditional mean of a continuous covariate in place of integration?
Name two assumptions, apart from correct model specification, needed to identify the regime-specific mean from observed data.

Solutions: (1) Use $A_t=d_t(H_t)$ inside the next regression, and average with the conditional distribution of $L_{t+1}$ given $H_t$ and that assigned treatment. (2) $Q_t=1(0.2)+4(0.5)+7(0.3)=0.2+2+2.1=4.3$. (3) In general, the average of a nonlinear function is not the function evaluated at the average; the continuous example gave $13/6$ by integration but 2 by substitution. (4) Examples include consistency, sequential exchangeability, treatment positivity, and integrability of the regime-specific outcome.

Deriving Sequential Outcome Regressions Under a Fixed Regime

What You'll Learn