Categorical Treatment Regimes and Clever Covariates

Regimes for Multilevel Treatments

At each decision time, a categorical treatment may have three or more possible levels. A treatment regime specifies which level to assign, possibly as a function of the history available at that decision. For example, a clinical rule might assign one treatment when a biomarker is high and another when it is low. In longitudinal targeted maximum likelihood estimation, that rule must be encoded precisely: the observed treatment must match the regime at every decision time, and the probability of each matching level must be used in the corresponding denominator.

Let decisions occur at times $t=0,\ldots,T$. Let $H_t$ denote the measured history immediately before treatment $A_t$, and suppose $A_t$ takes values in a finite set $\mathcal A_t$. Write $g_t(a\mid h)=P(A_t=a\mid H_t=h)$. A deterministic regime $d=(d_0,\ldots,d_T)$ consists of functions $d_t$ satisfying $d_t(h)\in\mathcal A_t$. Thus, the regime selects exactly one treatment level at each history.

Definition: The regime-match indicator through time $t$ is $I_t^d=\prod_{s=0}^{t} I\{A_s=d_s(H_s)\}$. The cumulative clever covariate for a deterministic categorical regime is $$ H_t^d=\prod_{s=0}^{t} \frac{I\{A_s=d_s(H_s)\}}{g_s\{d_s(H_s)\mid H_s\}}. $$ The indicator is one only for an observed treatment path that agrees with the regime at every decision through $t$. Each denominator is the conditional probability of the particular regime-selected category, not the probability of some broader “treated” group.

The formula is the categorical-treatment version of the cumulative covariate derived earlier in the material on deriving cumulative clever covariates. Its key feature is that the selected category may change with history. The regime is evaluated at the observed $H_s$ at each step; it is not generally a fixed sequence of categories.

Assumptions and the Regime-Specific Mean

Let $Y^d$ be the outcome that would be observed if the regime $d$ were followed, and let the target be $\psi^d=E[Y^d]$. The standard identification assumptions are consistency, no interference between individuals, sequential exchangeability, and positivity. For this tutorial, sequential exchangeability is understood strongly enough to cover the counterfactual future trajectory relevant to the regime, including future measured histories as well as the outcome. In particular, treatment assignment at time $t$, conditional on $H_t$, must not carry additional information about that future counterfactual trajectory. Positivity requires $g_t\{d_t(h)\mid h\}>0$ at histories $h$ that can occur under the regime. These assumptions permit the sequential outcome regressions below to identify the regime mean.

Define $V_{T+1}=Y$. Working backward, set $Q_t(h_t,a_t)=E[V_{t+1}\mid H_t=h_t,A_t=a_t]$, where for $t<T$, $V_{t+1}=Q_{t+1}\{H_{t+1},d_{t+1}(H_{t+1})\}$, and for $t=T$, $V_{T+1}=Y$. Define $V_t=Q_t\{H_t,d_t(H_t)\}$. These are the iterated outcome regressions along the regime; the target is $\psi^d=E[V_0]$.

Result: Under consistency, no interference, the stated sequential exchangeability, and positivity, the recursive regressions identify the regime-specific mean: $$ \psi^d=E\left[Q_0\{H_0,d_0(H_0)\}\right]. $$ The cumulative clever covariate at time $t$ is the factor that multiplies the residual in the corresponding longitudinal targeting score.

To justify the identification statement, start at the final decision. By consistency, among individuals with $A_T=d_T(H_T)$, the observed outcome is the outcome under the regime from that history onward. Sequential exchangeability allows its conditional mean to be represented by the observed conditional regression $Q_T$; positivity makes that conditional mean estimable at the selected category. Therefore the conditional mean under the final regime step is $V_T$. Now suppose the conditional mean under the regime after time $t$ is represented by $V_{t+1}$. Conditional on $H_t$, consistency identifies the next observed history under the selected treatment, and sequential exchangeability identifies its conditional distribution with the corresponding regime-specific distribution. Conditioning on $A_t=d_t(H_t)$ therefore gives $Q_t\{H_t,d_t(H_t)\}=V_t$. Repeating this argument backward to baseline and taking expectations gives $\psi^d=E[V_0]$. This identifies the outcome target; it does not assert that treatment weighting identifies the distribution of every future history under weaker, outcome-only exchangeability assumptions.

Why the Cumulative Covariate Has This Form

At a single decision, the regime retains observations with $A_t=d_t(H_t)$ and weights a matching observation by the inverse probability of receiving that category. In a longitudinal path, matching at the current time is not enough: all earlier decisions must also match. Multiplying the one-step factors gives $H_t^d=I_t^d/\prod_{s=0}^t g_s\{d_s(H_s)\mid H_s\}$. This is the relevant cumulative weight for the time-$t$ score.

For a given sequential regression, consider a fluctuation with a scalar parameter $\epsilon_t$. For a bounded outcome represented on the unit interval, a logistic fluctuation has the form $\operatorname{logit}Q_t^{\epsilon_t}=\operatorname{logit}Q_t+\epsilon_t H_t^d$. Its log-likelihood score at $\epsilon_t=0$, for a Bernoulli pseudo-outcome $V_{t+1}$, is proportional to $H_t^d(V_{t+1}-Q_t)$. Thus fitting the fluctuation solves the empirical version of the weighted residual equation.

Derivation: For one observation, the Bernoulli log-likelihood under the fluctuation is $V_{t+1}\log Q_t^{\epsilon_t}+(1-V_{t+1})\log(1-Q_t^{\epsilon_t})$. Since $\partial Q_t^{\epsilon_t}/\partial\epsilon_t =H_t^d Q_t^{\epsilon_t}(1-Q_t^{\epsilon_t})$, differentiating the log-likelihood gives $$ \frac{\partial \ell_t(\epsilon_t)}{\partial\epsilon_t} =H_t^d(V_{t+1}-Q_t^{\epsilon_t}). $$ At $\epsilon_t=0$, this is $H_t^d(V_{t+1}-Q_t)$. Summing over observations and solving the score equation targets the regime-relevant residual direction.

For continuous outcomes, a suitable fluctuation uses an identity link when that respects the outcome support; the score still has the cumulative clever covariate multiplying a residual. The exact fluctuation and support conditions are determined by the outcome type. The construction of $H_t^d$, however, comes from the categorical treatment mechanism and is unchanged.

Worked Examples

Worked Example: A History-Dependent Rule With Three Categories

Suppose there are two decisions and the treatment categories are A, B, and C. At time 0, the regime selects B. At time 1, it selects C if the measured history indicates a high biomarker and A otherwise. Consider a person whose observed time-0 treatment is B and whose time-1 history has a high biomarker. Suppose $g_0(B\mid H_0)=0.60$ and $g_1(C\mid H_1)=0.25$.

At time 0 the match indicator is $I(A_0=B)=1$, so $H_0^d=1/0.60=1.6667$. At time 1, the high biomarker makes the selected category C, and the observed treatment matches it. Hence $$ H_1^d=\frac{1}{0.60}\frac{1}{0.25} =\frac{1}{0.15}=6.6667. $$

If the observed time-1 treatment had instead been A, then $I(A_1=C)=0$ and $H_1^d=0$, regardless of the probability of A. The denominator must use $g_1(C\mid H_1)$, because C is the category specified by the regime at this history.

Worked Example: Calculating the Score Contribution

At one decision time, suppose an observation matches the regime, has cumulative clever covariate $H_t^d=4$, and has pseudo-outcome $V_{t+1}=0.70$. Its current fitted regression value is $Q_t=0.55$. At $\epsilon_t=0$, its contribution to the logistic fluctuation score is $$ 4(0.70-0.55)=4(0.15)=0.60. $$

If instead the observation’s treatment failed to match the regime at any decision through $t$, then $H_t^d=0$, and its contribution is zero even if its residual $V_{t+1}-Q_t$ is nonzero. The empirical score equation sums these contributions over the sample and finds a fluctuation parameter for which the total score is zero. A positive initial contribution does not by itself determine the sign of the fitted update, because the final score also includes all other observations and is evaluated at the updated predictions.

Worked Example: A Two-Step Regime Mean and Its Positivity Check

Suppose the time-0 regime selects category 1 from three possible categories. Under that category, the probability of a high biomarker at the next history is 0.40. At time 1, the regime selects category 2 for a high biomarker and category 0 otherwise. The regime-specific conditional mean outcomes are 0.80 for the high-biomarker/category-2 path and 0.30 for the low-biomarker/category-0 path.

The sequential regressions give the regime mean $$ \psi^d=0.40(0.80)+0.60(0.30) =0.32+0.18=0.50. $$ Suppose, at the histories in question, $g_0(1\mid H_0)=0.50$, $g_1(2\mid H_1=\text{high})=0.25$, and $g_1(0\mid H_1=\text{low})=0.60$. A matching high-biomarker path has cumulative weight $1/(0.50\times0.25)=8$; a matching low-biomarker path has weight $1/(0.50\times0.60)=10/3$, or about 3.333.

Both selected probabilities are positive, so the displayed weights are defined. The high-biomarker path is much more heavily weighted because its selected time-1 category is less common. The value 0.50 is calculated by the outcome-regression recursion; it is not obtained by averaging these two path weights.

Diagnostics, Alternatives, and Limitations

A categorical regime is easy to miscode when the selected category depends on history. Check that each $d_t(H_t)$ returns exactly one valid category, that the indicator compares the observed $A_t$ with that selected category, and that the denominator is the fitted probability for that same category at that history. Useful diagnostics include category-specific treatment counts, the distribution of $g_t\{d_t(H_t)\mid H_t\}$ among relevant histories, the fraction of observations whose path matches through each time, and the range and upper tail of the cumulative covariate. Very small selected-category probabilities produce large covariates and unstable updates. A zero probability violates positivity; replacing it with a small arbitrary number does not repair the causal identification failure.

The main alternatives answer the same regime-mean question differently. G-computation estimates the sequential outcome regressions and averages their regime-specific predictions. Inverse-probability weighting uses the cumulative weights to form a weighted outcome estimate. Longitudinal TMLE also uses outcome regressions, then updates them along the clever-covariate directions to target the parameter. These approaches rely on causal identification assumptions, but their finite-sample behavior can differ: weighting can be volatile under near-positivity violations, while regression-based estimates can be sensitive to outcome-model misspecification. TMLE does not remove the need for positivity or correct causal assumptions, and a targeting step cannot create information about rarely observed treatment categories.

The formula here is for deterministic interventions with discrete treatment levels. It is not the density-ratio formula for continuous treatments, nor the intervention mechanism for stochastic treatment policies. Those cases require different treatment-mechanism factors. Also, if the goal is to identify distributions of future histories beyond what is needed for the outcome target, the relevant exchangeability assumptions must cover those histories; outcome exchangeability alone should not be used to claim a general weighted-history identity.

Evaluate the regime.
At each observed history, compute the single category $d_t(H_t)$ selected by the rule.

Check the match.
Multiply the indicators $I\{A_s=d_s(H_s)\}$ through the current time; one mismatch makes the cumulative indicator zero.

Use matching-category probabilities.
For every matched time, divide by $g_s\{d_s(H_s)\mid H_s\}$, then multiply the factors.

Target the sequential regression.
Use the resulting $H_t^d$ in the fluctuation score for the time-$t$ regression and inspect the updated predictions and weights.

Common pitfall: Treating a three-level treatment as if the regime selected a binary “treated” indicator and putting $P(A_t\ne 0\mid H_t)$ in the denominator is generally wrong. If the regime selects category 2, the correct factor uses $I(A_t=2)/g_t(2\mid H_t)$. A second pitfall is to infer that the weights identify arbitrary future-history distributions from outcome-only exchangeability; that conclusion requires assumptions covering those counterfactual histories.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A regime selects category 0 at time 0 and category 2 at time 1 when history is high. An observation receives category 0 at time 0 and category 1 at time 1. What is $H_1^d$?
If the selected time-1 category has conditional probability 0.20 and the observation matches at that decision, what inverse-probability factor enters at time 1?
Why does the cumulative clever covariate include indicators and probabilities from earlier decision times?
State one diagnostic for weak positivity in a multilevel regime.
Does outcome-only sequential exchangeability justify using the treatment weights to identify the distribution of every future measured history? Explain.

Solutions. (1) $H_1^d=0$, because the time-1 treatment does not match category 2; the product of indicators is zero. (2) The factor is $1/0.20=5$, provided the observation also matches the regime through earlier decisions; the full cumulative covariate includes all earlier factors. (3) The regime path requires matching at every decision through time $t$, and each observed match is reweighted by the inverse probability of that selected category. (4) Examine the distribution or minimum of $g_t\{d_t(H_t)\mid H_t\}$ among regime-relevant histories, along with the upper tail of the cumulative weights. Small selected probabilities indicate weak positivity. (5) No. That claim requires exchangeability covering the relevant future counterfactual histories as well as the outcome; outcome-only exchangeability is not enough.