Clever Covariates for Stochastic Interventions

From Regime Indicators to Intervention Density Ratios

A deterministic treatment regime asks what the outcome would be if treatment were set to a specified value at each decision. A stochastic intervention instead specifies a probability distribution for treatment, possibly depending on the history observed before that decision. In longitudinal targeted maximum likelihood estimation (TMLE), this change replaces the familiar regime-match indicator in the clever covariate with a likelihood ratio: the intervention treatment density divided by the observed treatment density.

The ratio is not just a convenient weight. It is the change of measure that makes histories generated under the observed treatment mechanism represent histories under the intervention. Its validity depends on support: the intervention must not assign positive probability to treatment values that the observed treatment law cannot produce.

For a continuous treatment, “density” means density relative to a common dominating measure, usually Lebesgue measure. For a discrete treatment, it means probability mass. The ratio is evaluated at the treatment actually observed. Histories matter: even when the same treatment value is observed at two visits, its ratio can differ because the intervention and observed laws may depend on different histories.

Why the Ratio Is the Clever Covariate

To define the sequential regressions, let \(V_t(h_t)\) be the conditional mean of the outcome under the intervention from time \(t\) onward, given the history before treatment at time \(t\). Let \(Q_t(h_t,a_t)\) be the conditional expectation of the next intervention value given the current history and treatment. Specifically, for \(t<K\), \(Q_t(H_t,A_t)=E[V_{t+1}(H_{t+1})\mid H_t,A_t]\), where \(H_{t+1}\) includes the covariate history available before the next treatment. At the final visit, \(Q_K(H_K,A_K)=E[Y\mid H_K,A_K]\). Define

Here the integral also denotes a sum for discrete treatment. The observed-data law supplies the conditional covariate and outcome distributions in the \(Q_t\)'s; the intervention \(q_t\) supplies the treatment distributions in the \(V_t\)'s. Thus \(\psi\) is the mean outcome under the specified stochastic policy, under consistency and sequential exchangeability of treatment given the recorded histories. These causal assumptions give the intervention interpretation; the density-ratio calculations additionally require the support condition above.

The reason is the change-of-measure identity below. It says that weighting an observed treatment draw by its intervention-to-observed ratio produces expectation under the intervention treatment law. In a longitudinal history, the identity applies successively, so the ratios multiply.

Proof. Conditional on \(H_t=h\), the observed treatment is distributed according to \(g_t(da\mid h)\). Therefore,

For densities or mass functions, \(dq_t/dg_t\) is the stated ratio wherever \(g_t\) is positive. Applying this conditional identity at each decision gives the product \(W_t\); iterated expectation then replaces the observed treatment laws by the intervention laws through time \(t\). Integrability is required for these expectations to exist. \(\square\)

Worked Calculations

Derivation in the Targeting Equation

The ratio also appears directly in the efficient influence function. Under a nonparametric longitudinal model with the stated factorization, fixed \(q_t\), square-integrable influence-function terms, and the required absolute continuity, the efficient influence function for \(\psi\) is

where the final summand uses \(V_{K+1}=Y\). This expression has conditional residuals, not treatment residuals: \(Q_t\) is the conditional mean of the next intervention value given the current history and treatment. Consequently, each summand has conditional mean zero given \((H_t,A_t)\) before multiplication by \(W_t\).

A derivation follows by perturbing the conditional outcome and covariate-transition distributions in the observed-data factorization. For a perturbation at time \(t\), the derivative of the target is the intervention-weighted derivative of the conditional mean \(Q_t\). The change-of-measure proposition replaces the intervention treatment law at each preceding decision by the corresponding factor \(q_s/g_s\); iterating yields \(W_t\). The centered conditional residual \(V_{t+1}-Q_t\) represents the derivative with respect to that conditional distribution. Perturbations of the observed treatment mechanisms contribute zero because the target uses the fixed intervention laws \(q_t\), not the observed \(g_t\)'s. Perturbations of the baseline distribution contribute \(V_1(H_1)-\psi\). Summing these score contributions gives the displayed expression. Its terms have mean zero, and their conditional centering makes them orthogonal to score components for treatment mechanisms. These properties identify the gradient in the nonparametric model. \(\square\)

In a fluctuation of \(Q_t\), the score is obtained by multiplying its conditional residual by \(W_t\). For a binary outcome, for example, the logistic fluctuation score is proportional to \(W_t\{Y-Q_t^*\}\) at the terminal step; earlier steps use the corresponding downstream pseudo-outcome in place of \(Y\). Setting the empirical score close to zero targets the residual term in the influence function. The recursion and the choice of fluctuation submodel are the same principles used in longitudinal TMLE; the stochastic intervention changes the treatment component of the clever covariate to the density ratio.

Assumptions, Diagnostics, and a Common Pitfall

The ratio construction has two distinct requirements. First, the causal interpretation of \(\psi\) requires consistency and sequential exchangeability: conditional on \(H_t\), treatment assignment must be unconfounded for the relevant future counterfactuals. Second, the intervention must satisfy absolute continuity with respect to the observed treatment law. For stable estimation, stronger practical overlap is often needed: ratios should not be so large that a few observations dominate the fit.

Useful diagnostics include inspecting the distribution of each time-specific ratio and of the cumulative \(W_t\), checking extreme values and effective sample sizes, and comparing these across visits and histories. A ratio that is undefined or extremely large can signal a support violation or near-violation, a poorly estimated treatment density, or a policy that places substantial weight in sparsely observed regions. Truncating ratios may reduce variance, but changes the estimating procedure and can introduce bias; it does not repair a structural absence of support.

• Common mistake: treating every regime as a density-ratio intervention. For a discrete treatment and a deterministic regime \(A_t=d_t(H_t)\), the intervention mass is one at the assigned value and zero elsewhere. The ratio is therefore zero for a mismatch and \(1/g_t(d_t(H_t)\mid H_t)\) for a match, yielding the familiar indicator divided by the observed treatment probability.
• Correction for continuous treatment: a deterministic assignment is a point mass, generally singular with respect to a continuous observed treatment distribution. It has no density ratio of the stated kind. The discrete reduction must not be carried over to continuous treatment.
• Common mistake: using one history's denominator at another history. Every \(g_t(A_t\mid H_t)\) is evaluated at the observed current history. As the two-decision example shows, a different earlier treatment can lead to a different second-decision denominator.

Inverse-probability weighting uses the same ratios to reweight outcomes directly, while the g-computation formula averages sequential outcome regressions under \(q_t\). Longitudinal TMLE combines regression-based estimation with ratio-weighted targeting and a plug-in estimate. These approaches address the same intervention mean but have different finite-sample behavior; the clever covariate alone cannot overcome inadequate overlap or misspecified nuisance estimates.