Positivity for Dynamic Treatment Regimes

Why Dynamic-Regime Positivity Is a Support Question

A dynamic treatment regime chooses treatment using information available at each decision time. For example, a rule might prescribe one treatment when a patient’s current biomarker is high and another when it is low. The relevant positivity question is therefore not simply whether each treatment occurs somewhere in the study. It is whether the treatment prescribed by the rule is supported at the histories that the rule can reach.

There is an important order to this check. The observed treatment mechanism is defined only up to sets of probability zero under the observed history distribution. Before evaluating that mechanism at regime-relevant histories, we must establish that the regime’s history distribution is supported by the observed history distribution. Otherwise, the treatment probability at a history the regime can reach may depend on an arbitrary choice of conditional-probability version.

Definition: Consider decision times $t=0,\ldots,K$. Let $L_t$ be the covariate information available at time $t$, let $A_t$ be treatment at that time, and define the pre-treatment history $H_t=(L_0,A_0,L_1,\ldots,A_{t-1},L_t)$, with $H_0=L_0$. A deterministic dynamic treatment regime is a collection of measurable functions $d=(d_0,\ldots,d_K)$, where the prescribed treatment at time $t$ is $d_t(H_t)$. Write $P$ for the observed-data law, $P_{H_t}$ for its history marginal, and $g_t(a\mid h)=P(A_t=a\mid H_t=h)$ for a version of the observed treatment mechanism. For simplicity, assume treatments take values in a discrete set, so $g_t$ is a conditional mass function.

Define $Q_t^d$ to be the distribution of $H_t$ obtained by starting with the observed baseline distribution and, at each earlier decision, setting treatment to the regime’s prescribed value while using the observed conditional covariate distributions. Thus $Q_0^d=P_{L_0}$. To construct $Q_{t+1}^d$ from $Q_t^d$, set $A_t=d_t(H_t)$ and draw $L_{t+1}$ from the observed conditional law of $L_{t+1}$ given $H_t,A_t$. This construction is justified from the observed law only when the action-specific conditional law is supported there; the recursive condition below supplies that justification.

Definition: The observed-law positivity check for regime $d$ is recursive. At decision time $t$, first require $Q_t^d\ll P_{H_t}$, meaning every history set with zero observed probability also has zero $Q_t^d$-probability. Only then evaluate the observed treatment mechanism at the prescribed action and require $g_t(d_t(h)\mid h)>0$ for $Q_t^d$-almost every $h$. If either requirement fails, stop the observed-law recursion at that first failure; later observed treatment mechanisms must not be evaluated under a history distribution that is not dominated by the observed history law.

The order is part of the definition, not a technical afterthought. Conditional probabilities are unique only almost surely under the distribution of the variables being conditioned on. A numerical value assigned to $g_t(a\mid h)$ at an observed-null history can be changed without changing the observed law. The condition $Q_t^d\ll P_{H_t}$ prevents the positivity check from depending on such arbitrary values.

A Recursive Density Result

The following result makes the support condition concrete. It shows how a regime-history distribution is represented within the observed history law when each recursive check succeeds. Assume the longitudinal observed law admits regular conditional distributions for the covariate transitions, and that the variables take values in standard Borel spaces, so those conditional distributions exist. At each step, history dominance and positive probability for the regime-prescribed action ensure that the action-specific conditional covariate distribution is used only where its observed-law version is determined.

Theorem: Suppose the recursive observed-law positivity check holds through decision time $t-1$. Then $Q_t^d\ll P_{H_t}$, and a density of $Q_t^d$ with respect to $P_{H_t}$ is $$ w_t^d(H_t)= \prod_{j=0}^{t-1} \frac{\mathbb{1}\{A_j=d_j(H_j)\}} {g_j(A_j\mid H_j)} , $$ where the empty product for $t=0$ equals 1. The factors need only be defined on the observed-support set where their denominators are positive; the density can be assigned any value on the remaining $P_{H_t}$-null set.

Proof. At $t=0$, $Q_0^d=P_{H_0}$, so the density is 1. Suppose the density formula holds at time $t$, and the recursive check at time $t$ succeeds. For any nonnegative measurable function $f$ of $H_t,A_t,L_{t+1}$, the regime construction gives

E_{Q_{t+1}^d}[f] = \int Q_t^d(dh) \int q_{t+1}(dl\mid h,d_t(h)) f\bigl(h,d_t(h),l\bigr),

where $q_{t+1}(\cdot\mid h,a)$ is the observed conditional distribution of $L_{t+1}$ given $H_t=h,A_t=a$. Since $Q_t^d\ll P_{H_t}$ and $g_t(d_t(h)\mid h)>0$ $Q_t^d$-almost surely, the action-specific kernel in this integral is used only on observed-supported histories and actions. Substitute $dQ_t^d(h)=w_t^d(h)dP_{H_t}(h)$. Under the observed law, conditional on $H_t=h$, the joint probability of $A_t=d_t(h)$ and $L_{t+1}\in dl$ is $g_t(d_t(h)\mid h)q_{t+1}(dl\mid h,d_t(h))$. Therefore

E_{Q_{t+1}^d}[f] = E_P\left[ f(H_t,A_t,L_{t+1})\, w_t^d(H_t) \frac{\mathbb{1}\{A_t=d_t(H_t)\}} {g_t(A_t\mid H_t)} \right].

The multiplier is the stated product through time $t$, which proves the density formula at $t+1$. Its expectation under $P$ is 1: conditionally on $H_t$, the new factor has expectation $\sum_a g_t(a\mid H_t)\mathbb{1}\{a=d_t(H_t)\}/g_t(a\mid H_t)=1$ where the prescribed action has positive probability. Induction proves the result. $\square$

This density explains the familiar inverse-probability factors, but it also identifies what they require: at each step the prescribed action must be observed-supported on the regime’s current histories. A product formula does not repair a failure of support earlier in the recursion.

Worked Example: A Supported Rule That Uses the Current Covariate

There are two treatment decisions, at $t=0$ and $t=1$. Let the baseline covariate $X=L_0$ be binary with $P(X=0)=P(X=1)=1/2$. The rule sets $d_0(X)=1$. At the second decision it sets $d_1(H_1)=1$ if $L_1=1$, and $d_1(H_1)=0$ if $L_1=0$. Suppose the observed mechanism satisfies $g_0(1\mid X)=0.6$ for either value of $X$. Among histories with $A_0=1$, suppose $P(L_1=1\mid X=0,A_0=1)=0.25$ and $P(L_1=1\mid X=1,A_0=1)=0.75$. Finally, at histories with $A_0=1$, suppose $g_1(1\mid H_1)=0.7$ when $L_1=1$, and $g_1(0\mid H_1)=0.8$ when $L_1=0$.

At baseline, $Q_0^d=P_X$, so the history-dominance check holds. The prescribed first treatment has probability $0.6>0$ for each baseline history. Under the regime, $A_0=1$, and $Q_1^d(L_1=1)=\tfrac12(0.25)+\tfrac12(0.75)=0.5$; hence $Q_1^d(L_1=0)=0.5$. Each such history has observed support because $A_0=1$ occurred with positive probability for both values of $X$, and both values of $L_1$ have positive conditional probability at those histories.

At time 1, the prescribed action has probability $0.7$ on the $L_1=1$ histories and $0.8$ on the $L_1=0$ histories. Both are strictly positive, so the recursive check succeeds. For example, the regime density on a time-1 history with $A_0=1$ is $1/0.6$, while it is zero on histories with $A_0=0$. After the second treatment, the density factor is $1/(0.6\times0.7)$ when $L_1=1,A_1=1$, and $1/(0.6\times0.8)$ when $L_1=0,A_1=0$; it is zero when either observed treatment differs from the rule. These values encode the rule’s changing treatment choice rather than one fixed treatment sequence.

Worked Example: Stop at the First Unsupported History

Let $X$ be binary with $P(X=0)=P(X=1)=1/2$, and suppose the regime prescribes $d_0(X)=1$ for both values. In the observed data, $P(A_0=1\mid X=0)=0.6$, but $P(A_0=1\mid X=1)=0$.

The baseline distribution is $Q_0^d=P_X$, so $Q_0^d\ll P_X$. However, the prescribed action has probability zero at $X=1$, and $Q_0^d(X=1)=1/2>0$. The recursive check therefore fails at time 0. No observed-law regime distribution $Q_1^d$ can be constructed by this rule using the observed transition law at $X=1,A_0=1$: that treatment-covariate combination is not supported by the observed data.

Suppose someone assigns the value $0.4$ to $g_1(d_1(H_1)\mid H_1)$ at histories that arise after $X=1,A_0=1$. That assignment cannot make the regime supported. The values of a later conditional mechanism there are not determined by the observed law, because the histories themselves lie beyond the first failure. The correct diagnostic is to stop at time 0, not to carry on with an arbitrary version.

What the Condition Does—and Does Not—Require

The required histories are those assigned positive probability by $Q_t^d$, the history distribution reached under the regime’s earlier decisions and the observed covariate transitions. A history that occurs in the observed data but has zero $Q_t^d$-probability need not satisfy this regime-specific treatment-support check. For example, if the regime always sets $A_0=1$, observed histories with $A_0=0$ do not themselves need to support the regime’s next treatment decision.

A stronger condition sometimes used in practice requires the prescribed action to have positive probability at every history in the observed support. That condition can be convenient, but it asks more than the regime-specific condition: it may rule out a regime because of histories the regime cannot reach. Conversely, checking only that each treatment occurs somewhere, or that each complete treatment sequence has positive marginal probability, is not enough. A dynamic rule can prescribe different actions in different histories, and support must be checked conditionally at those histories.

Under consistency and sequential exchangeability, the regime’s mean outcome can be identified from the observed law when the required sequential support conditions hold. The density result explains the weighting representation: for a full observed trajectory, the regime contribution is zero unless every observed treatment matches the rule, and otherwise is the product of inverse treatment probabilities along that trajectory. The earlier material on identification gives the causal assumptions and identification formulas; the point here is that positivity makes the relevant observed-law comparisons well-defined.

Worked Example: Why Marginal Treatment Frequencies Can Mislead

Suppose $X$ is binary and both treatment values occur in the study overall. In fact, let $P(X=0)=P(X=1)=1/2$, with $A=1$ for everyone when $X=0$ and $A=0$ for everyone when $X=1$. The marginal probabilities are $P(A=1)=1/2$ and $P(A=0)=1/2$. Consider the rule that prescribes $A=1$ when $X=1$ and $A=0$ when $X=0$.

At $X=0$, the prescribed action is $0$, but $P(A=0\mid X=0)=0$. At $X=1$, the prescribed action is $1$, but $P(A=1\mid X=1)=0$. Since both baseline histories have regime probability $1/2$, positivity fails at both. The positive marginal frequencies of each treatment do not help: neither treatment is available at the histories where this rule asks for it.

Diagnostics and Common Pitfalls

In a real study, the recursive definition guides diagnostics, although finite data cannot prove population support. At each decision time, first describe or estimate the distribution of histories reached by the specified regime; then assess whether its prescribed action is observed at those histories. Useful warning signs include sparse treatment counts in relevant history strata, estimated treatment probabilities near zero, extreme inverse-probability weights, and substantial sensitivity of results to modest changes in the treatment model. For continuous histories, exact repeated-history counts are not useful; diagnostics must rely on the modeling strategy and available overlap. These checks indicate possible problems, not proof that positivity holds.

Mistake: checking only whether a treatment occurs overall. A treatment can be common overall and absent at the histories where the rule prescribes it. Check conditional support on the regime-reached histories.
Mistake: evaluating $g_t$ before checking history dominance. If $Q_t^d\not\ll P_{H_t}$, the value of $g_t$ on the unsupported part of $Q_t^d$ may depend on an arbitrary conditional-probability version. Stop at the first failed history-dominance or treatment-support check.
Mistake: treating a very small probability as a structural zero. A probability close to zero creates unstable estimation and may indicate near-violation, but it is not mathematically the same as a zero. The distinction does not make tiny probabilities harmless.

Positivity is necessary for identification by the observed-law comparisons used for a regime; it is not a guarantee of precise estimation. Even when all required probabilities are positive, very small probabilities can produce highly variable weights and make conclusions sensitive to modeling choices. Nor does passing the support check establish consistency or sequential exchangeability. Those are distinct assumptions.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What must be checked before evaluating $g_t(d_t(h)\mid h)$ under $Q_t^d$?
A regime assigns zero probability to a history, though that history occurs in the observed data. Must the prescribed treatment have positive probability there for this regime-specific check?
Suppose the prescribed treatment has probability 0.02 at every regime-reached history. Does the strict positivity condition hold, and what practical concern remains?
If a treatment occurs in half the sample overall but has probability zero at a history where the regime prescribes it, does marginal treatment frequency establish positivity?

Solutions. (1) First require $Q_t^d\ll P_{H_t}$, so the observed treatment mechanism is evaluated only where its version is determined by the observed law; then require the prescribed treatment probability to be positive $Q_t^d$-almost surely. (2) No. The regime-specific condition concerns histories with positive $Q_t^d$-probability; a history assigned zero regime probability is not reached by this rule. (3) Yes, because $0.02>0$, but inverse-probability weighting can be unstable and estimates may be imprecise. (4) No. Positivity is conditional on the histories reached by the rule, and a marginal frequency cannot compensate for a conditional zero.

Positivity for Dynamic Treatment Regimes

What You'll Learn