Dynamic Treatment Regimes As Sequential Decision Rules

From Treatment Schedules to Decision Rules

A treatment plan can be specified before follow-up begins, or it can make later decisions depend on what has happened so far. The second kind of plan is a dynamic treatment regime: at each decision time, it uses the information then available to prescribe an action. The defining feature is not that treatment changes over time, but that the prescribed action can change with the observed history.

Use the following time order. Let \(L_0\) denote baseline information. At time \(t\), the available history is \(H_t\), treatment \(A_t\) is given, and subsequent covariate information is recorded. For \(t=1,\ldots,K\), take \(H_t=(L_0,A_1,L_1,\ldots,A_{t-1},L_{t-1})\), with \(H_1=L_0\). Any variable observed before treatment \(A_t\) that is relevant to the decision is included in \(H_t\). In particular, \(H_t\) cannot include information observed only after \(A_t\).

Definition: A deterministic dynamic treatment regime is a sequence \(g=(g_1,\ldots,g_K)\) of measurable functions, where \(g_t\) maps the possible pre-treatment histories \(H_t\) to the treatment action space \(\mathcal A_t\). Under regime \(g\), the prescribed action at time \(t\) is \(A_t=g_t(H_t)\). The word deterministic means that the rule selects one action for each history; it does not mean that people with different histories must receive the same action.

A regime uses only the information available when its decision is made. It may depend on baseline characteristics, earlier treatments, and time-varying covariates already observed. It must not depend on future covariates or on the outcome that will occur after the treatment decision. For a rule to be mathematically defined, its functions must be measurable, so the events describing which action it selects are well-defined.

Static Schedules and Dynamic Rules

A static treatment schedule fixes the actions in advance, without reference to the evolving history. It can prescribe different actions at different times; for example, treatment \(a_1\) at the first decision and \(a_2\) at the second. Formally, it is the special case \(g_t(h)=a_t\) for every history \(h\) at time \(t\). Thus a dynamic regime is history-dependent when at least one of its decision functions takes different values for different histories.

The distinction concerns how actions are selected, not how many treatment levels exist. A rule that always assigns treatment 1 is static even if treatment 1 is highly effective. A rule that assigns treatment 1 when a measured symptom is high and treatment 0 otherwise is dynamic, even if both rules use the same two treatment options.

Worked Example: A Two-Decision Symptom Rule

Suppose \(A_t\in\{0,1\}\) at two decision times. Let \(L_0\) be a baseline severity score, and let \(L_1\) be a symptom score measured after the first treatment and before the second. Consider the rule \(g_1(L_0)=I(L_0\geq 6)\) and \(g_2(H_2)=I(L_1\geq 4)\), where \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise.

For a person with \(L_0=7\), the first prescribed action is \(g_1(7)=I(7\geq6)=1\). If the subsequently observed \(L_1=3\), the second action is \(g_2(H_2)=I(3\geq4)=0\). The prescribed sequence for that history is therefore \((1,0)\). If instead \(L_1=5\), the second prescribed action is \(I(5\geq4)=1\), giving \((1,1)\).

The second action is not fixed at baseline: it depends on the symptom measured before that decision. This is a dynamic regime. The fact that one realized path happens to be \((1,0)\) does not make the rule a static schedule; the same rule prescribes \((1,1)\) for another eligible history.

Worked Example: Classifying a Fixed Schedule

At three treatment times, a protocol prescribes \(A_1=1\), \(A_2=0\), and \(A_3=1\) for everyone, regardless of measured history. Its decision functions are \(g_1(h)=1\), \(g_2(h)=0\), and \(g_3(h)=1\) for every allowable history at the corresponding time.

Each function is constant in its history argument. The regime is therefore the static schedule \((1,0,1)\), not a history-dependent rule. It remains static if the histories are measured and recorded; they simply do not affect the prescribed actions.

By contrast, changing only the second rule to \(g_2(H_2)=I(L_1\geq4)\) makes the overall regime dynamic. The first and third actions remain fixed, but one history-dependent decision is enough for the sequence of rules to be dynamic.

How a Rule Generates an Action Path

A rule specifies an action from a history, and the resulting action can affect later covariates and hence later histories. To make this recursive description precise, suppose covariates evolve according to well-defined structural functions \(L_t=f_t(H_t,A_t,U_t)\), where \(U_t\) represents exogenous inputs. Given a baseline value and the exogenous inputs, applying \(g\) means setting \(A_t^g=g_t(H_t^g)\) and then obtaining the next covariate from its structural function. The superscript \(g\) denotes quantities generated under the regime.

Theorem: Suppose each \(g_t\) is a well-defined function on the histories that can occur at time \(t\), and each covariate transition \(f_t\) returns a well-defined next covariate for its input. For any fixed baseline value and fixed exogenous inputs, the recursion under \(g\) determines a unique action and covariate path through the \(K\) decisions.

Start with baseline history.
Set \(H_1^g=L_0\), which is known before the first treatment.

Prescribe the current action.
Once \(H_t^g\) is determined, set \(A_t^g=g_t(H_t^g)\). Because \(g_t\) is a function, this history gives one action.

Update the history.
For \(t<K\), use \(L_t^g=f_t(H_t^g,A_t^g,U_t)\) and append \(A_t^g,L_t^g\) to obtain \(H_{t+1}^g\).

Proof. At \(t=1\), \(H_1^g=L_0\) is fixed. The function \(g_1\) therefore determines one action \(A_1^g\), and \(f_1\) determines one \(L_1^g\) when \(t<K\). Suppose the history through time \(t\) has been uniquely determined. Then \(A_t^g=g_t(H_t^g)\) is uniquely determined; if \(t<K\), the specified transition function gives a unique \(L_t^g\), which determines \(H_{t+1}^g\). Induction over \(t=1,\ldots,K\) establishes the unique path. \(\square\)

This result is about the logical operation of a regime, not about whether its effects can be learned from observed data. Defining a rule requires no causal identification assumptions. Interpreting regime-specific counterfactual outcomes requires additional assumptions and a suitable causal model. In particular, consistency links an observed outcome to the counterfactual outcome for the treatment history actually received, and no interference rules out one person’s assigned treatment changing another person’s outcome. Those conditions do not turn a dynamic rule into a static one.

Worked Example: Tracing Actions Through a Rule

Let \(A_t\in\{0,1\}\) at two decisions. The rule assigns \(A_1^g=I(L_0\geq 5)\) and \(A_2^g=I(L_1\geq 3)\). Consider a person with \(L_0=4\). For illustration, suppose the covariate transition under the first action is \(L_1=2+A_1\).

First, \(A_1^g=I(4\geq5)=0\). The transition then gives \(L_1^g=2+0=2\). At the second decision, \(A_2^g=I(2\geq3)=0\). Thus the regime-generated treatment path is \((0,0)\).

For comparison, if \(L_0=6\), the first action is \(I(6\geq5)=1\), so the stated transition gives \(L_1^g=2+1=3\). The second action is \(I(3\geq3)=1\), and the path is \((1,1)\). The calculation shows why later prescriptions must be evaluated using the history generated under earlier regime actions, not by holding the person’s observed history fixed.

A Precise Criterion for Static Regimes

Theorem: A deterministic regime \(g=(g_1,\ldots,g_K)\) is a static schedule if and only if, for every time \(t\), \(g_t(h)=g_t(h')\) for every pair of allowable histories \(h,h'\) at that time.

Proof. If \(g\) is the static schedule \((a_1,\ldots,a_K)\), then \(g_t(h)=a_t=g_t(h')\) for every pair of histories, so each decision function is constant. Conversely, suppose each \(g_t\) has the same value for all allowable histories at time \(t\). Choose any allowable history \(h_t\) and set \(a_t=g_t(h_t)\). Constancy gives \(g_t(h)=a_t\) for every allowable \(h\). Doing this for each \(t\) yields a fixed schedule \((a_1,\ldots,a_K)\), so the regime is static. \(\square\)

The criterion concerns the domain of allowable histories, not merely the histories that happen to appear in one finite data set. A rule can vary across histories even if the sample contains only one of those histories. Conversely, a rule that is formally history-dependent but constant on its entire allowable domain acts as a static schedule.

Diagnostics, Comparisons, and Limitations

When writing or reviewing a regime, make the time ordering explicit. For each \(t\), list the information in \(H_t\), the action space \(\mathcal A_t\), and the rule \(g_t\). Then check that the rule returns an allowed action for every history in its domain and uses no future information. A decision table can expose errors that are difficult to see in prose.

Future-information error: a rule uses \(L_t\) to choose \(A_t\), although \(L_t\) is measured after \(A_t\). Correction: use only the pre-treatment history \(H_t\).
Indexing error: the rule for \(A_{t+1}\) is evaluated using the wrong covariate time. Correction: write the time order and construct each \(H_t\) before defining \(g_t\).
Static-versus-dynamic error: a fixed sequence is called dynamic simply because treatment occurs at several times. Correction: check whether any decision function actually varies with history.
Observed-path error: a later action is evaluated using a covariate observed under a different earlier treatment path. Correction: when describing the regime path, update the history under the regime’s prescribed actions.

A static schedule is easier to state because it ignores changing history. A deterministic dynamic regime can tailor decisions to measured information, but that flexibility does not guarantee better outcomes and does not by itself establish that the effects are identifiable. A stochastic regime is a different alternative: instead of selecting one action for each history, it specifies a probability distribution over actions given that history. The regime defined here is deterministic, even though the observed treatment received in a study may be random.

The definition also does not decide which rule should be preferred. It describes a policy, not a criterion for ranking policies or an estimation method. To assess a regime’s consequences from observational data requires causal assumptions and an appropriate analysis; the rule’s clear specification is a prerequisite, not a substitute.

Check Your Understanding

Answer each question using the distinction between a rule and a fixed schedule.

At two times, treatment is fixed as \((1,0)\) for everyone. Is this a deterministic dynamic regime, a static schedule, or both?
A rule assigns \(A_1=0\) for everyone and assigns \(A_2=1\) if \(L_1\geq2\), otherwise \(A_2=0\). Is the overall regime static or dynamic?
Why is a rule for \(A_t\) that uses a covariate measured after \(A_t\) not a valid decision rule based on \(H_t\)?
Under what conditions does a deterministic regime determine one action path for fixed baseline information and exogenous inputs?

Solutions. (1) It is both a deterministic regime and a static schedule: the functions \(g_1(h)=1\) and \(g_2(h)=0\) each prescribe one action, independent of history. (2) It is dynamic because \(g_2\) takes different actions for histories with \(L_1\geq2\) and \(L_1<2\). (3) The covariate is unavailable at the decision time, so it is not part of the pre-treatment history \(H_t\); using it would make the rule depend on future information. (4) Each decision function must be well-defined on the histories that can occur, and each covariate transition must return a well-defined next value. Recursion then uniquely determines the path.