Value Functions for Treatment Regimes

What a Regime’s Value Measures

A treatment regime specifies an action at each decision time as a function of the information available then. Its value summarizes the expected outcome if the target population were to follow that regime. It is therefore a population-level quantity, not an individual prediction and not, by itself, a comparison with another regime.

Let $K$ be the number of treatment times. Write $W$ for the complete baseline information measured before the first treatment. In particular, if the baseline is recorded as separate variables $X$ and $L_1$, then $W=(X,L_1)$; neither variable is left out of the baseline distribution. At time $t$, let $A_t$ be treatment, $R_t$ the reward or outcome accrued at that time, and $L_{t+1}$ the next covariate when $t<K$. The observed history just before treatment $A_t$ is $H_t=(W,A_1,R_1,L_2,\ldots,A_{t-1},R_{t-1},L_t)$, with $H_1=W$.

Definition: A deterministic treatment regime is a sequence $d=(d_1,\ldots,d_K)$ of decision rules, where $d_t(h_t)$ is the treatment assigned at time $t$ for history $h_t$. Let $R_t^d$ denote the potential reward at time $t$ under adherence to regime $d$, and let $S^d=\sum_{t=1}^K R_t^d$ be its cumulative outcome. For a specified target population with distribution $P$, the value of $d$ on this outcome scale is $$ V_P(d)=E_P[S^d]=E_P\left[\sum_{t=1}^K R_t^d\right]. $$

The subscript $P$ is essential: it identifies the population over which the expectation is taken. The outcome definition is essential too. A value has units and meaning inherited from the cumulative outcome—for example, expected symptom-free days or expected total cost. A contrast such as $V_P(d)-V_P(d')$ answers a different question: it compares two regimes in the same population and on the same scale.

Identification From Longitudinal Data

A value is defined in terms of potential outcomes, which are not all observed for any one person. To identify it from observed data, suppose the observed trajectory follows the ordering $W,A_1,R_1,L_2,\ldots,A_K,R_K$. For $t<K$, let $q_t(r_t,l_{t+1}\mid h_t,a_t)$ be the observed conditional distribution of the reward and next covariate given the history and treatment. At the final time, let $q_K(r_K\mid h_K,a_K)$ be the conditional distribution of the final reward. These distributions may be densities, probability mass functions, or probability kernels, according to the variable types.

The identification conditions are: consistency, so observed outcomes agree with the potential outcomes under the treatment actually received; sequential exchangeability, so at each time treatment assignment is conditionally independent of the future potential trajectory under the regime given the observed history; and positivity, so each treatment selected by the regime has positive probability at histories reached under that regime. Also assume well-defined interventions and no interference between individuals. Positivity is a support condition, not merely a requirement that every treatment occur somewhere in the sample.

Identification result: Under consistency, sequential exchangeability, and positivity, and assuming $E_P[|S^d|]<\infty$, the value is identified by averaging over the full baseline distribution and then the regime-specific sequence of outcome and covariate distributions. In the discrete case, $$ V_P(d)= \sum_w p_P(w) \sum_{r_1,l_2,\ldots,r_{K-1},l_K,r_K} \left\{\prod_{t=1}^{K-1} q_t(r_t,l_{t+1}\mid h_t,d_t(h_t))\right\} q_K(r_K\mid h_K,d_K(h_K)) \left(\sum_{t=1}^K r_t\right). $$ Here each $h_t$ is formed from the baseline value $w$ and the preceding rewards, covariates, and regime-assigned treatments. For continuous variables, the sums are replaced by integrals. If $K=1$, the product over $t=1,\ldots,K-1$ is empty and equals one.

This expression includes every baseline variable in $W$, as well as every intermediate covariate and reward that can affect later histories. The regime’s treatment is substituted at each step; the observed treatment distribution is not averaged over as if treatment were assigned naturally.

Worked Example: A Random Baseline Covariate Must Be Averaged

Suppose there is one treatment time, $X$ is constant, and the baseline covariate $L_1$ is Bernoulli with $P(L_1=1)=0.3$. Let the reward be $R_1=L_1$ regardless of treatment. A regime that assigns treatment $0$ to everyone therefore has $S^d=L_1$.

The value is $$ V_P(d)=E_P[L_1]=1(0.3)+0(0.7)=0.3. $$

In the one-time g-formula the baseline is $W=(X,L_1)$, so the calculation sums over $L_1$ as well as $X$. Since $X$ is constant, summing over $X$ changes nothing; averaging over $L_1$ is what produces $0.3$. If $L_1$ is omitted from the explicit baseline sum, its distribution must be integrated into the conditional outcome mechanism; omitting it without doing so can give an incorrect value.

Why the Identification Formula Holds

The formula follows by expanding the expectation of the counterfactual trajectory one time point at a time. Under the intervention $d$, the baseline has distribution $p_P(w)$. Given a history $h_t$, the treatment is fixed at $d_t(h_t)$. Sequential exchangeability and consistency then identify the conditional distribution of the next reward and, before the final time, next covariate with the corresponding observed conditional distribution $q_t$.

Start at baseline.
Average over $W\sim p_P$, which includes all variables measured before the first treatment.

Apply the regime.
At each generated history $h_t$, set $A_t=d_t(h_t)$, rather than averaging over the natural treatment assignment.

Generate the next variables.
Use $q_t$ for the reward and next covariate, or $q_K$ for the final reward, then average the cumulative reward.

More formally, iterated expectation under the intervention gives $E_P[S^d]=\int p_P(dw)\,E_P[\sum_t R_t^d\mid W=w]$. For any fixed baseline value, the conditional trajectory distribution factors into its successive conditional distributions. At time $t$, sequential exchangeability makes the counterfactual next-variable distribution under action $d_t(h_t)$ equal to the observed conditional distribution among those with that history and action; consistency identifies the observed variables with the corresponding potential variables. Positivity ensures these conditional distributions are defined on the histories required by the regime. Substituting these factors into the iterated expectation gives the displayed formula. The argument integrates over the joint baseline $W$, not just a selected subset of its components. $\square$

Worked Example: Computing a Two-Stage Value

Let $W$ be Bernoulli with $P(W=1)=0.4$. A regime assigns treatment $0$ at both times. At the first time, $R_1=W$, and $L_2$ is Bernoulli with probability $0.5$, independently of $W$. At the second time, $R_2=L_2$. Treatments have no other effect in this example.

The cumulative outcome is $S^d=W+L_2$. The first reward has expectation $E[W]=1(0.4)+0(0.6)=0.4$. The second has expectation $E[L_2]=1(0.5)+0(0.5)=0.5$. Therefore $$ V_P(d)=E[W+L_2]=0.4+0.5=0.9. $$

The same result follows by listing the joint baseline and intermediate possibilities. Their probabilities and cumulative outcomes are: $(W,L_2)=(0,0)$, probability $0.6(0.5)=0.30$, outcome $0$; $(0,1)$, probability $0.30$, outcome $1$; $(1,0)$, probability $0.4(0.5)=0.20$, outcome $1$; and $(1,1)$, probability $0.20$, outcome $2$. Thus $0(0.30)+1(0.30)+1(0.20)+2(0.20)=0.90$. Both baseline values contribute through $p_P(W)$.

Value Depends on the Target Population

Even with the same regime and the same potential outcomes for every type of person, changing the distribution of types can change the value. For a subgroup indicator $Z$, the law of total expectation gives $$ V_P(d)=\sum_z P(Z=z)E_P[S^d\mid Z=z], $$ when $Z$ has discrete support. Thus population composition matters whenever subgroup-specific expected outcomes differ. For a different target population, one may standardize subgroup-specific expectations to its subgroup proportions, but that calculation requires the conditional outcome mechanisms to be applicable to that target population.

Worked Example: A Change in Population Reverses the Ranking

Consider two baseline types, $Z=1$ and $Z=0$. Under regime $d_A$, cumulative outcomes are $10$ for type 1 and $0$ for type 0. Under regime $d_B$, outcomes are $6$ for both types.

In a population with $P(Z=1)=0.8$, the values are $V(d_A)=0.8(10)+0.2(0)=8$ and $V(d_B)=0.8(6)+0.2(6)=6$. Regime $d_A$ has the higher value by $2$.

In a second population with $P(Z=1)=0.2$, the values are $V(d_A)=0.2(10)+0.8(0)=2$ and $V(d_B)=0.2(6)+0.8(6)=6$. Now $d_B$ has the higher value by $4$. The subgroup-specific outcomes did not change; only the target population’s composition changed. Consequently, a value comparison is not automatically transportable to a population with different composition.

Value Depends on the Outcome Scale

The value also depends on how outcomes are measured or valued. If the cumulative outcome is changed by a positive affine transformation $g(s)=a+bs$, with $b>0$, then the transformed value is $a+bV_P(d)$. This transformation preserves the ranking of regimes when the same $a$ and $b$ are applied to every regime. A nonlinear transformation generally does not have this property: in general, $E[g(S^d)]\ne g(E[S^d])$.

Result: If $E_P[|S^d|]<\infty$ and $g(s)=a+bs$, then $$ E_P[g(S^d)]=a+bV_P(d). $$ For two regimes $d$ and $d'$, applying the same transformation gives $E_P[g(S^d)]-E_P[g(S^{d'})]=b\{V_P(d)-V_P(d')\}$; therefore the sign of their value contrast is preserved when $b>0$.

Proof. By linearity of expectation and the assumed integrability, $E_P[a+bS^d]=a+bE_P[S^d]=a+bV_P(d)$. Subtracting the corresponding identity for $d'$ cancels $a$ and leaves $b\{V_P(d)-V_P(d')\}$. Since $b>0$, multiplication does not change the sign. $\square$

Worked Example: A Nonlinear Scale Can Change the Ranking

Suppose regime $d_A$ produces cumulative outcome $0$ or $10$, each with probability $0.5$, while $d_B$ always produces $4$. On the original scale, $V(d_A)=0.5(0)+0.5(10)=5$, whereas $V(d_B)=4$. Thus $d_A$ has the greater expected cumulative outcome.

Now evaluate the square-root transformation $g(s)=\sqrt{s}$, which is defined for these nonnegative outcomes. The transformed expectations are $E[g(S^{d_A})]=0.5(0)+0.5\sqrt{10}\approx1.581$, and $E[g(S^{d_B})]=\sqrt{4}=2$. On this scale $d_B$ has the greater expected transformed outcome. The calculation compares $E[g(S^d)]$ with $E[g(S^{d'})]$; it does not replace either quantity with $g(E[S^d])$.

Interpretation, Diagnostics, and Alternatives

A value is an estimand, while the g-formula, inverse-probability weighting, and longitudinal targeted maximum likelihood estimation are alternative approaches to estimating it from data. They do not change what $V_P(d)$ means. Reporting a value should therefore specify the target population, the cumulative outcome and its scale, the regime, and the time horizon. A value alone does not state how much better a regime is than another; a contrast must name the comparator.

Several warning signs deserve attention. If a regime selects treatments rarely observed for some histories, positivity may fail or be practically weak, making estimates unstable and dependent on extrapolation. A baseline standardization that omits a pre-treatment variable can yield the wrong value if its distribution is not otherwise accounted for; the variable may be marginalized into the conditional outcome mechanism when that mechanism is correctly specified. A comparison standardized to one population should not be presented as though it applied automatically to another. Finally, if a nonlinear utility transformation is used, the estimand is expected transformed outcome, not transformed expected outcome.

These limitations are substantive as well as statistical. A single expected cumulative outcome can conceal variation across people, timing of benefits and harms, or outcomes not included in the chosen scale. Two regimes can have the same value but very different distributions of outcomes. Value is useful for a clearly defined population and objective, but it does not by itself resolve which outcome scale is ethically or clinically appropriate.

Quantity	Question answered	What must be specified
$V_P(d)$	What is the expected cumulative outcome under regime $d$?	Population, regime, time horizon, and outcome scale
$V_P(d)-V_P(d')$	How do two regimes differ in expected outcome?	The same population and scale for both regimes
$E_P[g(S^d)]$	What is expected outcome after applying transformation $g$?	The transformation and its interpretation

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If a baseline is recorded as $X$ and $L_1$, what distribution must be averaged over in the value formula?
Suppose a regime has expected subgroup outcomes $8$ and $2$. What is its value if the first subgroup has population proportion $0.25$?
Two regimes have values $3$ and $5$. What are their transformed values under $g(s)=2+3s$, and is their ranking preserved?
Why can a value estimated in one population fail to describe a population with different baseline composition?

Solutions. (1) Average over the joint baseline distribution of $W=(X,L_1)$, including both variables. (2) The value is $0.25(8)+0.75(2)=2+1.5=3.5$. (3) The transformed values are $2+3(3)=11$ and $2+3(5)=17$; the second remains larger because the common transformation has positive slope. (4) The population value weights subgroup-specific expected outcomes by subgroup proportions. If those proportions change and subgroup outcomes differ, the weighted average changes as well. A transport claim also requires that the relevant conditional outcome mechanisms apply to the new population.

Quantity	Question answered	What must be specified
\(V_P(d)\)	What is the expected cumulative outcome under regime \(d\)?	Population, regime, time horizon, and outcome scale
\(V_P(d)-V_P(d')\)	How do two regimes differ in expected outcome?	The same population and scale for both regimes
\(E_P[g(S^d)]\)	What is expected outcome after applying transformation \(g\)?	The transformation and its interpretation