Continuous Treatments And Constrained Q Maximization

From a Q-Function to a Continuous Action

For a continuous treatment, the action is a number—such as a dose, intensity, or duration—rather than one of a few labels. The decision problem is therefore an optimization problem: at each observed history, choose the feasible action with the greatest expected utility. The optimum may lie inside the treatment range, at a boundary, or at a boundary created by a clinical restriction.

Let $H$ denote the patient history available at a decision point, and let $a$ denote the treatment action. Use a reward convention, so larger values are preferred. At a terminal treatment stage, a Q-function can be written as the conditional expected outcome under action $a$. At an earlier stage, it can represent the expected outcome under $a$ followed by a specified continuation rule. The action chosen here maximizes this Q-function; the continuation rule is held fixed when comparing actions.

Definition: For history $h$, let $\mathcal{A}(h)$ be the set of actions permitted at that history, and let $Q(h,a)$ be the expected utility associated with taking action $a$ at $h$ and then following the specified continuation. An optimal action is any $$ a^*(h)\in\operatorname*{arg\,max}_{a\in\mathcal{A}(h)} Q(h,a). $$ If the maximizer is unique, $a^*(h)$ denotes that single action. Otherwise, the argmax is a set of tied maximizing actions.

The feasible set matters. A general dose range might be $0\leq a\leq 10$, while a patient's renal function or a concomitant medication may impose a smaller permitted range. A contraindication can also remove an interval of doses, leaving a disconnected set. Such restrictions belong in $\mathcal{A}(h)$ before optimization begins.

The interpretation of $Q$ as a causal expected outcome requires the relevant causal identification assumptions, including consistency and sequential exchangeability; estimating it from observational data also requires adequate treatment support for the actions being compared. This tutorial takes the Q-function as given and focuses on maximizing it. In particular, a numerical maximizer outside the data-supported range may be an extrapolation rather than a credible treatment recommendation.

Existence and Shape of the Maximizer

A maximizer is guaranteed under a familiar set of conditions. For a fixed history $h$, suppose $\mathcal{A}(h)$ is nonempty and compact, and $Q(h,a)$ is continuous in $a$ on that set. Then the extreme value theorem guarantees that $Q(h,a)$ attains a maximum. Compactness includes closed treatment bounds: an open interval such as $(0,1)$ need not contain its supremum.

Continuity and compactness guarantee existence, but not uniqueness. If $Q(h,\cdot)$ is strictly concave on a convex feasible interval, its maximizer is unique. For a differentiable concave function on $[L,U]$, the derivative helps locate the maximum: an interior optimum must have derivative zero; if the function is still increasing at $U$, the maximum is at $U$; if it is already decreasing at $L$, the maximum is at $L$. The boundary checks cannot be skipped.

Result: Suppose $Q(h,a)=c(h)+b(h)a-\frac{k}{2}a^2$, where $k>0$, and the allowed actions are the closed interval $[L,U]$, with $L\leq U$. Then the unique maximizing action is $$ a^*(h)=\min\!\left\{U,\max\!\left\{L,\frac{b(h)}{k}\right\}\right\}. $$

To derive this rule, complete the square, suppressing $h$ temporarily:

c+ba-\frac{k}{2}a^2 = c+\frac{b^2}{2k} -\frac{k}{2}\left(a-\frac{b}{k}\right)^2.

The first two terms do not depend on $a$, and $k/2>0$. Thus maximizing $Q$ is equivalent to minimizing the squared distance from $a$ to $b/k$. If $b/k\in[L,U]$, the distance is zero there. If $b/k<L$, distance increases as $a$ moves to the right from $L$, so $L$ is closest. If $b/k>U$, $U$ is closest. These cases give the stated clipped value. The strictly positive quadratic coefficient also makes the maximizer unique. $\square$

Worked Example: A Quadratic Q-Function With History-Dependent Optima

Suppose the available doses are $a\in[0,2]$ and $Q(h,a)=8+(3-2h)a-\frac{1}{2}a^2$. Find the optimal dose at $h=0$, $h=1$, and $h=2$.

For a fixed $h$, differentiate with respect to $a$: $Q_a(h,a)=3-2h-a$. The unconstrained stationary point is $a=3-2h$, and the quadratic coefficient is negative, so the function is strictly concave. Applying the bounded-interval rule gives $a^*(h)=\min\{2,\max\{0,3-2h\}\}$.

At $h=0$, the stationary point is 3, beyond the upper bound. The derivative is $3-a$, which is positive throughout $[0,2]$. Therefore the maximum is at $a=2$. Checking the endpoints with the history held fixed, $Q(0,0)=8$ and $Q(0,2)=8+6-2=12$, so the upper-bound action gives the larger value.

At $h=1$, the stationary point is $a=1$, which is feasible. Its value is $Q(1,1)=8+(3-2)1-\frac12=8.5$. The unique optimum is $a^*(1)=1$.

At $h=2$, the stationary point is $-1$, below the allowed range. The derivative is $-1-a<0$ for every $a\in[0,2]$, so $Q$ decreases over the interval and the optimum is $a^*(2)=0$, with $Q(2,0)=8$. The same treatment range thus yields different optimal actions at different histories.

Imposing Clinical Bounds and Restrictions

A clinical constraint changes the set over which the maximization is performed; it does not change the definition of a maximizer. For example, if a dose must be no greater than a safety ceiling $U(h)$, optimize over $[0,U(h)]$. If a minimum effective dose $L(h)$ is required whenever treatment is given, the choice may instead be $a=0$ or $a\in[L(h),U(h)]$. The latter set is not a single interval, so one should optimize on each permitted piece and compare the resulting values.

This distinction also separates a hard constraint from a penalty. A hard constraint removes disallowed actions from the candidate set. A penalty subtracts a cost from $Q$ and can leave an action available while making it less attractive. Unless that cost represents the intended utility, replacing a clinical prohibition with a penalty solves a different problem.

Worked Example: A Safety Ceiling Creates a Boundary Optimum

Suppose $Q(h,a)=10+5a-a^2$, with $a$ measured in dose units. The unconstrained treatment range is $[0,4]$, but a safety rule for the patient requires $a\in[0,2]$. Find and verify the constrained optimum.

The derivative is $5-2a$, so the unconstrained stationary point is $a=2.5$. It is outside the clinically permitted set. On $[0,2]$, the derivative is positive, since $5-2a\geq1>0$; hence $Q$ increases throughout the allowed interval and the constrained optimum is its upper endpoint, $a^*=2$.

The values confirm the distinction between the unconstrained and constrained problems. At the permitted endpoints, $Q(0)=10$ and $Q(2)=10+10-4=16$. The unconstrained vertex has $Q(2.5)=10+12.5-6.25=16.25$, but that dose violates the safety rule and is not a candidate. The best permitted action is therefore 2, with value 16.

Worked Example: A Contraindication Creates a Disconnected Feasible Set

Let the treatment range be $[0,3]$, but suppose doses strictly between 1 and 2 are contraindicated. Thus $\mathcal{A}=[0,1]\cup[2,3]$. For $Q(a)=6-(a-0.5)^2$, find the global maximizer over this set.

On $[0,1]$, the squared distance from $a$ to $0.5$ is minimized at $a=0.5$, which is permitted. It gives $Q(0.5)=6$, the largest value on that interval.

On $[2,3]$, every permitted action is to the right of $0.5$, so the distance is smallest at $a=2$. That interval's maximum is $Q(2)=6-(1.5)^2=3.75$. Comparing the two interval maxima, $6>3.75$, so the global maximizer is $a^*=0.5$. Optimizing only on $[0,3]$ would happen to yield the same answer here, but that shortcut is not generally valid: a global unconstrained optimum can fall in a prohibited region, requiring comparison of the permitted pieces.

How to Check a Proposed Action

A reliable calculation makes the candidate set and the boundary checks explicit. For a smooth, strictly concave Q-function on one closed interval, solve the derivative equation if possible, then compare the resulting stationary point with the interval endpoints. For more complicated or disconnected feasible sets, identify each permitted piece, find its best action, and compare the corresponding Q-values. The flow below is a useful calculation order.

Specify the feasible set.
Write the allowed dose range and any history-specific restrictions before differentiating.

Find interior candidates.
For a differentiable Q-function, solve $Q_a(h,a)=0$ within each permitted interval and check the function's shape.

Check boundaries and compare.
Evaluate Q at relevant permitted endpoints and stationary points; select the candidate with the largest value.

Several diagnostics follow directly. A reported action outside $\mathcal{A}(h)$ is infeasible, regardless of its modeled Q-value. For a concave quadratic on an interval, an alleged interior optimum that does not solve $Q_a=0$, or a boundary choice inconsistent with the derivative sign, indicates an algebra or coding error. For a nonconcave Q-function, a derivative root may be a local minimum or only a local maximum, so evaluate all relevant candidates rather than trusting one root-finding result.

Model support is a separate concern from mathematical feasibility. A dose can satisfy the clinical range yet have little or no observational support for patients with history $h$. In that case the fitted Q-function may be extrapolating, and its maximizing action can be unstable to modest changes in the model. Plotting fitted $Q(h,a)$ over the permitted range, marking observed dose support and the selected action, can expose this problem. A smooth curve or a precise numerical optimizer does not itself establish that the curve is credible.

Compared with categorical treatment decisions, continuous optimization cannot generally be done by checking a short list of treatment labels; it requires analyzing the shape of Q over a range. Compared with unconstrained optimization, constrained maximization respects the actual clinical action set and can force the best available choice to a boundary. Neither comparison changes the need for an adequate Q-model. Misspecification can distort the curve and hence the selected action, while a valid constrained optimum remains only as credible as the Q-values used to compute it.

Key takeaway: Optimize the Q-function over the actions that are actually permitted. For a concave quadratic, the unconstrained vertex is clipped to the allowed interval; with more complicated restrictions, optimize on every feasible piece and compare values. Always verify feasibility, boundary behavior, and support.

Check Your Understanding

Work each problem by identifying the feasible set and checking any boundary candidates. Short solutions follow the questions.

For $Q(a)=4+6a-2a^2$ on $[0,2]$, find the maximizing action and its Q-value.
For $Q(a)=9+2a-a^2$ on $[0,3]$, find the maximizing action and explain why checking the endpoints alone would not identify it.
For $Q(a)=5-(a-1)^2$ with feasible set $[0,0.5]\cup[1.5,2]$, find the global maximizing action.
State two assumptions that guarantee a maximum exists on a feasible action set, and one condition that gives uniqueness.

Solutions. For 1, complete the square or differentiate: $Q_a=6-4a=0$ gives $a=1.5$, which lies in $[0,2]$. The value is $Q(1.5)=4+9-4.5=8.5$. For 2, $Q_a=2-2a=0$ gives $a=1$, and $Q(1)=10$; endpoint values are $Q(0)=9$ and $Q(3)=6$. The maximum is interior, so endpoint checks alone would miss it. For 3, on $[0,0.5]$ the closest feasible point to 1 is 0.5, giving $Q=5-0.25=4.75$. On $[1.5,2]$ the closest is 1.5, also giving 4.75. Both 0.5 and 1.5 are global maximizers. For 4, a nonempty compact feasible set and a continuous Q-function guarantee that a maximum is attained. Strict concavity on a convex feasible interval guarantees uniqueness.

Continuous Treatments And Constrained Q Maximization

What You'll Learn