Clinically Constrained Treatment Regimes

Why clinical constraints change the decision problem

A fitted Q-function may assign its largest value to an action that a clinician cannot prescribe. A medication may be contraindicated for a particular patient, a dose may exceed a safety limit, or a treatment plan may use more of an individual’s remaining resource than is available. The decision rule must therefore maximize expected outcome over actions that are both clinically admissible and available at that point in the treatment history.

This is not a change to the objective being modeled. It is a change to the set over which the objective is optimized. In a sequential problem, constraints can also change what belongs in the state: if later choices depend on how much of a patient-specific budget remains, that remaining budget must be represented in the history used for later decisions.

Feasible actions and constrained Q-learning

The feasible set may encode patient-specific eligibility, such as excluding a drug when a contraindication is present; a dose interval, such as \(0\leq a\leq d_{\max}(h)\); or a treatment option available only at particular histories. It should describe the actions the clinical protocol allows, not merely the actions that happened to be observed in a data set. Whether observed data are sufficient to estimate the value of those actions is a separate support and identification question.

For a finite horizon ending at stage \(T\), write \(Y\) for the final outcome. In the population defined by the causal decision problem, define the terminal Q-function as the expected outcome under action \(a\) at the terminal history:

For earlier stages, use the optimal continuation value at the next stage:

These equations specify the decision recursion, not a particular estimation method. They presume the conditional laws under interventions and the expectations used in the recursion are well-defined. At each stage, the maximization is over the clinically permitted actions, not over a larger unconstrained action space.

Eligibility constraints in a categorical decision

With categorical treatment, clinical restrictions can remove options rather than alter a dose interval. The Q-values still compare expected outcomes on a common scale; the feasible set determines which comparisons are relevant to the recommendation.

Backward maximization and measurable rules

Pointwise maximization is not alone enough to establish that a valid regime exists. An action attaining the maximum at each history must also be selectable as a measurable function of history. The following sufficient conditions make that requirement explicit.

The compactness and continuity assumptions are convenient sufficient conditions, not the only possible ones. More general measurable-selection results can be used when their hypotheses hold. Nonempty feasible sets and attained pointwise maxima by themselves do not guarantee a measurable regime.

Proof. Define \(V_T(h)=\max_{a\in\mathcal{A}_T(h)}Q_T(h,a)\). The compact feasible sections and continuity in \(a\) ensure that this maximum is attained; the measurable maximum theorem gives a measurable maximizing selector \(g_T^*\). Recursively for \(t<T\), define \(Q_t\) using the conditional law of \(H_{t+1}\) and the already defined \(V_{t+1}\), and set \(V_t(h)=\max_{a\in\mathcal{A}_t(h)}Q_t(h,a)\). By the stated recursive measurability and maximum-theorem conditions, this maximum is attained and has a measurable selector \(g_t^*\).

To compare regimes, let \(g\) be any measurable feasible regime. At the terminal stage, for each relevant \(h\), \(Q_T(h,g_T(h))\leq V_T(h)=Q_T(h,g_T^*(h))\). Now suppose the optimal continuation value \(V_{t+1}\) is at least the conditional expected outcome under \(g\)'s continuation, at every relevant next history. Conditional on \(H_t=h\) and setting \(A_t=g_t(h)\), taking expectations of that inequality gives an expected outcome no larger than \(Q_t(h,g_t(h))\). Since \(g_t(h)\in\mathcal{A}_t(h)\), \(Q_t(h,g_t(h))\leq V_t(h)\). By construction, choosing \(g_t^*(h)\) attains \(V_t(h)\), and its continuation uses the maximizing selectors at all later stages. Backward induction therefore shows that the conditional expected outcome under \(g\) is no greater than under \(g^*\) at the initial history. Averaging over the initial-history distribution preserves the inequality, so the regime \(g^*\) has expected outcome at least as large as every feasible \(g\). \(\square\)

Resource limits across treatment stages

A resource restriction that applies across a patient’s stages is not generally a fixed action cap at each stage. If current treatment changes what can be given later, the remaining resource must be part of the state. Backward Q-learning can then optimize subject to the budget by including that state in the history and restricting each next action to what remains feasible.

A shared resource constraint across different patients is different. If one patient’s treatment uses a pool that limits what can be assigned to others, their decisions are coupled. Optimizing each patient’s Q-function independently need not satisfy the population-level budget or optimize the constrained aggregate objective. Such a problem needs a joint allocation formulation, or an appropriate augmented resource state or constrained optimization method; it is not solved simply by applying an individual feasible-set restriction to each patient.

Diagnostics, alternatives, and limitations

A constrained decision rule should be checked for both clinical validity and optimization validity. Useful checks include the following:

• Feasibility audit. For histories used to make recommendations, verify that the selected action belongs to the documented feasible set, including boundary rules and patient-specific exclusions.
• Unrestricted-versus-restricted comparison. Record whether the unrestricted maximizing action was excluded and which admissible action replaced it. A large difference in modeled Q-value can signal a consequential constraint or a model scale problem worth examining.
• Resource accounting. For a cumulative individual budget, check that every recommended sequence respects the total limit, not just each stage’s separate dose cap.
• Support and model checks. A clinically feasible action may still be poorly represented in the observed data. Sparse observations or unstable fitted Q-values near a constraint boundary can make recommendations unreliable even when the rule is mathematically feasible.
• Measurability and implementation. Confirm that the decision can be computed from information available at the decision time and that the feasible set and selector are well-defined as history changes.

The main alternative is unrestricted Q maximization, which is appropriate only when every action in the optimization domain is genuinely allowable. Clipping an unconstrained continuous recommendation to a dose limit can agree with constrained maximization for a concave, unimodal Q-function on an interval, as in the quadratic example. It is not a general substitute: with a nonconcave function, an irregular feasible set, or history-dependent options, the best feasible action need not be obtained by simple clipping. Explicitly maximizing over \(\mathcal{A}_t(h)\) remains the general formulation.

Clinical constraints also do not repair a misspecified Q-model, guarantee adequate data support for each allowed action, or resolve conflicts between a patient’s individual optimum and a shared population resource target. The recommendation is optimal relative to the modeled outcome, the stated feasible sets, and the assumptions that justify the Q-functions. Those qualifications should accompany interpretation of any constrained rule.