Safety-Constrained Bandit Allocation

Safety Is a Constraint, Not a Reward

A bandit allocation rule can favor treatments that have produced greater clinical benefit, but reward alone does not capture whether their adverse outcomes are acceptable. A safety-constrained formulation makes the distinction explicit: first specify which allocations meet a safety requirement, then maximize expected reward among those allocations. A policy with the highest expected reward is not acceptable under this formulation if it violates the safety constraint.

The definitions below use a fixed set of \(K\) treatment arms. The reward \(R_t\) is coded so that larger values represent greater clinical benefit, following the earlier material on defining rewards for clinical benefit. Let \(Z_t=1\) indicate an adverse outcome for patient \(t\), and \(Z_t=0\) otherwise. For a simple stationary model, treatment arm \(a\) has expected reward \(\mu_a=E[R_t\mid A_t=a]\) and adverse-outcome risk \(q_a=P(Z_t=1\mid A_t=a)\). The adverse outcome is binary here; a more detailed harm measure can replace \(Z_t\) if its scale and safety limit are defined appropriately.

Two Forms of an Expected-Safety Constraint

A per-assignment constraint limits the expected risk for each enrollment. If the requirement is that the expected risk for the next patient be no more than \(\tau\), then an arm selected for that patient must satisfy \(q_a\leq\tau\). A cumulative constraint instead limits the expected number of adverse outcomes over a horizon of \(T\) patients, for example \(E[\sum_{t=1}^{T} Z_t]\leq B\), where \(B\) is a prespecified expected-outcome budget. Equivalently, an expected average-risk limit is \(E[\sum_{t=1}^{T}Z_t]/T\leq\tau\). These are different design requirements; the protocol must say which one is intended.

For a fixed allocation in which a proportion \(x_a\) of patients receive arm \(a\), the proportions satisfy \(x_a\geq0\) and \(\sum_{a=1}^{K}x_a=1\). Under the stationary arm-specific model, its expected reward per patient is \(\sum_a x_a\mu_a\), and its expected adverse-outcome rate is \(\sum_a x_aq_a\). Thus an average-risk formulation is the constrained optimization problem

This formulation permits a treatment whose risk exceeds \(\tau\) to receive some allocation if its use is offset by allocation to lower-risk treatments and the average constraint is still met. That may be suitable for an explicitly average-risk policy, but it is not the same as requiring every treatment used to satisfy \(q_a\leq\tau\). A per-assignment limit rules out any arm above the limit; an average-risk limit need not.

Feasibility Comes Before Optimization

Feasibility asks whether any allowed allocation satisfies the safety requirement. With unrestricted allocation shares and a single average-risk constraint, feasibility has a simple condition: at least one arm must have \(q_a\leq\tau\). To see why, every weighted average \(\sum_a x_aq_a\) is at least \(\min_a q_a\), since each \(q_a\geq\min_j q_j\) and the nonnegative weights sum to one. Conversely, assigning everyone to an arm with minimum risk achieves risk \(\min_a q_a\). Therefore the feasible set is nonempty exactly when \(\min_a q_a\leq\tau\). Additional requirements, such as minimum allocation shares, can change this condition.

Once feasibility has been established, reward is optimized over the feasible set. The safety requirement is not automatically a penalty that can be traded away for enough reward. A penalized objective such as expected reward minus a multiple of expected harm may prefer an unsafe allocation for some penalty values. It is not a substitute for the stated hard constraint unless the penalty and optimization method are specifically shown to enforce that constraint.

Sequential Policies and Expected Cumulative Harm

A bandit policy can choose treatments using the observed history, so the arm assigned at time \(t\), denoted \(A_t\), may vary with previous assignments and outcomes. Under the stationary model, if the conditional adverse-outcome risk for arm \(a\) is \(q_a\), then \(E[Z_t\mid A_t,\text{history before outcome }t]=q_{A_t}\). By the rule of iterated expectation, \(E[Z_t]=E[q_{A_t}]\). Linearity of expectation then gives the expected total harm under policy \(\pi\):

This identity does not require adverse outcomes to be independent across patients. It does require the stated conditional risks to describe the outcomes generated by the policy. If risk changes over time, depends on patient characteristics, or is affected by earlier treatment, a fixed arm-specific \(q_a\) may not be an adequate model; the expected harm must then be defined using the relevant conditional risks.

Uncertainty, Diagnostics, and Limits

In an actual bandit problem, the risks \(q_a\) are not known at the start. A constraint computed from estimated risks is only as reliable as those estimates and the outcome model. If risks are estimated using observed data, an arm with few assignments may have substantial uncertainty even when its estimated risk is below the limit. If a Bayesian model is used, a constraint on posterior expected risk is a statement about the risk averaged over that model's current uncertainty; it is not automatically a guarantee that the true risk is below the limit. The analysis should state which quantity is constrained and how uncertainty is handled.

Useful diagnostics include checking the number of observed outcomes by arm, comparing observed adverse-outcome rates with the risk values used by the policy, and reviewing whether outcomes are delayed or incompletely recorded. Very sparse arm data, unexpected excess harm, changes in patient mix, or evidence that risk changes over time are reasons to question a stationary risk model. A safety calculation can also fail if the adverse outcome is defined inconsistently or if relevant harms are omitted from the outcome measure. Monitoring can identify warning signs, but it does not make an incorrectly specified expected-risk constraint protective.

An expected-risk constraint should not be described as a guarantee that each patient is safe or that the realized total number of adverse outcomes will stay below a budget. A chance constraint, such as \(P(\sum_t Z_t>B)\leq\delta\), instead limits the probability of exceeding a specified total; it addresses tail risk rather than only the expectation. These are distinct criteria and generally require different calculations. Likewise, an average constraint does not enforce a separate per-arm risk cap. The appropriate criterion depends on the clinical and design requirement, not just on which is easiest to optimize.

The formulation is also limited by its assumptions. Fixed arm-specific means and risks may not represent treatment-by-patient differences, time trends, or effects of treatment history. A reward that combines benefit and harm into one score can obscure a safety requirement unless a separate harm constraint is retained. Finally, mathematical feasibility is conditional on the model and the allowed allocations: it is not a clinical judgment that the feasible treatments are acceptable for every patient.

Common Mistakes

• Choosing the highest-reward arm first. The best unconstrained arm may be infeasible. Check the safety criterion before comparing reward within the feasible set.
• Treating an expected limit as a realized guarantee. An expected total of at most \(B\) does not mean the observed total cannot exceed \(B\).
• Confusing average risk with a per-arm cap. A mixture can meet an average-risk constraint even when one included arm exceeds the average limit; that does not make the arm individually compliant.
• Calling an estimated constraint a guarantee. Plug-in or posterior-average calculations depend on the risk model and data. They do not establish the true risk without additional assumptions or uncertainty control.
• Optimizing before checking feasibility. If no allowed allocation satisfies the safety requirement, there is no reward-maximizing feasible allocation; the constraint or design must be reconsidered.