Regret and Opportunity Loss in Dose Decisions

Why Measure Regret?

A dosing policy can have high average utility and still make consequentially poor decisions for some patients. Regret measures the utility lost when a policy’s recommendation is compared with the best decision available for that same patient. It therefore answers a patient-level question: how much better could the decision have been, given the chosen utility and the feasible dose options?

This is a benchmark comparison, not a claim that a real clinician or algorithm could always identify the best dose for an individual. The benchmark is often an idealized, patient-specific decision that uses information unavailable to the policy. That makes regret useful for describing opportunity loss, but it also means the benchmark and its assumptions must be stated clearly.

Definition: Let $x$ denote a patient’s relevant characteristics, $A(x)$ the set of doses feasible for that patient, and $u(a,x)$ the utility of dose $a$ for patient $x$, with larger utility preferred. For a policy $\pi$ recommending dose $\pi(z)$ using available information $z$, patient-level regret is $$ R(x,z;\pi)=\max_{a\in A(x)}u(a,x)-u(\pi(z),x), $$ provided $\pi(z)\in A(x)$. The maximum is the utility of the best feasible dose for that patient. Regret is the resulting utility difference, also called opportunity loss.

In dosing evaluation, $u(a,x)$ will often be a conditional expected utility: the expected clinical value of dose $a$ for a patient with characteristics $x$, integrating over uncertain outcomes such as efficacy and toxicity. Using expected utility rather than one realized outcome keeps random outcome variation from being mistaken for a policy’s decision quality. The utility scale may be a defined clinical score, a monetary value, or another explicitly specified scale. Regret has the same units as that utility.

The benchmark is patient-specific: the maximizing dose can differ across patients. It is also constrained: it maximizes utility only over $A(x)$, not over every imaginable dose. A feasible set might reflect dose options, contraindications, or other constraints already specified for the evaluation. A recommendation outside the set is not simply a high-regret recommendation; it violates the policy’s feasibility requirement and should be reported as such.

Calculating Patient-Level and Summary Regret

For each patient, calculate the best utility among feasible doses, subtract the utility of the policy’s recommendation, and retain the patient’s regret. Since the recommendation is itself feasible, its utility cannot exceed the maximum over the same set. Regret is therefore nonnegative. A patient with regret zero receives a dose tied for the best feasible utility; a positive value is the utility difference from that benchmark.

For a target population of patients, mean regret is the population average of patient-level regret. In a sample of $n$ patients with equal weights, its estimate is the arithmetic mean. If patients represent different proportions of the target population, use stated nonnegative weights that sum to one. Worst-case regret is the largest regret over a defined patient set or population support. A maximum from a finite evaluation sample is an observed sample maximum, not automatically the population’s true worst case.

\text{Mean regret}=\mathbb{E}\{R(X,Z;\pi)\}, \qquad \widehat{\text{Mean regret}}=\frac{1}{n}\sum_{i=1}^{n}R_i, \qquad \text{Observed worst-case regret}=\max_{1\leq i\leq n}R_i.

Worked Example: Regret for Three Hypothetical Patients

Suppose three hypothetical patients can receive 50, 75, or 100 mg. Utilities are in hypothetical clinical utility points, with larger values preferred. A policy recommends 75 mg for Patients 1 and 3, and 100 mg for Patient 2.

Patient	Utility at 50 mg	Utility at 75 mg	Utility at 100 mg	Policy dose
1	8	10	7	75 mg
2	6	9	8	100 mg
3	5	7	11	75 mg

For Patient 1, the best feasible utility is 10 points at 75 mg. The policy also gives 10 points, so regret is $10-10=0$ points. For Patient 2, the best utility is 9 points at 75 mg, whereas the policy’s 100 mg recommendation gives 8 points. Regret is $9-8=1$ point. For Patient 3, the best utility is 11 points at 100 mg, while the policy’s 75 mg recommendation gives 7 points. Regret is $11-7=4$ points.

The equal-weight mean regret is $$ \frac{0+1+4}{3}=\frac{5}{3}=1.667\text{ utility points per patient}. $$ The observed worst-case regret among these three patients is 4 utility points, for Patient 3. The mean describes average loss in this evaluated group; the maximum identifies its largest individual loss. Neither number alone describes the distribution of regret in a broader population.

Why Utility, Not Dose Distance, Determines Regret

Regret measures a difference in utility, not a difference in milligrams from an optimal dose. Two doses equally far from the best dose can cause very different opportunity losses. In particular, when utility penalizes underdosing and overdosing differently, equal dose deviations need not produce equal regret. This is one reason to define the utility scale and its clinical interpretation before calculating regret.

Worked Example: Equal Dose Deviations, Unequal Regret

For a hypothetical patient, suppose the feasible doses are 50, 75, and 100 mg, with utilities of 96, 100, and 70 points, respectively. The best feasible dose is 75 mg, with utility 100 points. Consider two recommendations, each 25 mg from that dose.

If the policy recommends 50 mg, its regret is $100-96=4$ utility points. If it recommends 100 mg, its regret is $100-70=30$ utility points. The two recommendations are equally distant from 75 mg, but the second gives up 26 more utility points. Thus, dose-distance summaries cannot substitute for utility-based regret when clinical consequences are asymmetric.

The numbers are hypothetical utility points, not measured clinical outcomes. Their interpretation depends on how the utility scale was defined; without that definition, the arithmetic is possible but the clinical meaning of a 4- or 30-point loss is not established.

Derivation: Mean Regret and Expected Utility

The link between average regret and policy utility follows directly from the definition. Write $u^*(X)=\max_{a\in A(X)}u(a,X)$ for the patient-specific benchmark. For a policy that recommends a feasible dose, regret equals $u^*(X)-u(\pi(Z),X)$. Taking expectations over the same target population gives

\begin{aligned} \mathbb{E}\{R(X,Z;\pi)\} &=\mathbb{E}\{u^*(X)-u(\pi(Z),X)\}\\ &=\mathbb{E}\{u^*(X)\}-\mathbb{E}\{u(\pi(Z),X)\}. \end{aligned}

The second equality uses linearity of expectation. The patient-specific benchmark is fixed across policies being compared, as long as they use the same utility definition, feasible sets, and target population. Consequently, for policies $\pi_1$ and $\pi_2$,

\mathbb{E}\{R(X,Z;\pi_1)\}-\mathbb{E}\{R(X,Z;\pi_2)\} = \mathbb{E}\{u(\pi_2(Z),X)\}-\mathbb{E}\{u(\pi_1(Z),X)\}.

Thus, the policy with greater expected utility has lower mean regret under these shared conditions. This identity does not say that the policies have the same patient-level pattern of loss. It also does not imply that the policy with lower mean regret has lower worst-case regret.

Worked Example: Mean and Worst-Case Regret Can Rank Policies Differently

Consider four hypothetical patients and two policies, evaluated using the same utility scale, feasible doses, and patient-specific benchmarks. Suppose Policy A has regrets of 0, 0, 0, and 8 points; Policy B has regrets of 3, 3, 3, and 3 points.

Policy A’s mean regret is $(0+0+0+8)/4=2$ points, and its observed worst-case regret is 8 points. Policy B’s mean regret is $(3+3+3+3)/4=3$ points, and its observed worst-case regret is 3 points. Mean regret favors Policy A by 1 point per patient, while worst-case regret favors Policy B by 5 points.

Neither summary is erroneous: they answer different questions. Policy A has lower average opportunity loss in this group, but one patient has a larger loss. Policy B has a smaller maximum loss, at the cost of greater average loss. A policy comparison should state which summary is being used rather than implying that one ranking answers every decision question.

Assumptions, Diagnostics, and Limitations

Regret is only as meaningful as the benchmark and utility calculation behind it. Before interpreting a result, verify the following conditions:

Common utility definition: Utilities for all candidate doses and policies use the same scale and clinical trade-offs. A difference in points is interpretable only in the context of that scale.
Correct feasible set: The maximization and policy recommendation use the same patient-specific constraints. If safety or other feasibility rules apply, they must be reflected in $A(x)$.
Valid utility inputs: The utility values correspond to the patient characteristics and outcomes being evaluated. If they are estimated, estimation uncertainty may affect the benchmark and the regret.
Defined target population: Mean and worst-case summaries require a stated evaluation group or population support. A sample average depends on who was included and how they were weighted.

Practical checks can reveal calculation or interpretation problems. A negative regret usually signals a mismatch: perhaps the benchmark was maximized over a different set of doses, utilities were mixed across scales, or a subtraction was reversed. Check that every policy recommendation is feasible before calculating regret. Ties for the maximum should produce zero regret for any tied dose.

Worst-case regret is particularly sensitive to the evaluated set. A single extreme utility value, a sparse sample, or an expanded set of possible patient profiles can change the maximum substantially. Report whether it is a sample maximum or a maximum over a specified supported set, and examine the patient and dose that produce it. If utility values or patient profiles are simulated or estimated, numerical error and uncertainty can also affect the reported maximum.

Regret has useful invariance properties, but they do not remove the need for a meaningful utility scale. Adding the same constant to every dose utility for a patient leaves regret unchanged, because the constant cancels in the difference. Multiplying every utility by the same positive constant multiplies all regrets by that constant; policy rankings by mean regret are preserved, but the reported units change. Comparing policies with different utility definitions is not valid merely because their regret numbers look similar.

Regret also has limitations. The patient-specific maximum is an oracle benchmark if it relies on information the policy cannot observe, so a nonzero regret need not indicate an avoidable operational error. If the utility model is misspecified, the benchmark can be misleading. And a mean or maximum compresses a distribution of patient-level losses: two policies with the same mean may affect different patients. Regret complements, rather than replaces, reporting the underlying utility choices, feasibility rules, and distribution of patient-level results.

Key takeaway: Patient-level regret is the utility of the best feasible dose for that patient minus the utility of the policy’s feasible recommendation. Mean regret summarizes average opportunity loss; worst-case regret identifies the largest loss in a defined evaluation set. Both depend on a common, defensible utility scale and benchmark.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A patient’s feasible doses have utilities 4, 9, and 7 points. A policy recommends the dose with utility 7. What is the patient’s regret?
Why must a patient-specific benchmark maximize utility over the feasible dose set rather than over all conceivable doses?
Two policies have equal mean regret, but one has greater observed worst-case regret. What can be concluded about their average and largest observed losses?
For every patient, the same constant $c$ is added to the utility of every feasible dose. Show whether regret changes.

Solutions. (1) The maximum utility is 9 points, so regret is $9-7=2$ points. (2) The benchmark is defined as the best decision the patient could feasibly receive; an unavailable or prohibited dose is not a valid opportunity for that patient. (3) The policies have the same average opportunity loss in the evaluation set, but the policy with the greater observed maximum has a larger loss for at least one evaluated patient. This does not establish which has the smaller population maximum. (4) Regret is unchanged: $(\max_a u(a,x)+c)-(u(\pi(z),x)+c)=\max_a u(a,x)-u(\pi(z),x)$, because the same added constant cancels.