Patient-Level Heterogeneity in Policy Benefit

Why an Average Can Hide Patient-Level Losses

When comparing two dosing policies, an average utility difference answers an important question: which policy has greater expected utility in the target population? But it does not answer a different question: how evenly is that benefit distributed across patients? A policy can improve average utility while having lower predicted utility for some individuals. In a clinical decision, the size and frequency of those losses matter alongside the population average.

The focus here is the distribution of predicted individual utility differences, not a new way to choose a policy from its average utility. The earlier material on expected utility defines the population-level comparison; the earlier material on subgroup-specific utility and harm considers variation between groups. Patient-level heterogeneity asks whether predicted benefit varies among individual patients, including among patients who share the same recorded subgroup.

Definition: Let \(U_i^A\) and \(U_i^B\) denote patient \(i\)'s predicted expected utility under dosing policies \(A\) and \(B\), on the same utility scale. Define the patient-level predicted utility contrast as \(\Delta_i=U_i^A-U_i^B\). A positive \(\Delta_i\) favors policy \(A\), a negative \(\Delta_i\) favors policy \(B\), and zero indicates equal predicted utility. These are model-based contrasts for the same patient under alternative policies, not outcomes observed under both policies.

Utility may combine multiple outcomes, such as target attainment and toxicity, using a prespecified value scale. A contrast of \(+5\) utility points means that policy \(A\) has five more predicted utility points than policy \(B\) for that patient under the stated model and utility definition. It is not automatically a five-percentage-point improvement or a clinical benefit of a particular magnitude. That interpretation depends on the scale and on whether a difference of five points is meaningful.

Define the Contrast Before Summarizing It

A patient-level comparison is meaningful only if both policy predictions refer to the same patient, outcome horizon, utility definition, and information available at the time of decision. For a target population with patients indexed by \(i=1,\ldots,n\), the empirical average contrast is \(\bar{\Delta}=n^{-1}\sum_{i=1}^n\Delta_i\). If the patients represent a target population with unequal sampling or transport weights, the average should instead use the prespecified weights and be interpreted for that target population.

The average should be accompanied by summaries that reveal the distribution. Useful quantities include the median, a spread measure such as the sample standard deviation or interquartile range, selected quantiles, and the proportion with \(\Delta_i<0\). If a clinically meaningful loss threshold \(\delta>0\) has been specified on the utility scale, also report the proportion with \(\Delta_i<-\delta\). The threshold must be justified on that scale; it should not be chosen after seeing which value makes a policy look safest.

\bar{\Delta}=\frac{1}{n}\sum_{i=1}^{n}\Delta_i,\qquad \widehat{q}_{\mathrm{harm}}=\frac{1}{n}\sum_{i=1}^{n} I(\Delta_i<0),\qquad \widehat{q}_{\mathrm{meaningful\ loss}}=\frac{1}{n}\sum_{i=1}^{n} I(\Delta_i<-\delta)

Here \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The first quantity summarizes average predicted utility change; the second is the empirical fraction of patients for whom the contrast favors \(B\); the third is the fraction with a loss exceeding the prespecified threshold. These proportions describe the evaluated set unless an inferential procedure supports generalization to a broader population.

Key distinction: A negative predicted contrast means the model assigns lower expected utility to policy \(A\) than to policy \(B\) for that patient. It does not establish that the patient would experience a bad outcome under \(A\), nor does it prove that the prediction is correct. The contrast is a prediction about expected utility, conditional on the model, data, and utility specification.

Worked Examples

Worked Example: Positive Average Benefit With Individual Losses

Hypothetical data. A model produces predicted utility contrasts, in utility points, for eight patients when comparing policy \(A\) with policy \(B\): \(8, 6, 4, 3, 2, 1, -5, -7\). Assume all eight predictions use the same utility definition and horizon. Summarize the average, spread, median, and fraction with negative predicted benefit.

The sum is \(8+6+4+3+2+1-5-7=12\), so the mean contrast is \(\bar{\Delta}=12/8=1.50\) utility points. On average in this set, policy \(A\) has 1.50 more predicted utility points per patient than policy \(B\).

Two of the eight contrasts are negative, so \(\widehat{q}_{\mathrm{harm}}=2/8=0.25\), or 25%. This means that one quarter of these patients have lower predicted expected utility under \(A\) than under \(B\); it does not mean that one quarter will necessarily experience a harmful clinical outcome.

In sorted order the contrasts are \(-7,-5,1,2,3,4,6,8\). The median is \((2+3)/2=2.50\) utility points. For the sample standard deviation, the sum of squared contrasts is \(64+36+16+9+4+1+25+49=204\). Thus \(s_{\Delta}=\sqrt{[204-12^2/8]/7}=\sqrt{26.571}=5.155\) utility points. The positive mean does not describe the two negative contrasts; reporting the mean alone would conceal this feature of the predictions.

Worked Example: Constructing Contrasts From Predicted Outcomes

Hypothetical predictions. Suppose utility is defined as \(100\) points for target attainment, less \(60\) points for toxicity. For illustration, predicted expected utility for a patient under policy \(\pi\) is calculated as \(U_i^\pi=100P_i^\pi(\text{target})-60P_i^\pi(\text{toxicity})\). Apply this same rule to both policies for three patients.

For patient 1, policy \(A\) predicts target probability \(0.80\) and toxicity probability \(0.10\), so \(U_1^A=100(0.80)-60(0.10)=80-6=74\) points. Under \(B\), the respective probabilities are \(0.70\) and \(0.08\), giving \(U_1^B=70-4.8=65.2\) points. Therefore \(\Delta_1=74-65.2=+8.80\) points.

For patient 2, \(A\) predicts probabilities \(0.65\) and \(0.25\), giving \(U_2^A=65-15=50\). Policy \(B\) predicts \(0.75\) and \(0.10\), giving \(U_2^B=75-6=69\). Thus \(\Delta_2=50-69=-19.0\) points.

For patient 3, \(A\) predicts \(0.90\) and \(0.30\), so \(U_3^A=90-18=72\). Policy \(B\) predicts \(0.80\) and \(0.12\), so \(U_3^B=80-7.2=72.8\). Thus \(\Delta_3=72-72.8=-0.800\) points.

The mean contrast is \((8.80-19.0-0.800)/3=-11.0/3=-3.67\) utility points, and two of three contrasts are negative. In this example, the average favors \(B\), but the patient-level calculations show both the magnitude and direction of variation. The arithmetic assumes the stated utility formula is appropriate and that the predicted probabilities are valid for each policy; changing the outcome values or model predictions could change the result.

Why the Mean Cannot Recover the Harm Fraction

The mean contrast is the mean of the patient-level contrasts, so it preserves their average but not their distribution. In particular, it does not determine how many contrasts are negative. This follows directly from averaging: positive and negative values can offset one another, and many different sets of values can have the same sum.

\frac{1}{n}\sum_{i=1}^{n}\Delta_i \quad\text{does not determine}\quad \frac{1}{n}\sum_{i=1}^{n}I(\Delta_i<0)

For example, the eight contrasts in the first example have mean \(1.50\) and two negative values. A different set of eight contrasts could have the same mean but no negative values, or several negative values balanced by larger positive ones. Therefore, the average utility contrast and a patient-level harm summary answer complementary questions and should not be treated as substitutes.

Specify the comparison.
State the two policies, target population, outcome horizon, utility scale, and patient information used for prediction.

Pair predictions within patient.
For each patient, calculate \(U_i^A\) and \(U_i^B\) under the same assumptions, then form \(\Delta_i=U_i^A-U_i^B\).

Summarize location and distribution.
Report the average and spread, and directly calculate the fraction below zero or below a prespecified meaningful-loss threshold.

Check robustness and communicate scope.
Assess prediction and sampling uncertainty, inspect where predictions are least supported, and state whether results apply to the evaluated sample or a target population.

Heterogeneity, Uncertainty, and Diagnostics

Variation in point predictions across patients is not the same as uncertainty about those predictions. Heterogeneity describes differences in the predicted contrasts \(\Delta_i\) across patient profiles. Model uncertainty describes uncertainty in the estimated contrast for a patient, due, for example, to limited data or uncertain model parameters. Outcome variability is another distinct concept: even if a patient's expected utility is known, the eventual outcome can differ from its expectation.

Worked Example: A Positive Estimate With Model Uncertainty

Hypothetical calculation. For one patient, suppose four equally weighted parameter draws from a fitted model produce contrasts of \(-4, 0, 2,\) and \(6\) utility points. Treat these draws only as a small illustrative approximation to uncertainty in that patient's expected-utility contrast.

The mean across draws is \((-4+0+2+6)/4=1.00\) utility point. One of four draws gives a negative contrast, so the fraction of draws below zero is \(1/4=25\%\). The model's average contrast favors \(A\), while the draws show that the sign is uncertain under this illustrative parameter distribution.

This 25% is not the proportion of patients harmed: it concerns uncertainty for one patient's contrast across model draws. Nor is it a general probability that policy \(A\) harms that patient, absent the model and probability interpretation being justified. A real analysis should use an adequate number of draws and describe how they were generated and what sources of uncertainty they represent.

Several checks help determine whether a reported distribution is trustworthy. First, assess whether the component outcome predictions are calibrated in relevant parts of the patient population; good average calibration can still mask errors in particular regions. Second, examine whether the policy contrasts are driven by extrapolation, sparse covariate patterns, or a small number of extreme patients. Third, repeat the calculation across appropriate parameter or resampling draws so that uncertainty in the mean and in the harm fraction is visible. The resampling or model-based procedure must preserve the paired comparison: both policy predictions for a given patient belong together.

The evaluation also relies on a common, clinically defensible utility scale and a model that meaningfully predicts outcomes under each policy. The predicted contrasts are counterfactual comparisons: the same patient is evaluated under alternatives, although only one policy could ordinarily be used for that patient at a given decision. Their interpretation depends on the modeling and identification assumptions supporting the policy predictions. If those assumptions are weak, apparent patient-level precision is misleading.

How This Complements Other Summaries

Summary	Question answered	What it can miss
Mean utility contrast	What is the average predicted utility difference?	How many patients have negative or substantial losses
Subgroup averages	Do predefined groups differ in average utility or harm?	Variation among individuals within each group
Patient-level contrast distribution	How do predicted benefits and losses vary across individuals?	Whether the predictions are calibrated or causally identified
Predictive accuracy summaries	How well do predicted outcomes agree with observed outcomes?	Whether policy differences produce meaningful utility gains

These approaches are complementary. A patient-level distribution does not establish that a dosing policy is safe, and a small estimated fraction with negative contrasts is not a guarantee that no patient will lose. Conversely, an average benefit does not show that every patient benefits. Summaries should therefore be presented together with their assumptions, the utility scale, the evaluated population, and uncertainty. This evaluation is distinct from imposing a rule that prohibits a recommendation whenever a specified safety condition is met.

Common Pitfalls

Calling a positive mean universal benefit. The wrong reasoning is that a policy with positive average contrast must benefit every patient. The correction is to report the distribution and the fraction with negative contrasts separately.
Calling every negative prediction an observed adverse event. A negative \(\Delta_i\) is a modeled difference in expected utility, not proof of an outcome that occurred or will occur.
Confusing parameter uncertainty with patient heterogeneity. A range across model draws for one patient is uncertainty about that patient's contrast; a range across patient-specific contrasts is variation between patients.
Comparing unpaired predictions. Contrasts should compare policies for the same patients under a common utility definition. Comparing unrelated patient sets can mix policy differences with differences in patient composition.
Using a harm threshold without justification. A threshold such as \(\Delta_i<-5\) has meaning only if five points is interpretable on the chosen utility scale and the threshold was specified for a defensible reason.

Key takeaway: Report both average predicted utility and patient-level variation. The mean describes average benefit; the distribution, including the fraction with negative or meaningfully negative contrasts, shows whether that benefit is shared or conceals predicted losses for some patients.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For contrasts of \(6, 2, -3,\) and \(-1\) utility points, calculate the mean and the fraction with negative predicted benefit.
In one sentence, explain why a positive average contrast does not establish that all patients benefit.
A single patient's predicted contrast is positive in most but not all parameter draws. Is that across-patient heterogeneity or model uncertainty? Explain.
Why should the two policy predictions be calculated for the same patients when forming patient-level contrasts?

Solutions. (1) The sum is \(6+2-3-1=4\), so the mean is \(4/4=1.00\) utility point. Two of four contrasts are negative, giving \(2/4=0.50\), or 50%. (2) Positive contrasts can be offset by negative contrasts in the average, so the mean does not determine how many patients have \(\Delta_i<0\). (3) This is model uncertainty for that patient's contrast across parameter draws; patient heterogeneity instead concerns differences in predicted contrasts across patients. (4) Pairing holds patient characteristics constant in the comparison, so the contrast reflects the difference between policy predictions rather than differences in the patients being compared.