Why Average Treatment Effect Is Not Enough
A dose can produce the same average treatment effect in two populations while giving very different experiences to individual patients. One population might have fairly similar responses; another might contain patients who benefit greatly alongside patients who benefit little or are harmed. Whether those patterns lead to different population utility depends on how utility is defined. In particular, if utility is nonlinear in an individual’s outcomes, the average utility cannot generally be recovered by applying the utility function to the average outcome.
The relevant distinction is between the average effect of a dose and the average of patient-level utilities under that dose. The first summarizes a mean response contrast. The second averages each patient’s utility over a specified target population. Dose decisions based on utility therefore require attention to the distribution of individual effects, not just its mean.
The sign of an outcome contrast must be defined in context. For a symptom score where lower is better, one might define improvement as \(Y_i(d_0)-Y_i(d)\), so positive values represent benefit. Below, examples state their direction explicitly. Utility may depend on more than one outcome, such as efficacy and toxicity, and may also depend on a patient’s baseline characteristics. In that case, a single scalar treatment effect does not contain all the information needed to determine utility.
Assumptions Behind Population Utility
A population utility calculation is meaningful only after specifying whose outcomes are being averaged, which doses are compared, and how outcomes map to utility. If the intended target is a particular clinical population, the expectation \(E_P\) must use that population’s distribution of baseline characteristics and relevant response variation. Reweighting to a different population can change the result, even when the dose-response model is unchanged.
A causal interpretation also requires that the potential outcome \(Y_i(d)\) represent the outcome that patient \(i\) would have under dose \(d\), and that the study or model supports predictions for the doses and patients being evaluated. In a randomized dose study, random assignment supports comparisons between dose groups; a model is still needed to predict patient-level outcomes and their distribution. For observational data, confounding and lack of overlap across dose levels can undermine those predictions. Standard causal assumptions include consistency, no interference between patients, and adequate exchangeability and dose support for the target population.
Finally, variation in treatment effects must be distinguished from residual outcome variation. A noisy endpoint can vary substantially among patients even if their underlying dose effects are identical. Conversely, effects can genuinely vary across patients even when each patient’s outcome is measured precisely. A model that only describes the marginal outcome variance does not automatically identify the distribution of individual treatment effects.
Why Utility Curvature Changes the Consequence of Heterogeneity
Suppose, for simplicity, that a patient’s utility gain is a function \(u(\Delta_i)\) of that patient’s treatment effect alone. Population utility gain is \(E_P[u(\Delta_i)]\), whereas utility evaluated at the average treatment effect is \(u(E_P[\Delta_i])\). These quantities are equal for a linear utility function, but need not agree for a nonlinear one.
The key justification is Jensen’s inequality. If \(u\) is concave over the range of effects in the target population, then
If \(u\) is convex, the inequality is reversed. For a twice-differentiable function, concavity means \(u''(x)\leq 0\) throughout the relevant range; convexity means \(u''(x)\geq 0\). One way to see why curvature matters is to expand \(u(\Delta_i)\) around the mean effect \(\mu_\Delta=E_P[\Delta_i]\). The first-order term averages to zero because \(E_P[\Delta_i-\mu_\Delta]=0\); the second-order term is approximately \(\frac{1}{2}u''(\mu_\Delta)\operatorname{Var}_P(\Delta_i)\). Thus, locally, greater effect variance lowers average utility for concave \(u\) and raises it for convex \(u\). The approximation is local; Jensen’s inequality gives the stated direction under global concavity or convexity without requiring this approximation.
For a linear utility \(u(\Delta)=a+b\Delta\), \(E_P[u(\Delta_i)]=a+bE_P[\Delta_i]\) exactly. In that special case, the distribution of effects adds no information about average utility beyond its mean. But threshold utilities, diminishing returns, and utility that jointly depends on benefit and harm are not generally linear in treatment effect. Their average can depend on the distribution’s spread, shape, subgroup composition, and dependence among outcomes.
Worked Example: Diminishing Returns With a Heterogeneous Effect
Suppose a hypothetical dose produces symptom-score improvements of 2 points for half the target population and 8 points for the other half. Improvement is defined so that a positive value is beneficial. Use the utility gain \(u(\Delta)=1-\exp(-\Delta/5)\), which increases with benefit but has diminishing returns.
The average treatment effect is
The population mean utility gain must be calculated by applying utility to each group’s effect and then averaging:
Applying the utility function to the average effect instead gives \(u(5)=1-e^{-1}=0.6321\). The difference is about \(0.0682\) utility units. Because \(u\) is concave, Jensen’s inequality predicts that the average of individual utilities is no greater than utility at the average effect. The two values answer different questions; \(0.6321\) is not the population mean utility gain for this heterogeneous population.
Thresholds and the Distribution of Effects
A threshold utility makes the distinction especially clear. If utility is assigned only when an individual’s response reaches a specified target, the fraction of patients above that threshold determines mean utility. The mean response alone does not determine that fraction. This is one reason a dose can have an attractive average effect but a low probability of delivering a clinically meaningful benefit to a particular patient.
Worked Example: Same Mean Effect, Different Target Attainment
Consider two hypothetical dose-response patterns for improvement in a symptom score. Define utility as 1 for an individual improvement of at least 4 points and 0 otherwise.
Under Pattern A, 40% of patients improve by 6 points and 60% improve by 2 points. The average improvement is
Only the first group reaches the 4-point threshold, so population mean utility is \(0.40(1)+0.60(0)=0.40\). Under Pattern B, every patient improves by 3.6 points. Its average improvement is also 3.6 points, but no one reaches the threshold, so population mean utility is 0. The mean effects match, while target-attainment utility differs by 0.40 utility units. This difference follows from the assumed threshold rule and effect distributions; the example does not establish which pattern would occur in a real population.
Joint Heterogeneity in Benefit and Harm
When utility depends on multiple outcomes, knowing each outcome’s marginal distribution may still be insufficient. The relationship between outcomes across patients matters. For example, the same mean benefit and mean toxicity can arise when benefit tends to occur in patients without toxicity or when the greatest benefit tends to occur in patients who also experience toxicity. A nonlinear joint utility can assign different population values to those patterns.
Worked Example: The Same Marginal Outcomes, Different Joint Utility
For illustration, define a hypothetical utility that equals 1 only when a patient has a benefit score of at least 5 and no toxicity, and equals 0 otherwise. Each of two dose-response patterns has two equally common patient types.
In Pattern A, half the patients have benefit 8 and toxicity 0; the other half have benefit 2 and toxicity 1. Mean benefit is \(0.5(8)+0.5(2)=5\), and mean toxicity is \(0.5(0)+0.5(1)=0.5\). Half meet both utility conditions, so mean utility is 0.5.
In Pattern B, half have benefit 8 and toxicity 1; the other half have benefit 2 and toxicity 0. Mean benefit remains 5 and mean toxicity remains 0.5. But no patient meets both conditions, so mean utility is 0. The marginal averages alone cannot distinguish the patterns; their joint distribution can. This calculation depends on the stated illustrative utility rule, not on a general claim that one particular relationship between benefit and harm is preferable.
A Practical Evaluation Sequence
When heterogeneity may affect a dose’s population utility, keep the target population and the patient-level utility definition fixed while evaluating the distribution of predicted outcomes. A useful sequence is:
Define the population, dose, comparator or reference dose, outcome direction, and time horizon.
State how each relevant patient outcome maps to utility, including any thresholds or joint benefit-harm conditions.
Use a model that can represent relevant differences among patients and, when needed, dependence between their benefit and harm outcomes.
Compute or estimate utility for the target population and average those values; do not substitute utility evaluated at the average outcome unless the utility is linear in the relevant quantity.
Examine whether conclusions change with plausible response distributions, subgroup weights, and utility definitions.
Diagnostics, Alternatives, and Limitations
Diagnostics should address both the response model and the population utility calculation. Check whether predictions reproduce observed outcome distributions across doses and across relevant covariate groups, not only their overall means. For multiple outcomes, examine whether the model captures their joint behavior. Where data are sparse, flag extrapolation to poorly represented dose levels or patient profiles. A model may predict the mean well while missing the tails or subgroup patterns that drive threshold utility.
Comparing average treatment effects is appropriate when the decision objective is explicitly linear in that effect. Comparing outcomes in subgroups can help reveal where model-predicted effects differ, but a few subgroup means do not necessarily capture within-group variation or joint outcome dependence. Population utility based on the full patient-level distribution addresses nonlinear utility directly, but it relies more heavily on the model and on the specified utility function. These approaches answer related but distinct questions; none should be presented as a substitute for the others without stating the objective.
A major limitation is that individual treatment effects are counterfactual: the same patient cannot simultaneously reveal their outcome under every dose. Consequently, estimates of effect heterogeneity are more assumption-dependent than estimates of an average effect in a randomized comparison. Apparent variation can also reflect measurement error, model misspecification, or sampling noise rather than true effect differences. Report the target population, model assumptions, utility rule, and relevant uncertainty, and avoid presenting model-predicted individual effects as directly observed facts.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Half of a hypothetical population has an improvement of 1 point and half has an improvement of 5 points. For \(u(\Delta)=1-\exp(-\Delta/3)\), calculate the mean effect and mean utility. How do they compare with utility evaluated at the mean effect?
- A threshold utility equals 1 for improvement of at least 3 points and 0 otherwise. If 25% improve by 1 point and 75% by 4 points, calculate the average improvement and population mean utility.
- Give a condition under which two populations with the same mean treatment effect have the same mean utility even if their effect distributions differ.
- Why can two populations with the same marginal mean benefit and mean toxicity have different population utility?
Solutions. (1) The mean effect is \(0.5(1)+0.5(5)=3\) points. Mean utility is \(0.5(1-e^{-1/3})+0.5(1-e^{-5/3})=0.5473\). Utility at the mean effect is \(1-e^{-1}=0.6321\), which is larger because the utility function is concave. (2) Average improvement is \(0.25(1)+0.75(4)=3.25\) points. Only the 4-point group reaches the threshold, so mean utility is 0.75. (3) If utility is linear in treatment effect, \(u(\Delta)=a+b\Delta\), then mean utility is \(a+bE[\Delta]\); populations with the same mean effect have the same mean utility, assuming the same utility function. (4) Their benefit and toxicity can be associated differently across patients. A joint, nonlinear utility may value the combinations differently even when each outcome’s marginal mean is unchanged.