Subgroup-Specific Utility and Harm

Why Evaluate Utility and Harm by Subgroup?

An overall average can conceal clinically important variation in a dosing policy’s benefits and harms. A policy might improve expected utility in one clinically defined group but offer little benefit in another; it might also reduce a safety event in one group while increasing it elsewhere. Subgroup evaluation makes these patterns visible, but small subgroup samples can produce striking estimates by chance. The central task is therefore to estimate each subgroup’s outcomes and compare the subgroup effects directly, with uncertainty made explicit.

The target is a group-level policy contrast, not a claim that every patient in a subgroup benefits equally. Utility and harm are distinct outcomes unless the utility definition explicitly combines them. A policy can have favorable average utility and still have a safety concern that requires separate attention.

These quantities are causal policy effects when the comparison supports causal interpretation. For a randomized policy comparison, that requires valid randomization, consistency (the policy received corresponds to the policy being evaluated), and no interference between participants. Each policy must have a positive probability of assignment in every subgroup being studied. Subgroup membership should be determined from baseline information, not from a post-treatment outcome or a variable changed by the policy. In a nonrandomized comparison, treatment exchangeability conditional on measured confounders is also needed; an unadjusted difference alone does not ensure it.

Estimate Effects Within Groups, Then Compare Effects

For a randomized comparison, estimate utility in each policy arm within each subgroup. The difference between those arm means estimates \(\tau_g\). For independent participants in the two arms, an approximate standard error is obtained from the sample variances. This is a large-sample approximation; it is most credible when the subgroup samples are adequate and the utility distribution is not dominated by extreme values.

Here \(\bar{U}_{ag}\), \(s_{ag}\), and \(n_{ag}\) are the sample mean, sample standard deviation, and sample size in policy arm \(a\) and subgroup \(g\). A large-sample 95% confidence interval is \(\widehat{\tau}_g\pm1.96SE(\widehat{\tau}_g)\). It describes a range of effects compatible with the data under the model and sampling assumptions; it is not a 95% probability statement about a fixed effect.

A subgroup effect being statistically distinguishable from zero does not establish that it differs from the effect in another subgroup. To assess effect modification, compare the effects themselves. For two mutually exclusive subgroups, define the utility interaction contrast as \(\psi=\tau_1-\tau_2\). If the subgroup estimates come from disjoint participants and the arm assignments are independent, their sampling errors are independent, giving:

The same principle applies to harm risk differences. If subgroups overlap, or estimates share participants or fitted parameters, the covariance between estimates must be included: \(Var(\widehat{\tau}_1-\widehat{\tau}_2)=Var(\widehat{\tau}_1)+Var(\widehat{\tau}_2)-2Cov(\widehat{\tau}_1,\widehat{\tau}_2)\). A model with a policy-by-subgroup interaction can estimate the same contrast, provided its coding and outcome scale match the intended estimand.

Testing Whether Effects Differ

Consider the same hypothetical trial, with a second, mutually exclusive subgroup of 64 patients per arm. In that subgroup, mean utility is 7.4 points under the new policy and 6.9 under usual dosing; the standard deviations are again 2.4 and 2.0 points. The subgroup effect is \(0.5\) points, with the same standard error, 0.3905 points. The relevant question about subgroup variation is whether the estimated 1.6-point effect differs from the estimated 0.5-point effect.

This direct contrast avoids a common error: declaring effect modification because one subgroup’s effect has a p-value below 0.05 and another’s does not. The difference between “statistically significant” and “not statistically significant” is not itself necessarily statistically significant. The interaction contrast, with its own standard error and interval, addresses the question directly.

Evaluate Harm on a Clinically Interpretable Scale

For a binary harm outcome, estimate the harm risk in each policy arm and subgroup. The risk difference is often useful because it expresses the absolute change in risk; a negative value favors the evaluated policy when the event is harmful. With independent groups, an approximate standard error for a risk difference is:

This normal approximation can perform poorly when event counts are very small or estimated risks are near zero or one. In such settings, use an interval method appropriate for sparse binomial data and report the event counts, not just the estimated risks. A risk difference and a relative risk answer different questions: the absolute impact depends on the baseline risk, while the relative contrast describes a proportional change. State the scale used before comparing subgroups.

Interpretation, Diagnostics, and Limits

Subgroup estimates are most useful when interpreted together: show each policy-arm outcome, each subgroup effect, and the direct between-subgroup contrast with uncertainty. Also show denominators and event counts for harms. A pooled policy effect can be clinically relevant, but it does not answer whether the policy’s effect varies across groups. Conversely, subgroup-specific estimates do not replace the overall result.

Before interpreting apparent differences, check whether the subgroup definitions were clinically motivated and specified before examining outcomes, whether enough participants and events are present, and whether the same outcome definition and follow-up apply across groups. Examine missingness and protocol deviations by policy and subgroup. Sparse events, wide intervals, unstable estimates, and sensitivity to a few observations are warning signs that the data do not support precise subgroup conclusions. If a regression model is used, check that the interaction is on the intended scale: an interaction on a risk-difference scale need not imply an interaction on a relative-risk scale.

When many subgroups or outcomes are examined, some contrasts may look unusual by chance. Prespecifying a small set of clinically justified contrasts, reporting all planned comparisons, and treating unplanned findings as exploratory reduce the risk of overinterpreting selective results. Multiplicity adjustments may be appropriate for confirmatory families of tests, but they do not repair biased subgroup definitions or inadequate sample size.

A subgroup-average contrast does not imply that each member experiences the subgroup average effect. It describes a mean or risk contrast for the defined group, subject to the design and measurement assumptions. Nor does a nonsignificant interaction prove that effects are identical: an interval spanning both clinically important differences and no difference indicates limited precision. These analyses inform policy evaluation; they do not by themselves establish an individual patient’s outcome under both policies.

The main alternatives answer related but different questions. A pooled estimate summarizes average impact across the study population; it can obscure subgroup variation. Separate subgroup estimates are easy to communicate but invite informal comparisons. A regression interaction model can estimate contrasts efficiently and adjust for prespecified covariates, but depends on its functional form, coding, and chosen effect scale. For clinically relevant decisions, report absolute utility and harm effects on interpretable scales, and use relative measures as complements when useful.