Reporting Subgroup and Effect-Modification Analyses

Why subgroup reporting needs its own contrast

An overall treatment-effect estimate can conceal differences across groups, but finding a difference requires a direct comparison of subgroup-specific effects. A statistically significant estimate in one subgroup and a nonsignificant estimate in another do not, by themselves, show that the effects differ. The uncertainty in the difference between effects is what matters.

In an observational comparative study, subgroup results also depend on who is represented in each subgroup and how treatment and comparator outcomes were estimated. A useful report therefore identifies the subgroup definition, states whether it was prespecified or exploratory, names the effect scale, and gives both subgroup-specific estimates and a direct interaction estimate with uncertainty. These results describe effect modification on the stated scale; they do not by themselves establish that treatment caused the observed differences.

Definition: Let \(T=1\) denote the treatment strategy and \(T=0\) the comparator, \(Y(1)\) and \(Y(0)\) the potential outcomes, and \(G\) a subgroup variable. A subgroup-specific average treatment effect is \(\tau(g)=E[Y(1)-Y(0)\mid G=g]\) in the stated target population. For two subgroups, the additive-scale interaction contrast is \(\delta=\tau(1)-\tau(0)\). Effect modification on this scale means the subgroup-specific effects differ in their average outcome difference. The contrast, and therefore whether effects appear to vary, depends on the effect scale.

A subgroup variable might be a baseline characteristic such as age category, disease severity, or a biomarker group. It should be defined in operational terms: for example, “age at cohort entry, classified as younger than 65 versus 65 or older,” rather than simply “age.” State when the variable was measured and how missing or ambiguous values were handled. A post-treatment characteristic is generally not interchangeable with a baseline subgroup: conditioning on it can change the question and may introduce bias.

Prespecification and the reporting target

Prespecified analyses are those whose subgroup definition, contrast, and analysis plan were set before examining the relevant outcome results. Exploratory analyses are generated or materially changed after examining the data. Reporting which applies helps readers assess how much weight to place on an apparent pattern. Prespecification does not guarantee validity, and an exploratory result is not automatically uninformative; the distinction clarifies how the analysis arose and how cautiously it should be interpreted.

For each analysis, report the subgroup variable and cut points, the subgroup-specific population or analysis sample, the outcome and follow-up window, the treatment contrast, and the effect scale. State whether the estimates are adjusted and identify the adjustment approach sufficiently to understand what was estimated. In a propensity-score analysis, also make clear whether the subgroup estimates target the same matched or weighted population as the overall estimate or a subgroup-specific population. Do not imply that an estimate applies to people who were excluded or lacked treatment overlap.

If a causal interpretation is intended, it requires the relevant causal assumptions within the target population, including consistency, conditional exchangeability given the measured confounders, and positivity for the treatment strategies being compared. These assumptions can be especially doubtful in small or poorly overlapping subgroups. An interaction analysis does not repair confounding or make unsupported subgroup comparisons valid.

Worked Example: Reporting an Additive Interaction Contrast

Suppose a hypothetical adjusted observational analysis estimates a treatment effect on hospital stay, in days, for two prespecified age groups. Negative values indicate fewer days in the treatment group. The subgroup estimates and their standard errors are: younger than 65, \(-2.0\) days with \(SE=0.8\); age 65 or older, \(-5.0\) days with \(SE=1.2\). For this illustration, assume the two subgroup estimates are independent and their uncertainty estimates are valid.

The interaction contrast, defined as older minus younger, is

\hat{\delta}=\hat{\tau}(1)-\hat{\tau}(0)=(-5.0)-(-2.0)=-3.0\text{ days}.

Thus the estimated reduction in hospital stay is 3.0 days greater in the older subgroup on the additive scale. Under the stated independence assumption, the standard error of the difference is

SE(\hat{\delta})=\sqrt{SE(\hat{\tau}(1))^2+SE(\hat{\tau}(0))^2} =\sqrt{1.2^2+0.8^2}=\sqrt{2.08}=1.442\text{ days}.

A normal-approximation 95% confidence interval is

-3.0\mathbin{\pm}1.96(1.442)=(-5.83,-0.174)\text{ days}.

This interval excludes zero; under the model and approximation used, the data provide evidence of an additive-scale difference between subgroup effects. A report should still identify the estimates as adjusted observational estimates, describe their target populations, and avoid treating the interval as proof of a causal difference.

Estimating an interaction and its uncertainty

For a binary subgroup variable \(G\), an identity-link model for a continuous outcome can express the interaction directly:

E(Y\mid T,G,X)=\beta_0+\beta_1T+\beta_2G+\beta_3(TG)+f(X),

where \(X\) denotes adjustment covariates and \(f(X)\) is the specified adjustment term. On this outcome scale, the modeled treatment contrast for \(G=0\) is \(\beta_1\), while the contrast for \(G=1\) is \(\beta_1+\beta_3\). Their difference is \(\beta_3\). Therefore, the interaction coefficient estimates the additive interaction contrast, provided the model and estimand support that interpretation. The uncertainty for \(\beta_1+\beta_3\) depends on the covariance between those coefficient estimates; it is not generally correct to add their standard errors.

In an observational analysis, a causal interpretation additionally requires the causal assumptions relevant to the comparison, and adequate covariate adjustment within the subgroups. Statistical inference also relies on a suitable variance estimator for the design and analysis. Matching, weighting, repeated observations, or clustered data can induce dependence; standard errors that ignore it may be misleading. Use an uncertainty procedure appropriate to the estimator, and explain it briefly. If estimates for subgroups are correlated, the variance of their difference must include that covariance:

\operatorname{Var}\{\hat{\tau}(1)-\hat{\tau}(0)\} =\operatorname{Var}\{\hat{\tau}(1)\} +\operatorname{Var}\{\hat{\tau}(0)\} -2\operatorname{Cov}\{\hat{\tau}(1),\hat{\tau}(0)\}.

Reporting principle: Give the interaction contrast and its uncertainty directly. A table of subgroup-specific estimates is useful, but it is not a substitute for the estimate of their difference and its standard error, confidence interval, or other stated uncertainty measure.

Worked Example: Why Separate Significance Tests Do Not Test Interaction

Consider a second hypothetical analysis of an outcome measured in days. In subgroup A, the estimated treatment effect is \(-4.0\) days with \(SE=1.8\). In subgroup B, it is \(-1.5\) days with \(SE=1.2\). Assume independent estimates and use normal approximations for illustration.

The subgroup A test statistic is \(-4.0/1.8=-2.22\), with a two-sided \(p\)-value of about 0.026. For subgroup B, the statistic is \(-1.5/1.2=-1.25\), with a two-sided \(p\)-value of about 0.211. One estimate is conventionally significant and the other is not.

That difference in significance is not a test of whether the effects differ. The direct interaction estimate is

\hat{\delta}=(-1.5)-(-4.0)=2.5\text{ days}, \qquad SE(\hat{\delta})=\sqrt{1.2^2+1.8^2}=\sqrt{4.68}=2.163\text{ days}.

The interaction statistic is \(2.5/2.163=1.16\), giving a two-sided normal-approximation \(p\)-value of about 0.248. The 95% confidence interval is \(2.5\mathbin{\pm}1.96(2.163)=(-1.74,6.74)\) days. Thus this analysis does not provide strong evidence that the subgroup effects differ, even though the within-subgroup tests have different significance labels. The interval also shows that appreciable differences in either direction remain compatible with these estimates.

Scale, diagnostics, and limitations

Effect modification is scale-specific. For a binary outcome, treatment effects might be reported as risk differences, risk ratios, or odds ratios; equality on one scale does not imply equality on another. For example, hypothetical risks of 10% under the comparator and 5% under treatment in one subgroup correspond to a risk difference of \(-5\) percentage points and a risk ratio of 0.50. Risks of 30% and 20% in another subgroup correspond to a difference of \(-10\) percentage points and a ratio of 0.67. The effects differ on both scales here, but by different amounts. A report must name the scale rather than using “interaction” as though it had only one meaning.

Check subgroup-specific sample sizes, outcome events, treatment and comparator representation, covariate balance, and overlap. Very small cells can produce unstable estimates, wide intervals, or model fitting problems. If a subgroup has little or no support for one treatment strategy, its contrast may be poorly identified or not estimable for the intended population. Report that limitation rather than presenting a precise-looking estimate without qualification. Also check whether the subgroup cut points or definitions were selected after inspecting results, and state any such data-driven choices.

Subgroup analyses are often numerous. Testing many characteristics and cut points increases the chance of finding an apparently unusual result by chance. Report the number and scope of analyses, distinguish planned from exploratory findings, and avoid highlighting only favorable contrasts. A nominal confidence interval or \(p\)-value does not automatically account for searching across many subgroups. The result should be interpreted in light of that search and its uncertainty.

Worked Example: A Transparent Exploratory Finding

Suppose investigators examine a treatment effect by baseline biomarker category after the primary analysis and find an estimated risk difference of \(-8\) percentage points in the high-biomarker group and \(-2\) percentage points in the low-biomarker group. The interaction estimate, high minus low, is \(-6\) percentage points, with a 95% confidence interval from \(-13\) to \(1\) percentage point.

A suitable summary is: “In an exploratory analysis defined after the primary analysis, the estimated risk difference was \(-8\) percentage points in the high-biomarker group and \(-2\) percentage points in the low-biomarker group; the additive interaction estimate was \(-6\) percentage points (95% CI \(-13\) to \(1\)). The interval includes zero, so these data do not establish an additive-scale difference, and the finding requires cautious interpretation.” This states the analysis status, subgroup estimates, direct contrast, scale, and uncertainty without turning the pattern into a confirmed subgroup effect.

Common reporting errors

Comparing significance labels: “Significant in one group, not significant in the other” is not evidence that the group effects differ. Report and interpret the interaction contrast.
Omitting the scale: An interaction on the risk-difference scale need not match an interaction on the ratio scale. Name the scale for every effect and contrast.
Leaving out how groups were formed: Give the variable, measurement timing, categories or cut points, and whether the analysis was prespecified or exploratory.
Overstating a subgroup result: An estimate in a small, poorly supported, or exploratory subgroup can be highly uncertain. Include uncertainty and describe limitations.
Implying causation from adjustment alone: State observational estimates as estimates from the specified analysis, and reserve causal language for interpretations whose assumptions are justified.

Define the analysis.
Identify the subgroup variable, timing, categories, outcome, follow-up, target population, and whether the analysis was prespecified.

State the estimand and scale.
Give the treatment contrast within each subgroup and define the interaction contrast as their difference on the chosen scale.

Report estimates and uncertainty.
Show subgroup estimates and a direct interaction estimate with an appropriate standard error or confidence interval.

Qualify the interpretation.
Discuss sample support, overlap, multiplicity, exploratory choices, and the limits of causal interpretation.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A treatment estimate has \(p=0.03\) in subgroup A and \(p=0.20\) in subgroup B. What additional result is needed to assess whether the effects differ?
Two subgroup effects are \(-3\) and \(-1\) units. Define the interaction contrast as subgroup B minus subgroup A. What is its estimate, and what does its sign mean?
Why should a report distinguish prespecified subgroup analyses from exploratory ones?
A subgroup has almost no comparator observations. What should the analyst report about its estimated treatment contrast?
Can one conclude that effects are the same on all scales if the estimated risk difference is zero? Explain briefly.

Solutions. (1) Report the direct interaction contrast—the difference between subgroup effects—with its uncertainty; separate \(p\)-values do not test that difference. (2) The contrast is \((-1)-(-3)=2\) units, indicating the subgroup B estimate is 2 units higher than subgroup A on the stated scale. Whether this is persuasive evidence of heterogeneity depends on its uncertainty. (3) The distinction tells readers whether the subgroup definition and analysis were set before examining the results, which affects how exploratory patterns should be interpreted. (4) Report the limited treatment-comparator support and that the contrast may be unstable or not estimable for the intended population; do not imply a reliable comparison. (5) No. A zero risk difference concerns the additive scale; effects can still differ on a ratio scale, and the uncertainty of each estimate also matters.