Why Interaction Evidence Can Be Fragile
An estimated treatment effect can differ across biomarker strata without the data providing a reliable estimate of how much it differs. This distinction matters in adaptive trials: subgroup-specific effects may inform whether a population is worth studying further, but a noisy or fragile interaction estimate is a poor basis for asserting that treatment works differently by biomarker status. The diagnostics here focus on the evidence itself—its precision, sensitivity to observations, and dependence on the effect scale.
The earlier material on treatment-effect heterogeneity establishes the role of comparing subgroup effects directly. The key quantity is the interaction contrast, not whether one subgroup has a statistically significant treatment effect while another does not. Its interpretation depends on the outcome scale and the model used to estimate it.
For example, in a linear model with binary treatment indicator \(T\) and marker indicator \(M\), both coded 0 or 1, the conditional mean can be written as \(E(Y\mid T,M)=\beta_0+\beta_TT+\beta_MM+\delta TM\). The treatment effect at \(M=0\) is \(\beta_T\), and at \(M=1\) it is \(\beta_T+\delta\), so the difference is \(\delta\). For a binary outcome, a model with a logit link instead defines interaction on the log-odds scale. The same numerical conclusion need not hold on both scales.
Assumptions and a Precision Check
A simple comparison of subgroup treatment effects is most directly interpretable when treatment is randomized within each biomarker stratum, the marker definition and outcome are consistently measured, and the compared participants have the same follow-up and analysis rules. Randomization supports a causal treatment comparison within a stratum, subject to issues such as missing outcomes and adherence to the analysis plan. It does not guarantee that the interaction estimate is precise. Each stratum must contain enough information in both treatment groups to estimate its effect credibly.
For independent stratum-specific estimates, the variance of their difference is the sum of their variances. Thus, if estimated standard errors are \(SE_1\) and \(SE_0\), the standard error of the interaction contrast is \(\sqrt{SE_1^2+SE_0^2}\). This calculation presumes the estimates are independent; shared participants, correlated outcomes, or a more complex model may require a covariance term. A confidence interval combines the interaction estimate with its uncertainty. For approximately normal estimates, a 95% interval is \(\hat{\delta}\pm1.96SE(\hat{\delta})\); with small samples and estimated variances, an appropriate \(t\) critical value and degrees of freedom may be used instead.
Worked Example: A Wide Interval in a Small Stratum
Suppose a hypothetical randomized trial estimates treatment effects of 8 score points in marker-positive participants and 5 score points in marker-negative participants. The estimated interaction is therefore \(8-5=3\) score points. Based on the within-stratum standard errors and a Welch approximation, suppose the interaction standard error is 2.210 score points and \(df=16.5\). For a two-sided 95% \(t\) interval, the critical value is approximately 2.115.
The interval is \(\hat{\delta}\pm t^*SE(\hat{\delta})=3\pm2.115(2.210)\). The margin of error is \(2.115(2.210)=4.674\) score points, giving an interval of approximately \((-1.674,7.674)\) score points.
This interval includes zero, but it also includes positive differences as large as about 7.7 score points. The data therefore do not establish a difference in treatment effect, and they do not establish that the effects are similar. In repeated use of this interval procedure, 95% of intervals constructed by the procedure would cover the true interaction under its assumptions; this particular interval is not a 95% probability statement about the fixed true interaction.
The precision check is not merely whether an interval crosses zero. Ask whether it excludes interaction magnitudes that would matter clinically. A wide interval crossing zero signals weak information: effects consistent with little heterogeneity may be compatible with effects that would change a clinical or design decision. Conversely, an interval tightly concentrated near zero can provide useful evidence against large heterogeneity even if no formal equivalence analysis was planned.
Influence and Sparse-Stratum Diagnostics
A subgroup estimate can be unstable because a stratum has few participants, few outcome events, or unusually variable outcomes. Sparse cells are especially concerning when one or two observations contribute a large fraction of the treatment-group information. Examine the number of participants and events in each treatment-by-marker cell, not just the total trial sample size. Also review outcome distributions, missingness, protocol deviations, and the balance created by randomization within strata.
Influence checks ask whether the estimated interaction changes materially under a justified perturbation, such as deleting one observation temporarily, checking a data correction, or using a prespecified robust analysis. They are diagnostic, not permission to remove an inconvenient observation. Confirm a suspected data error against source records; otherwise, report the primary result and sensitivity result transparently. Large changes indicate dependence on a small amount of information and should temper interpretation.
Worked Example: One Outcome Drives the Interaction
Consider hypothetical continuous outcomes in a very small randomized trial. Among marker-positive participants, treatment outcomes are 6, 7, 8, and 30 points; control outcomes are 4, 5, 6, and 7 points. Among marker-negative participants, suppose the treatment mean is 4 points and the control mean is 3 points.
In the marker-positive stratum, the treatment mean is \((6+7+8+30)/4=12.75\), and the control mean is \((4+5+6+7)/4=5.50\). The treatment-effect estimate is \(12.75-5.50=7.25\) points. In the marker-negative stratum, it is \(4-3=1\) point. The interaction estimate is therefore \(7.25-1=6.25\) points.
Now inspect the sensitivity to the outcome of 30. If it is temporarily omitted for this diagnostic calculation, the marker-positive treatment mean becomes \((6+7+8)/3=7\), and its treatment effect becomes \(7-5.50=1.50\) points. The resulting interaction is \(1.50-1=0.50\) points. A change from 6.25 to 0.50 points is substantial: the original estimate is highly sensitive to one observation.
This check does not show that the observation is invalid, nor does it justify excluding it from the primary analysis. It shows that the interaction is weakly supported by the small data set. Verify the value, examine the outcome definition and distribution, and report the sensitivity without presenting the deletion result as the preferred estimate unless a valid, prespecified rule requires it.
Sparse-stratum diagnostics should also consider model fit. For continuous outcomes, look for unusual residuals, unequal variability, and a mean model that misses important features. For binary outcomes, a stratum with few events can yield imprecise or unstable estimates, especially under models whose approximations rely on adequate information. If a model produces extreme estimates, large standard errors, convergence warnings, or results that shift greatly with a modest modeling change, treat these as warning signs rather than cosmetic problems.
Scale Dependence: What Does “Different Effects” Mean?
Heterogeneity is always relative to an effect scale. On an additive scale, the interaction compares differences in outcome means or risks. On a relative scale, it may compare risk ratios, odds ratios, or another ratio measure. A treatment can have the same relative effect in two groups but different absolute effects, or the reverse. There is no scale-free statement that effects are “the same”; select the scale that matches the clinical question and planned analysis, then describe that choice explicitly.
Worked Example: The Interaction Changes With the Scale
Suppose a hypothetical trial observes the following response risks: among marker-positive participants, 60% with treatment and 40% with control; among marker-negative participants, 30% with treatment and 20% with control. On the risk-difference scale, the effects are \(0.60-0.40=0.20\) and \(0.30-0.20=0.10\). The additive interaction is \(0.20-0.10=0.10\), or 10 percentage points.
On the risk-ratio scale, the effects are \(0.60/0.40=1.50\) and \(0.30/0.20=1.50\). Their ratio is \(1.50/1.50=1.00\), so there is no interaction on this relative scale. The odds ratios differ: in marker-positive participants, the odds ratio is \((0.60/0.40)/(0.40/0.60)=2.25\); in marker-negative participants, it is \((0.30/0.70)/(0.20/0.80)\approx1.714\). Thus, even these same risks yield different interaction descriptions depending on scale.
The example is descriptive and does not establish that any of these differences are estimated precisely. In a real analysis, uncertainty would be calculated for the selected interaction measure. The scale should be chosen for clinical interpretation and specified before examining subgroup results, rather than selected because it gives the strongest-looking finding.
Weak Evidence Is Not Evidence of No Heterogeneity
A nonsignificant interaction test means that the data did not provide sufficient evidence against the null hypothesis of no interaction on the tested scale, at the chosen significance level and under the model. It does not mean that the null is true. Nor does it mean the subgroup effects are identical. A small trial may have little ability to distinguish a real but moderate difference from random variation, particularly when one stratum is rare.
To support a claim that clinically meaningful heterogeneity is absent, define a clinically acceptable margin \(\Delta>0\) on the chosen scale. A confidence interval for the interaction that lies wholly within \((-\Delta,\Delta)\) is evidence that effects differ by less than that margin, under the model and assumptions. If the interval extends beyond either boundary, the data have not ruled out a clinically important difference. This is a different question from whether a test rejects a point null of exactly zero.
Comparing Diagnostic Approaches and Their Limits
| Approach | What it helps assess | Main limitation |
|---|---|---|
| Interaction estimate and confidence interval | Direction, magnitude, and precision on a stated scale | May be wide or depend on model assumptions |
| Cell counts and outcome summaries | Sparsity, imbalance, and unusual distributions | Descriptive checks do not quantify interaction uncertainty |
| Influence and sensitivity analyses | Dependence on particular observations or analysis choices | Alternative analyses do not replace the prespecified primary analysis |
| Equivalence-style margin assessment | Whether important heterogeneity is excluded | Requires a justified margin and adequate precision |
No single diagnostic resolves every limitation. Confidence intervals may rely on approximate sampling distributions; with very sparse binary data, those approximations can be unreliable. Robust methods may reduce sensitivity to some distributional features but often lose precision and do not repair a lack of information. A model that borrows information across strata can stabilize estimates, but its assumptions about how effects relate across strata must be credible. Diagnostics should therefore be interpreted together, with the design, endpoint, and clinical context in view.
Common Mistake
Wrong reasoning: “The treatment effect is significant in marker-positive participants but not significant in marker-negative participants, so the treatment works differently by marker status.” The two within-stratum tests address whether each separate effect is zero; their different significance labels do not test whether the effects differ. The correction is to estimate and assess the interaction contrast directly, with its uncertainty, and to state the scale. Then check whether the result is precise, robust to influential observations, and clinically meaningful. A fragile estimate should be described as uncertain, not promoted to a definitive subgroup claim.
Define the subgroup treatment effects and their interaction contrast on the clinically relevant prespecified scale.
Review treatment-by-marker sample sizes, event counts where relevant, outcome distributions, and missingness.
Interpret the interaction interval, inspect influential observations, and assess whether reasonable diagnostics change the conclusion.
Distinguish weak evidence, evidence of clinically important heterogeneity, and precise evidence that important heterogeneity is unlikely.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An interaction estimate is 1.2 units with a 95% confidence interval from -3.0 to 5.4 units. What can and cannot be concluded about heterogeneity?
- Why is a significant treatment effect in one biomarker stratum and a nonsignificant effect in another not, by itself, evidence of interaction?
- In a binary-outcome trial, the estimated treatment risk differences are 0.15 in one stratum and 0.05 in the other. What is the additive interaction estimate, and what additional information is needed before claiming it is convincing?
- A single unusual outcome changes an interaction estimate substantially when omitted in a diagnostic calculation. Should it automatically be excluded?
Solutions. 1. The interval includes zero, so the data do not establish a nonzero interaction at the corresponding two-sided 5% level. It also includes differences up to 5.4 units in one direction and -3.0 in the other, so without a clinical margin it cannot rule out important heterogeneity. 2. Those tests assess each stratum effect against zero, not the difference between effects. Estimate the interaction directly and evaluate its uncertainty. 3. The additive interaction is \(0.15-0.05=0.10\), or 10 percentage points. Its confidence interval, cell information, assumptions, and clinical importance are needed to judge evidence. 4. No. Verify the observation and report the influence diagnostic; exclude it only under a justified data correction or valid prespecified rule, not simply because its removal gives a preferred result.