Why Diagnose the Pattern of Treatment Effects?
A treatment may have different effects for patients with different baseline biomarker values. A model can represent that variation with a simple pattern, such as a treatment effect that changes linearly as the biomarker increases. But a fitted interaction term does not establish that the assumed pattern is adequate. A curve may bend, the effect may change only in part of the biomarker range, or a few observations may dominate the apparent trend. Diagnostics help assess whether the model gives a reasonable description of the data over the range that matters.
This tutorial uses plots and estimates as diagnostic tools, not as automatic rules for selecting a subgroup or claiming that a biomarker causes treatment response. Their value is in showing where an assumed pattern appears to fit, where it does not, and how much uncertainty accompanies that impression.
The conditional contrast is defined on the outcome scale used for \(Y\). For a continuous outcome measured in points, for example, \(\tau(z)=-3\) means the active-treatment mean is 3 points lower than the control mean at biomarker value \(z\). The direction of benefit depends on how the outcome is defined. Diagnostics should therefore be interpreted in context, with the outcome scale and biomarker units stated.
Interaction Plots and the Assumed Pattern
An interaction plot displays treatment-arm outcome means across biomarker values, or displays estimated treatment effects directly. If separate arm means are plotted, their vertical difference at each \(z\) represents the estimated treatment effect there. The plot can show whether those differences appear constant, steadily changing, curved, or irregular. A scatterplot of individual outcomes can also be useful, but raw outcome variation may obscure the mean pattern; summaries or fitted mean curves should be accompanied by uncertainty information when available.
A common model for a continuous outcome and a continuous biomarker is the linear treatment-by-biomarker model
Under this model, the control mean is \(\beta_0+\beta_Zz\), while the treatment mean is \(\beta_0+\beta_T+(\beta_Z+\beta_{TZ})z\). Subtracting the control mean from the treatment mean gives the model-implied effect pattern:
Thus, the interaction coefficient \(\beta_{TZ}\) is the change in treatment effect for a one-unit increase in \(Z\), measured in outcome units per biomarker unit. This is a particular shape assumption: the treatment effect is linear in \(z\). A plot can reveal visible departures from that shape, but a plot alone cannot prove that the true effect follows any specific alternative curve.
Worked Example: Reading an Interaction Plot From Arm Means
In a hypothetical randomized trial, the endpoint is a symptom score in points, with larger values indicating worse symptoms. The following are hypothetical mean scores at three biomarker values. Suppose each mean is based on the same number of patients in each arm at that value.
| Biomarker \(z\) (units) | Control mean (points) | Treatment mean (points) | Treatment minus control (points) |
|---|---|---|---|
| 0 | 10 | 8 | -2 |
| 1 | 11 | 8 | -3 |
| 2 | 12 | 8 | -4 |
The treatment effects are calculated by subtracting the control mean from the treatment mean: \(8-10=-2\), \(8-11=-3\), and \(8-12=-4\) points. The plotted arm means have a widening separation as \(z\) increases. Across the displayed range, the estimated effect decreases by 1 point per one-unit increase in the biomarker.
The control means follow \(10+z\), and the treatment means are constant at 8. Their difference is \(8-(10+z)=-2-z\) points. In the linear interaction model, this pattern corresponds to \(\beta_T=-2\) points at \(z=0\) and \(\beta_{TZ}=-1\) point per biomarker unit. This description is limited to the displayed range and does not show whether the pattern is precise: sample sizes, variability, and uncertainty intervals would be needed to assess that.
Residual Checks for an Assumed Mean Pattern
A residual is an observed outcome minus its model-predicted mean. Residual plots can help locate systematic mismatch that an interaction plot may not make obvious. For this topic, useful displays include residuals against biomarker value, fitted outcome, and treatment arm, and treatment-arm residual summaries across the biomarker range. A smooth trend or repeated curvature in residuals can indicate that a proposed mean pattern leaves structure unexplained. A widening residual spread can point to nonconstant variance, which affects standard errors and interval calculations if ignored.
The interpretation of residuals requires care. For a fixed prediction function \(g(T,Z)\), or predictions trained on independent data and evaluated on new observations, let \(m(T,Z)=E[Y\mid T,Z]\). Then, conditional on \(T,Z\),
This follows from linearity of conditional expectation: \(E[Y-g(T,Z)\mid T,Z]=E[Y\mid T,Z]-E[g(T,Z)\mid T,Z]\); because \(g\) is fixed given \(T,Z\), the second term is \(g(T,Z)\). A systematic conditional residual mean therefore reflects a difference between the true conditional mean and the fixed prediction.
For predictions from a model fitted on the same observations being plotted, however, \(g\) depends on those outcomes. Conditioning only on \(T,Z\) does not make that fitted function fixed. The displayed equality with the realized fitted value is not generally valid: the conditional expectation involves \(E[g(T,Z)\mid T,Z]\). Moreover, fitting can force residuals to satisfy constraints; for example, ordinary least squares residuals are orthogonal to every fitted design column; when a group-indicator column is included, the residuals sum to zero within that group. Same-sample residual plots remain useful descriptive checks, but their patterns can be muted or shaped by fitting. Independent validation data or cross-fitted predictions can make residual diagnostics more informative about prediction error beyond the fitting sample.
Worked Example: Detecting a Residual Pattern in Validation Data
A hypothetical trial team fits a linear treatment-by-biomarker mean model using a training sample. It then evaluates predictions on a separate validation sample. Among treatment-arm patients, validation-sample mean outcomes and the model-predicted means at four biomarker values are as follows. The outcome is a score in points, and the entries are group averages.
| \(z\) (units) | Observed validation mean (points) | Predicted mean (points) | Residual mean (points) |
|---|---|---|---|
| 0 | 20 | 20 | 0 |
| 1 | 21 | 22 | -1 |
| 2 | 24 | 24 | 0 |
| 3 | 29 | 26 | 3 |
Each residual mean is observed minus predicted: for \(z=1\), \(21-22=-1\) point; for \(z=3\), \(29-26=3\) points. The residuals are not scattered around zero without an apparent pattern: the model overpredicts at \(z=1\), matches at \(z=2\), and underpredicts by 3 points at \(z=3\). This suggests that the linear mean pattern may understate the rise in treatment-arm outcomes at the upper end of the displayed range. It is a diagnostic signal, not proof of a particular curve. The averages also conceal within-group variability, so their precision should be examined before treating small departures as meaningful.
Because the predictions came from separate training data, the fitted function is independent of these validation outcomes, conditional on the predictors, which supports the fixed-prediction interpretation above. If the same four means had been used to fit the model, the residual pattern would instead be a same-sample diagnostic subject to the constraints and dependence induced by fitting.
Subgroup-Specific Estimates as a Check
Subgroup-specific treatment-effect estimates provide another view of the pattern. For a continuous biomarker, groups might be defined by prespecified intervals; for a binary biomarker, its two strata may be shown directly. In each group, calculate the treatment-minus-control contrast and show its uncertainty. These estimates can expose a mismatch between a smooth model and the observed data, but they are less precise when data are divided into smaller groups. Arbitrary cutpoints also discard information and can make the apparent pattern depend on how the biomarker was divided.
Worked Example: Comparing Effects Across Two Biomarker Strata
Consider a hypothetical randomized trial with a binary baseline biomarker: negative or positive. The outcome is measured in points, with larger values better. In each biomarker stratum, 50 patients are assigned to each arm. The hypothetical within-arm sample standard deviation is 10 points in every group.
| Biomarker stratum | Control mean (points) | Treatment mean (points) | Estimated effect (points) |
|---|---|---|---|
| Negative | 50 | 49 | \(49-50=-1\) |
| Positive | 50 | 43 | \(43-50=-7\) |
For independent randomized arms within a stratum, the standard error of a difference in means is estimated by \(\sqrt{s_1^2/n_1+s_0^2/n_0}\). Here, in either stratum,
The estimated effects differ by \((-7)-(-1)=-6\) points, a descriptive indication that the effect estimates are not alike across these strata. Under independent stratum estimates, the standard error of their difference is \(\sqrt{2.00^2+2.00^2}=2.83\) points. A normal-approximation 95% interval for the difference is
This interval summarizes uncertainty in the difference under the stated approximation; it does not establish a biological explanation or validate a subgroup-selection rule. The estimates also need to be interpreted in light of the trial design, analysis plan, and any multiplicity from examining many possible subgroups.
How to Combine the Diagnostics
No single display answers whether an effect pattern is adequate. A useful assessment checks whether the arm means and treatment-effect estimates tell a coherent story, whether residuals show structure left unexplained by the assumed model, and whether apparent departures are large relative to their uncertainty. Examine the biomarker range actually represented in the trial; sparse data at the extremes make both curves and subgroup estimates unstable. Also check whether a small number of observations, influential points, or a change in residual spread is driving the visual impression.
For example, specify whether the treatment effect is assumed constant, linear in the biomarker, or represented by another prespecified form. Identify the outcome scale and biomarker range.
Plot both treatment arms or their contrast across biomarker values. Look for a systematic shape that conflicts with the proposed pattern, while considering uncertainty and data density.
Look for residual trends, curvature, changing spread, and unusual observations. Distinguish same-sample diagnostic residuals from predictions evaluated on independent or cross-fitted observations.
Use interpretable strata and show estimates with uncertainty. Treat disagreement as a reason to investigate model adequacy, not as an automatic basis for selecting a subgroup.
Alternatives, Limitations, and Common Pitfalls
A linear interaction model is compact and uses the biomarker continuously, but it can miss curvature. A flexible smooth or a model with a prespecified nonlinear term can represent more shapes, although added flexibility may increase uncertainty and sensitivity to sparse regions. Subgroup-specific estimates are easier to communicate for clinically meaningful strata, but grouping a continuous biomarker loses within-group information and can hide variation within each interval. Interaction plots give a direct visual comparison, while residual plots focus on departures from a chosen fitted mean. These approaches complement one another; none replaces scientific judgment, uncertainty assessment, or a design-appropriate analysis.
- Pitfall: treating the fitted line as evidence that the shape is correct. An interaction coefficient estimates a slope under the model; it does not demonstrate that the effect is linear throughout the biomarker range. Check arm means, residuals, and subgroup estimates for patterns the line misses.
- Pitfall: reading a same-sample residual as an unbiased estimate of lack of fit. Fitting uses the outcomes and can constrain the residuals. Use same-sample plots descriptively, or evaluate predictions on independent or cross-fitted observations when assessing out-of-sample residual patterns.
- Pitfall: declaring heterogeneity because one subgroup estimate is precise or appears different. Precision within one subgroup is not a comparison between subgroups. Inspect the estimated differences and their uncertainty, and account for the analysis plan and the number of groups examined.
- Pitfall: interpreting lack of visible residual structure as proof of adequacy. Diagnostics can have little ability to reveal departures when subgroup counts are small, the biomarker range is narrow, or outcome noise is large. “No obvious pattern” is not proof that effects are constant or linear.
Ultimately, diagnostics support a reasoned assessment of model adequacy; they do not establish the true treatment-effect function. Results can be sensitive to the outcome scale, measured biomarker range, missing data, influential observations, and random variation. A pattern noticed after examining the data should be treated as exploratory unless its assessment is supported by the trial’s prespecified analysis plan and appropriate evidence.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A model specifies \(\tau(z)=2+0.5z\) points. What is the model-implied treatment effect at \(z=4\), and what does the interaction slope mean?
- In validation data, a model predicts a mean of 15 points at one biomarker value, while the observed validation mean is 17 points. Calculate the residual and interpret its sign.
- Two independent biomarker strata have estimated treatment effects of \(-2\) and \(-5\) points, each with \(SE=1.5\) points. What is the estimated difference (second minus first), and its standard error?
- Why might subgroup estimates show a jagged pattern even when the underlying treatment effect changes smoothly?
Solutions. 1. The effect is \(2+0.5(4)=4\) points. The slope means the model-implied treatment effect increases by 0.5 outcome points for each one-unit increase in the biomarker. 2. The residual is \(17-15=2\) points; the observed validation mean is 2 points higher than predicted, so the model underpredicts at that value. 3. The estimated difference is \((-5)-(-2)=-3\) points. Under independence, its standard error is \(\sqrt{1.5^2+1.5^2}=\sqrt{4.5}=2.12\) points. 4. Each subgroup estimate has sampling variability, and dividing the data reduces the number of observations contributing to each estimate. Sparse groups can therefore fluctuate substantially around a smooth underlying pattern.