Testing Treatment-By-Subgroup Interactions

Why Test Equality Across Subgroups?

A treatment can have a detectable effect in one subgroup and an uncertain effect in another without the effects being detectably different from each other. The question “does treatment work in this subgroup?” is not the same as “does the treatment effect differ across subgroups?” The second question requires a direct test of a contrast between subgroup effects.

Before testing, specify the subgroup estimands and the effect scale. As established in the earlier material on effect modification and subgroup estimands, treatment effects can differ across subgroups on one scale but not another. A test of equal risk differences, for example, is not a test of equal risk ratios. The target populations for the subgroup effects should also be defined before estimation.

Definition: Let \(\tau_s\) denote the treatment effect in subgroup \(s\), defined for a specified target population and on a specified scale. For \(K\) subgroups, the joint null hypothesis of equal effects is \(H_0:\tau_1=\tau_2=\cdots=\tau_K\). The alternative is that at least one subgroup effect differs from another. This is an omnibus test: rejecting the null establishes evidence against equality, but does not by itself identify which subgroup effects differ.

For two subgroups, the null can be written \(H_0:\tau_1-\tau_2=0\). With more than two subgroups, equality is a collection of restrictions. For instance, comparing each of three effects with the first gives the two restrictions \(\tau_1-\tau_2=0\) and \(\tau_1-\tau_3=0\). These restrictions test the same overall equality null as any other set of \(K-1\) independent contrasts.

Formulating the Joint Test

Write the subgroup effects as a vector \(\boldsymbol{\tau}=(\tau_1,\ldots,\tau_K)^\mathsf{T}\), and let \(\widehat{\boldsymbol{\tau}}\) be its estimated value. A contrast matrix \(C\) contains \(K-1\) linearly independent comparisons. Equal effects are equivalent to \(C\boldsymbol{\tau}=\boldsymbol{0}\). The estimated contrasts are \(C\widehat{\boldsymbol{\tau}}\), and their estimated covariance matrix is \(C\widehat{V}C^\mathsf{T}\), where \(\widehat{V}\) is the covariance matrix of the subgroup effect estimates.

If the estimates are approximately jointly normal, the covariance estimate is suitable, and the contrast covariance matrix has full rank, the Wald statistic is

W=(C\widehat{\boldsymbol{\tau}})^\mathsf{T} (C\widehat{V}C^\mathsf{T})^{-1} (C\widehat{\boldsymbol{\tau}})

Under \(H_0\), \(W\) is approximately distributed as a chi-square random variable with \(K-1\) degrees of freedom. The p-value is the probability, under this null reference distribution, of obtaining a statistic at least as large as the observed \(W\). A small p-value is evidence against equal effects; it is not the probability that the null hypothesis is true.

Justification: If \(\widehat{\boldsymbol{\tau}}\) is approximately normal with mean \(\boldsymbol{\tau}\) and covariance \(V\), then the linear transformation \(C\widehat{\boldsymbol{\tau}}\) is approximately normal with mean \(C\boldsymbol{\tau}\) and covariance \(CVC^\mathsf{T}\). Under the null, its mean is zero. Standardizing this vector by its covariance matrix gives a squared length with a chi-square distribution on \(K-1\) independent dimensions. Replacing \(V\) by its estimate gives the Wald test in large samples.

The covariance matters. Subgroup estimates may be correlated, for example when they come from a shared model or overlapping data. Even for disjoint subgroups, the contrasts themselves share estimated effects and therefore need not be independent. Use the covariance of the contrasts, not a collection of separate standard errors treated as if they answered the joint question.

Worked Examples

Worked Example: Comparing Risk Differences in Two Subgroups

Suppose a hypothetical randomized trial assigns treatment or control within two prespecified subgroups, A and B. Each arm contains 400 people in each subgroup, and the binary outcome is an adverse event. In subgroup A, 80 of 400 treated participants and 128 of 400 control participants experience the event. In subgroup B, the corresponding counts are 112 of 400 and 120 of 400. Test equality of the subgroup risk differences.

The estimated risk difference is treatment risk minus control risk. Thus, in A it is \(80/400-128/400=0.20-0.32=-0.12\), or \(-12\) percentage points. In B it is \(112/400-120/400=0.28-0.30=-0.02\), or \(-2\) percentage points. The estimated difference between subgroup effects is \(-0.12-(-0.02)=-0.10\), or \(-10\) percentage points.

For independent binomial groups, estimate the variance of a risk difference by adding the estimated variances of the two sample proportions. The treatment-effect variance in A is

\widehat{\operatorname{Var}}(\widehat{\tau}_A) =\frac{0.20(0.80)}{400}+\frac{0.32(0.68)}{400} =0.000400+0.000544=0.000944.

The corresponding estimate for B is

\widehat{\operatorname{Var}}(\widehat{\tau}_B) =\frac{0.28(0.72)}{400}+\frac{0.30(0.70)}{400} =0.000504+0.000525=0.001029.

The two subgroup estimates are independent because they use disjoint randomized groups. Therefore the variance of their difference is \(0.000944+0.001029=0.001973\), and its standard error is \(\sqrt{0.001973}=0.04442\). The Wald statistic on the standard-normal scale is

z=\frac{-0.10}{0.04442}=-2.251.

The two-sided p-value is approximately \(0.0244\). At a 5% significance level, this is evidence that the risk differences differ between the two subgroups. Because treatment was randomized within each subgroup, the within-subgroup risk differences estimate causal effects for the trial’s subgroup populations, subject to consistency, adequate outcome measurement, and no relevant interference. This conclusion concerns heterogeneity on the risk-difference scale.

For comparison, the within-A test of zero risk difference gives \(-0.12/\sqrt{0.000944}=-3.906\), while the within-B test gives \(-0.02/\sqrt{0.001029}=-0.623\). Those tests address whether each effect is zero, not whether the two effects are equal. The direct contrast test answers the equality question.

Worked Example: A Three-Subgroup Wald Test

In a hypothetical study, estimated subgroup risk differences, expressed in percentage points, are \(3\), \(2\), and \(1\). Suppose their estimated covariance matrix is the identity matrix in squared percentage points: each estimate has standard error \(1\) percentage point, and the estimates are uncorrelated. Test whether all three population risk differences are equal using the first subgroup as the reference.

The contrast matrix and estimated contrast vector are

C= \begin{pmatrix} 1&-1&0\\ 1&0&-1 \end{pmatrix}, \qquad C\widehat{\boldsymbol{\tau}}= \begin{pmatrix} 3-2\\ 3-1 \end{pmatrix} = \begin{pmatrix} 1\\ 2 \end{pmatrix}.

Since \(\widehat{V}=I\), the contrast covariance matrix is \(CIC^\mathsf{T}=CC^\mathsf{T}\). Its off-diagonal entry is \(1\): the two contrasts both contain the estimated first subgroup effect with coefficient \(1\). Direct multiplication gives

C\widehat{V}C^\mathsf{T} = \begin{pmatrix} 2&1\\ 1&2 \end{pmatrix}, \qquad (C\widehat{V}C^\mathsf{T})^{-1} = \frac{1}{3} \begin{pmatrix} 2&-1\\ -1&2 \end{pmatrix}.

Therefore,

W= \begin{pmatrix}1&2\end{pmatrix} \frac{1}{3} \begin{pmatrix}2&-1\\-1&2\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} =2.

There are \(K-1=2\) restrictions, so the reference distribution is chi-square with 2 degrees of freedom. The p-value is \(P(\chi^2_2\geq 2)=e^{-1}\approx0.368\). These estimates do not provide strong evidence against equality, even though their numerical values are not identical. The shared reference estimate is why the two contrasts have covariance \(1\), rather than zero.

Worked Example: Testing a Logistic Regression Interaction

Consider a hypothetical randomized trial with a binary outcome and a binary subgroup indicator \(S\), where \(S=0\) is the reference subgroup. Treatment \(A\) is coded 0 for control and 1 for treatment. A logistic model is fit:

\operatorname{logit}\{P(Y=1\mid A,S)\} =\beta_0+\beta_A A+\beta_S S+\beta_{AS}AS.

Assume the logit mean model is correctly specified and the standard error accounts appropriately for the sampling design. On the odds-ratio scale, equal treatment effects across these two subgroups correspond to \(H_0:\beta_{AS}=0\). Suppose the fitted interaction coefficient is \(0.60\) with standard error \(0.25\).

The Wald statistic is \(z=0.60/0.25=2.40\), giving a two-sided p-value of approximately \(0.0164\). This is evidence against equal treatment odds ratios across the subgroups under the model. The ratio of the subgroup-specific treatment odds ratios is \(\exp(0.60)=1.822\): the treatment odds ratio in \(S=1\) is estimated to be about 1.82 times the treatment odds ratio in \(S=0\). The interpretation is on the odds-ratio scale; it does not establish equality or inequality of effects on the risk-difference scale.

Assumptions and Interpretation

A statistical interaction test and a causal conclusion require different considerations. The test quantifies sampling uncertainty in estimated effects or model coefficients. For those estimates to represent causal effects, the estimands must be clearly defined and the treatment comparisons must be identified causally. Random assignment within the subgroups can support that identification; in observational settings, identification requires appropriate control of confounding along with the relevant consistency and positivity conditions. The estimator and its covariance must also be appropriate for the design and outcome.

The Wald approximation relies on sufficiently regular, adequately precise estimates and a correctly estimated covariance matrix. Sparse data or unstable model fits can make the approximation unreliable. A failure to reject equality is not proof that the subgroup effects are equal: it may reflect limited information. Conversely, a small p-value does not show that the difference is clinically important. Statistical evidence and the size and practical meaning of the estimated contrast are separate considerations.

Key takeaway: To test treatment-effect heterogeneity, compare subgroup effects directly on a prespecified scale. The joint null is equality across all subgroups, and its test must use the covariance of the effect contrasts. Separate tests of whether each subgroup effect is zero do not test this null.

Common Mistake

Wrong reasoning: “The treatment effect is statistically significant in subgroup A but not significant in subgroup B, so treatment effects differ.” This compares two p-value labels, not the effects themselves. One estimate can be more precise than the other, and a significant result in one group plus a nonsignificant result in another does not imply a significant difference between them.

Correction: Form the contrast \(\widehat{\tau}_A-\widehat{\tau}_B\), calculate its standard error using the covariance of the subgroup estimates, and test whether the contrast is zero. For several subgroups, test the full set of independent equality restrictions jointly. State the effect scale so the conclusion has a clear meaning.

Check Your Understanding

Use the definitions and testing logic in this tutorial to answer the following questions.

For three subgroups, how many independent restrictions are needed to test equality of all subgroup effects?
Why does “significant in one subgroup but not another” fail to establish a treatment-by-subgroup interaction?
In a Wald test, what role does \(C\widehat{V}C^\mathsf{T}\) play?
In a logistic model with a treatment-by-subgroup product term, what scale of treatment-effect heterogeneity does that term describe?
What does failure to reject the joint equality null allow you to conclude, and what does it not establish?