Deriving Tests for Prespecified Effect Heterogeneity

Why Test Prespecified Effect Heterogeneity?

An estimated treatment effect can differ across subgroups, but observed differences alone do not establish that the population effects differ. Sampling variation matters, and so does the question being asked: a joint test of all treatment interactions is not the same as a test of one contrast between two subgroup effects. Both require the estimated covariance among the relevant coefficients.

This tutorial develops Wald tests for hypotheses specified in advance. The tests are useful when a linear interaction model represents treatment effects across a finite set of subgroups. The inference concerns the chosen model and the specified target: changing the contrast or the population used to average subgroup effects can change the result.

Model and Hypotheses

Suppose the treatment indicator \(D_i\) is binary and each unit belongs to exactly one of \(G\) prespecified subgroups. Use subgroup 1 as the reference category. A linear interaction model for the conditional mean can be written as

where \(S_i\) denotes subgroup membership, \(X_i\) denotes other included covariates, and \(m(\cdot)\) collects the non-treatment terms. The implied treatment-effect parameter for subgroup 1 is \(\tau_1\); for subgroup \(g\geq 2\), it is \(\tau_g=\tau_1+\gamma_g\). Thus \(\gamma_g\) is the difference between subgroup \(g\)'s effect and subgroup 1's effect.

Interpreting these parameters causally requires the relevant identification conditions, such as random assignment or appropriate conditional exchangeability, consistency, and positivity, as well as a correctly specified conditional-mean model for the estimand at issue. The Wald calculations below concern sampling uncertainty in the model parameters; they do not by themselves establish those causal conditions.

Let \(\hat\beta\) be the full coefficient estimator and let \(\widehat V\) estimate its covariance matrix. Write the null restrictions as \(R\beta=r\), where \(R\) is a specified matrix and \(r\) is a specified vector. The estimated restriction is \(R\hat\beta-r\), with estimated covariance \(R\widehat V R'\). If that covariance is nonsingular, the Wald statistic is

For \(q\) linearly independent restrictions, under the null and standard large-sample conditions, \(W\) has an asymptotic \(\chi^2_q\) distribution. These conditions include asymptotic normality of the estimator and consistent covariance estimation; the restriction covariance must have full rank. The covariance estimate may be model-based or robust, but it must be appropriate for the sampling or assignment structure.

Joint Tests of Interaction Coefficients

In the model above, the null that all specified subgroup effects equal the reference subgroup effect is

This is a joint test of \(q=G-1\) restrictions. Rejection indicates evidence that at least one of these prespecified effects differs from the reference effect; it does not identify which subgroup differs, and it does not imply that every subgroup effect differs. The test is not a test that the treatment effect is zero in every subgroup. That different null would also restrict the reference treatment coefficient.

Testing a Contrast Between Subgroup Effects

A contrast is a specified linear combination of parameters. For example, the difference between subgroup 2 and subgroup 3 effects is \(\tau_2-\tau_3=\gamma_2-\gamma_3\). More generally, if the target is \(c'\beta\), its estimate is \(c'\hat\beta\), and its estimated variance is \(c'\widehat Vc\). A two-sided Wald test of \(H_0:c'\beta=0\) uses the squared statistic \((c'\hat\beta)^2/(c'\widehat Vc)\), asymptotically compared with \(\chi^2_1\). Equivalently, the signed statistic \(c'\hat\beta/\sqrt{c'\widehat Vc}\) is compared with a standard normal distribution.

Reference Populations and Aggregated Effects

Subgroup effects can also be combined into an average for a specified reference population. Let \(\hat{\boldsymbol\tau}\) collect the subgroup effect estimates and let \(w_g\) be fixed target-population weights, with \(w_g\geq0\) and \(\sum_g w_g=1\). The weighted estimate is \(\hat\theta_w=\mathbf{w}'\hat{\boldsymbol\tau}\). Its variance is \(\mathbf{w}'\widehat V_\tau\mathbf{w}\), where \(\widehat V_\tau\) is the full estimated covariance matrix of the subgroup effects. The weights define the target: they are not automatically the proportions in the analysis sample.

A useful justification follows from the variance of a linear transformation. If \(\hat{\boldsymbol\tau}\) has covariance matrix \(V_\tau\), then

Expanding this expression gives \(\sum_g w_g^2\operatorname{Var}(\hat\tau_g)+2\sum_{g<h}w_gw_h\operatorname{Cov}(\hat\tau_g,\hat\tau_h)\). The covariance terms therefore matter whenever subgroup estimates are statistically related.

Why Covariance and the Target Matter

The covariance matrix is part of the inferential result, not a secondary detail. For a contrast, positive covariance can reduce the variance of a difference; negative covariance can increase it. For a joint test, covariance affects the geometry of the estimated coefficient vector relative to its uncertainty. Using only standard errors, or silently setting covariances to zero, can therefore change a test statistic, interval, or conclusion.

The reference population is equally substantive. A weighted average answers a question about effects for the population represented by the specified weights. A sample-weighted average answers a different question if the sample subgroup proportions differ from the target population proportions. State the target weights and the units of the outcome when reporting an aggregate effect.

Common Pitfall

For transparent reporting, name the restrictions or contrast, give the coefficient estimate and covariance method, report the test statistic with degrees of freedom and p-value, and identify the target population for any weighted average. A statistically significant joint test says that at least one restricted coefficient departs from its null value; a prespecified contrast gives a more focused answer to a particular comparison.