Contrasts for Comparing Group Effects Across Strata

Why Compare Group Differences Across Strata?

A group difference can depend on the level of a second categorical factor. For example, a hypothetical program might be associated with a larger mean score difference in one age stratum than in another. The quantity of interest is not either group difference alone, but the difference between those group differences. A contrast gives a precise way to define that comparison and to quantify its uncertainty.

This tutorial uses cell means: the population mean outcome for each combination of group and stratum. The same calculations can be expressed using coefficients in a regression model. The essential task is to specify which cell means enter the comparison and with what weights.

For two strata, a common contrast is \(C=D_1-D_2\). Written directly in terms of cell means, this is

The signs indicate exactly what is being compared. A positive value means the group \(A\)-minus-\(B\) difference is larger in stratum 1 than in stratum 2. The units are the same as the outcome’s units. A contrast is not automatically an average over strata: its weights determine the comparison.

Estimating a Contrast and Its Standard Error

Replace each population cell mean with its sample estimate, \(\hat{\mu}_{gs}\), to obtain the estimated contrast \(\hat{C}\). To quantify uncertainty, consider a categorical-factor model that estimates a separate mean for every group-by-stratum cell. Assume observations are independent, the cell-mean model is appropriate, and the errors have a common variance \(\sigma^2\). For an exact \(t\)-based interval and test, also assume normally distributed errors. Under these assumptions, the estimated cell means from disjoint cells are independent, and the estimated common error variance is the model’s residual mean square, \(MSE\).

If cell \(gs\) has \(n_{gs}\) observations, then the estimated variance of the cell mean is \(MSE/n_{gs}\); its true variance is \(\sigma^2/n_{gs}\). Write a contrast as \(C=\sum_{g,s}c_{gs}\mu_{gs}\), where the \(c_{gs}\) are its cell-mean coefficients. Its estimated standard error is

This formula squares each coefficient, so negative signs in the contrast do not cancel variance. It also shows why both residual variability and the cell sizes matter. With unequal variances or dependent observations, this particular formula need not apply; the uncertainty calculation must match the model and sampling structure.

Worked Comparisons

How the Uncertainty Formula Follows

The standard-error calculation follows from the variance of a weighted sum. For independent cell-mean estimates, the variance of \(\hat{C}=\sum_{g,s}c_{gs}\hat{\mu}_{gs}\) is the sum of their variances multiplied by squared coefficients. Each cell mean has estimated variance \(MSE/n_{gs}\). Substitution gives

Taking the square root gives \(SE(\hat{C})\). Under the model assumptions, the contrast estimate is normally distributed around \(C\), and the pooled residual variance is estimated independently using \(N-K\) residual degrees of freedom. Dividing the centered estimate by its estimated standard error therefore gives the stated \(t\) statistic. This is why the same standard error supports both a test and a confidence interval.

Choosing and Interpreting the Contrast

A contrast should reflect the comparison of interest, not be selected only because it produces a convenient calculation. For two groups and two strata, \(D_1-D_2\) has a direct interpretation as the change in the group difference from one stratum to the other. With more strata, several scientifically meaningful contrasts may be possible: comparing two particular strata, comparing one stratum with an average of others, or comparing two weighted averages. State the weights explicitly so the reader can see what is being compared.

A contrast test addresses the particular linear combination specified in its null hypothesis. It does not automatically test every possible difference among group effects. A single contrast can be nonzero even if another contrast is zero, and a nonsignificant result does not prove exact equality. Conclusions should name the groups, strata, outcome, units, and uncertainty in context.

Common Pitfall

A frequent error is to compare two separate \(p\)-values and infer that the group effects differ because one is significant and the other is not. That reasoning does not test the difference between the effects. The correct procedure estimates a contrast such as \(D_1-D_2\), calculates its standard error using all four relevant cell means and their covariance structure, and tests or forms an interval for that contrast directly. Also take care not to omit coefficients from a weighted comparison: halving two stratum differences and subtracting a third defines a different quantity from simply subtracting one difference from another.