A Worked Analysis of Group Differences Across Two Factors

Why Compare Group Differences Across Two Factors?

A group difference may not be the same in every setting. For example, a new tutoring approach might be associated with a larger improvement in remote classes than in classroom sessions. Comparing the approach groups in only one setting would not answer whether their difference changes across settings. A two-by-two analysis addresses that question by comparing the group difference at both levels of a second factor.

The key quantity is a difference-in-differences: take the difference between the two approach groups in one setting, take the same difference in the other setting, and subtract. This contrast measures how much the approach-group difference varies across settings. Its estimate, standard error, test, and confidence interval all depend on the four cell means.

Definition: In a two-by-two design, two categorical factors each have two levels, producing four cells. A cell mean is the mean outcome for one combination of factor levels. The interaction contrast (or difference-in-differences) is the difference between the group difference at one level of the second factor and the group difference at its other level.

Model and Interaction Contrast

Let the factors be tutoring approach, usual or new, and class format, classroom or remote. Write \(\mu_{a,f}\) for the population mean score in the cell with approach \(a\) and format \(f\). The cell-means model is

Y_{a,f,k}=\mu_{a,f}+\varepsilon_{a,f,k},

where \(Y_{a,f,k}\) is the score for the \(k\)th participant in cell \((a,f)\), and \(\varepsilon_{a,f,k}\) is that participant's error from the cell mean. The interaction contrast, oriented as the new-versus-usual difference in remote classes minus the new-versus-usual difference in classroom sessions, is

\Delta=(\mu_{\text{new,remote}}-\mu_{\text{usual,remote}}) -(\mu_{\text{new,classroom}}-\mu_{\text{usual,classroom}}).

If \(\Delta=0\), the new-versus-usual difference is equal across the two formats. If \(\Delta\) is positive, that difference is larger in remote classes than in classroom sessions, in the units of the outcome. A nonzero contrast describes an interaction for these two factors: the difference associated with one factor varies across the levels of the other.

For the usual pooled-error t test and confidence interval, assume the errors are independent, normally distributed with mean zero, and have a common variance \(\sigma^2\) across the four cells. Independence should be plausible from how observations were collected; normality and equal variance are model conditions to assess using the data, not facts guaranteed by the design. Random assignment, when it is used and carried out appropriately, can support causal comparisons, but the statistical interaction by itself does not establish causation.

A Hypothetical Two-by-Two Example

Suppose, hypothetically, participants are assigned to one of two tutoring approaches and one of two class formats. Their outcome is a score on a common assessment, measured in points. The table gives the number of participants and mean score in each cell. It also gives the within-cell sum of squared deviations from that cell's mean, which is used to estimate the common error variance.

Approach	Format	Sample size	Sample mean (points)	Within-cell sum of squares
Usual	Classroom	10	70	144
New	Classroom	8	76	128
Usual	Remote	12	68	160
New	Remote	10	80	144

Worked Example: Estimate the Four Cell Means

The four sample means are estimates of their corresponding population cell means. For instance, the sample mean for new tutoring in remote classes is 80 points, so the estimated population mean in that cell is 80 points. The four estimates, in the order usual-classroom, new-classroom, usual-remote, and new-remote, are 70, 76, 68, and 80 points.

First compare the approaches within classroom sessions:

\text{new minus usual}=76-70=6\text{ points}.

Then compare the approaches within remote classes:

\text{new minus usual}=80-68=12\text{ points}.

The observed approach-group difference is 6 points larger in remote classes than in classroom sessions. This is a description of the sample means. To estimate the population interaction contrast and assess its uncertainty, combine the two differences and use the residual variation within cells.

Estimating the Contrast and Its Standard Error

Let \(\bar{Y}_{a,f}\) denote the sample mean in cell \((a,f)\), and let \(n_{a,f}\) denote that cell's sample size. The estimated interaction contrast is obtained by replacing each population mean in \(\Delta\) with its sample mean. Under the common-variance model, the variance of a cell sample mean is \(\sigma^2/n_{a,f}\). Since the four cell means come from independent groups, the variance of their contrast is the sum of those four variances: each mean has coefficient \(+1\) or \(-1\), and squaring either coefficient gives 1. In practice, estimate \(\sigma^2\) with the pooled within-cell mean square, \(MSE\).

Standard error: For independent cells with common error variance, the estimated standard error of the two-by-two interaction contrast is

SE(\widehat{\Delta})= \sqrt{MSE\left( \frac{1}{n_{\text{new,remote}}} +\frac{1}{n_{\text{usual,remote}}} +\frac{1}{n_{\text{new,classroom}}} +\frac{1}{n_{\text{usual,classroom}}} \right)}.

In the table, the total sample size is \(N=10+8+12+10=40\). The cell-means model estimates four means, so the residual degrees of freedom are \(df=N-4=36\). The total within-cell sum of squares is \(144+128+160+144=576\), giving \(MSE=576/36=16\) points squared.

Worked Example: Calculate the Difference-in-Differences and Its Standard Error

Using the sample means, the estimated contrast is

\widehat{\Delta}=(80-68)-(76-70)=12-6=6\text{ points}.

To find its standard error, substitute \(MSE=16\) and the four cell sizes into the formula:

\begin{aligned} SE(\widehat{\Delta}) &=\sqrt{16\left(\frac{1}{10}+\frac{1}{12}+\frac{1}{8}+\frac{1}{10}\right)}\\ &=\sqrt{16(0.4083)}\\ &=\sqrt{6.533}\\ &=2.556\text{ points}. \end{aligned}

The estimate is 6 points, with an estimated standard error of 2.556 points. The standard error describes the estimated sampling variability of the contrast under the model; it is not the standard deviation of individual assessment scores.

Testing and Quantifying Uncertainty

To test for an interaction between these two factors in the population, use the hypotheses \(H_0:\Delta=0\) and \(H_a:\Delta\ne0\). Under the stated normal, independent, common-variance model, the test statistic follows a t distribution with \(df=N-4\) when \(H_0\) is true. The test statistic is the estimated contrast minus its null value, divided by its standard error. A two-sided test is appropriate when the question is whether the differences vary, regardless of direction.

Worked Example: Test the Interaction and Construct a Confidence Interval

For the hypothetical data, test \(H_0:\Delta=0\) against \(H_a:\Delta\ne0\) at \(\alpha=0.05\). Using the contrast estimate of 6 points and its standard error of 2.556 points:

t=\frac{\widehat{\Delta}-0}{SE(\widehat{\Delta})} =\frac{6}{2.556}=2.347,\qquad df=36.

The two-sided p-value is approximately 0.0245. Because \(0.0245<0.05\), reject \(H_0\) at the 5% significance level. The data provide evidence that the new-versus-usual mean-score difference is not the same in the two class formats. The estimated direction is a larger difference for remote classes; the test alone does not show that tutoring approach caused this pattern.

A 95% confidence interval for \(\Delta\) uses the critical value \(t^*=2.028\) for 36 degrees of freedom:

\begin{aligned} \widehat{\Delta}\mathbin{\pm}t^*SE(\widehat{\Delta}) &=6\mathbin{\pm}2.028(2.556)\\ &=6\mathbin{\pm}5.184\\ &=(0.816,\ 11.184)\text{ points}. \end{aligned}

In this hypothetical setting, the interval estimates that the new-versus-usual mean-score difference is about 0.816 to 11.184 points larger in remote classes than in classroom sessions. In repeated samples under the model, the interval procedure would cover the true contrast 95% of the time; this is not a claim that the fixed population contrast has a 95% probability of lying in this particular interval.

Why the Contrast and Its Uncertainty Matter

The four cell means make the comparison transparent, but interpreting any one of them in isolation can miss the question of interest. Here, comparing the new and usual approaches only in classroom sessions gives 6 points; comparing them only in remote classes gives 12 points. The interaction contrast directly quantifies the change between those differences, in points. Its confidence interval communicates both the estimated size and the uncertainty of that change.

The calculation also shows why sample sizes and residual variation matter. Larger cell sample sizes reduce the reciprocal terms in the standard-error formula, while greater within-cell variation increases \(MSE\). A precise-looking difference-in-differences estimate can still be uncertain if the cell samples are small or the outcomes vary substantially within cells.

Common mistake: It is incorrect to conclude that an interaction exists because the approach difference is statistically significant in one format but not statistically significant in the other. Those are separate tests, and a significant result in one group and a nonsignificant result in another does not itself test whether the two differences differ. The direct test is \(H_0:\Delta=0\), using the difference-in-differences and its standard error.

Before reporting the test, check whether the observations can reasonably be treated as independent and whether the common-variance and normal-error conditions are plausible. A residual plot can help identify unequal spread or unusual patterns, and the group sizes and data collection process help assess whether independence is credible. If participants were randomized to both factors, describe that design accurately; if the data are observational, report the interaction as an association rather than a causal effect.

Key takeaway: In a two-by-two analysis, estimate the four cell means, subtract the group differences to form a difference-in-differences, and use the within-cell residual variance and cell sizes to quantify its uncertainty. Interpret the contrast in the outcome's units and in the context of both factors.

Check Your Understanding

Answer each question using the definitions and calculations above.

If the new-versus-usual difference is 4 points in classroom sessions and 9 points in remote classes, what is the interaction contrast when defined as remote minus classroom?
What does a zero population interaction contrast mean for the two approach-group differences?
Why does the standard-error formula add the four reciprocal cell sizes?
For a two-by-two cell-means model with 52 observations, what are the residual degrees of freedom?
Why is it not sufficient to find a significant approach difference in one format and a nonsignificant difference in the other to establish an interaction?