Expected Mean Squares in the Two-Period Crossover ANOVA

Why Expected Mean Squares Matter

An ANOVA table does more than partition variability into named sources. Its expected mean squares show which variance components and fixed effects contribute, on average, to each mean square. Comparing the right pair of mean squares is essential: the treatment statistic must use a denominator that reflects the random variation in the treatment contrast, not simply the largest or most familiar residual term.

In a balanced two-period crossover, each subject receives both treatments, and equal numbers of subjects are assigned to each of the two treatment sequences. A subject’s overall response level can vary substantially between subjects, but that level is shared across the subject’s two periods. It therefore does not contribute to the treatment comparison in the same way as the subject’s period-to-period variation. The expected mean squares make this distinction explicit.

Model and Assumptions

Let \(Y_{jik}\) be the log-scale endpoint for subject \(i\) in sequence \(j\), measured in period \(k\). There are two sequences, TR and RT, and \(n\) subjects in each sequence. Thus there are \(2n\) subjects and \(4n\) observations. Each subject receives treatment T once and treatment R once. Use the additive model

Here \(\mu\) is the overall mean; \(S_j\) is the fixed sequence effect; \(U_{i(j)}\) is the random effect for subject \(i\) within sequence \(j\); \(P_k\) is the fixed period effect; and \(T_{\ell(j,k)}\) is the fixed treatment effect assigned in that sequence and period. The subject effects are independent with mean zero and variance \(\sigma_U^2\). The errors are independent with mean zero and variance \(\sigma^2\), and are independent of the subject effects. The fixed effects use sum-to-zero coding: \(S_1+S_2=0\), \(P_1+P_2=0\), and \(T_T+T_R=0\).

The model assumes equal group sizes, no carryover contribution to the measured response, additive sequence, period, and treatment effects, and a common residual variance. For the exact F tests described here, the random subject effects and residual errors are normally distributed. The balanced design makes sequence, period, and treatment contrasts orthogonal under this model. Serious departures—such as unequal sequence sizes, missing measurements, or treatment-related variance differences—can change the simple mean-square relationships.

ANOVA Sources and Expected Mean Squares

The balanced ANOVA separates the sequence contrast, subject differences within sequence, period contrast, treatment contrast, and the remaining subject-by-period variation. The source “subject(sequence) by period” means variation in a subject’s two-period pattern that remains after accounting for the model’s sequence, period, and treatment terms. It is the treatment-test error term.

For example, let the treatment contrast be \(\delta=T_T-T_R\). Under sum-to-zero coding, \(T_T=\delta/2\) and \(T_R=-\delta/2\), so \(\sum_\ell T_\ell^2=\delta^2/2\). The treatment EMS is therefore \(\sigma^2+n\delta^2\). Under the null hypothesis \(H_0:\delta=0\), its expectation is \(\sigma^2\), the same as the subject(sequence)-by-period error EMS. This matching of expectations motivates the treatment F ratio.

Why the Treatment Denominator Is Within-Subject Error

A short derivation using subject-level treatment differences shows why the subject effect cancels. Define \(d_{ji}\) as that subject’s log response on T minus the log response on R. The subject’s \(U_{i(j)}\) appears once in each response, so it subtracts out. The period contrast can remain, but its sign is opposite in the two sequences. Write that contrast as \(c\) for TR and \(-c\) for RT. Then

The variance \(2\sigma^2\) follows because \(e_{ji}\) is the difference between two independent residual errors, each with variance \(\sigma^2\). With equal sequence sizes, averaging the two sequence means cancels \(c\) and estimates \(\delta\). The residual sum of squares in the subject-by-period part of the ANOVA can be written

where \(\bar d_j\) is the mean difference in sequence \(j\). Centering within each sequence removes both \(\delta\) and the sequence-specific period contribution. Each centered difference has variance \(2\sigma^2\); multiplying its sum of squares by \(1/2\) gives expected residual sum of squares \((2n-2)\sigma^2\), with \(2n-2\) degrees of freedom. Thus the error mean square has expectation \(\sigma^2\).

By contrast, the mean square for subjects within sequence contains the random subject effect. Its EMS is \(\sigma^2+2\sigma_U^2\). That source describes between-subject differences in overall level, which cancel in the paired treatment difference. It is not the appropriate denominator for testing treatment.

The numerator and denominator have the same expected value under \(H_0\). Under the normality and independence assumptions, the ratio has the stated F distribution. A large value is evidence against equal treatment means on the log scale; it is not evidence that a null hypothesis has a particular probability of being true.

Using the Denominator Correctly

The next hypothetical ANOVA summary illustrates why the two subject-related mean squares must not be confused. Suppose there are \(n=10\) subjects per sequence, and the fitted table gives \(MS_{\mathrm{treatment}}=0.0120\), \(MS_{\mathrm{subject(sequence)\ by\ period}}=0.00300\), and \(MS_{\mathrm{subject(sequence)}}=0.0280\), all in squared log units. The correct treatment statistic is

At the 5% level, the upper critical value for \(F_{1,18}\) is approximately 4.414. Since \(4.00<4.414\), this statistic does not reject \(H_0\) at \(\alpha=0.05\). Using the subject-within-sequence mean square instead would give \(0.0120/0.0280=0.429\), but that is not the treatment test: the denominator includes between-subject variation that cancels from the within-subject contrast.

Diagnostics, Comparisons, and Limitations

Before relying on this decomposition, verify that subjects are assigned to both sequences in equal numbers and that each included subject has both period measurements. Check the treatment and period labels against the assigned sequences, since a coding error can reverse or corrupt the paired differences. On the fitted model’s residuals, look for striking non-normality, influential observations, sequence-specific spread, and patterns suggesting that residual variance differs by treatment or period. These checks can reveal model concerns, but a small study has limited ability to diagnose them.

The paired-difference view is a useful way to see the cancellation of the subject intercept and the role of the two sequence means. The ANOVA view makes the full source decomposition and its degrees of freedom explicit. In a complete balanced study, these views agree on the treatment contrast and its within-subject error. The simple ANOVA formulas are less suitable when the data are unbalanced or incomplete; they also depend on the additive model and the absence of carryover. A large sequence mean square is not, on its own, a substitute for checking those assumptions or a reason to change the treatment denominator.

Common Mistakes

• Using subject(sequence) as the treatment denominator. That mean square contains the between-subject variance component \(2\sigma_U^2\). Treatment is a within-subject contrast, so its denominator is subject(sequence) by period.
• Calling the treatment EMS equal to the residual EMS under every condition. They match under \(H_0\); under a nonzero fixed treatment contrast, the treatment EMS also contains \(n\delta^2\).
• Ignoring the sequence balance. Equal sequence sizes allow the period contribution to cancel in the average of the sequence-specific treatment differences. Unequal sizes need not have this simple cancellation.
• Treating an F statistic as the probability that treatments are equal. The F statistic is a ratio of mean squares. Its p-value is calculated under \(H_0\) and the model assumptions; it is not the probability that \(H_0\) is true.