Subjects Nested Within Sequence as a Random Effect

Why Subjects Are Nested Within Sequence

In a two-period crossover study, each participant follows one sequence, such as test then reference (TR) or reference then test (RT). The participant contributes measurements in both periods, but participants in one sequence are not the same participants as those in the other. Thus, a participant is nested within sequence: the participant belongs to exactly one sequence, and their measurements share a persistent subject-specific component.

For a pharmacokinetic endpoint analyzed on the log scale, this component represents a subject’s tendency to have higher or lower log exposure than other subjects in the same sequence, after accounting for the design effects. It is not an additional treatment effect. The earlier material on sequence, period, and treatment effects in crossover ANOVA distinguishes those design effects; here, the focus is how to represent the subject-to-subject heterogeneity in the model.

The Crossover Model

Let \(Y_{ijk}\) be the log-transformed endpoint for subject \(j\) in sequence \(i\) during period \(k\), where \(i\) is TR or RT and \(k=1,2\). Let \(d(i,k)\) denote the treatment received by sequence \(i\) in period \(k\). A random-intercept model is

Here, \(\mu\) is the overall mean; \(\alpha_i\), \(\pi_k\), and \(\tau_{d(i,k)}\) are the fixed sequence, period, and treatment contributions; \(S_{j(i)}\) is the random subject effect; and \(\varepsilon_{ijk}\) is the remaining measurement-level error. With two treatments, the sequence and treatment effects require suitable constraints, such as effects summing to zero, so that the model is identifiable.

The usual assumptions for this model are that subject effects have mean zero and common variance, residual errors have mean zero and common variance, and the subject effects and residual errors are mutually independent. A standard normal formulation is

Subject effects are independent across subjects, and residual errors are independent across periods and subjects under this simple model. Both random components are independent of the fixed design effects. These assumptions mean the subject effect is a shared additive shift: it raises or lowers both measurements for the same subject by the same amount on the log scale. Normality is a distributional assumption used for model-based inference; the variance interpretation itself is the spread of the subject effects around zero.

The sequence effect remains a fixed design effect in this formulation. A random subject effect within sequence does not mean that sequence itself is random. Nor does “nested” mean that measurements are nested in different subjects within each period: the same subject is observed in both periods, and their subject effect is shared across those observations.

What the Subject Variance Represents

The variance \(\sigma_S^2\) quantifies how much subjects differ in their underlying log-endpoint levels, conditional on the fixed sequence, period, and treatment terms. A larger value indicates greater between-subject heterogeneity. It does not measure the average treatment difference, and it is not the residual variance \(\sigma_e^2\), which represents remaining period-level variation after the subject effect and fixed effects have been accounted for.

Because a subject effect is shared across periods, it also creates dependence between measurements from the same subject. Under the assumptions above, the marginal variance of one observation and the covariance of two observations from the same subject in different periods are

The covariance follows because the only random component common to both observations is \(S_{j(i)}\). Independent period-specific residuals contribute no covariance. For observations from different subjects, the covariance is zero under this model. When the variances are positive, the within-subject correlation is \(\sigma_S^2/(\sigma_S^2+\sigma_e^2)\).

Why the Shared Effect Matters for a Crossover

A useful way to see the role of the random effect is to subtract the two measurements from one subject. Write the fixed contribution in each period as \(F_{ijk}=\mu+\alpha_i+\pi_k+\tau_{d(i,k)}\). Then

The subject effect cancels exactly. This is the algebraic reason that a crossover comparison can control for a subject’s persistent level: the treatment contrast is formed from measurements on the same person. The cancellation does not remove period effects or period-specific error, and it relies on the additive random-intercept structure. It also does not mean that the subject variance is unimportant; it describes the level differences and within-subject dependence in the original observations.

Assumptions, Diagnostics, and Alternatives

The random-intercept model is useful when a subject’s persistent shift is a reasonable representation of their repeated measurements. Its assumptions should be checked against the design and data rather than treated as automatic.

• Correct subject identity and nesting. Each subject must be assigned to one sequence, and the same identifier must be used for both period records. Reusing a subject number across sequences without distinguishing sequence can incorrectly link different participants.
• Common subject variance. The model assumes the subject-level spread is the same in TR and RT. Large differences in adjusted spreads by sequence may indicate that this assumption is inadequate or that the model’s other terms are misspecified.
• Shared additive shift. The same subject effect is assumed to apply to both treatments and periods. Strong evidence that subjects have different treatment responses, rather than just different overall levels, is not captured by a random intercept alone.
• Residual behavior. Residuals should be centered around zero, with no obvious pattern against fitted values or period. Marked skewness, outliers, or unequal residual spreads can undermine the normal, common-variance model and its standard inferential procedures.

A common alternative is to treat each subject as a fixed blocking effect. That approach estimates a separate constant for each observed subject, rather than describing a population variance of subject effects. It can represent subject-specific levels, but it does not directly estimate \(\sigma_S^2\). At the other extreme, treating all period observations as independent omits the shared subject component and can misrepresent their covariance. The random-intercept model lies between these descriptions: it accounts for repeated observations and models subject differences as draws from a population distribution.

The random-intercept model has limits. With only two observations per subject, the data provide limited information about more elaborate subject-specific covariance patterns. The model also cannot represent subject-specific treatment slopes unless such terms are added, and it can be sensitive to a small number of unusual subjects or to misspecified residual assumptions. The variance component describes heterogeneity under the stated model; it is not a direct measure of the range of individual treatment effects.

Common Mistakes

• Calling the subject effect a treatment effect. The random intercept is a persistent subject-level deviation shared across periods. Treatment differences are represented by the treatment terms.
• Confusing \(\sigma_S^2\) with \(\sigma_e^2\). The former is between-subject variance; the latter is residual period-level variance. The marginal variance of an observation includes both.
• Assuming nesting means independent period records. The subject is nested in sequence, but that subject’s two observations remain linked through the shared random effect.
• Interpreting the random effect as a causal explanation. A variance component describes variation represented by the model; it does not identify why subjects differ.