Coding Treatment and Subject Effects in a Scaled Model

Why the model needs careful coding

A replicate crossover study records more than one observation from at least some participants. The analysis must therefore represent both the systematic structure of the design and the dependence among observations from the same participant. In reference-scaled analysis, it must also identify the within-subject residual variance associated specifically with reference-product observations. A model that merely includes a “subject” term and an overall residual variance can obscure that distinction.

The purpose here is to specify the model, interpret its terms, and see why the reference-specific residual variance is identifiable from repeated reference observations under the stated assumptions. The model specification is not itself a variance-component estimation procedure.

Definition: Let $Y_{spj}$ be the log-transformed endpoint for subject $j$ assigned to sequence $s$, measured in period $p$. Let $T_{sp}=1$ when the observation is from the test product and $T_{sp}=0$ when it is from the reference product. A treatment-coded random-intercept model is $$ Y_{spj}=\mu+\gamma_s+\pi_p+\tau T_{sp}+b_{j(s)}+\varepsilon_{spj}. $$ Here $\mu$ is an intercept, $\gamma_s$ is the fixed sequence effect, $\pi_p$ is the fixed period effect, $\tau$ is the fixed test-minus-reference treatment effect on the log scale, $b_{j(s)}$ is the random subject effect for subject $j$ nested within sequence $s$, and $\varepsilon_{spj}$ is the observation-level residual.

Assumptions and interpretation of the terms

To make the variance components precise, assume that subject effects are independent and normally distributed, $b_{j(s)}\sim N(0,\sigma_b^2)$. Conditional on those subject effects, residuals are independent across observations and subjects, with $$ \varepsilon_{spj}\sim \begin{cases} N(0,\sigma_{W,R}^2), & \text{for a reference observation},\\ N(0,\sigma_{W,T}^2), & \text{for a test observation}. \end{cases} $$ The residuals are independent of the subject effects. Thus $\sigma_{W,R}^2$ is the within-subject residual variance for reference observations, while $\sigma_{W,T}^2$ is the corresponding variance for test observations. These variances are on the log scale. The model does not require them to be equal.

The fixed effects describe mean structure. Sequence is a between-subject design factor: a subject follows one assigned sequence. Period represents systematic differences among measurement occasions, and treatment represents the adjusted mean difference between test and reference. Subject nested within sequence represents subject-to-subject differences after accounting for those fixed effects. It is a random intercept shared by all observations from that subject.

With treatment coded as above, $\tau$ is the adjusted difference in mean log response for test versus reference, conditional on the included sequence and period effects. Exponentiating it gives the test-to-reference ratio of geometric means, $\exp(\tau)$. The random-intercept variance $\sigma_b^2$ measures between-subject heterogeneity; it is not the within-subject reference variance. Keeping these meanings separate is central to the specification.

For identifiability, the fixed-effect coding must avoid redundant columns. One convenient choice is reference-level coding: use one sequence as the baseline, one period as the baseline, and reference treatment as the baseline. Alternatively, impose sum-to-zero constraints. These choices change the coefficient labels and intercept, but not fitted cell means or the treatment contrast when the model is full rank.

Worked Example: Coding a Partial-Replicate Design

Suppose a hypothetical three-period study uses sequences RRT, RTR, and TRR, where R denotes reference and T denotes test. Choose RRT as the baseline sequence and period 1 as the baseline period. Let $I(\mathrm{RTR})$ and $I(\mathrm{TRR})$ be sequence indicators, and let $I(p=2)$ and $I(p=3)$ be period indicators. Then a fixed-effects mean can be written $$ E(Y_{spj}\mid b_{j(s)})=\mu+\gamma_{\mathrm{RTR}}I(\mathrm{RTR})+\gamma_{\mathrm{TRR}}I(\mathrm{TRR})+\pi_2I(p=2)+\pi_3I(p=3)+\tau T_{sp}+b_{j(s)}. $$

For a subject in sequence TRR, the period-1 test observation has fixed-effects mean $\mu+\gamma_{\mathrm{TRR}}+\tau$; its conditional mean additionally includes that subject’s $b_{j(s)}$. The period-2 reference observation has mean $\mu+\gamma_{\mathrm{TRR}}+\pi_2+b_{j(s)}$, and the period-3 reference observation has mean $\mu+\gamma_{\mathrm{TRR}}+\pi_3+b_{j(s)}$. The test indicator is zero for both reference observations.

For illustration, suppose the hypothetical fitted coefficients are $\mu=4.50$, $\gamma_{\mathrm{TRR}}=0.02$, $\pi_2=-0.01$, $\pi_3=0.015$, and $\tau=-0.04$, all in log-response units. Ignoring the subject deviation to report population-level fitted means, the three means are $4.50+0.02-0.04=4.48$ for period-1 test, $4.50+0.02-0.01=4.51$ for period-2 reference, and $4.50+0.02+0.015=4.535$ for period-3 reference. The test-minus-reference treatment coefficient is $-0.04$; it is not the difference between two raw observations from different periods.

How repeated reference observations identify their residual variance

The subject random intercept creates covariance among a subject’s repeated observations. Under the assumptions above, any two observations from the same subject have covariance $\sigma_b^2$, because they share $b_{j(s)}$ and their residuals are independent. Their variances differ according to treatment: a reference observation has variance $\sigma_b^2+\sigma_{W,R}^2$, while a test observation has variance $\sigma_b^2+\sigma_{W,T}^2$.

Consider the two reference observations from a subject in sequence RRT, in periods 1 and 2. Subtract their model equations. The subject effect and sequence effect cancel; the treatment effect is zero for both observations. The difference in period effects remains, so adjust for it: $$ D_R=(Y_{s,1,j}-Y_{s,2,j})-(\pi_1-\pi_2) =\varepsilon_{s,1,j}-\varepsilon_{s,2,j}. $$ Because the two residuals are independent, each has variance $\sigma_{W,R}^2$, and therefore $$ \operatorname{Var}(D_R)=\operatorname{Var}(\varepsilon_{s,1,j})+\operatorname{Var}(\varepsilon_{s,2,j})=2\sigma_{W,R}^2. $$ This establishes why repeated reference observations carry information about reference within-subject variability, separately from the shared subject intercept. In an actual analysis, the period effects and variance components are estimated from the full design; one pairwise difference is not, by itself, a reliable estimate of the population variance.

Worked Example: Adjusting a Pair of Reference Observations

A hypothetical RRT subject has log responses 4.20 in period 1 and 4.08 in period 2. Suppose the estimated period effects are $\pi_1=0.03$ and $\pi_2=-0.01$. The unadjusted difference is $4.20-4.08=0.12$ log units. The period difference is $0.03-(-0.01)=0.04$, so the period-adjusted reference contrast is $$ D_R=(4.20-4.08)-\{0.03-(-0.01)\}=0.12-0.04=0.08 $$ log units.

Under the model, this adjusted contrast equals the difference between the two reference residuals. Its population variance is $2\sigma_{W,R}^2$, not $\sigma_b^2+2\sigma_{W,R}^2$, because the same subject intercept occurs in both measurements and cancels. The single observed contrast of 0.08 log units does not determine $\sigma_{W,R}^2$; it illustrates the cancellation that makes repeated reference measurements informative.

Covariance structure and model checks

The model’s covariance structure can be written explicitly. For a subject observed in periods 1, 2, and 3, the covariance matrix is the subject-intercept contribution plus the residual variances on the diagonal: $$ V_j=\sigma_b^2 \begin{pmatrix} 1&1&1\\ 1&1&1\\ 1&1&1 \end{pmatrix} + \begin{pmatrix} \sigma_{W,1}^2&0&0\\ 0&\sigma_{W,2}^2&0\\ 0&0&\sigma_{W,3}^2 \end{pmatrix}, \qquad \sigma_{W,p}^2= \begin{cases} \sigma_{W,R}^2,&\text{if period }p\text{ is reference},\\ \sigma_{W,T}^2,&\text{if period }p\text{ is test}. \end{cases} $$

Worked Example: Building a Subject Covariance Matrix

For a hypothetical RRT subject, assume $\sigma_b^2=0.09$, $\sigma_{W,R}^2=0.04$, and $\sigma_{W,T}^2=0.0625$, in squared log-response units. Periods 1 and 2 are reference; period 3 is test. The marginal variance of each reference observation is $0.09+0.04=0.13$, and that of the test observation is $0.09+0.0625=0.1525$. The covariance between any two observations from this subject is $0.09$. Thus $$ V_j= \begin{pmatrix} 0.13&0.09&0.09\\ 0.09&0.13&0.09\\ 0.09&0.09&0.1525 \end{pmatrix}. $$

For the two reference periods, their difference has variance $0.13+0.13-2(0.09)=0.08$, which equals $2\sigma_{W,R}^2=2(0.04)=0.08$. This agrees with the adjusted-contrast derivation and shows why the marginal variance of a reference observation is not itself the reference residual variance.

Before interpreting model terms, check that the specification matches the design and data. Treatment and period must be correctly assigned for every observation; sequence must reflect the assigned order; and subjects must be identified and nested in their assigned sequence. The design must contain repeated reference observations for the reference-specific residual variance to be informed by within-subject replication. A rank-deficient fixed-effects design, mistaken treatment labels, or treating repeated observations as independent can invalidate the intended interpretation.

The assumptions also have substantive consequences. The random-intercept model assumes a common subject deviation across that subject’s periods, while the residuals are conditionally independent. Carryover, treatment-by-period interactions, changing residual variation across periods, or additional within-subject correlation not captured by the intercept may make this covariance structure inadequate. These are reasons to investigate the design and model fit, not reasons to relabel $\sigma_b^2$ as reference variability.

Alternatives, limitations, and common mistakes

A common-residual model sets $\sigma_{W,R}^2=\sigma_{W,T}^2$. It is simpler, but it does not preserve a separate reference residual variance. That restriction should not be used when the analysis needs to represent reference variability distinctly unless it is justified for the purpose and supported by the applicable analysis plan. At the other extreme, a more flexible covariance model can allow additional covariance patterns, but it requires enough repeated information to estimate its parameters and may not yield the same direct interpretation of a reference-specific residual component.

The random-intercept model is a structured working model, not a universal description of every replicate dataset. It assumes a single subject-level random deviation and independent, treatment-specific residuals. It does not by itself address missing observations, outliers, small-sample inference, or the choice of estimator for variance components. Those matters affect implementation and inference, but they do not change the meanings of the terms in the specification presented here.

Mistake: treating $\sigma_b^2$ as the reference within-subject variance. The subject effect is shared across periods; $\sigma_{W,R}^2$ is the residual variance for reference observations after accounting for fixed effects and subject effect.
Mistake: assigning the same residual variance to every observation by default. If reference variability is the target, explicitly define the reference residual variance and state whether test residual variance is separate or constrained equal.
Mistake: interpreting $\tau$ as an unadjusted raw difference. It is the fixed test-minus-reference log-scale contrast adjusted for the model’s period and sequence terms.
Mistake: assuming sequence and subject are interchangeable. Sequence is a fixed design effect between groups; subject is a random effect nested within the sequence followed.

Key takeaway: In the treatment-coded replicate model, $T_{sp}$ identifies test observations, $\tau$ is the adjusted log-scale treatment contrast, $b_{j(s)}$ accounts for shared subject-level variation, and $\sigma_{W,R}^2$ is the residual variance specific to reference observations. These terms describe different features and should not be conflated.

Check Your Understanding

Answer each question using the model definitions above. Brief solutions follow the questions.

In the equation, what does $T_{sp}=0$ indicate, and which variance is assigned to that observation’s residual?
Why does the subject random intercept cancel when two reference observations from the same subject are subtracted?
A reference observation has marginal variance $\sigma_b^2+\sigma_{W,R}^2$. Which part is the reference-specific within-subject residual variance?
What does $\tau$ represent when test is coded as 1 and reference as 0?

Solutions. (1) It indicates a reference observation, whose residual variance is $\sigma_{W,R}^2$. (2) The same $b_{j(s)}$ is added to both observations, so their difference contains $b_{j(s)}-b_{j(s)}=0$. (3) The reference-specific residual variance is $\sigma_{W,R}^2$; $\sigma_b^2$ is between-subject variance. (4) It is the adjusted test-minus-reference difference in mean log response, with $\exp(\tau)$ giving the corresponding geometric mean ratio.