Estimating Within-Test Variability in Full Replicates

Why Repeating the Test Formulation Matters

A full-replicate crossover administers both the test formulation and the reference formulation more than once to each participant. In a four-period schedule such as TRTR or RTRT, every participant receives test twice and reference twice. Those repeated administrations provide within-participant contrasts for each formulation, including the test formulation.

This tutorial focuses on what the repeated test observations tell us: the variability of a participant’s response across separate test administrations, after accounting for systematic differences such as period effects. This is not the same as the difference between the average test and reference responses. Nor is it simply the variability among participants’ test responses, which also reflects genuine differences between participants.

The key design feature is that the same participant contributes two test responses. Their contrast removes stable participant-level response differences, so the remaining variation can inform the within-test variance. In a reference-only replicate, participants repeat reference but receive test only once. That design provides repeated reference responses, but it does not provide the corresponding within-participant test contrast.

Model and Assumptions

Let \(Y_{ij}\) be the log-transformed endpoint for participant \(i\) in period \(j\). A useful conceptual model is

Here \(\mu\) is an overall mean, \(\gamma_{s(i)}\) represents the participant’s sequence, \(\pi_j\) is the period effect, and \(\tau_{F_{ij}}\) is the mean effect of the formulation given in that period. The term \(b_i\) captures a stable participant-level response, and \(u_{i,F_{ij}}\) represents a stable participant-by-formulation deviation. The occasion-specific error \(\varepsilon_{ij}\) represents within-participant variation not explained by those terms. For test periods, assume its variance is \(\sigma_{wT}^{2}\).

This model makes the logic of the contrast explicit. For a participant’s two test periods \(a\) and \(b\), subtract the test response in period \(b\) from that in period \(a\), and remove the corresponding fitted period contrast. The common mean, stable participant response, and stable participant-by-test deviation cancel. Under the assumptions below, the adjusted difference is the difference between the two occasion-specific errors.

• Systematic effects are accounted for. Sequence, period, and formulation effects that matter for the design are represented in the analysis. A period trend left in the contrast can be mistaken for within-test variation.
• No unmodeled carryover distorts the test responses. The response in a test period must not retain a relevant effect from an earlier formulation that the model fails to account for.
• The within-test error variance is stable across the test administrations being combined. If variability differs materially by period or sequence, a single pooled value may not describe all administrations well.
• Occasion-specific test errors are independent with common variance. This assumption gives the simple difference-variance formula below. If errors are correlated, the difference has a different variance.
• Participants are independent of one another. The contrast from one participant should not provide duplicated information about another participant’s response.

A full-replicate schedule makes a test contrast available, but does not by itself guarantee that these assumptions hold. The design and the model must still be checked against the study conduct and response patterns.

Deriving the Variance Estimate

Suppose the fixed effects have been removed appropriately and let \(D_{iT}\) be participant \(i\)’s adjusted difference between their two test responses. Under the model, \(D_{iT}=\varepsilon_{ia}-\varepsilon_{ib}\), where the two test-period errors are independent and each has variance \(\sigma_{wT}^{2}\). Therefore,

If the adjusted differences are independent across \(n\) participants and have a common mean, their sample variance estimates \(\operatorname{Var}(D_{iT})\). Dividing that sample variance by two estimates the within-test variance:

The subtraction of \(\bar{D}_T\) allows for a nonzero average adjusted difference, for example due to estimation noise or residual imbalance. If the period adjustments are known and the adjusted differences have a known zero mean, a corresponding estimator can instead use \(\sum D_{iT}^{2}/(2n)\). In actual analyses, period and other fixed effects are usually estimated from the study data, so the simple calculation may not capture all uncertainty or degrees-of-freedom adjustments. A suitable crossover model can estimate the variance while accounting for that fitted structure.

The derivation also shows why an unadjusted difference is not automatically a variance measure. If the two test periods have different means, the raw difference includes that systematic contrast. Likewise, if the two errors are correlated, the variance is not generally \(2\sigma_{wT}^{2}\). The simple estimator is justified only when the stated model and assumptions are appropriate.

Estimating Variability From Adjusted Differences

For the simple estimator, first calculate one appropriately adjusted test difference per participant. Then calculate the sample variance of those differences and divide by two. The within-test standard deviation is the square root of the variance estimate. Because the endpoint is on a log scale, a commonly reported coefficient of variation is

This conversion expresses variability on the original, ratio scale; it does not change the underlying analysis scale. The formula is intended for log-transformed positive endpoints under the usual lognormal interpretation. It should not be applied to a variance estimated on an untransformed scale as if that variance were a log-scale variance.

What the Design Identifies—and What It Does Not

In a full replicate, each participant has a repeated test response and a repeated reference response. Thus the design can provide separate within-participant information for both formulations, subject to the model and data completeness. The estimates are formulation-specific: a within-reference estimate is not automatically an estimate of within-test variability.

In a reference-only replicate, the repeated reference administrations support a reference contrast, but each participant has only one test administration. There is no within-participant test difference to use in the derivation above. A test-specific within-participant variance is therefore not directly identified by replication in that design. It cannot be recovered simply by relabeling the reference estimate as the test estimate; that would require an additional assumption that the formulation-specific variability is the same.

Full replication does not make every variance question answerable without qualification. For example, a participant who misses one of their two test periods has no complete test pair for the direct difference calculation. A mixed-model analysis may use incomplete records under its modeling assumptions, but the simple paired-contrast formula cannot treat a missing response as observed. The number of complete pairs, missingness patterns, and the fitted model all matter.

Diagnostics, Alternatives, and Limitations

Before interpreting an estimate, examine whether the data and study conduct support the model. Useful checks include plotting adjusted replicate differences by sequence and by the pair of periods involved; checking for a systematic nonzero mean difference; inspecting the spread for unusually influential participants; and reviewing missingness and protocol deviations. A sequence-specific pattern may signal that a single pooled estimate is concealing different behavior. A period pattern in unadjusted contrasts may indicate that period effects were not adequately handled.

A mixed-effects model is the main alternative to calculating a variance from adjusted participant-level differences. It can represent sequence, period, formulation, and participant structure together, and can use records that do not form complete pairs under its assumptions. Its variance-component estimate need not equal the simple estimate obtained by dividing a sample variance by two, particularly when fixed effects are estimated, observations are incomplete, or the model includes additional covariance structure. The paired-difference calculation remains useful for understanding the source of information and checking the scale of a model-based result.

The simple estimator has further limitations. It assumes equal occasion-specific variance across the test administrations being pooled and independent errors within a participant. If the repeated errors are correlated, then \(\operatorname{Var}(\varepsilon_{ia}-\varepsilon_{ib})=2\sigma_{wT}^{2}-2\operatorname{Cov}(\varepsilon_{ia},\varepsilon_{ib})\), not necessarily \(2\sigma_{wT}^{2}\). It also assumes that systematic effects and carryover have been handled adequately. These are reasons to investigate the design and model, not reasons to infer variability from an unadjusted raw difference.

Common mistake: “The test response was measured twice, so the standard deviation of the two values is the within-test standard deviation.” This ignores participant effects, period effects, and the fact that a difference of two independent errors has twice the error variance. The correction is to form a within-participant contrast that accounts for systematic effects, estimate the variance of those contrasts across participants, and divide by two under the stated assumptions. A second mistake is to use repeated reference responses as if they were repeated test responses; the reference-only replicate does not directly identify within-test variability.