Why Preliminary Carryover Tests Can Mislead

What a Preliminary Carryover Test Can Tell You

A preliminary carryover test may look like a simple safeguard: test for carryover, then decide whether the main crossover analysis is appropriate. The difficulty is that the test can be insensitive to carryover that matters, and its result depends on the contrast and model used. In a two-period, two-sequence study, a sequence difference in observed outcomes can reflect more than residual effects from the previous treatment. A nonsignificant test therefore does not certify that carryover is absent.

The earlier material on differential carryover distinguishes the residual effects of the two formulations. Here the focus is narrower: why statistical tests used to detect those effects can mislead. The central issues are limited power, unequal sequence sizes, and the interpretation imposed by the model.

Why the Tested Contrast Matters

Consider a two-period study with sequences AB and BA. Let \(\tau_A\) and \(\tau_B\) denote the direct effects of treatments A and B, and let \(\lambda_A\) and \(\lambda_B\) denote their first-order carryover effects into period 2. Define the direct-effect contrast as \(\delta=\tau_B-\tau_A\), and define the differential carryover contrast as \(\kappa=\lambda_A-\lambda_B\). For this illustration, suppose there is a common period effect, subject-specific effects have the same expectation in the randomized sequences, and the outcome model has no additional sequence-specific mean difference.

In period 2, participants in sequence AB receive B after A, while participants in BA receive A after B. The expected period-2 sequence contrast, AB minus BA, is therefore:

Thus, a period-2 sequence difference does not isolate carryover: it combines the direct treatment contrast and differential carryover. Another possible test uses the change in the sequence contrast from period 1 to period 2. In period 1, AB minus BA has expected contrast \(\tau_A-\tau_B=-\delta\). Subtracting that contrast from the period-2 contrast gives:

That contrast also combines direct and carryover effects. The particular mixture depends on the statistic and model. A test is evidence about its null hypothesis, not a direct assay of a physical residual concentration or a general test of every possible carryover mechanism.

This distinction is especially important in a two-period design, where direct and carryover effects cannot generally be separated using the observed treatment-period means alone without additional assumptions or information. A test may be useful under a specified model, but the model does not make a mixed contrast into a pure carryover measure.

Low Power: Failure to Detect Is Not Evidence of Absence

Power is the probability that a specified test rejects \(H_0\) when a particular alternative is true. It depends on the significance level \(\alpha\), sample size, variability, the size of the effect being tested, and the test procedure. A carryover test with low power can often fail to detect a nonzero effect. A large p-value does not measure the probability that the null hypothesis is true, nor does it show that the effect is small enough to be unimportant.

The following approximation makes the role of sample size and variability explicit. Suppose the target is a difference between independent period-2 sequence means, the outcome is normally distributed within each sequence with a common known standard deviation \(\sigma\), and there are \(n_{AB}\) and \(n_{BA}\) participants. Under those assumptions, the standard error of the difference is \(\sigma\sqrt{1/n_{AB}+1/n_{BA}}\). For a two-sided test with known \(\sigma\), the standardized statistic has a standard normal distribution under \(H_0\). If the true contrast is \(\theta\), its mean under the alternative is \(\theta/SE\). These assumptions are for illustrating power; real analyses generally estimate variability and may use a different model.

Sequence Imbalance Reduces Precision

For a two-sequence comparison with independent observations and common variance \(\sigma^2\), the variance of the difference in sample means is \(\sigma^2(1/n_{AB}+1/n_{BA})\). With a fixed total sample size, this variance is smallest when sequence sizes are equal. Imbalance does not automatically bias a randomized comparison, but it increases uncertainty and can reduce the chance of detecting a given contrast. It may also make results more sensitive to model choices when sequence groups differ in other ways.

Assumptions, Diagnostics, and Alternatives

A conventional test is only as interpretable as its assumptions and the contrast it tests. For a pooled two-sample test of period-2 means, relevant assumptions include independent participants across sequences, a common within-sequence variance, and a suitable normal model for the outcome or enough information for the approximation used. The test also requires a clearly defined outcome and a prespecified contrast. A repeated-measures model has its own assumptions about the within-participant structure and the included period, treatment, sequence, and carryover terms.

Diagnostics should therefore go beyond reading a p-value. Check the numbers assigned to each sequence, the spread and distribution of outcomes within each sequence, unusual observations, missingness, and whether the fitted model matches the design. Compare the observed period-specific summaries with the model’s implied pattern. If plausible models give materially different carryover conclusions, that is evidence of model dependence, not a reason to select whichever model gives the smallest p-value. Predose concentrations and pharmacokinetic reasoning can provide complementary information, but they answer different questions from a statistical test of outcome contrasts.

A preliminary test is also not a reliable switch for deciding whether to use the planned crossover analysis. Choosing an analysis only after seeing whether a preliminary p-value crosses 0.05 makes the final procedure depend on the data twice. It can distort the operating characteristics of the overall decision process, and a nonsignificant preliminary result may simply reflect low power. Conversely, a significant result may reflect the direct treatment contrast or model misspecification rather than carryover alone.

The more defensible alternatives are to reduce the opportunity for carryover through design, use pharmacokinetic knowledge to choose an adequate washout, prespecify the primary model, and report sensitivity analyses that show how conclusions change under plausible assumptions. A first-period-only comparison avoids prior-period carryover because there is no earlier study treatment in period 1, but it discards the within-participant crossover information and answers a less precise parallel-group comparison. It is not automatically a superior analysis merely because a preliminary test is significant. The choice should be justified by design and scientific aims, rather than triggered mechanically by a p-value.

Common Pitfall

Wrong reasoning: “The carryover test had \(p>0.05\), so there is no carryover and the crossover analysis is safe.”

Correction: First identify the exact contrast and model. Ask whether the test has adequate power for effects that would matter, whether sequence sizes and variability support useful precision, and whether the contrast separates direct effects from carryover. A nonsignificant result is not an equivalence conclusion; establishing that effects are sufficiently small would require a prespecified margin and an analysis designed to assess that question. Most importantly, a test of a mixed contrast cannot by itself establish that its carryover component is zero.