Per-Protocol Comparisons and Crossover Pitfalls

Why the Analysis Group Matters

Random assignment creates comparable groups at the point of assignment. It does not guarantee that the groups remain comparable after participants decide whether to start, stop, or switch treatments. A comparison organized by what participants actually received can therefore answer a different question from the one answered by comparing their randomized assignments.

This distinction matters in ordinary parallel-group trials and becomes especially visible in crossover trials, where participants are assigned to treatment sequences and may receive different treatments across periods. The central discipline is to name the comparison being made before interpreting its estimate.

Definition: Let \(Z_i\) denote participant \(i\)'s randomized assignment, \(D_i\) the treatment actually received, and \(Y_i\) the observed outcome. An intention-to-treat (ITT) comparison compares outcomes by \(Z_i\), regardless of \(D_i\). An as-treated comparison compares outcomes by \(D_i\). A per-protocol comparison compares participants classified as having followed their assigned treatment protocol with those classified as following the comparison protocol. The latter two classifications use post-assignment behavior.

When outcomes are observed for everyone and assignment is randomized, the difference in mean outcomes by assignment estimates the average effect of assignment to the treatment strategy rather than the comparison strategy. This is often called the intention-to-treat effect. It remains an effect of assignment even when some participants do not follow their assignments. It is not automatically the effect of receiving the treatment.

\widehat{\tau}_{\mathrm{ITT}} = \bar{Y}_{Z=1}-\bar{Y}_{Z=0}

For a treatment-receipt comparison, randomization does not in general make the groups \(D_i=1\) and \(D_i=0\) comparable. For example, participants who feel worse may be more likely to start treatment, while participants who feel better may be more likely to stop. The same post-assignment behavior can be related to prognosis, access, motivation, or side effects. Comparing by receipt then mixes any treatment effect with differences between the people who did and did not receive treatment.

A Simple Decomposition of the Problem

Let \(X_i\) be a prognostic baseline characteristic, such as disease severity. The observed outcome mean among those receiving treatment averages over the distribution of \(X_i\) in that group; the mean among those not receiving treatment averages over a potentially different distribution. For discrete \(X_i\), the difference can be written as:

E[Y_i\mid D_i=1]-E[Y_i\mid D_i=0] = \sum_x E[Y_i\mid D_i=1,X_i=x]P(X_i=x\mid D_i=1) - \sum_x E[Y_i\mid D_i=0,X_i=x]P(X_i=x\mid D_i=0).

For continuous \(X_i\), the corresponding expression uses integrals over the respective distributions:

E[Y_i\mid D_i=1]-E[Y_i\mid D_i=0] = \int E[Y_i\mid D_i=1,X_i=x]\,dP(x\mid D_i=1) - \int E[Y_i\mid D_i=0,X_i=x]\,dP(x\mid D_i=0).

Even if treatment has the same effect at every value of \(X_i\), these group means need not differ by that effect: the distributions \(P(X_i\mid D_i=1)\) and \(P(X_i\mid D_i=0)\) may differ. Randomization balances \(X_i\) across assignment groups in expectation, not necessarily across groups formed later by treatment receipt or adherence.

Worked Example: Assignment Versus Treatment Received

Suppose a hypothetical trial randomizes 200 people: 100 to an active treatment and 100 to control. By chance, each assignment group contains 50 people with mild and 50 with severe illness. Among those assigned active treatment, 60 receive it; among those assigned control, 20 obtain it outside the trial. Suppose the outcome is symptom score, where lower is better. The hypothetical outcome means by severity and actual treatment are as follows.

Severity	Number receiving treatment	Mean score if treated	Number not receiving treatment	Mean score if untreated
Mild	60	8	40	10
Severe	20	18	80	20

Within each severity category, treatment is associated with a 2-point lower mean score: \(8-10=-2\) for mild illness and \(18-20=-2\) for severe illness. But severity is distributed very differently among people who received treatment and those who did not.

The as-treated mean among the 80 treatment recipients is \( (60\times8+20\times18)/80=840/80=10.50 \) points. The mean among the 120 nonrecipients is \( (40\times10+80\times20)/120=2{,}000/120=16.67 \) points. Thus the as-treated contrast is \(10.50-16.67=-6.17\) points, not the within-severity treatment difference of \(-2\) points. Treatment recipients include many more mildly ill participants.

Now compare by randomized assignment. In the active-assignment group, the outcome total is \(45\times8+15\times18+5\times10+35\times20=1{,}380\), giving a mean of \(1{,}380/100=13.80\). In the control-assignment group, the total is \(15\times8+5\times18+35\times10+45\times20=1{,}460\), giving a mean of \(1{,}460/100=14.60\). The ITT contrast is \(13.80-14.60=-0.80\) points. It is smaller in magnitude than the treatment-receipt contrast because assignment changed the probability of receiving treatment, but many participants did not receive the treatment specified by their assignment.

What a Per-Protocol Comparison Does—and Does Not—Show

In the same trial, define adherence to mean receiving the assigned regimen: receiving active treatment when assigned active and not receiving it when assigned control. The active-assignment adherers are 45 mildly ill and 15 severely ill participants. Their mean outcome is \( (45\times8+15\times18)/60=630/60=10.50 \) points. The control-assignment adherers are 35 mildly ill and 45 severely ill participants. Their mean is \( (35\times10+45\times20)/80=1{,}250/80=15.625 \) points. The simple per-protocol contrast is therefore \(10.50-15.625=-5.125\) points.

This contrast is not an alternative randomized comparison: the adherers were selected after assignment. Only \(15/60=25\%\) of active-assignment adherers are severely ill, compared with \(45/80=56.25\%\) of control-assignment adherers. That severity imbalance alone can make the observed contrast more favorable than the within-severity treatment difference. A per-protocol estimate may be useful for a question about following a specified strategy, but simply excluding nonadherers does not identify that strategy's causal effect.

Key distinction: ITT preserves the randomized groups and estimates the effect of assignment, subject to outcome observation and the trial's design. A naive as-treated or per-protocol comparison changes group membership using post-assignment behavior. To interpret such an estimate causally requires additional assumptions and an analysis that addresses selection into adherence; randomization alone is not enough.

The issue is not that every adherence-based analysis must be biased by a fixed amount or in a particular direction. The direction and size depend on why participants adhere, switch, or stop, and on how those factors relate to outcomes. The general warning is narrower: the original randomization does not itself justify treating post-randomization groups as exchangeable.

Crossover Trials: Assignment Is to a Sequence

In a two-period crossover trial, participants are randomized to sequences such as AB or BA, where A and B are treatments. Each participant is intended to receive both treatments, one in each period. The randomized unit is the sequence; the within-person treatment contrast is a different comparison and relies on additional design assumptions.

A simple model makes the assumptions visible. Write the outcome for participant \(i\) in period \(p\) as:

Y_{ip}=\alpha_i+\pi_p+\tau A_{ip} +\lambda\,I(p=2\ \text{and period 1 treatment was A})+\varepsilon_{ip},

Here \(\alpha_i\) is a participant-specific level, \(\pi_p\) is a period effect, \(A_{ip}=1\) for treatment A and 0 for B, \(\tau\) is the direct effect of A relative to B, and \(\lambda\) is a carryover effect from receiving A in period 1. This illustrative model assumes the direct effect is constant across participants and periods. A usual crossover interpretation also needs appropriate control of carryover, suitable washout, and outcome observation that does not create consequential selection.

Worked Example: Period Effects and Carryover in a Crossover

Consider a hypothetical, balanced crossover with equal numbers assigned to AB and BA. Suppose the direct effect of A relative to B is \(\tau=-2\) outcome points, period 2 raises scores by \(\pi_2-\pi_1=3\) points, and receiving A in period 1 has a carryover effect of \(\lambda=4\) points in period 2. Lower outcomes are better for the direct treatment effect in this example.

For an AB participant, the within-person contrast A minus B is the period 1 outcome minus the period 2 outcome. Substituting the model terms, the participant-level contrast averages to \( \tau-(\pi_2-\pi_1)-\lambda=-2-3-4=-9 \) points. For a BA participant, A occurs in period 2 and B in period 1, so A minus B averages to \( \tau+(\pi_2-\pi_1)=-2+3=1 \) point. Averaging the two sequence-specific contrasts gives \( (-9+1)/2=-4 \) points.

The direct effect in the stipulated model is \(-2\), not \(-4\). In a balanced design, the additive period effect cancels when these oriented contrasts are averaged, but the carryover term does not: the average is \(\tau-\lambda/2=-2-4/2=-4\). Thus randomization to sequence does not make the within-person treatment contrast a valid direct-effect estimate when carryover is present. The ITT comparison by randomized sequence remains a comparison of assignment to AB versus BA; it is not automatically an estimate of the direct effect of A versus B.

Crossover analyses can also lose the protection of the design when actual exposure or completion differs from the assigned sequence. For instance, comparing all observed A-period outcomes with all observed B-period outcomes may change the mix of periods or participants if people skip a treatment period, switch treatments, or have outcomes recorded selectively. A simple pooled comparison by treatment received can then combine treatment, period, carryover, and participant-selection differences. The appropriate analysis depends on the estimand and design; merely labeling a comparison “within-person” does not remove these problems.

A Practical Sequence for Interpretation

Name the randomized assignment.
In a parallel trial, this is usually the assigned treatment group. In a crossover trial, it is generally the assigned treatment sequence.

Name the target contrast.
Decide whether the question concerns assignment, treatment received, adherence to a protocol, or a direct treatment effect across periods.

Check how analysis groups were formed.
If group membership depends on post-assignment adherence, switching, or completion, do not assume the original randomization balances those groups.

State the extra assumptions.
For an adherence-based causal interpretation, explain how selection is addressed. For a crossover direct-effect interpretation, examine period, carryover, and sequence-specific adherence.

Common Pitfalls

“Only adherers count, so randomization is preserved.” Restricting to adherers conditions on behavior after assignment. The remaining groups need not be comparable.
“The as-treated contrast is the treatment effect.” It is an observed difference by receipt. It may reflect both treatment and differences in who received it.
“Each crossover participant serves as their own control, so there is no confounding.” Participant-level differences can cancel in a within-person contrast, but period effects, carryover, changing adherence, and missing outcomes can still matter.
“ITT estimates the direct effect of treatment received.” ITT estimates the effect of assignment to the randomized strategy. With nonadherence, that is generally a different estimand.

Key takeaway: Randomization protects comparisons organized by randomized assignment, not automatically comparisons defined by later treatment receipt or adherence. In crossover trials, assignment is to a sequence, and interpreting within-person contrasts as direct treatment effects requires attention to period and carryover effects as well as adherence.

Check Your Understanding

For each question, distinguish the comparison being made from the causal effect it might be intended to represent.

In a randomized trial, why can people who received treatment differ systematically from people who did not, even though assignment was randomized?
A trial reports a lower mean outcome among participants who followed their assigned regimen than among those who did not. What additional concern arises if adherence depends on baseline prognosis?
In a crossover trial, what is randomized: treatment receipt in each observed period or assignment to a treatment sequence?
Under the crossover model in this tutorial, what does the average of the two oriented sequence contrasts equal when the period effect is additive and carryover from A is \(\lambda\)?
Why is an ITT estimate not generally the same as the causal effect of actually receiving treatment when some participants do not follow their assignments?