Why Separate Visit-Wise Regressions Miss A Regime Contrast

Two Different Questions

In a two-visit study, let $A_1$ and $A_2$ denote treatment at visits 1 and 2, let $L$ be a covariate measured between the visits, and let $Y$ be the final outcome. A separate visit-wise regression of outcome on treatment fits one regression using $A_1$ and $Y$, and another using $A_2$ and $Y$. In this tutorial, these are unadjusted regressions with an intercept and one treatment predictor each.

A complete-regime comparison asks a different question: what is the difference in mean outcome under the treatment sequence $(A_1,A_2)=(1,1)$ versus the sequence $(0,0)$? Write this target as $\mu(1,1)-\mu(0,0)$, where $\mu(a_1,a_2)$ is the mean outcome under an intervention that sets both visits to $a_1,a_2$. The separate regressions compare people observed to have different treatment values at one visit; they do not, by themselves, compare populations assigned to two complete treatment sequences.

That distinction matters especially when earlier treatment changes a later covariate, the covariate affects later treatment, and both covariate and treatment predict the outcome. A coefficient can be calculated perfectly and still answer the wrong question.

Worked Two-Visit Example With Feedback

Consider the following fully specified data-generating model. Treatment at the first visit is randomized, with $P(A_1=1)=0.5$. The intermediate covariate is binary and has probabilities

P(L=1\mid A_1=0)=0.2,\qquad P(L=1\mid A_1=1)=0.8.

Later treatment depends on the covariate:

P(A_2=1\mid L=0)=0.2,\qquad P(A_2=1\mid L=1)=0.8.

Finally, the outcome follows the conditional mean model $E[Y\mid A_1,A_2,L]=2A_1+3A_2+4L$. For clarity, take $Y$ to equal this expression exactly. Thus the calculations below do not depend on residual variation.

Worked Example: Comparing Two Visit-Wise Slopes With the Regime Contrast

First calculate the regime means. Under regime $(0,0)$, $A_1=A_2=0$, while $P(L=1)=0.2$. Therefore $E[L]=0.2$ and $\mu(0,0)=4(0.2)=0.8$. Under regime $(1,1)$, $P(L=1)=0.8$, so $\mu(1,1)=2+3+4(0.8)=8.2$. The complete-regime contrast is $8.2-0.8=7.4$.

Now calculate the separate regression of $Y$ on $A_1$. Since $A_1$ is binary, its unadjusted regression slope is the difference in observed group means. For $A_1=0$, $E[L\mid A_1=0]=0.2$ and $P(A_2=1\mid A_1=0)=0.2(0.8)+0.8(0.2)=0.32$. Thus $E[Y\mid A_1=0]=2(0)+3(0.32)+4(0.2)=1.76$.

For $A_1=1$, $E[L\mid A_1=1]=0.8$ and $P(A_2=1\mid A_1=1)=0.2(0.2)+0.8(0.8)=0.68$. Hence $E[Y\mid A_1=1]=2+3(0.68)+4(0.8)=7.24$. The $A_1$-only regression slope is $7.24-1.76=5.48$, not $7.4$.

For the regression on $A_2$, first note that $P(A_2=1)=0.5$. Among those with $A_2=1$, the joint probabilities for $(A_1=1,L=1)$ and $(A_1=1,L=0)$ are $0.5(0.8)(0.8)=0.32$ and $0.5(0.2)(0.2)=0.02$. Dividing their sum by $0.5$ gives $P(A_1=1\mid A_2=1)=0.68$; also $P(L=1\mid A_2=1)=0.8$. Therefore $E[Y\mid A_2=1]=2(0.68)+3+4(0.8)=7.56$.

For $A_2=0$, the corresponding joint probabilities are $P(A_1=1,L=1,A_2=0)=0.5(0.8)(0.2)=0.08$ and $P(A_1=1,L=0,A_2=0)=0.5(0.2)(0.8)=0.08$. Consequently $P(A_1=1\mid A_2=0)=0.32$, and $P(L=1\mid A_2=0)=0.2$. Thus $E[Y\mid A_2=0]=2(0.32)+4(0.2)=1.44$, giving an $A_2$-only slope of $7.56-1.44=6.12$.

The two slopes, $5.48$ and $6.12$, are neither the regime contrast $7.4$ nor estimates of separate pieces that can safely be added. The $A_1$ comparison includes the changed distribution of $L$ and the resulting change in $A_2$. The $A_2$ comparison is between groups with different distributions of both $L$ and $A_1$.

What a Binary-Treatment Regression Slope Measures

Result: In a regression with an intercept and a single binary predictor $A$, the least-squares slope is $E[Y\mid A=1]-E[Y\mid A=0]$, provided both treatment groups have positive probability and the group means exist.

To verify this, let $p=P(A=1)$, with $0<p<1$, and let $m_a=E[Y\mid A=a]$. The least-squares slope is $\operatorname{Cov}(A,Y)/\operatorname{Var}(A)$. Because $E[AY]=pm_1$ and $E[Y]=pm_1+(1-p)m_0$,

\operatorname{Cov}(A,Y) =pm_1-p\{pm_1+(1-p)m_0\} =p(1-p)(m_1-m_0).

Also, $\operatorname{Var}(A)=p(1-p)$. Dividing gives the stated difference in group means. This result identifies exactly what the separate regressions calculate: observed, visit-specific mean differences. It does not make them complete-regime contrasts. $\square$

In the feedback example, the $A_1$-only slope is a difference between the observed $A_1$ groups, including their different distributions of later treatment. The $A_2$-only slope is a difference between the observed $A_2$ groups, including their different earlier treatment and covariate histories. Neither regression holds the other visit’s treatment fixed.

Interactions Can Matter Even Without Confounding

Worked Example: Randomized Treatments With an Interaction

Suppose $A_1$ and $A_2$ are independent randomized binary treatments, each with probability $0.25$, and $Y=2A_1+3A_2+6A_1A_2$. The two complete regimes have means $\mu(0,0)=0$ and $\mu(1,1)=2+3+6=11$, so their contrast is $11$.

In the separate regression on $A_1$, independence gives $E[A_2\mid A_1=a]=0.25$ for either value of $a$. Therefore $E[Y\mid A_1=1]-E[Y\mid A_1=0]=2+6(0.25)=3.5$. Similarly, the $A_2$-only slope is $3+6(0.25)=4.5$. These slopes depend on the probability of receiving the other treatment; their sum is $8$, not the regime contrast $11$.

This example has no confounding: treatment assignments are independent. The discrepancy arises because the outcome includes an interaction. Even in a randomized study, separate visit-wise slopes need not represent the effect of setting both treatments to a specified sequence.

Deriving the Contrast Under an Additive Outcome Model

Result: Suppose the conditional mean model is $E[Y\mid A_1,A_2,L]=\beta_0+\beta_1A_1+\beta_2A_2+\gamma L$, and suppose its mean is correctly specified under each regime. Then $$ \mu(a_1,a_2)=\beta_0+\beta_1a_1+\beta_2a_2+\gamma E[L\mid \text{regime }(a_1,a_2)]. $$

The formula follows by taking the expectation of the specified conditional mean under the regime. Since $a_1$ and $a_2$ are fixed by that regime, linearity of expectation gives

\mu(a_1,a_2) =\beta_0+\beta_1a_1+\beta_2a_2 +\gamma E[L\mid \text{regime }(a_1,a_2)].

Subtracting the expression for a second regime $(a'_1,a'_2)$ yields

\mu(a_1,a_2)-\mu(a'_1,a'_2) =\beta_1(a_1-a'_1)+\beta_2(a_2-a'_2) +\gamma\left\{E[L\mid \text{regime }(a_1,a_2)]- E[L\mid \text{regime }(a'_1,a'_2)]\right\}.

The last term is essential whenever the regime changes the distribution of $L$ and $L$ predicts $Y$. The formula does not claim that the model is always correct or that its parameters are identified from observational data without further assumptions. It shows, under the stated model, why a regime contrast can contain a covariate-distribution contribution in addition to the treatment terms. $\square$

Worked Example: Why the Conditional Treatment Terms Are Not Enough

Return to the feedback example, where the conditional mean has $\beta_1=2$, $\beta_2=3$, and $\gamma=4$. If one compares $(1,1)$ with $(0,0)$ while leaving $L$ fixed at the same value in both predictions, the difference is only $2+3=5$. But the regime changes the mean of $L$ from $0.2$ to $0.8$, so the omitted contribution is $4(0.8-0.2)=2.4$. Adding it gives $5+2.4=7.4$, agreeing with the direct regime calculation.

A conditional outcome regression can therefore be a useful component of a regime calculation, but reading treatment coefficients alone is not the same as calculating regime means. The covariate distribution must be handled in a way that matches the target comparison.

Interpretation, Alternatives, and Diagnostics

Separate regressions can be useful for describing observed associations at each visit. Their main limitation is not that ordinary least squares fails to calculate them; it is that the coefficients correspond to different observed comparisons. By contrast, a treatment-history regression represents treatment combinations explicitly, while standardization calculates predicted outcomes under specified treatment sequences and averages over the covariate distributions appropriate to those sequences. Marginal structural models provide another framework for marginal regime contrasts. These approaches require their own modeling and identification assumptions; changing the regression formula alone does not guarantee a causal interpretation.

Several checks help reveal when visit-wise slopes are unlikely to answer a regime question:

Ask whether a treatment at an earlier visit changes later covariates or treatment. If so, the later treatment groups may differ in their earlier histories.
Ask whether the outcome depends on treatment combinations. An interaction can make a visit-specific slope depend on the distribution of treatment at another visit.
State the target as a complete sequence before interpreting any coefficient. A contrast such as $(1,1)$ versus $(0,0)$ is not automatically represented by either one-visit slope.
Check which covariates and treatment histories are included in a regression and what population is being averaged over. A coefficient conditional on covariates is not automatically a marginal regime contrast.

A common mistake is to label the first-visit coefficient “the effect of treatment at visit 1,” label the second-visit coefficient “the effect at visit 2,” and add them to obtain the effect of treating at both visits. That reasoning fails in the feedback example: the slopes are $5.48$ and $6.12$, while the regime contrast is $7.4$. It also fails in the randomized interaction example, where the slopes sum to $8$ and the regime contrast is $11$. The correction is to define the complete regimes of interest and calculate their mean outcomes under a method that targets those sequences.

Check Your Understanding

Use the examples and results above to answer each question. Brief solutions follow the questions.

In the feedback example, what are the separate $A_1$-only and $A_2$-only regression slopes, and what is the $(1,1)$ versus $(0,0)$ regime contrast?
Why does the $A_2$-only regression compare groups with different distributions of earlier variables in the feedback example?
In the randomized interaction example, calculate the regime contrast and explain why the sum of the two visit-wise slopes does not equal it.
Under the additive conditional mean model, what extra term beyond the treatment terms enters the contrast between two regimes?

Solutions. (1) The slopes are $5.48$ and $6.12$; the regime contrast is $7.4$. (2) $A_2$ is assigned depending on $L$, and $L$'s distribution depends on $A_1$, so the $A_2$ groups do not have the same distributions of $L$ and $A_1$. (3) The contrast is $11-0=11$; the slopes are $3.5$ and $4.5$, summing to $8$, because the interaction contribution to each observed slope depends on the other treatment’s assignment probability. (4) The extra term is $\gamma$ times the difference between the regimes’ mean values of $L$.