Regression of Y on X Versus Regression of X on Y

Two Directions, Two Conditional Questions

A regression of \(Y\) on \(X\) describes how the conditional mean of \(Y\) changes as \(X\) varies. A regression of \(X\) on \(Y\) asks a different question: how the conditional mean of \(X\) changes as \(Y\) varies. The same pair of variables and the same observations may be used in both analyses, but the direction determines which variable is being predicted.

It is tempting to think that the second regression is obtained by solving the first equation for the other variable. In general, that is incorrect. The two regression slopes use the same covariance but divide by different variances. Consequently, the lines generally have different slopes and neither is the algebraic inverse of the other.

Definition: Let \(X\) and \(Y\) have finite means, positive variances, and covariance \(\operatorname{Cov}(X,Y)\). A linear conditional-mean model for \(Y\) given \(X\) specifies \(E[Y\mid X=x]=\alpha_{Y\mid X}+\beta_{Y\mid X}x\). A linear conditional-mean model for \(X\) given \(Y\) specifies \(E[X\mid Y=y]=\alpha_{X\mid Y}+\beta_{X\mid Y}y\). These are separate conditional-mean statements, not one equation written in two ways.

The formulas below give the coefficients when the relevant conditional mean is linear. For a jointly normal pair, both conditional means are linear, so both formulas give exact conditional-mean lines. For a general distribution, a conditional mean need not be linear; the covariance formulas alone do not establish that it is.

Slopes and Intercepts in Each Direction

Suppose first that \(E[Y\mid X]\) is linear. Its slope is the covariance of \(X\) and \(Y\) divided by the variance of \(X\). Reversing the roles gives a different denominator for the other regression.

\beta_{Y\mid X}=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}, \qquad \beta_{X\mid Y}=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(Y)}.

Each intercept is chosen so that the conditional-mean line passes through the point of means \((E[X],E[Y])\). Thus

\alpha_{Y\mid X}=E[Y]-\beta_{Y\mid X}E[X], \qquad \alpha_{X\mid Y}=E[X]-\beta_{X\mid Y}E[Y].

The point-of-means check is useful both conceptually and computationally. Averaging \(E[Y\mid X]\) over the distribution of \(X\) must give \(E[Y]\); similarly, averaging \(E[X\mid Y]\) must give \(E[X]\). A proposed intercept that makes its line miss the means is therefore wrong.

Worked Example: Computing Both Lines From Moments

Suppose \(E[X]=4\), \(E[Y]=10\), \(\operatorname{Var}(X)=9\), \(\operatorname{Var}(Y)=16\), and \(\operatorname{Cov}(X,Y)=6\). Assume both conditional means are linear. Find both conditional-mean equations and check their values at the means.

Method. Use covariance divided by the variance of the predictor for each slope, then choose the intercept so the line passes through \((4,10)\).

For \(Y\) on \(X\), the predictor is \(X\), so

\beta_{Y\mid X}=\frac{6}{9}=\frac{2}{3}, \qquad \alpha_{Y\mid X}=10-\frac{2}{3}(4) =10-\frac{8}{3} =\frac{22}{3}.

Therefore \(E[Y\mid X=x]=\frac{22}{3}+\frac{2}{3}x\). At \(x=4\), this gives \(\frac{22}{3}+\frac{8}{3}=10\), as required.

For \(X\) on \(Y\), the predictor is \(Y\), so

\beta_{X\mid Y}=\frac{6}{16}=\frac{3}{8}, \qquad \alpha_{X\mid Y}=4-\frac{3}{8}(10) =4-\frac{30}{8} =\frac{1}{4}.

Thus \(E[X\mid Y=y]=\frac{1}{4}+\frac{3}{8}y\). At \(y=10\), the value is \(\frac{1}{4}+\frac{30}{8}=4\), again the required mean.

Check and trap avoided. The two slopes have product \((2/3)(3/8)=1/4\), not 1. The intercepts are not interchangeable: using 6 as the first intercept would give \(6+(2/3)(4)=26/3\), not the required mean 10. Calculating each intercept from its own mean is essential.

Why the Slope Formula Holds

A short derivation shows why the covariance appears. Assume \(E[Y\mid X]=\alpha+\beta X\), and write \(\mu_X=E[X]\) and \(\mu_Y=E[Y]\). Taking expectations of the conditional mean gives \(\mu_Y=\alpha+\beta\mu_X\). Now condition on \(X\) inside the covariance:

\begin{aligned} \operatorname{Cov}(X,Y) &=E[(X-\mu_X)(Y-\mu_Y)]\\ &=E\!\left[(X-\mu_X)E[Y-\mu_Y\mid X]\right]\\ &=E\!\left[(X-\mu_X)\{\alpha+\beta X-\mu_Y\}\right]\\ &=E\!\left[(X-\mu_X)\beta(X-\mu_X)\right]\\ &=\beta\operatorname{Var}(X). \end{aligned}

In the second line, \(X-\mu_X\) is fixed once we condition on \(X\), so it can be taken outside the conditional expectation. In the fourth line, \(\alpha+\beta\mu_X=\mu_Y\), which leaves \(\beta(X-\mu_X)\). Since \(\operatorname{Var}(X)>0\), division gives \(\beta_{Y\mid X}=\operatorname{Cov}(X,Y)/\operatorname{Var}(X)\). Applying the same argument with \(X\) and \(Y\) interchanged gives the other slope. This derivation depends on the specified linear conditional mean; it does not prove that every conditional mean is linear.

How the Two Slopes Are Related

Let \(\sigma_X\) and \(\sigma_Y\) be the positive standard deviations of \(X\) and \(Y\), and let \(\rho=\operatorname{Corr}(X,Y)\). Since \(\operatorname{Cov}(X,Y)=\rho\sigma_X\sigma_Y\), the slopes can also be written as

\beta_{Y\mid X}=\rho\frac{\sigma_Y}{\sigma_X}, \qquad \beta_{X\mid Y}=\rho\frac{\sigma_X}{\sigma_Y}, \qquad \beta_{Y\mid X}\beta_{X\mid Y}=\rho^2.

The slopes have the same sign as the correlation, but their magnitudes depend on the units and variability of the predictor in each direction. The product is \(\rho^2\), which is less than 1 when \(|\rho|<1\). By contrast, algebraically solving a line with nonzero slope for its original predictor gives an inverse slope, so the product of the original and inverse slopes is 1. This is why the two operations generally do not agree.

When \(|\rho|=1\) and both variances are positive, the variables are perfectly linearly related: one is an exact linear function of the other, and the two conditional-mean lines are algebraic inverses. When \(\rho=0\), both slopes are zero. Neither horizontal conditional-mean line can be inverted into the other. These special cases make clear that a regression direction is part of the model, not a cosmetic choice.

Worked Example: Conditional Means for a Jointly Normal Pair

Suppose \(X\) and \(Y\) are jointly normal with \(E[X]=20\), \(E[Y]=8\), standard deviations \(\sigma_X=4\) and \(\sigma_Y=2\), and correlation \(\rho=-0.5\). Find both conditional-mean lines and compare the reverse regression with the algebraic inverse of the first line.

Method. Joint normality ensures that both conditional means are linear. First calculate the covariance:

\operatorname{Cov}(X,Y)=\rho\sigma_X\sigma_Y =(-0.5)(4)(2)=-4.

For \(Y\) on \(X\), \(\operatorname{Var}(X)=4^2=16\). Hence

\beta_{Y\mid X}=\frac{-4}{16}=-0.25, \qquad \alpha_{Y\mid X}=8-(-0.25)(20)=8+5=13.

Therefore \(E[Y\mid X=x]=13-0.25x\). Checking at \(x=20\) gives \(13-5=8\).

For \(X\) on \(Y\), \(\operatorname{Var}(Y)=2^2=4\), so

\beta_{X\mid Y}=\frac{-4}{4}=-1, \qquad \alpha_{X\mid Y}=20-(-1)(8)=28.

Thus \(E[X\mid Y=y]=28-y\), which gives \(28-8=20\) at the means. The slopes multiply to \((-0.25)(-1)=0.25=\rho^2\).

Comparison and trap avoided. Solving \(y=13-0.25x\) for \(x\) gives \(x=52-4y\), whose slope is \(-4\). That is not the conditional-mean line \(x=28-y\). For instance, at \(y=10\), the algebraic inverse gives \(x=12\), while the reverse conditional mean gives \(E[X\mid Y=10]=18\). The inverse answers which \(x\) lies on the first line at that \(y\); the reverse regression answers the conditional-mean question given \(Y=10\).

Worked Example: Distinct Predictions in Two Directions

In a hypothetical study, let \(X\) be annual hours of technical training and \(Y\) be an annual performance score. Suppose the pair is jointly normal with \(E[X]=12\), \(E[Y]=30\), \(\sigma_X=3\), \(\sigma_Y=10\), and \(\rho=0.6\). Find both conditional-mean lines, then use each for its stated prediction.

Method. Joint normality makes the conditional means linear. The covariance is

\operatorname{Cov}(X,Y)=(0.6)(3)(10)=18.

The \(Y\)-on-\(X\) slope is \(18/3^2=18/9=2\), and its intercept is \(30-2(12)=6\). Therefore \(E[Y\mid X=x]=6+2x\). At the mean \(x=12\), the line gives \(6+24=30\).

The \(X\)-on-\(Y\) slope is \(18/10^2=18/100=0.18\), and its intercept is \(12-(0.18)(30)=12-5.4=6.6\). Therefore \(E[X\mid Y=y]=6.6+0.18y\). At \(y=30\), this gives \(6.6+5.4=12\).

At \(x=15\), the first line predicts the conditional mean score \(6+2(15)=36\). At \(y=36\), the reverse line predicts the conditional mean training hours \(6.6+0.18(36)=6.6+6.48=13.08\). It does not return 15. The two predictions condition on different information and are not designed to undo each other.

Check and trap avoided. The slopes multiply to \(2(0.18)=0.36=\rho^2\), rather than 1. Taking the reciprocal of 2 to obtain 0.5 would incorrectly treat the reverse conditional mean as the inverse of the first line.

Choosing and Interpreting the Direction

Choose the direction from the question being asked, not from which equation looks easier to manipulate. If the objective is to estimate the mean response among cases with a specified \(X\), use the conditional mean of \(Y\) given \(X\). If the objective is instead to estimate the mean \(X\) among cases with a specified \(Y\), use the conditional mean of \(X\) given \(Y\). In a model where both conditional means are linear, their slopes and intercepts must still be calculated separately.

A reliable calculation sequence is to name the response and predictor, compute the covariance, divide by the predictor variance to obtain that direction's slope, and use the corresponding means to find the intercept. Finish by checking that the line passes through the point of means. Also check whether the problem gives a model that supports an exact linear conditional mean, such as joint normality, or explicitly assumes conditional linearity. Without such a condition, the moment formulas do not by themselves show that the true conditional mean is a straight line.

Key takeaway: The \(Y\)-on-\(X\) slope is \(\operatorname{Cov}(X,Y)/\operatorname{Var}(X)\); the \(X\)-on-\(Y\) slope is \(\operatorname{Cov}(X,Y)/\operatorname{Var}(Y)\). Each intercept is calculated from its own conditional-mean equation, and the two lines generally are not algebraic inverses.

Check Your Understanding

For each problem, distinguish the direction of conditioning, calculate the requested quantities, and check any line at the means.

Suppose both conditional means are linear, with \(E[X]=5\), \(E[Y]=12\), \(\operatorname{Var}(X)=4\), \(\operatorname{Var}(Y)=25\), and \(\operatorname{Cov}(X,Y)=6\). Find both conditional-mean lines and verify each at its predictor mean. (Answer: \(E[Y\mid X=x]=4.5+1.5x\); \(E[X\mid Y=y]=2.12+0.24y\).)
Let \((X,Y)\) be jointly normal with means \(E[X]=10\), \(E[Y]=40\), standard deviations \(2\) and \(5\), and correlation \(-0.4\). Calculate the covariance and both conditional-mean lines; verify that each passes through the means. (Answer: \(\operatorname{Cov}(X,Y)=-4\); \(E[Y\mid X=x]=50-x\); \(E[X\mid Y=y]=16.4-0.16y\).)
Assume both conditional means are linear and \(E[X]=3\), \(E[Y]=7\), \(\operatorname{Var}(X)=16\), \(\operatorname{Var}(Y)=9\), and \(\operatorname{Cov}(X,Y)=-6\). Find both slopes and intercepts, then calculate their slope product and compare it with the squared correlation. (Answer: \(E[Y\mid X=x]=65/8-(3/8)x\); \(E[X\mid Y=y]=23/3-(2/3)y\); slope product \(1/4=\rho^2\), where \(\rho=-1/2\).)
Let \(X\) and \(Y\) be jointly normal with \(E[X]=50\), \(E[Y]=20\), \(\sigma_X=10\), \(\sigma_Y=4\), and \(\rho=0.75\). Find both conditional-mean lines. Use the \(Y\)-on-\(X\) line at \(x=60\), and the \(X\)-on-\(Y\) line at \(y=23\). Compare the latter prediction with the value obtained by algebraically inverting the first line. (Answer: \(E[Y\mid X=x]=5+0.3x\), giving 23 at \(x=60\); \(E[X\mid Y=y]=12.5+1.875y\), giving 55.625 at \(y=23\); inversion of the first line gives 60, not 55.625.)
Suppose a jointly normal pair has means \(E[X]=6\), \(E[Y]=9\), and conditional-mean slopes \(\beta_{Y\mid X}=0.8\) and \(\beta_{X\mid Y}=0.45\). Determine the correlation and, if \(\sigma_X=3\), find \(\sigma_Y\), the covariance, and both intercepts. (Answer: \(\rho=0.6\); \(\sigma_Y=4\); \(\operatorname{Cov}(X,Y)=7.2\); \(\alpha_{Y\mid X}=4.2\); \(\alpha_{X\mid Y}=1.95\).)

What You'll Learn