The Linearity Condition for Inverse Regression

Why the Predictor Distribution Matters

Inverse-regression methods examine how the predictors $X$ vary after conditioning on a response $Y$. Their population summaries often involve conditional means. But the central dimension-reduction structure is about which linear combinations of $X$ carry information about $Y$, so a useful method needs a connection between conditional means and those combinations. The linearity condition supplies that connection: it says that the mean of the full predictor vector, given a reduced projection, varies affinely with that projection.

This is a condition on the distribution of the predictors, not a claim that the response itself is a linear function of them. In particular, a nonlinear regression relationship between $Y$ and $X$ can still be studied under the linearity condition.

Definition: Let $X\in\mathbb{R}^p$ have mean $\mu=E[X]$ and positive-definite covariance matrix $\Sigma=\operatorname{Var}(X)$. Let $B\in\mathbb{R}^{p\times d}$ have full column rank, where $1\leq d\leq p$. The linearity condition for $B$ is that $E[X\mid B^T X]$ is an affine function of $B^T X$. Equivalently, $$ E[X\mid B^T X] = \mu+\Sigma B(B^T\Sigma B)^{-1}(B^T X-B^T\mu) \quad\text{almost surely}. $$ The usual global linearity condition requires this property for every full-rank projection matrix $B$ of the dimensions under consideration.

The inverse exists because $B$ has full column rank and $\Sigma$ is positive definite: for any nonzero $v\in\mathbb{R}^d$, $v^TB^T\Sigma Bv=(Bv)^T\Sigma(Bv)>0$. Thus $B^T\Sigma B$ is positive definite. The formula also specifies which affine function is meant; the slope is determined by the covariance of $X$ with its projection.

Why the Formula Has This Particular Slope

An affine conditional mean, if it exists, must have the same slope as the linear least-squares predictor of $X$ from $B^TX$. This does not mean that linearity of the conditional mean follows from least squares. Least squares determines the best affine predictor whether or not the actual conditional mean is affine.

Result: Suppose $E[\|X\|^2]<\infty$, $\Sigma$ is positive definite, and $B$ has full column rank. If $E[X\mid B^TX]$ is affine in $B^TX$, then it must equal the formula in the definition.

Proof. Put $T=B^TX$. By the assumed affine form, write $E[X\mid T]=a+DT$, for a vector $a$ and a matrix $D$. Taking expectations gives $\mu=a+D B^T\mu$, so $E[X\mid T]=\mu+D(T-B^T\mu)$. Taking covariance with $T$, and using the defining property of conditional expectation, gives

\operatorname{Cov}(X,T) = \operatorname{Cov}(E[X\mid T],T) = D\operatorname{Var}(T).

Directly from $T=B^TX$, $\operatorname{Cov}(X,T)=\Sigma B$ and $\operatorname{Var}(T)=B^T\Sigma B$. The latter matrix is invertible, so $D=\Sigma B(B^T\Sigma B)^{-1}$. Substitution gives the claimed formula. $\square$

Worked Example: A Gaussian Projection

Let $X\sim N_3(\mu,\Sigma)$, with $\mu=(1,-1,2)^T$, $\Sigma=\begin{pmatrix}4&1&0\\1&2&0\\0&0&1\end{pmatrix}$, and $B=(1,1,0)^T$. Find $E[X\mid X_1+X_2=t]$.

The matrix is positive definite: its leading principal minors are $4$, $4(2)-1=7$, and $7$, all positive. Thus the covariance assumptions hold. Also $B^T\mu=1-1=0$, and

\Sigma B= \begin{pmatrix}5\\3\\0\end{pmatrix}, \qquad B^T\Sigma B=(1,1,0) \begin{pmatrix}5\\3\\0\end{pmatrix}=8.

A multivariate normal vector satisfies the linearity condition. Applying its formula, for $T=X_1+X_2$,

E[X\mid T=t] = \begin{pmatrix}1\\-1\\2\end{pmatrix} + \frac{1}{8} \begin{pmatrix}5\\3\\0\end{pmatrix}(t-0) = \begin{pmatrix}1+5t/8\\-1+3t/8\\2\end{pmatrix}.

The conditional mean changes only in the first two coordinates, at rates determined by their covariances with $T$. The third coordinate remains at its marginal mean because its covariance with $T$ is zero.

Elliptical Predictors Satisfy the Condition

The linearity condition is not limited to normal predictors. It holds for elliptical predictors with finite second moments and nonsingular covariance. One way to see why is to express such a predictor as an affine transformation of a spherical random vector. Spherical symmetry makes the component perpendicular to a given projection have conditional mean zero.

Result: Suppose $X=\mu+\Sigma^{1/2}Z$, where $\Sigma$ is positive definite and $Z\in\mathbb{R}^p$ is spherically symmetric with $E[Z]=0$ and $\operatorname{Var}(Z)=I_p$. Here spherical symmetry means that $QZ$ has the same distribution as $Z$ for every orthogonal matrix $Q$. Then, for every full-column-rank $B$, $$ E[X\mid B^TX] = \mu+\Sigma B(B^T\Sigma B)^{-1}(B^TX-B^T\mu). $$ Thus these finite-moment elliptical predictors satisfy the global linearity condition.

Proof. Define $C=\Sigma^{1/2}B$, which has full column rank, and $P=C(C^TC)^{-1}C^T$, the orthogonal projection onto the column space of $C$. The observed projection is $B^TX=B^T\mu+C^TZ$. Its random part $C^TZ$ is unchanged by any orthogonal transformation that fixes the column space of $C$.

In particular, the reflection $R=2P-I_p$ fixes that column space and changes the sign of $(I_p-P)Z$. By spherical symmetry, $RZ$ has the same distribution as $Z$. Consequently, for every bounded measurable function $g$,

$$ E[g(C^TZ)(I_p-P)Z] = E[g(C^TRZ)(I_p-P)RZ] = -E[g(C^TZ)(I_p-P)Z]. $$

The expectations exist because $Z$ has a finite first moment. The equality implies that each is zero. By the characterization of conditional expectation through integrals against bounded measurable functions, $E[(I_p-P)Z\mid C^TZ]=0$. Since $PZ=C(C^TC)^{-1}C^TZ$ is determined by $C^TZ$, it follows that

E[Z\mid C^TZ]=C(C^TC)^{-1}C^TZ.

Now $C^TC=B^T\Sigma B$, and $X=\mu+\Sigma^{1/2}Z$. Transforming the last expression back to $X$ gives

E[X\mid B^TX] = \mu+\Sigma^{1/2}C(C^TC)^{-1}C^TZ = \mu+\Sigma B(B^T\Sigma B)^{-1}(B^TX-B^T\mu).

This proves the result. $\square$

Worked Example: A Non-Gaussian Elliptical Predictor

Let $X=(X_1,X_2)^T$ be uniformly distributed on the circle of radius $\sqrt{2}$ centered at the origin. This distribution is spherical, has mean zero, and has covariance $I_2$: by rotational symmetry the coordinate variances are equal, and their sum is $E[X_1^2+X_2^2]=2$, so each variance is $1$.

Take $B=(1,0)^T$, so $B^TX=X_1$. For any $-\sqrt{2}<t<\sqrt{2}$, the two points on the circle with first coordinate $t$ have second coordinates $\sqrt{2-t^2}$ and $-\sqrt{2-t^2}$. Conditional on $X_1=t$, these two points have equal probability by reflection symmetry. Therefore

E[X\mid X_1=t] = \begin{pmatrix}t\\0\end{pmatrix}.

The linearity formula gives the same answer: $\mu=0$, $\Sigma=I_2$, and $B^T\Sigma B=1$, so it gives $B X_1=(X_1,0)^T$. This predictor is not Gaussian, yet it satisfies the condition.

How Linearity Supports Inverse Regression

Suppose $B^TX$ is a sufficient reduction in the sense that $Y$ and $X$ are conditionally independent given $B^TX$, and assume the expectations below exist. Conditional independence means that, once $B^TX$ is known, observing $Y$ provides no further information about $X$. If the linearity condition holds for $B$, then the conditional mean of $X$ given the response has a particularly structured form.

Result: Under the conditions above, $$ E[X\mid Y] = \mu+\Sigma B(B^T\Sigma B)^{-1} \bigl(E[B^TX\mid Y]-B^T\mu\bigr). $$

Derivation. Conditional independence gives $E[X\mid B^TX,Y]=E[X\mid B^TX]$. Apply iterated conditional expectation, then use the linearity condition:

\begin{aligned} E[X\mid Y] &=E\bigl[E[X\mid B^TX,Y]\mid Y\bigr]\\ &=E\bigl[E[X\mid B^TX]\mid Y\bigr]\\ &=\mu+\Sigma B(B^T\Sigma B)^{-1} \bigl(E[B^TX\mid Y]-B^T\mu\bigr). \end{aligned}

The response-conditioned predictor mean therefore changes only through the reduced coordinates, after accounting for the covariance scaling. This is why inverse-regression methods can use changes in conditional predictor means as evidence about relevant predictor directions. The identity is a population statement: by itself, it does not promise that a particular sample summary will estimate those directions accurately, nor that every relevant direction will be visible in conditional means.

Worked Example: A Nonlinear Response With a Linear Inverse Mean

Let $X=(X_1,X_2)^T\sim N_2(0,I_2)$, and define $Y=X_1^2+\varepsilon$, where $\varepsilon$ is independent of $X$ and has mean zero. The response depends nonlinearly on $X_1$, but it depends on $X$ only through $B^TX=X_1$, with $B=(1,0)^T$. Thus $Y$ and $X$ are conditionally independent given $X_1$.

The predictor distribution is spherical, so the linearity condition holds. Also, by symmetry of the standard normal distribution and the fact that $Y$ depends on $X_1$ only through its square, $E[X_1\mid Y]=0$. Independence gives $E[X_2\mid Y]=E[X_2]=0$. Hence $E[X\mid Y]=(0,0)^T$, which agrees with the inverse-regression identity: $E[B^TX\mid Y]=E[X_1\mid Y]=0$, while $\mu=0$.

This example illustrates an important limitation as well as the condition’s role. Linearity makes the inverse-mean identity valid, but a conditional mean can still fail to display a direction when the response relationship is symmetric. Whether a particular inverse-regression summary detects a direction depends on the response mechanism as well as on the predictor distribution.

When the Condition Fails

Outside predictor families with suitable symmetry, the condition must be assessed rather than assumed. Consider $U$ taking each of $-2,-1,1,2$ with probability $1/4$, and define $X=(U,U^2-2.5)^T$. Then $E[X]=0$, and direct calculation gives

\operatorname{Var}(X) = \begin{pmatrix}2.5&0\\0&2.25\end{pmatrix}.

For $B=(1,0)^T$, conditioning on $B^TX=U$ determines both coordinates, so $E[X\mid U]=(U,U^2-2.5)^T$. This is not affine in $U$: the second coordinate takes the same value, $1.5$, at $U=2$ and $U=-2$, whereas a nonconstant affine function cannot take the same value at two distinct inputs. The formula based on the covariance would instead give $(U,0)^T$, because $\Sigma B/(B^T\Sigma B)=(1,0)^T$. The condition genuinely fails.

Common Mistake

Wrong reasoning: “The best linear predictor of $X$ from $B^TX$ always equals $E[X\mid B^TX]$, so the linearity condition is automatic.”
Correction: Least squares always determines the best affine predictor, but the conditional mean can be nonlinear. The covariance calculation determines the slope only after affine conditionality has been assumed. The example with $X=(U,U^2-2.5)^T$ shows the difference.
Another limitation: Elliptical predictors with finite, positive-definite covariance satisfy the condition, but arbitrary finite-moment predictors need not. A method relying on the condition may therefore be unreliable when predictor symmetry is implausible.

Key takeaway: The linearity condition states that the predictor mean given a reduced projection is affine, with slope fixed by $\Sigma B(B^T\Sigma B)^{-1}$. Finite-moment elliptical predictors satisfy it by symmetry. In inverse regression, it connects response-conditioned predictor means to the reduced coordinates, but it does not guarantee that every relevant direction will be detected.

Check Your Understanding

Use the definitions and results above to answer these questions.

Why is $B^T\Sigma B$ invertible when $\Sigma$ is positive definite and $B$ has full column rank?
What is the distinction between the best affine predictor of $X$ from $B^TX$ and the conditional mean $E[X\mid B^TX]$?
Which symmetry argument makes the conditional mean of the component perpendicular to a projection equal to zero for spherical predictors?
In the circle example, why does $E[X_2\mid X_1=t]=0$ even though $X_2$ is not determined by $X_1=t$?
Does the inverse-regression identity guarantee that conditional means will reveal every direction relevant to the response? Explain.