Inverse Regression Means as Subspace Information

Why Inverse Means Are Useful—and Limited

In regression, it is natural to ask how the response changes with the predictors. Inverse regression asks a complementary question: among observations with a given response, where is the average predictor vector? If those conditional averages vary along a few directions, they can reveal a low-dimensional structure in a high-dimensional predictor.

The information in these averages is not the whole conditional distribution of the predictors given the response. Averages can cancel, even when the response depends strongly on the predictors. The precise relationship also depends on the coordinate system: with correlated, unstandardized predictors, the raw centered means need not lie in the central subspace itself.

Setup and the Linearity Condition

Let $X$ be a $p$-dimensional random predictor with mean $\mu=E[X]$ and positive-definite covariance matrix $\Sigma=\operatorname{Var}(X)$. Let $Y$ be a response. Assume the central subspace $\mathcal S_{Y\mid X}$ exists, and let the columns of a $p\times d$ matrix $B$ form a basis for it. Thus $Y$ is conditionally independent of $X$ given $B^T X$, and $d$ is the dimension of this subspace.

Definition: The linearity condition for the central-subspace projection is the following conditional-mean identity: $$ E[X\mid B^T X] = \mu+\Sigma B(B^T\Sigma B)^{-1}(B^T X-B^T\mu). $$ It says that the conditional mean of $X$ given the central coordinates is an affine function of those coordinates, with the linear-regression coefficient shown. The inverse exists because $\Sigma$ is positive definite and $B$ has linearly independent columns.

This identity is the relevant form of the linearity condition for the derivation below. It holds, for example, when $X$ has a multivariate normal distribution. More generally, it is an assumption, not a consequence of the existence of a central subspace alone.

It is helpful to distinguish directions in the original coordinates from directions after standardizing predictors. Define $Z=\Sigma^{-1/2}(X-\mu)$, where $\Sigma^{-1/2}$ is the symmetric inverse square root of $\Sigma$. The central subspace in these standardized coordinates is spanned by the columns of $\Sigma^{1/2}B$, since $B^T X=(\Sigma^{1/2}B)^T Z+B^T\mu$.

What the Conditional Means Must Contain

Theorem: Suppose $E\|X\|^2<\infty$, $\Sigma$ is positive definite, the central subspace has basis $B$, and the stated linearity condition holds. Then $$ E[X\mid Y]-\mu\ \in\ \operatorname{span}(\Sigma B) \quad\text{almost surely}, $$ and $$ E[Z\mid Y]\ \in\ \operatorname{span}(\Sigma^{1/2}B) \quad\text{almost surely}. $$ Equivalently, $\Sigma^{-1}(E[X\mid Y]-\mu)$ lies in $\mathcal S_{Y\mid X}$ almost surely.

The conclusion for raw $X$ is a containment in $\operatorname{span}(\Sigma B)$, not necessarily in $\operatorname{span}(B)$. For standardized $Z$, the mean deviations do lie in the central subspace expressed in standardized coordinates.

Proof. Write $U=B^T X$. Because $Y$ is conditionally independent of $X$ given $U$, the conditional mean of $X$ given both $U$ and $Y$ equals its conditional mean given $U$. The tower property therefore gives $$ E[X\mid Y]=E[E[X\mid U]\mid Y]. $$ Substitute the linearity-condition identity into the right-hand side. The matrices and vectors not involving $X$ can be taken outside the conditional expectation, giving $$ E[X\mid Y] = \mu+\Sigma B(B^T\Sigma B)^{-1} \bigl(E[B^T X\mid Y]-B^T\mu\bigr). $$ Subtracting $\mu$ shows that the result is a linear combination of the columns of $\Sigma B$, as claimed. Multiplication by $\Sigma^{-1}$ gives the equivalent original-coordinate statement. Multiplication by $\Sigma^{-1/2}$ gives $$ E[Z\mid Y] =\Sigma^{-1/2}(E[X\mid Y]-\mu) \in \operatorname{span}(\Sigma^{1/2}B), $$ which proves the standardized-coordinate claim. $\square$

Worked Example: A Conditional Mean Reveals the Relevant Direction

Let $X=(X_1,X_2)^T\sim N(0,I_2)$, and let $Y=X_1+\varepsilon$, where $\varepsilon\sim N(0,3)$ is independent of $X$. Find the inverse conditional mean and its span.

The response depends on $X$ through $X_1$ alone, so $B=(1,0)^T$ spans the central subspace. The predictors are standard normal, so the linearity condition holds. Since $(X,Y)$ is jointly normal, the conditional-mean formula for jointly normal variables gives $$ E[X\mid Y] = \operatorname{Cov}(X,Y)\operatorname{Var}(Y)^{-1}Y. $$ Here $\operatorname{Cov}(X,Y)=(1,0)^T$ and $\operatorname{Var}(Y)=1+3=4$. Consequently, $$ E[X\mid Y]= \begin{pmatrix}Y/4\\0\end{pmatrix}. $$ As $Y$ is nondegenerate, this conditional mean varies along the first coordinate, so its span is exactly the central subspace $\operatorname{span}(B)$. In this example, inverse means reveal the entire one-dimensional central subspace.

How Much Directional Information Do the Means Carry?

The theorem gives an upper bound on what inverse means can reveal: their variation cannot point outside the central subspace after the appropriate standardization. They may, however, span only part of it—or none of it. A convenient way to quantify their span is through the covariance of the standardized conditional mean.

Result: Let $M=E[Z\mid Y]$, with $E\|Z\|^2<\infty$, and define $\Gamma=E[MM^T]$. Since $E[Z]=0$, $E[M]=0$, so $\Gamma=\operatorname{Var}(E[Z\mid Y])$. The range of $\Gamma$ is the linear span of the values taken by $M$, up to events of probability zero. In particular, $\operatorname{rank}(\Gamma)$ is the dimension of the subspace spanned by the centered conditional means.

Proof. The tower property gives $E[M]=E[Z]=0$. For any vector $v\in\mathbb R^p$, $$ v^T\Gamma v=E[(v^T M)^2]. $$ This quantity is zero exactly when $v^T M=0$ almost surely. Thus the null space of $\Gamma$ consists precisely of vectors orthogonal to the almost-sure span of $M$. Because $\Gamma$ is symmetric, its range is the orthogonal complement of its null space. This is exactly that span, and its dimension is the rank of $\Gamma$. $\square$

Combining this result with the theorem shows that the rank cannot exceed $d$, the central-subspace dimension. If the rank is smaller than $d$, then first conditional means alone do not span the central subspace. The gap is real: it indicates information that these means do not carry, not evidence that the corresponding regression directions are irrelevant.

Worked Example: Correlation Changes the Raw Mean Direction

Let $X$ be bivariate normal with mean zero and covariance $$ \Sigma= \begin{pmatrix} 1&1/2\\ 1/2&1 \end{pmatrix}, $$ and take $Y=X_1$. Determine the central direction and the inverse mean direction.

The response is determined by $X_1$, so the central subspace is spanned by $B=(1,0)^T$. The normal distribution satisfies the linearity condition. Here $B^T\Sigma B=1$, and $$ E[X\mid X_1] = \Sigma B X_1 = \begin{pmatrix}X_1\\X_1/2\end{pmatrix}. $$ The raw mean deviations lie along $(1,1/2)^T$, which is not the central direction $(1,0)^T$. The covariance correction recovers the central direction: $$ \Sigma^{-1}= \begin{pmatrix}4/3&-2/3\\-2/3&4/3\end{pmatrix}, \qquad \Sigma^{-1}(E[X\mid Y]-\mu) = \begin{pmatrix}X_1\\0\end{pmatrix}. $$ Thus the distinction between $\operatorname{span}(\Sigma B)$ and $\operatorname{span}(B)$ is essential when predictors are correlated.

When Conditional Means Miss a Relevant Direction

A conditional mean summarizes a conditional distribution by its first moment. If the conditional distribution changes symmetrically around its unconditional mean, that summary can remain constant. A constant mean then has zero span, even though $Y$ can depend on $X$ and the central subspace can be nontrivial.

Worked Example: A Symmetric Effect With Zero Inverse Mean

Let $X=(X_1,X_2)^T\sim N(0,I_2)$, and let $Y=X_1^2$. The response depends only on $X_1$, so $\operatorname{span}((1,0)^T)$ is a dimension-one sufficient subspace. It is minimal: $Y$ is nonconstant, so it cannot be independent of all of $X$. Indeed, if $Y$, a function of $X$, were independent of $X$, then every event $A$ determined by $Y$ would be independent of itself, forcing $P(Y\in A)=P(Y\in A)^2$. This is impossible for, for example, $A=\{Y\leq 1\}$, whose probability is strictly between zero and one.

The distribution of $X_1$ conditional on $X_1^2=y$, for $y>0$, assigns equal probability to $\sqrt y$ and $-\sqrt y$. Hence $E[X_1\mid Y=y]=0$. Also, $X_2$ is independent of $Y$ and has mean zero, so $E[X_2\mid Y]=0$. Therefore $$ E[X\mid Y]= \begin{pmatrix}0\\0\end{pmatrix} \quad\text{almost surely}. $$ The conditional means span no direction, although the central subspace has dimension one. The response depends on the magnitude of the first coordinate, but the two signs cancel in the conditional average.

This example identifies the limitation precisely: inverse means contain information about changes in conditional location, not every possible change in conditional distribution. When their span is smaller than the central subspace, other features of the inverse conditional distribution may still vary with $Y$.

Common Mistake and Practical Interpretation

Claiming raw conditional means always lie in the central subspace. Under the stated condition, the raw centered means lie in $\operatorname{span}(\Sigma B)$. The central-subspace statement applies after whitening, or after applying $\Sigma^{-1}$ to the raw mean deviations.
Equating zero inverse means with no regression relationship. The quadratic example has a nontrivial central subspace despite $E[X\mid Y]=E[X]$. A constant conditional mean only says that the first conditional moment does not vary.
Assuming the linearity condition follows from central sufficiency. Sufficiency states that $Y$ depends on $X$ through the central coordinates; it does not by itself impose the displayed conditional-mean identity. The identity is an additional distributional condition.

In practice, inverse conditional means are useful as a source of directional information when their span is sufficiently rich. Their rank is a measure of the dimension represented by those means, and the theorem constrains that span under the linearity condition. A smaller rank does not establish a smaller central subspace: it may reflect cancellation or dependence expressed through features beyond the conditional mean.

Check Your Understanding

Use the assumptions and distinctions developed above to answer these questions.

Under the linearity condition, in which subspace do $E[X\mid Y]-\mu$ and $E[Z\mid Y]$ lie?
Why can the raw inverse mean direction differ from the central direction when $\Sigma$ is not the identity matrix?
What does the rank of $\operatorname{Var}(E[Z\mid Y])$ measure, and what upper bound does the theorem give it?
In the model $Y=X_1^2$ with standard normal predictors, why is $E[X\mid Y]=0$ compatible with a nontrivial central subspace?
Does a conditional-mean span smaller than the central subspace prove that the missing directions are irrelevant? Explain.