Directional Regression and Its Population Matrix

Why Use Pairwise Inverse-Regression Information?

Inverse-regression methods study how the distribution of predictors changes after conditioning on the response. A conditional mean can reveal a direction when the response changes that mean, but it can miss effects that leave the mean unchanged. Directional regression uses a richer object: the conditional second moment of the difference between two independent predictors whose responses are observed. This brings conditional means and conditional covariances into one population operator.

Throughout, let $Z\in\mathbb{R}^p$ be a standardized predictor, so $E[Z]=0$ and $\operatorname{Cov}(Z)=I_p$, and let $Y$ be a response. Assume $E[\|Z\|^4]<\infty$, which ensures the matrix operator below has finite entries. Let $(Z',Y')$ be an independent copy of $(Z,Y)$. Write $m(y)=E[Z\mid Y=y]$ and $C(y)=\operatorname{Cov}(Z\mid Y=y)$, wherever these conditional moments are defined.

Definition: The directional regression population operator is $$ M_{\mathrm{DR}} = E\left[ \left\{ 2I_p-E\left[(Z-Z')(Z-Z')^T\mid Y,Y'\right] \right\}^2 \right]. $$ The square is a matrix product: $A^2=AA$ for a square matrix $A$. The inner conditional expectation is taken given the two responses; the outer expectation averages over their joint distribution.

The factor $2I_p$ is tied to standardization. Without standardization, the unconditional second moment of a predictor difference would involve the predictor covariance matrix rather than $2I_p$. In this tutorial the target is the standardized population operator, not a sample estimate.

Deriving the Standardized Population Form

Conditional on $Y$ and $Y'$, the predictors $Z$ and $Z'$ are independent: the two observed pairs were drawn independently. Their conditional means are $m(Y)$ and $m(Y')$, and their conditional covariance matrices are $C(Y)$ and $C(Y')$. The conditional mean of the difference is therefore $m(Y)-m(Y')$, and its conditional covariance is $C(Y)+C(Y')$. For any random vector $W$, its second moment is its covariance plus the outer product of its mean. Applying that identity gives the required conditional second moment.

E\left[(Z-Z')(Z-Z')^T\mid Y,Y'\right] = C(Y)+C(Y')+ \{m(Y)-m(Y')\}\{m(Y)-m(Y')\}^T .

Consequently, an equivalent population form, expressed entirely through inverse-regression means and covariances, is

M_{\mathrm{DR}} = E\left[ \left\{ 2I_p-C(Y)-C(Y') -\{m(Y)-m(Y')\}\{m(Y)-m(Y')\}^T \right\}^2 \right].

Standardization also implies two useful moment identities. By iterated expectation, $E[m(Y)]=E[Z]=0$. Decomposing the second moment conditionally gives $E[C(Y)+m(Y)m(Y)^T]=E[ZZ^T]=I_p$. These identities are useful checks on calculations, but the pairwise formula above still depends on the conditional moments at both responses; it cannot in general be replaced by a function of only the average conditional covariance.

Result: $M_{\mathrm{DR}}$ is symmetric and positive semidefinite. In particular, for every $a\in\mathbb{R}^p$, $$ a^T M_{\mathrm{DR}}a = E\left[\left\| \left\{ 2I_p-E[(Z-Z')(Z-Z')^T\mid Y,Y'] \right\}a \right\|^2\right] \geq 0. $$

To verify the result, the inner conditional second-moment matrix is symmetric, so the matrix in braces is symmetric as well. If it is denoted by $A$, then $M_{\mathrm{DR}}=E[A^2]$, and $a^TA^2a=(Aa)^T(Aa)=\|Aa\|^2$. Averaging this nonnegative quantity proves positive semidefiniteness. This also shows why the order of operations matters: the definition averages the squared conditional matrix, not the square of its average.

What Subspace Does the Operator Target?

In sufficient dimension reduction, the central subspace is the smallest predictor subspace that retains all information in $Z$ about $Y$. If it has basis matrix $B$, the defining conditional-independence statement is $Y\perp Z\mid B^TZ$, and its column space is the central subspace. Directional regression is designed so that the range of its population matrix can contain directions in this subspace. That connection requires conditions on inverse moments; standardizing predictors alone does not establish it.

The following sufficient condition makes the range relationship explicit. It asks that predictor components orthogonal to a target subspace have zero conditional mean and retain their unconditional cross-moments with the full predictor after conditioning on the response. This is a moment condition, not a consequence of standardization by itself.

Theorem: Let $\mathcal{S}$ be a subspace of $\mathbb{R}^p$. Suppose that, for every $v\in\mathcal{S}^{\perp}$, $$ E[v^TZ\mid Y]=0 \quad\text{and}\quad E[Z(v^TZ)\mid Y]=v \quad\text{almost surely}. $$ Then $M_{\mathrm{DR}}v=0$ for every $v\in\mathcal{S}^{\perp}$, and therefore $\operatorname{range}(M_{\mathrm{DR}})\subseteq\mathcal{S}$.

Proof. Fix $v\in\mathcal{S}^{\perp}$. The first condition says $m(Y)^Tv=0$. By the definition of conditional covariance, $$ C(Y)v = E[Z(v^TZ)\mid Y]-m(Y)E[v^TZ\mid Y] = v- m(Y)\,0 = v. $$ The same identities hold at $Y'$. Hence $C(Y)v=C(Y')v=v$, and $\{m(Y)-m(Y')\}^Tv=0$. Multiplying the conditional difference matrix by $v$ now gives $$ E[(Z-Z')(Z-Z')^T\mid Y,Y']v = v+v+\{m(Y)-m(Y')\}\,0 = 2v. $$ Thus the matrix inside braces in the definition of $M_{\mathrm{DR}}$ sends $v$ to zero for almost every response pair. Its square also sends $v$ to zero, so averaging yields $M_{\mathrm{DR}}v=0$. Since $M_{\mathrm{DR}}$ is symmetric, its range is orthogonal to its null space. The null space contains $\mathcal{S}^{\perp}$, so its range lies in $(\mathcal{S}^{\perp})^\perp=\mathcal{S}$. $\square$

When $\mathcal{S}$ is the central subspace, this result gives a sufficient route to the usual target-subspace interpretation. It is a containment statement, not an automatic equality: the operator may have additional zero eigenvalues within $\mathcal{S}$. Recovering the entire central subspace requires enough signal in every relevant direction, beyond the condition just proved.

Worked Population Calculations

Worked Example: A Linear Response in One Dimension

Let $Z\sim N(0,1)$ and $Y=Z$. Since $Y$ determines $Z$, conditioning on $Y,Y'$ makes $Z-Z'=Y-Y'$ known. The difference $D=Y-Y'$ has distribution $N(0,2)$, so $E[D^2]=2$ and $E[D^4]=3(2^2)=12$. The scalar operator is

M_{\mathrm{DR}} =E[(2-D^2)^2] =4-4E[D^2]+E[D^4] =4-8+12 =8.

The predictor space is one-dimensional and the response varies with its only direction; the positive operator has range $\mathbb{R}$. As an arithmetic check using the same moments, the polynomial $1-D^2/2+D^4/4$ has expectation $1-2/2+12/4=3$. This is a polynomial moment calculation, not the expansion of $E[(1-D^2/2)^2]$.

Worked Example: A Symmetric Response

Again let $Z\sim N(0,1)$, but now set $Y=Z^2$. Given $Y=y$, the two possible values of $Z$ are $\sqrt y$ and $-\sqrt y$, with equal probabilities (for $y>0$). Thus $m(Y)=0$, while $C(Y)=Y$, because the conditional second moment is $Y$. The conditional difference second moment is $Y+Y'$, so

M_{\mathrm{DR}}=E[(2-Y-Y')^2].

Here $Y$ and $Y'$ are independent chi-square variables with one degree of freedom. Each has mean $1$ and variance $2$. Therefore $E[Y+Y']=2$ and $\operatorname{Var}(Y+Y')=4$. Centering the square at the mean gives $$ E[(2-Y-Y')^2] = \operatorname{Var}(Y+Y')+\{2-E[Y+Y']\}^2 = 4+0 = 4. $$

The inverse-regression mean is identically zero, so a method based only on changes in that mean has no signal here. Directional regression still has a positive operator because the conditional variance changes with $Y$.

Worked Example: A Discrete Two-Predictor Model

Let $Z=(Z_1,Z_2)^T$, where $Z_1$ and $Z_2$ are independent and each takes values $-1$ and $1$ with equal probability. This predictor has mean zero and covariance $I_2$. Set $Y=Z_1$, so the response depends only on the first coordinate. Given $Y$, $Z_1=Y$, while $Z_2$ remains independent with mean zero and variance one. For an independent response copy $Y'$,

E[(Z-Z')(Z-Z')^T\mid Y,Y'] = \begin{pmatrix} (Y-Y')^2 & 0\\ 0 & 2 \end{pmatrix}.

The off-diagonal entries are zero because the second coordinate has conditional mean zero and is independent of the first. The directional matrix inside braces is consequently $$ \begin{pmatrix} 2-(Y-Y')^2 & 0\\ 0 & 0 \end{pmatrix}. $$ If $Y=Y'$, which occurs with probability $1/2$, its first diagonal entry is $2$. If $Y\ne Y'$, also with probability $1/2$, then $(Y-Y')^2=4$, and the entry is $-2$. Squaring either value gives $4$, so $$ M_{\mathrm{DR}} = \begin{pmatrix} 4&0\\ 0&0 \end{pmatrix}. $$ Its range is the span of $(1,0)^T$, precisely the response-relevant direction in this model.

Interpretation and Common Pitfalls

The operator combines pairwise changes in conditional means and conditional covariances through the matrix of conditional predictor differences. Its eigenvectors with nonzero eigenvalues identify directions represented by this population matrix; its range is the corresponding target subspace. The actual reduction subspace depends on the model and the inverse-moment conditions, so an eigenvector calculation should be interpreted alongside the assumptions that justify its target.

Squaring after averaging. The definition is $E[A^2]$, where $A$ is the conditional matrix. Replacing it by $\{E[A]\}^2$ changes the operator, since in general $E[A^2]\ne\{E[A]\}^2$.
Ignoring standardization. The $2I_p$ term assumes $E[Z]=0$ and $\operatorname{Cov}(Z)=I_p$. Using it unchanged with unstandardized predictors does not give the stated population operator.
Claiming automatic exhaustiveness. A range-containment result does not prove that every direction in the central subspace has a positive eigenvalue. Zero eigenvalues can remain in relevant directions.

Key takeaway: Directional regression averages the square of a conditional pairwise-difference matrix. Under suitable inverse-moment conditions, the range of this positive-semidefinite population operator lies in the central subspace, while equality additionally requires that all relevant directions be detected.

Check Your Understanding

Answer these questions by using the conditional-moment form and the examples above.

Why are $Z$ and $Z'$ conditionally independent given $Y,Y'$?
Write the conditional difference second moment in terms of $m(Y)$, $m(Y')$, $C(Y)$, and $C(Y')$.
Explain why positive semidefiniteness follows from $M_{\mathrm{DR}}=E[A^2]$ when $A$ is symmetric.
In the symmetric-response example, why is the directional regression operator positive even though $E[Z\mid Y]=0$?
What additional conclusion, beyond range containment, is needed to identify the whole central subspace from the operator?