Why SIR Recovers Directions Under Linearity

Why Linearity Makes SIR's Directions Relevant

The population SIR operator is built from the conditional mean of a standardized predictor given the response. Its eigenvectors can therefore reveal directions in the predictor space only when those inverse regression means are constrained to the directions that matter for the response. The linearity condition supplies that constraint: it makes each inverse regression mean belong to the standardized central subspace. A separate coverage condition is needed to ensure that the means actually span the whole subspace.

The distinction is important. Linearity gives a guarantee against irrelevant directions, but it does not guarantee that every relevant direction appears in the operator. The result below separates these two claims and states exactly what additional condition gives equality.

Standardized Central Subspace and Linearity

Let \(X\) be a \(p\)-dimensional random vector with mean \(\mu\) and positive-definite covariance matrix \(\Sigma\), and suppose \(E[\|X\|^2]<\infty\). Define the standardized predictor \(Z=\Sigma^{-1/2}(X-\mu)\), so that \(E[Z]=0\) and \(\operatorname{Cov}(Z)=I_p\). Assume a central subspace exists, and let \(B\) be a \(p\times d\) full-column-rank basis matrix for it. In particular, the response \(Y\) is conditionally independent of \(X\) given \(B^T X\). The corresponding directions in standardized coordinates are the columns of \(A=\Sigma^{1/2}B\); write \(\mathcal S_Z=\operatorname{span}(A)\).

Let \(P_A=A(A^TA)^{-1}A^T\) be the orthogonal projection onto \(\mathcal S_Z\). Since \(B^TX=B^T\mu+A^TZ\), conditioning on \(B^TX\) is equivalent to conditioning on \(A^TZ\).

Definition: The linearity condition in standardized coordinates is \(E[Z\mid A^TZ]=P_AZ\) almost surely. It says that the conditional mean of the standardized predictor given the central-subspace indices equals its linear projection onto those indices. The population SIR operator is \(M_{\mathrm{SIR}}=E[m(Y)m(Y)^T]\), where \(m(Y)=E[Z\mid Y]\). These expectations exist under the stated second-moment assumption.

The linearity condition is a condition on the predictor distribution relative to the central directions. It is not a claim that the response itself is linear in those directions. For example, the response may depend on a nonlinear function of \(B^TX\); what matters here is the conditional mean of \(Z\) given \(A^TZ\).

The Range-Containment Result

Theorem: Suppose \(Y\) is conditionally independent of \(X\) given \(B^TX\), the central subspace has basis \(B\), the moments above exist, and the linearity condition holds. Then \(\operatorname{range}(M_{\mathrm{SIR}})\subseteq\mathcal S_Z\).

To prove the result, put \(U=A^TZ\). Conditional independence implies that conditioning on \(Y\) does not change the conditional distribution of \(Z\) once \(U\) is given. Thus \(E[Z\mid Y,U]=E[Z\mid U]\) almost surely. The tower property and linearity then give

\begin{aligned} m(Y) &=E[Z\mid Y]\\ &=E\!\left[E[Z\mid Y,U]\mid Y\right]\\ &=E\!\left[E[Z\mid U]\mid Y\right]\\ &=E[P_AZ\mid Y]\\ &=P_AE[Z\mid Y] =P_Am(Y). \end{aligned}

Therefore \(m(Y)\) belongs to \(\mathcal S_Z\) almost surely. In particular, \(P_Am(Y)=m(Y)\). Taking outer products and expectations yields

M_{\mathrm{SIR}} =E[P_Am(Y)m(Y)^TP_A] =P_AM_{\mathrm{SIR}}P_A.

For any vector \(v\), this identity implies \(M_{\mathrm{SIR}}v=P_AM_{\mathrm{SIR}}P_Av\), which belongs to \(\mathcal S_Z\). Hence the range is contained in \(\mathcal S_Z\), as claimed. \(\square\)

The argument uses both ingredients. Conditional independence allows the conditioning on \(Y\) to be removed after conditioning on the central indices. Linearity then turns the resulting conditional mean into a projection onto the central subspace. Without the first ingredient, the indices need not capture all response dependence; without the second, the inverse mean need not lie in the proposed subspace.

Worked Example: A Gaussian Single-Index Response

Let \(X\sim N_2(\mu,\Sigma)\), where \(\mu=(2,-1)^T\), \(\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}\), and \(B=(1,1)^T\). Set \(T=B^T(X-\mu)\) and let \(Y=T\). The response depends on \(X\) only through \(B^TX\), so the central subspace is \(\operatorname{span}(B)\). Calculate the population SIR operator and check its range.

First, \(\Sigma\) is positive definite: its leading diagonal entry is \(4>0\), and its determinant is \(4(2)-1(1)=7>0\). Also,

\Sigma B= \begin{pmatrix}4&1\\1&2\end{pmatrix} \begin{pmatrix}1\\1\end{pmatrix} = \begin{pmatrix}5\\3\end{pmatrix}, \qquad B^T\Sigma B=(1,1)\begin{pmatrix}5\\3\end{pmatrix}=8.

Thus \(T\sim N(0,8)\). Joint normal conditioning gives

E[X\mid T]=\mu+\frac{\Sigma B}{B^T\Sigma B}T =\mu+\frac18\begin{pmatrix}5\\3\end{pmatrix}T.

Since \(Z=\Sigma^{-1/2}(X-\mu)\), this becomes \(E[Z\mid T]=\frac18\Sigma^{1/2}B\,T\). Because \(Y=T\), \(m(Y)=\frac18\Sigma^{1/2}B\,T\). Using \(E[T^2]=\operatorname{Var}(T)=8\),

M_{\mathrm{SIR}} =\frac{1}{64}\Sigma^{1/2}B\,E[T^2]\,B^T\Sigma^{1/2} =\frac18\Sigma^{1/2}BB^T\Sigma^{1/2}.

This is a positive multiple of \(vv^T\), where \(v=\Sigma^{1/2}B\neq0\). Its range is therefore \(\operatorname{span}(v)=\mathcal S_Z\). In this example the inverse mean varies along the central direction, so the containment is an equality.

Coverage: When Containment Becomes Equality

Containment alone allows the operator to have rank smaller than the central-subspace dimension. The precise extra requirement is that the inverse regression means vary enough to span every central direction. The following fact makes the criterion exact.

Theorem: Under the assumptions of the range-containment result, let \(m(Y)=E[Z\mid Y]\). Then \(\operatorname{range}(M_{\mathrm{SIR}})=\operatorname{span}_{\mathrm{a.s.}}\{m(Y)\}\), where the right-hand side is the smallest fixed linear subspace containing \(m(Y)\) almost surely. Consequently, \(\operatorname{range}(M_{\mathrm{SIR}})=\mathcal S_Z\) if and only if the inverse regression means span \(\mathcal S_Z\) almost surely.

For a complete justification, \(M_{\mathrm{SIR}}\) is symmetric and nonnegative definite because for every \(v\in\mathbb R^p\),

v^TM_{\mathrm{SIR}}v =E[(v^Tm(Y))^2]\geq 0.

This quantity is zero exactly when \(v^Tm(Y)=0\) almost surely: a nonnegative random variable has expectation zero if and only if it is zero almost surely. Hence the null space of \(M_{\mathrm{SIR}}\) consists exactly of vectors orthogonal to every inverse mean, up to almost-sure equality. For a symmetric matrix, its range is the orthogonal complement of its null space. Taking orthogonal complements proves the stated range identity. Combining it with containment proves the equality criterion. \(\square\)

Definition: The inverse-mean coverage condition holds when \(\operatorname{span}_{\mathrm{a.s.}}\{E[Z\mid Y]\}=\mathcal S_Z\). Under linearity and the central-subspace assumptions, it is equivalent to \(\operatorname{rank}(M_{\mathrm{SIR}})=d\), and therefore to SIR recovering the entire standardized central subspace at the population level.

An equivalent positive-definiteness formulation is useful. For every nonzero \(v\in\mathcal S_Z\), coverage requires \(E[(v^Tm(Y))^2]>0\). If this expectation were zero for some nonzero central direction, the inverse means would all be orthogonal to that direction and could not span the subspace. Since the operator is nonnegative definite and its range is already contained in a \(d\)-dimensional space, rank \(d\) is precisely the condition for its range to equal that space.

Worked Example: Symmetry Causes Coverage Failure

Use the Gaussian predictor and \(T\) from the preceding example, but now let \(Y=T^2\). The response still depends only on \(B^TX\), and its central subspace is one-dimensional. The Gaussian predictor satisfies the linearity condition. Determine the SIR operator.

The conditional mean given \(T\) is \(E[Z\mid T]=\frac18\Sigma^{1/2}B\,T\). A centered normal variable has a distribution symmetric about zero, and conditioning on \(T^2\) leaves the two signs equally likely (apart from the probability-zero event \(T=0\)). Therefore \(E[T\mid T^2]=0\), and

m(Y)=E[Z\mid T^2] =E[E[Z\mid T]\mid T^2] =\frac18\Sigma^{1/2}B\,E[T\mid T^2] =0.

It follows that \(M_{\mathrm{SIR}}=0\) and its range is \(\{0\}\), strictly smaller than \(\mathcal S_Z\). Linearity still guarantees containment, but coverage fails: the inverse means do not vary at all. This is why SIR can miss a genuine direction when the response depends on it symmetrically.

Worked Example: Full Coverage With a Two-Dimensional Response

Let \(Z\sim N_2(0,I_2)\) and take the response to be the vector \(Y=Z\). The standardized central subspace is all of \(\mathbb R^2\), and \(A=I_2\) gives the linearity condition \(E[Z\mid A^TZ]=Z\). Compute the operator and verify coverage.

Since \(Y=Z\), the conditional mean is \(m(Y)=Y\). Therefore

M_{\mathrm{SIR}}=E[YY^T]=E[ZZ^T]=I_2.

The operator has rank \(2\) and range \(\mathbb R^2=\mathcal S_Z\). The inverse means span both coordinate directions, so the coverage condition holds. This example illustrates the equality criterion directly; the theorem does not require the response to be scalar-valued.

What the Result Does—and Does Not—Guarantee

The range-containment theorem is a population statement under stated assumptions. It says that, if the linearity condition holds, SIR's nonzero operator directions cannot extend outside the standardized central subspace. Equality needs coverage as well. Neither claim says that a finite-sample estimate will recover those directions exactly; estimating the operator and its eigenspace introduces sampling variation.

A common mistake is to conclude from linearity alone that SIR must recover the full central subspace. The reasoning is wrong because “every inverse mean lies in the subspace” does not imply “the inverse means span the subspace.” The \(Y=T^2\) example satisfies linearity, yet all inverse means vanish. The correction is to check both conditions: linearity for containment, and inverse-mean coverage for equality.

A second pitfall is to confuse a direction in the original predictor coordinates with its standardized counterpart. If \(B\) spans the central subspace for \(X\), then the corresponding standardized directions are spanned by \(\Sigma^{1/2}B\), not generally by \(B\) itself. Keeping the coordinate system explicit is essential when interpreting SIR eigenvectors.

In practice, the distinction helps interpret a low-rank SIR operator. Under the model assumptions, a missing direction can reflect failure of coverage even when the linearity condition is reasonable. The operator's range is a subspace supported by inverse means; it is not automatically a complete representation of all response-relevant directions.

Check Your Understanding

Use the assumptions and results above to answer these questions.

Why does conditional independence given the central indices allow \(E[Z\mid Y,U]\) to be replaced by \(E[Z\mid U]\)?
State the linearity condition in standardized coordinates and explain what it implies for \(m(Y)\).
Why does \(M_{\mathrm{SIR}}=P_AM_{\mathrm{SIR}}P_A\) imply range containment?
Give two equivalent ways to state the inverse-mean coverage condition.
In the \(Y=T^2\) example, which condition fails, and what is the resulting SIR operator?