A Reliable Workflow for Moment-Based SDR
A dimension-reduction calculation is not complete when an eigenvector has been found. One must also identify the regression subspace the problem is trying to recover, choose an operator suited to the regression pattern, check the operator’s assumptions, and determine whether its nonzero eigenspace is large enough to recover the target. These steps matter because different inverse-moment methods can succeed or fail on the same model.
Throughout, let \(Z\in\mathbb R^p\) be a standardized predictor vector with \(E[Z]=0\) and \(\operatorname{Cov}(Z)=I_p\), and let \(Y\) be a response. Assume the moments used below exist. A subspace \(\mathcal S\subseteq\mathbb R^p\) is a dimension-reduction subspace if \(Y\) is conditionally independent of \(Z\) given the projection \(P_{\mathcal S}Z\). When a unique smallest such subspace exists, it is the central subspace, denoted \(\mathcal S_{Y\mid Z}\); its dimension \(d\) is the structural dimension.
In standardized coordinates, the linearity condition relative to a candidate target \(\mathcal S\) is \(E[Z\mid P_{\mathcal S}Z]=P_{\mathcal S}Z\). The constant-variance condition relative to that target is \(\operatorname{Var}((I_p-P_{\mathcal S})Z\mid P_{\mathcal S}Z)=I_p-P_{\mathcal S}\). These are assumptions about the predictor distribution, not consequences of the regression equation. They are particularly natural for multivariate normal predictors.
Choose the Operator for the Regression Pattern
The three population operators most useful in this workshop summarize different features of the conditional distribution of predictors given the response. SIR uses conditional means, SAVE uses conditional covariance matrices, and PHD uses a response-weighted second moment. In every case, inspect the operator’s rank as well as its eigenvectors.
SIR is a natural first choice when the conditional mean of the predictors changes with the response. Symmetric effects can make those means cancel, however, so a zero SIR matrix need not mean that the response is unrelated to the predictors. SAVE can detect changes in conditional covariance that SIR misses. PHD can detect response-weighted second-moment structure, but its containment claim requires care: linearity alone is not enough.
Express how the response depends on a projection of the predictors, and identify the candidate subspace and its dimension.
Use SIR for inverse-mean changes, SAVE when inverse covariances may carry signal, and PHD when a response-weighted second moment is informative.
Verify moment existence and the predictor-distribution conditions required for containment. In particular, PHD needs more than linearity for its usual containment guarantee.
Compare the nonzero eigenspace with the target, then inspect the gap between relevant eigenvalues and zero to assess sensitivity.
Worked Example: SIR Recovers a Threshold Direction
Worked Example: SIR Recovers a Threshold Direction
Let \(Z=(Z_1,Z_2)^T\sim N(0,I_2)\), and define \(Y=1\) when \(Z_1>0\) and \(Y=0\) otherwise. The regression depends only on \(Z_1\), so the target is \(\mathcal S=\operatorname{span}(e_1)\), where \(e_1=(1,0)^T\).
By symmetry, \(P(Y=1)=P(Y=0)=1/2\). For a standard normal variable, the density is \(\phi(z)=(2\pi)^{-1/2}e^{-z^2/2}\), and \[ E[Z_1\mid Z_1>0]=\frac{\int_0^\infty z\phi(z)\,dz}{1/2} =\frac{\phi(0)}{1/2} =\sqrt{\frac{2}{\pi}}. \] The negative-half expectation is its negative. Since \(Z_2\) is independent of \(Z_1\) and has mean zero, it has conditional mean zero in both response groups. Thus, writing \(a=\sqrt{2/\pi}\), \[ E[Z\mid Y=1]=ae_1,\qquad E[Z\mid Y=0]=-ae_1. \]
The overall mean of these conditional means is zero. Therefore \[ M_{\mathrm{SIR}} =\tfrac12(ae_1)(ae_1)^T+\tfrac12(-ae_1)(-ae_1)^T =a^2 e_1e_1^T =\begin{pmatrix}2/\pi&0\\0&0\end{pmatrix}. \] Its only nonzero eigenvalue is \(2/\pi\), with eigenvector \(e_1\). The rank is one, matching the target dimension, and the nonzero eigenspace equals \(\mathcal S\). Here the method recovers the target exactly.
Why SIR’s Range Is Contained in the Target
The preceding example illustrates a general containment argument. State the assumptions explicitly: suppose \(\mathcal S\) is a dimension-reduction subspace, the linearity condition holds relative to \(\mathcal S\), and the conditional expectations defining SIR exist. Then SIR’s range is contained in \(\mathcal S\).
To prove containment, conditional independence gives \(E[Z\mid Y,P_{\mathcal S}Z]=E[Z\mid P_{\mathcal S}Z]\). The linearity condition makes the latter equal to \(P_{\mathcal S}Z\). Taking a conditional expectation given \(Y\) yields
Consequently, every centered conditional-mean vector \(E[Z\mid Y]-E[Z]\) lies in \(\mathcal S\), since \(E[Z]=0\). The covariance \(M_{\mathrm{SIR}}\) is an expectation of outer products of these vectors. For any \(v\in\mathbb R^p\), \(M_{\mathrm{SIR}}v\) is an expectation of vectors in \(\mathcal S\), and hence belongs to \(\mathcal S\). This proves the claimed range containment. The proof also shows why a zero or low-rank operator may leave part of the target unrecovered. \(\square\)
Worked Example: SAVE Finds a Symmetric Effect
Worked Example: SAVE Finds a Symmetric Effect
Let \(Z\sim N(0,I_2)\), and set \(Y=Z_1^2\). The response depends on \(Z_1\) through a symmetric function. The target is \(\operatorname{span}(e_1)\), but \(E[Z\mid Y]=0\): conditional on \(Z_1^2\), the two possible signs of \(Z_1\) have equal probability, and \(Z_2\) remains independent with mean zero. Thus \(M_{\mathrm{SIR}}=0\), so SIR does not identify the direction in this model.
Given \(Y=y\), the first coordinate has conditional mean zero and variance \(y\). The second coordinate remains independent with variance one. Hence \[ \operatorname{Cov}(Z\mid Y)= \begin{pmatrix}Y&0\\0&1\end{pmatrix}, \qquad (I_2-\operatorname{Cov}(Z\mid Y))^2 =\begin{pmatrix}(1-Y)^2&0\\0&0\end{pmatrix}. \] Since \(Y=Z_1^2\), \(E[Y]=1\) and \(E[Y^2]=E[Z_1^4]=3\). Therefore \(E[(1-Y)^2]=1-2+3=2\), and \[ M_{\mathrm{SAVE}}=\begin{pmatrix}2&0\\0&0\end{pmatrix}. \] The range is \(\operatorname{span}(e_1)\), and the nonzero eigenvalue is \(2\). The normal predictor distribution satisfies the linearity and constant-variance conditions relative to this target; the conditional covariance calculation directly displays the signal SAVE uses.
PHD: Containment Requires a Stronger Check
For PHD, a useful sufficient set of conditions is conditional independence given the target, linearity, and constant variance relative to the target, along with finite moments such as \(E[|Y-EY|\|Z\|^2]<\infty\). Under these conditions the centered PHD range is contained in the target. The constant-variance condition is essential to this standard argument; linearity by itself does not justify the conclusion.
Here is the derivation. Write \(R=(I_p-P)Z\). Given \(PZ\), the conditional mean of \(Z\) is \(PZ\); the conditional covariance of \(R\) is \(I_p-P\). Because \(PZ\) is fixed given itself and \(E[R\mid PZ]=0\), the conditional second moment is
Conditional independence lets us factor the response from \(ZZ^T\) when conditioning on \(PZ\). Thus
The first term inside this expectation has both left and right factors \(P\), so its range lies in \(\mathcal S\). The second term has expectation \(E[(Y-EY)(I_p-P)]=0\), because \(I_p-P\) is fixed and \(E[Y-EY]=0\). Hence \(M_{\mathrm{PHD}}\) has range contained in \(\mathcal S\), as claimed. \(\square\)
Worked Example: PHD Recovers a Quadratic Direction
Again let \(Z\sim N(0,I_2)\), but now use the centered response \(Y=Z_1^2-1\). The central subspace is \(\operatorname{span}(e_1)\), and \(E[Y]=0\). The PHD matrix is \(E[YZZ^T]\). Its first diagonal entry is \[ E[(Z_1^2-1)Z_1^2]=E[Z_1^4]-E[Z_1^2]=3-1=2. \] Its second diagonal entry is \[ E[(Z_1^2-1)Z_2^2] =E[Z_1^2-1]E[Z_2^2]=0\cdot1=0, \] using independence. The off-diagonal entry is \(E[(Z_1^2-1)Z_1Z_2]=0\), since \(E[Z_2]=0\) and \(Z_2\) is independent of \(Z_1\). Therefore \[ M_{\mathrm{PHD}}=\begin{pmatrix}2&0\\0&0\end{pmatrix}. \] Its range is exactly the one-dimensional target. Normality ensures both stated predictor conditions, and the required moments are finite.
Dimension, Stability, and a PHD Pitfall
For a symmetric SDR matrix, compute its eigenvalues and inspect the eigenspace associated with nonzero eigenvalues. If the target has dimension \(d\), containment plus rank \(d\) gives exact recovery. If the rank is smaller, the operator has failed to reveal all target directions, even if every direction it does reveal is valid. In the three examples above, the recovered one-dimensional operators have eigenvalue gaps \(2/\pi\), \(2\), and \(2\), respectively, between their signal eigenvalue and zero.
In data, estimated matrices are perturbed versions of their population counterparts. A small gap between the \(d\)th signal eigenvalue and the remaining eigenvalues makes the estimated eigenspace sensitive to perturbations; a substantial gap supports more stable estimation. This is an eigenspace statement, not evidence that a particular basis vector is uniquely meaningful: any orthonormal basis spanning the same target describes the same reduction.
A particularly important pitfall is to assert PHD containment from linearity alone. The following finite-support example shows why that reasoning fails. Let \(Z_1\) be equally likely to be \(1\) or \(-1\). Given \(Z_1=1\), let \(Z_2\) be equally likely to be \(\sqrt2\) or \(-\sqrt2\); given \(Z_1=-1\), let \(Z_2=0\). Then \(E[Z]=0\), \(\operatorname{Cov}(Z)=I_2\), and \(E[Z\mid Z_1]=Z_1e_1\), so linearity holds for \(\mathcal S=\operatorname{span}(e_1)\). Set \(Y=Z_1\); the central subspace is exactly that span. Yet the centered PHD matrix is
Indeed, the first two entries vanish by symmetry of \(Z_1\) and the conditional symmetry of \(Z_2\). For the last entry, the \(Z_1=1\) case contributes \((1/2)(2)=1\), while the \(Z_1=-1\) case contributes zero. Thus the PHD range is \(\operatorname{span}(e_2)\), outside the target. The conditional variance of \(Z_2\) is \(2\) when \(Z_1=1\) and \(0\) when \(Z_1=-1\), so the constant-variance condition fails. The correction is to verify the required conditions before interpreting a PHD eigenspace as a target direction.
In practice, use a conditional-mean operator when inverse means change, use a covariance-sensitive operator when the regression is symmetric, and use PHD only with an explicit assumption check. Then report both the eigenspace and what its rank can establish. A direction is a statistical reduction direction under the stated model and conditions; it is not, by itself, a causal effect or a guarantee of complete dimension recovery.
Check Your Understanding
For each question, distinguish what the operator calculates from what assumptions allow you to conclude.
- In the threshold example, why does the SIR matrix have rank one, and what is its nonzero eigenvalue?
- For \(Y=Z_1^2\) with standard normal predictors, explain why SIR is zero while SAVE is not.
- State the extra predictor condition, beyond linearity, used in the PHD containment derivation.
- What additional fact must be checked, beyond range containment, to conclude that an operator recovers a \(d\)-dimensional target exactly?
- In the finite-support PHD counterexample, which conditional variance changes with \(Z_1\), and how does that affect the interpretation of the PHD range?