Choosing Among SIR, SAVE, and PHD

Three Methods, Three Views of the Response–Predictor Relationship

SIR, SAVE, and principal Hessian directions (PHD) all use low-order moments to find directions in predictor space that carry information about a response. They do not, however, inspect the same feature of the relationship. SIR looks for changes in the conditional mean of the predictors given the response. SAVE also uses the conditional covariance. PHD weights predictor second moments by the centered response. As a result, a regression direction can be clear to one method and invisible to another.

To compare the population methods, let \(X\in\mathbb{R}^p\) have mean \(\mu\) and positive-definite covariance matrix \(\Sigma\), and define the standardized predictor \(Z=\Sigma^{-1/2}(X-\mu)\). Thus \(E[Z]=0\) and \(\operatorname{Cov}(Z)=I_p\). Write \(C(Y)=\operatorname{Cov}(Z\mid Y)\). The formulas below concern population operators; sample estimates introduce additional variation that is not the issue here.

Definition: The population SIR operator is \(M_{\mathrm{SIR}}=\operatorname{Cov}(E[Z\mid Y])\). The population SAVE operator is \(M_{\mathrm{SAVE}}=E[(I_p-C(Y))^2]\). The standardized PHD matrix is \(H_{\mathrm{PHD}}=E[(Y-E[Y])(ZZ^\mathsf{T}-I_p)]\), when these expectations exist. The eigenvectors associated with nonzero eigenvalues provide candidate directions in standardized predictor space.

The subtraction of \(I_p\) in the PHD definition does not change the matrix if the definition is written as \(E[(Y-E[Y])ZZ^\mathsf{T}]\), because \(E[Y-E[Y]]=0\). It makes the centering of the second moment explicit. In applications, directions in standardized coordinates must be translated back to the original predictor scale.

What Information Does Each Operator Use?

For SIR, \(E[Z\mid Y]\) records how the average standardized predictor changes across response values. If the conditional mean is constant, SIR has no signal, even if the conditional distribution changes in its spread or shape. SAVE compares the conditional covariance \(C(Y)\) with the unconditional covariance \(I_p\); squaring the symmetric difference makes the operator positive semidefinite and retains covariance changes regardless of their sign. PHD instead asks whether centered response values co-vary with entries of \(ZZ^\mathsf{T}\), the predictor’s second-moment matrix.

A nonzero operator is not automatically a guarantee of complete recovery. The usual containment claims require conditions on the predictor distribution and regression structure. For example, SIR’s standard containment argument uses the linearity condition, which places \(E[Z\mid B^\mathsf{T}Z]\) in the span of \(B\). SAVE’s analogous argument also needs a suitable conditional-covariance condition. PHD’s containment arguments commonly use an elliptical predictor distribution, such as a multivariate normal distribution, together with a regression mean depending on \(B^\mathsf{T}Z\). These conditions concern whether the operator points within the central subspace; whether its nonzero eigenvectors span all of that subspace is a further question.

Result: \(M_{\mathrm{SIR}}=0\) if and only if \(E[Z\mid Y]=0\) almost surely. Also, \(M_{\mathrm{SAVE}}=0\) if and only if \(C(Y)=I_p\) almost surely. Neither statement says that the corresponding method must be nonzero whenever the response depends on the predictors.

Proof. Let \(m(Y)=E[Z\mid Y]\). Since \(E[m(Y)]=E[Z]=0\), \(M_{\mathrm{SIR}}=E[m(Y)m(Y)^\mathsf{T}]\). Its trace is \(E[\lVert m(Y)\rVert^2]\), which is zero exactly when \(m(Y)=0\) almost surely. For SAVE, put \(A(Y)=I_p-C(Y)\), a symmetric matrix. The trace of \(M_{\mathrm{SAVE}}=E[A(Y)^2]\) is \(E[\operatorname{tr}(A(Y)^2)]\). For a symmetric matrix, \(\operatorname{tr}(A^2)\) is the sum of the squares of its entries, so it is nonnegative and equals zero exactly when \(A=0\). These trace characterizations prove both equivalences. \(\square\)

Worked Examples: A Quadratic Effect SIR Misses

Worked Example: A Quadratic Effect SIR Misses

Let \(Z=(Z_1,Z_2)^\mathsf{T}\) have independent standard normal coordinates, and let \(Y=Z_1^2\). The regression depends on the direction \(e_1=(1,0)^\mathsf{T}\).

SIR. Given \(Y=y>0\), \(Z_1\) is equally likely to be \(\sqrt y\) or \(-\sqrt y\), so \(E[Z_1\mid Y]=0\). Independence gives \(E[Z_2\mid Y]=0\) as well. Therefore \(E[Z\mid Y]=0\) and \(M_{\mathrm{SIR}}=0\): SIR does not reveal the direction.

SAVE. Given \(Y\), the first coordinate is fixed up to its sign, while \(Z_2\) remains independent standard normal. Thus \(C(Y)=\operatorname{diag}(Y,1)\). It follows that \(M_{\mathrm{SAVE}}=E[\operatorname{diag}((1-Y)^2,0)]=\operatorname{diag}(2,0)\), because \(Y=Z_1^2\), \(E[Z_1^2]=1\), and \(E[Z_1^4]=3\), so \(E[(1-Z_1^2)^2]=1-2+3=2\). SAVE identifies \(e_1\).

PHD. The off-diagonal entry is zero by independence and centering. Its first diagonal entry is \(E[(Z_1^2-1)(Z_1^2-1)]=E[(Z_1^2-1)^2]=2\). Its second diagonal entry is \(E[(Z_1^2-1)(Z_2^2-1)]=0\), again by independence. Hence \(H_{\mathrm{PHD}}=\operatorname{diag}(2,0)\), which also identifies \(e_1\).

This example isolates a key distinction: a symmetric effect can change conditional spread while leaving the inverse conditional mean unchanged.

Worked Examples: A Cubic Effect and a Linear Effect

Worked Example: A Cubic Effect PHD Misses

Keep \(Z_1,Z_2\) independent standard normal, but set \(Y=Z_1^3\). The map from \(Z_1\) to \(Y\) is one-to-one, so the regression direction is \(e_1\).

SIR. Since \(Z_1\) is determined by \(Y\), \(E[Z_1\mid Y]=Z_1\); and \(E[Z_2\mid Y]=0\). Therefore \(M_{\mathrm{SIR}}=\operatorname{diag}(1,0)\).

SAVE. Conditional on \(Y\), \(Z_1\) is fixed and \(Z_2\) remains standard normal, so \(C(Y)=\operatorname{diag}(0,1)\). Thus \(M_{\mathrm{SAVE}}=\operatorname{diag}(1,0)\).

PHD. The first diagonal entry is \(E[Z_1^3(Z_1^2-1)]=E[Z_1^5]-E[Z_1^3]=0\), because odd moments of a centered normal variable vanish. The second diagonal entry is \(E[Z_1^3(Z_2^2-1)]=0\) by independence, and the off-diagonal entry is \(E[Z_1^4Z_2]=0\). Consequently \(H_{\mathrm{PHD}}=0\), despite the strong dependence of \(Y\) on \(Z_1\).

The failure is a moment cancellation: this PHD matrix uses response-weighted second moments, which do not register this odd cubic regression function under the symmetric normal distribution.

Worked Example: A Noisy Linear Effect

Let \(Z_1,Z_2\) be independent standard normal variables, let \(\varepsilon\sim N(0,\tau^2)\) be independent of them, with \(\tau^2>0\), and set \(Y=Z_1+\varepsilon\). This is a one-direction regression along \(e_1\).

The pair \((Z_1,Y)\) is jointly normal. Its conditional mean and variance are \(E[Z_1\mid Y]=Y/(1+\tau^2)\) and \(\operatorname{Var}(Z_1\mid Y)=\tau^2/(1+\tau^2)\). The second predictor coordinate remains independent of \(Y\). Since \(\operatorname{Var}(Y)=1+\tau^2\), \(M_{\mathrm{SIR}}=\operatorname{diag}(1/(1+\tau^2),0)\).

The conditional covariance is constant: \(C(Y)=\operatorname{diag}(\tau^2/(1+\tau^2),1)\). Therefore \(M_{\mathrm{SAVE}}=\operatorname{diag}(1/(1+\tau^2)^2,0)\). Both methods identify \(e_1\).

For PHD, every entry is an expectation of a centered response times a centered quadratic expression in \(Z\). The joint vector \((Y,Z_1,Z_2)\) is centered Gaussian, and all its third-order centered moments are zero. For example, \(E[Y(Z_1^2-1)]=0\) and \(E[YZ_1Z_2]=0\); the other entries vanish for the same reason. Thus \(H_{\mathrm{PHD}}=0\). A linear mean effect can therefore be visible to SIR and SAVE but invisible to this PHD moment.

As a numerical check, if \(\tau^2=1\), the nonzero SIR eigenvalue is \(1/2\), the nonzero SAVE eigenvalue is \(1/4\), and the PHD matrix is zero. The difference in eigenvalue size does not change which direction the first two operators identify.

How to Choose Among the Methods

These examples do not establish a universal ranking. They show that the methods are complementary moment probes. SIR is a natural first choice when changes in inverse conditional means are plausible. SAVE can add information when the conditional spread changes but the conditional mean does not. PHD can detect response-weighted quadratic structure, but its moment can cancel for other regression shapes, including the cubic example.

In practice, the choice should follow the likely regression structure and the assumptions supporting the population interpretation. If symmetry makes inverse means uninformative, relying on SIR alone is risky. If conditional covariance is expected to vary, SAVE is a relevant complement. PHD supplies a different second-moment view rather than a general replacement for either method. Examining multiple operators can reveal sensitivity to the form of the regression, but their directions should still be interpreted with the relevant predictor and regression conditions in mind.

Common pitfall: “The response depends on a predictor direction, so every inverse-regression method must find it.” This is false: dependence alone does not ensure that a particular moment operator is nonzero. In the quadratic example SIR is zero, and in the cubic and noisy linear examples PHD is zero. Check the actual conditional mean, conditional covariance, or response-weighted second moment used by the method before interpreting a zero operator as evidence of no regression effect.

Check Your Understanding

For each question, identify which moment the method uses and explain the implication for direction recovery.

What conditional feature of the predictors is summarized by the SIR operator?
Why can SAVE be nonzero when the SIR operator is zero?
In the quadratic example, which conditional moment changes with the response, and which one remains zero?
Why does the PHD matrix vanish in the noisy linear Gaussian example?
What conditions are relevant when claiming that an operator’s directions lie within the central subspace?