The SAVE Operator From Conditional Covariance

Why Look at Conditional Covariance?

Sliced inverse regression (SIR) uses changes in the conditional mean of the predictors given the response. That can be informative when the response shifts the predictor mean, but it can miss a symmetric relationship: for example, a response depending on the square of a centered predictor can leave the conditional mean unchanged. Conditional covariance offers another view. It detects changes in the spread and orientation of predictors across response values.

SAVE, short for sliced average variance estimation, turns those changes into a positive semidefinite matrix. Its eigenvectors with nonzero eigenvalues identify directions along which conditional covariance differs from the standardized marginal covariance. Under additional conditions, those directions lie in the central subspace. Without those conditions, SAVE remains a well-defined summary, but its eigenspaces need not recover all regression directions.

Definition of the Population SAVE Operator

Let $X\in\mathbb{R}^p$ have mean $\mu$ and positive-definite covariance matrix $\Sigma$. Define the standardized predictor $Z=\Sigma^{-1/2}(X-\mu)$, so $E[Z]=0$ and $\operatorname{Var}(Z)=I_p$. Let $Y$ be the response, and assume $E[\|Z\|^4]<\infty$. Write $\operatorname{Cov}(Z\mid Y)$ for the conditional covariance matrix, understood as a function of $Y$.

Definition: The population SAVE matrix is $$ M_{\mathrm{SAVE}}=E\left[\left\{I_p-\operatorname{Cov}(Z\mid Y)\right\}^2\right], $$ where the square means matrix multiplication: $A^2=AA$. The expectation is taken entry by entry over the distribution of $Y$.

The identity matrix appears because the marginal covariance of $Z$ is $I_p$. Thus the matrix inside the square records how conditional covariance differs from marginal covariance. The fourth-moment assumption ensures the entries of this expectation are finite: conditional covariance is a conditional second moment minus a conditional mean product, and its squared entries have finite expectations under this assumption.

This is a population definition, not a rule for choosing a finite-sample number of directions. The matrix is symmetric, and its eigenvectors describe predictor directions. A positive eigenvalue indicates that the corresponding eigenspace is affected by at least some conditional-covariance changes. The zero-eigenvalue space consists of directions that are invisible to this particular operator.

Basic Geometry and a Recovery Result

Theorem: Under the finite-fourth-moment condition above, $M_{\mathrm{SAVE}}$ is positive semidefinite. For every $a\in\mathbb{R}^p$, $$ a^\top M_{\mathrm{SAVE}}a =E\left[\left\|\{I_p-\operatorname{Cov}(Z\mid Y)\}a\right\|^2\right]\geq 0. $$ Consequently, its eigenvalues are nonnegative, and $a^\top M_{\mathrm{SAVE}}a=0$ exactly when $\{I_p-\operatorname{Cov}(Z\mid Y)\}a=0$ almost surely.

Proof. Put $A(Y)=I_p-\operatorname{Cov}(Z\mid Y)$. Each $A(Y)$ is symmetric, so $A(Y)^2=A(Y)^\top A(Y)$. Therefore

a^\top M_{\mathrm{SAVE}}a =E[a^\top A(Y)^\top A(Y)a] =E[\|A(Y)a\|^2]\geq 0.

A nonnegative random variable has expectation zero if and only if it is zero almost surely. Applying this fact to $\|A(Y)a\|^2$ proves the equality statement. Thus $M_{\mathrm{SAVE}}$ is positive semidefinite, and the usual spectral theorem gives nonnegative eigenvalues. $\square$

The quadratic form also gives a directional interpretation: it averages the squared size of the conditional-covariance deviation acting on $a$. A large value means that the covariance deviations have substantial action in that direction. This interpretation concerns covariance changes; it does not say that every direction relevant to the full distribution of $Y$ must have a positive SAVE eigenvalue.

To state a useful recovery result, let $S$ be the central subspace for $Y$ given $Z$, and let $P$ be the orthogonal projection onto $S$. By definition, $Y$ is conditionally independent of $Z$ given $PZ$, and $S$ is the smallest subspace with this property when the central subspace exists. The following conditions are sufficient for SAVE to target $S$:

Linearity condition: with $R=(I_p-P)Z$, $E[R\mid PZ]=0$.
Constant variance condition: $\operatorname{Cov}(R\mid PZ)=I_p-P$ almost surely.

These are conditions on the predictor distribution relative to the central subspace, not automatic consequences of standardization. The second condition says that the component orthogonal to $S$ has constant conditional covariance given the component in $S$.

Theorem: If $S$ exists and the linearity and constant variance conditions hold, then $$ \operatorname{range}(M_{\mathrm{SAVE}})\subseteq S. $$ If, in addition, $\operatorname{rank}(M_{\mathrm{SAVE}})=\dim(S)$, then $\operatorname{range}(M_{\mathrm{SAVE}})=S$.

Proof. Since $Y$ is conditionally independent of $Z$ given $PZ$, it is also conditionally independent of $R$ given $PZ$. The linearity condition therefore gives

E[R\mid PZ,Y]=E[R\mid PZ]=0.

Taking conditional expectations given $Y$ yields $E[R\mid Y]=0$. The constant variance condition and the zero conditional mean give

E[RR^\top\mid PZ]=I_p-P, \qquad E[RR^\top\mid Y]=I_p-P.

For the second equality, condition first on $PZ$ and $Y$; conditional independence makes the conditional second moment equal to $I_p-P$, which is constant. The cross-covariance between $R$ and $PZ$ given $Y$ is zero: conditional independence and $E[R\mid PZ,Y]=0$ imply $E[R(PZ)^\top\mid Y]=0$, while $E[R\mid Y]=0$. Thus $\operatorname{Cov}(Z\mid Y)$ has no cross-blocks between $S$ and its orthogonal complement, and its block on the complement is $I_p-P$. It follows that

\{I_p-\operatorname{Cov}(Z\mid Y)\}(I_p-P)=0

almost surely. Hence $M_{\mathrm{SAVE}}(I_p-P)=0$, so its range is contained in $S$, because $M_{\mathrm{SAVE}}$ is symmetric. If its rank equals $\dim(S)$, that containment between subspaces of equal dimension is equality. $\square$

This result separates two claims that should not be conflated. The assumptions ensure that SAVE does not point outside the central subspace. They do not by themselves ensure that every central direction produces a nonzero eigenvalue. The additional rank condition is what makes the range equal to the entire central subspace. When equality holds, the eigenvectors associated with the positive eigenvalues span the regression subspace; individual eigenvectors need not be uniquely interpretable if eigenvalues are repeated.

Worked Examples

Worked Example: A Symmetric Effect Missed by Conditional Means

Let $Z=(Z_1,Z_2)^\top\sim N(0,I_2)$, and set $Y=Z_1^2$. Find the population SAVE matrix.

For $Y=y>0$, the conditional distribution of $Z_1$ places equal probability on $\sqrt y$ and $-\sqrt y$. Its conditional mean is zero and its conditional variance is $y$. The variable $Z_2$ is independent of $Y$, so its conditional variance is one, and the conditional cross-covariance is zero. Thus

\operatorname{Cov}(Z\mid Y)= \begin{pmatrix} Y&0\\ 0&1 \end{pmatrix}, \qquad I_2-\operatorname{Cov}(Z\mid Y)= \begin{pmatrix} 1-Y&0\\ 0&0 \end{pmatrix}.

Squaring and taking expectations gives

M_{\mathrm{SAVE}}= \begin{pmatrix} E[(1-Y)^2]&0\\ 0&0 \end{pmatrix} = \begin{pmatrix} E[(1-Z_1^2)^2]&0\\ 0&0 \end{pmatrix} = \begin{pmatrix} 1-2E[Z_1^2]+E[Z_1^4]&0\\ 0&0 \end{pmatrix} = \begin{pmatrix} 2&0\\ 0&0 \end{pmatrix}.

Here $E[Z_1^2]=1$ and $E[Z_1^4]=3$. The positive-eigenvalue eigenspace is the span of $e_1=(1,0)^\top$, the direction used by the regression. In contrast, $E[Z\mid Y]=0$, so a method based only on inverse conditional means sees no change.

Worked Example: A Threshold Response

Let $Z\sim N(0,I_2)$ and $Y=1\{Z_1>0\}$. Compute SAVE by considering the two possible response values.

Conditional on $Y=1$, $Z_1$ has a standard normal distribution truncated to positive values. Its mean is $\sqrt{2/\pi}$, and its second moment is $1$, so its variance is $1-2/\pi$. Conditional on $Y=0$, symmetry gives mean $-\sqrt{2/\pi}$ and the same variance. In both groups $Z_2$ remains independent with variance $1$. Consequently,

\operatorname{Cov}(Z\mid Y)= \begin{pmatrix} 1-2/\pi&0\\ 0&1 \end{pmatrix} \quad\text{almost surely}.

The deviation from $I_2$ is $\operatorname{diag}(2/\pi,0)$ in either group. Its square is $\operatorname{diag}(4/\pi^2,0)$, so averaging over $Y$ leaves

M_{\mathrm{SAVE}}= \begin{pmatrix} 4/\pi^2&0\\ 0&0 \end{pmatrix}.

The only positive-eigenvalue direction is again $e_1$. Unlike the preceding example, this response changes the conditional mean too; the calculation here shows specifically how a change in conditional variance contributes to SAVE.

Worked Example: A Noisy Linear Regression

Let $Z_1,Z_2,\varepsilon$ be mutually independent, with $Z_1,Z_2\sim N(0,1)$, $\varepsilon\sim N(0,3)$, and $Y=Z_1+\varepsilon$. Find SAVE.

The pair $(Z_1,Y)$ is jointly normal, with $\operatorname{Var}(Y)=1+3=4$ and $\operatorname{Cov}(Z_1,Y)=1$. The conditional variance formula for a jointly normal pair gives

\operatorname{Var}(Z_1\mid Y) =1-\frac{\operatorname{Cov}(Z_1,Y)^2}{\operatorname{Var}(Y)} =1-\frac14=\frac34.

The independent variable $Z_2$ retains conditional variance $1$ and has zero conditional covariance with $Z_1$. Therefore

\operatorname{Cov}(Z\mid Y)= \begin{pmatrix} 3/4&0\\ 0&1 \end{pmatrix}, \qquad M_{\mathrm{SAVE}}= \begin{pmatrix} (1/4)^2&0\\ 0&0 \end{pmatrix} = \begin{pmatrix} 1/16&0\\ 0&0 \end{pmatrix}.

The positive-eigenvalue eigenspace is the span of $e_1$, the regression direction. The eigenvalue is smaller than in the symmetric example: here the conditional variance change is only $1/4$, and SAVE squares that deviation.

Reading the Eigenspaces Carefully

If $M_{\mathrm{SAVE}}v=\lambda v$ with $\lambda>0$, then $v$ belongs to the range of SAVE and represents a direction with nonzero average conditional-covariance deviation. A collection of eigenvectors for positive eigenvalues spans that range. Under the theorem's assumptions and rank condition, this is the central subspace. Without the assumptions, it is safer to describe the result as a covariance-based target rather than as guaranteed recovery of the full regression subspace.

The calculation is made in standardized coordinates. If $v$ is a direction in $Z$-coordinates, the corresponding linear predictor in the original $X$-coordinates is proportional to $v^\top\Sigma^{-1/2}(X-\mu)$, with coefficient vector $\Sigma^{-1/2}v$. This change of coordinates matters when interpreting estimated directions on the original measurement scales.

Common Pitfall

A tempting but incorrect argument is: “SAVE squares the covariance deviation, so any response-dependent direction must have a positive eigenvalue.” Squaring prevents positive and negative scalar deviations from canceling in the same direction, but it does not guarantee that every feature of the conditional distribution appears in its covariance. For example, if $\operatorname{Cov}(Z\mid Y)=I_p$ almost surely, then $M_{\mathrm{SAVE}}=0$, even if $Y$ depends on higher-order features of $Z$. The correction is to interpret SAVE specifically as a conditional-covariance method, check the assumptions behind any central-subspace claim, and avoid treating a zero eigenvalue as proof of conditional independence.

Check Your Understanding

Use the definition and results above to answer these questions.

Why is the identity matrix in the SAVE definition appropriate after standardizing $X$?
For a symmetric matrix $A$, explain why $a^\top A^2a$ cannot be negative.
In the model $Y=Z_1^2$ with $Z\sim N(0,I_2)$, why does the conditional mean fail to reveal the direction that SAVE identifies?
What do the linearity and constant variance conditions each control in the range-containment theorem?
Why does $\operatorname{range}(M_{\mathrm{SAVE}})\subseteq S$ not, by itself, imply that SAVE recovers all of $S$?