Deriving SAVE's Conditional-Covariance Identity

From Conditional Covariance to Conditional Moments

SAVE uses changes in the conditional covariance of a standardized predictor vector to identify directions associated with the response. Its definition is compact, but interpreting it correctly requires care: conditional covariance is not simply a conditional second moment. The conditional mean must also be accounted for. This tutorial derives that identity and establishes a range guarantee under explicit assumptions.

Let \(X\) be a \(p\)-dimensional predictor with finite, positive-definite covariance matrix. Standardize it as \(Z=\operatorname{Cov}(X)^{-1/2}(X-E[X])\), so \(E[Z]=0\) and \(\operatorname{Cov}(Z)=I_p\). Assume \(Z\) has finite second moments. Write \(\operatorname{Cov}(Z\mid Y)\) for the random conditional covariance matrix given the response \(Y\).

Definition: The population SAVE operator for standardized predictors is \(\mathcal S_{\mathrm{SAVE}}=E[(I_p-\operatorname{Cov}(Z\mid Y))^2]\). The square is a matrix product. The expectation is entrywise, and is well-defined when the entries of the squared conditional-covariance matrix are integrable.

For each response value, the conditional covariance is the conditional second moment about the conditional mean. Define \(m(Y)=E[Z\mid Y]\) and \(R(Y)=E[ZZ'\mid Y]\). The law of conditional covariance gives \(\operatorname{Cov}(Z\mid Y)=R(Y)-m(Y)m(Y)'\). Substituting this expression yields the identity SAVE uses at the level of conditional moments.

Theorem: If \(E[\|\operatorname{Cov}(Z\mid Y)\|_F^2]<\infty\), then \(\mathcal S_{\mathrm{SAVE}}=E[(I_p-R(Y)+m(Y)m(Y)')^2]\). Here \(\|\cdot\|_F\) is the Frobenius norm.

Proof. By the definition of conditional covariance,

\operatorname{Cov}(Z\mid Y) =E[(Z-m(Y))(Z-m(Y))'\mid Y] =R(Y)-m(Y)m(Y)'.

Therefore \(I_p-\operatorname{Cov}(Z\mid Y)=I_p-R(Y)+m(Y)m(Y)'\) almost surely. Replacing the term inside the square in the definition of SAVE by this equal matrix proves the identity. The stated integrability condition ensures the expectation exists. \(\square\)

Worked Conditional-Moment Calculations

Worked Example: A Linear Response

Let \(Z=(U,V)'\), where \(U\) and \(V\) are independent \(N(0,1)\) variables, and let \(Y=U\). For a realized response \(Y=y\), the conditional mean and second moment are

m(y)=\begin{pmatrix}y\\0\end{pmatrix}, \qquad R(y)=\begin{pmatrix}y^2&0\\0&1\end{pmatrix}.

Thus

\operatorname{Cov}(Z\mid Y=y) =R(y)-m(y)m(y)' =\begin{pmatrix}0&0\\0&1\end{pmatrix}.

The SAVE integrand is constant:

(I_2-\operatorname{Cov}(Z\mid Y=y))^2 =\begin{pmatrix}1&0\\0&0\end{pmatrix}.

Using conditional moments gives the same answer. In fact, \(I_2-R(y)+m(y)m(y)'=\operatorname{diag}(1,0)\), so its square and expectation are both \(\operatorname{diag}(1,0)\). The conditional mean term cancels the \(y^2\) term in \(R(y)\).

Worked Example: A Symmetric Response

Keep independent standard normal \(U,V\), but set \(Y=U^2\). Given \(Y=y>0\), \(U\) is equally likely to be \(\sqrt y\) or \(-\sqrt y\). Consequently \(E[U\mid Y]=0\) and \(E[U^2\mid Y]=Y\). Independence gives \(E[V\mid Y]=0\), \(E[V^2\mid Y]=1\), and zero conditional cross moment. Hence

m(Y)=\begin{pmatrix}0\\0\end{pmatrix}, \qquad R(Y)=\operatorname{Cov}(Z\mid Y) =\begin{pmatrix}Y&0\\0&1\end{pmatrix}.

It follows that

\mathcal S_{\mathrm{SAVE}} =E\left[\begin{pmatrix}(1-Y)^2&0\\0&0\end{pmatrix}\right] =\begin{pmatrix}2&0\\0&0\end{pmatrix}.

For the last equality, a standard normal variable has \(E[U^2]=1\) and \(E[U^4]=3\). Therefore \(E[(1-U^2)^2]=1-2E[U^2]+E[U^4]=1-2+3=2\). This calculation shows how a conditional covariance change can register a symmetric response even though \(E[U\mid Y]=0\).

When SAVE's Range Lies in the Central Subspace

The central subspace \(\mathcal S_{Y\mid Z}\), when it exists, is the smallest subspace \(\mathcal S\) such that \(Y\) and \(Z\) are conditionally independent given the projection of \(Z\) onto \(\mathcal S\). The following range result is a population statement. It requires more than the definition of SAVE.

Suppose the central subspace has dimension \(k\). Let \(B\) be a \(p\times k\) matrix with orthonormal columns spanning it, and let \(B_0\) have orthonormal columns spanning its orthogonal complement. Define \(U=B'Z\) and \(V=B_0'Z\). Then \(Z=BU+B_0V\). The conditions used below are:

Sufficiency: \(Y\) and \(Z\) are conditionally independent given \(U\).
Linearity: \(E[V\mid U]=0\).
Constant conditional variance: \(\operatorname{Cov}(V\mid U)=I_{p-k}\) almost surely.

In standardized coordinates, the last two conditions are the relevant complement-space consequences of the usual linearity and constant-variance conditions. Assume the conditional moments exist and the SAVE expectation is finite.

Theorem: Under sufficiency, linearity, and constant conditional variance as stated above, \(\operatorname{range}(\mathcal S_{\mathrm{SAVE}})\subseteq\mathcal S_{Y\mid Z}\).

Proof. Conditional independence implies that, given \(U\) and \(Y\), the conditional distribution of \(V\) is its distribution given \(U\). Thus \(E[V\mid U,Y]=E[V\mid U]=0\) and \(\operatorname{Cov}(V\mid U,Y)=\operatorname{Cov}(V\mid U)=I_{p-k}\), almost surely. Taking conditional expectations given \(Y\) gives

E[V\mid Y]=0,\qquad \operatorname{Cov}(V\mid Y) =E[\operatorname{Cov}(V\mid U,Y)\mid Y] +\operatorname{Cov}(E[V\mid U,Y]\mid Y) =I_{p-k}.

The conditional cross-covariance of \(U\) and \(V\) given \(Y\) is also zero. Indeed, \(E[V\mid U,Y]=0\), so iterated conditioning gives \(E[UV'\mid Y]=E[U E[V'\mid U,Y]\mid Y]=0\). Since \(E[V\mid Y]=0\), this is the conditional cross-covariance. Therefore, in the orthonormal coordinates \((U,V)\), the conditional covariance of \(Z\) is block diagonal:

\operatorname{Cov}\left( \begin{pmatrix}U\\V\end{pmatrix}\middle|Y\right) = \begin{pmatrix} \operatorname{Cov}(U\mid Y)&0\\ 0&I_{p-k} \end{pmatrix}.

Because \([B\ B_0]\) is orthogonal, the corresponding matrix \(I_p-\operatorname{Cov}(Z\mid Y)\), expressed in these coordinates, is block diagonal with blocks \(I_k-\operatorname{Cov}(U\mid Y)\) and \(0\). Its square still has a zero complement-space block. Taking expectations preserves that zero block. Consequently the SAVE operator has no range component in \(\operatorname{span}(B_0)\), which proves the asserted containment in \(\operatorname{span}(B)=\mathcal S_{Y\mid Z}\). \(\square\)

Why the Conditions Matter

Worked Example: Linearity Without Constant Conditional Variance

Let \(U\) and \(\varepsilon\) be independent standard normal variables, set \(V=\sqrt{1+\tfrac12\sin U}\,\varepsilon\), let \(Z=(U,V)'\), and take \(Y=U\). The square root is real because \(1+\tfrac12\sin U\geq\tfrac12\). Symmetry gives \(E[\sin U]=0\), so \(E[V]=0\) and \(E[V^2]=E[1+\tfrac12\sin U]=1\). Also \(E[UV]=0\), since \(E[V\mid U]=0\). Thus \(Z\) is standardized, and

E[Z\mid U]=\begin{pmatrix}U\\0\end{pmatrix}, \qquad \operatorname{Cov}(V\mid U)=1+\tfrac12\sin U.

The linearity condition holds, but the constant conditional variance condition fails. Here \(Y=U\), so \(\operatorname{span}(1,0)'\) is sufficient; it is the central subspace. In fact, the joint density of \((U,V)\) is positive throughout the plane, so a nonzero one-dimensional linear projection involving \(V\) cannot determine \(U\); the only one-dimensional sufficient direction is therefore \(\operatorname{span}(1,0)'\).

Given \(Y=U\), the conditional covariance of \(Z\) is \(\operatorname{diag}(0,1+\tfrac12\sin U)\). Hence

\mathcal S_{\mathrm{SAVE}} =E\left[ \begin{pmatrix}1&0\\0&-\tfrac12\sin U\end{pmatrix}^{\!2} \right] =\begin{pmatrix} 1&0\\ 0&\tfrac14 E[\sin^2 U] \end{pmatrix} =\begin{pmatrix} 1&0\\ 0&\dfrac{1-e^{-2}}{8} \end{pmatrix}.

To check the final value, \(E[\cos(2U)]=e^{-2}\) for \(U\sim N(0,1)\), and \(\sin^2 U=(1-\cos(2U))/2\). The second diagonal entry is positive, so the SAVE range is not contained in this central subspace. This example isolates the role of constant conditional variance.

A common error is to replace \(\operatorname{Cov}(Z\mid Y)\) by \(E[ZZ'\mid Y]\). That substitution silently drops \(m(Y)m(Y)'\), and is wrong unless \(E[Z\mid Y]=0\) almost surely. Another error is to infer the range guarantee from linearity alone. The heteroscedastic example shows that conditional variation in a complement direction can create a nonzero SAVE component outside the central subspace. The range theorem is therefore a conditional statement with assumptions, not an automatic property of the operator.

Key takeaway: SAVE is the expected square of the difference between the identity and the conditional covariance. In conditional-moment form, that covariance is \(R(Y)-m(Y)m(Y)'\). Under sufficiency, linearity, and constant conditional variance, the SAVE operator has range contained in the central subspace.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

Write conditional covariance in terms of \(R(Y)\) and \(m(Y)\), and explain what term would be omitted by using the conditional second moment alone.
In the symmetric-response example, why is \(E[U\mid Y]=0\) even though \(E[U^2\mid Y]\) is generally positive?
Which conditional moments of the complement coordinate \(V\) are needed in the proof that SAVE has no complement-space range component?
Where does conditional independence of \(Y\) and \(Z\) given \(U\) enter the range proof?
In the heteroscedastic example, which assumption fails, and what calculation shows that the SAVE range extends outside the central subspace?

Deriving SAVE's Conditional-Covariance Identity

What You'll Learn