Computing Sample SIR From Slices

From Observations to Estimated SIR Directions

Sliced inverse regression (SIR) turns a multivariate data set into a small matrix whose leading eigenvectors summarize how the predictor distribution changes across response slices. In practice, the calculation has several linked steps: center and whiten the predictors, partition the observed responses, average the whitened predictors within each slice, and combine those averages with slice weights. An error in any step changes the matrix and can change the estimated directions.

This tutorial works through those computations for a fixed choice of slices. The resulting eigenvectors describe directions in whitened predictor coordinates; a final transformation is needed to express them as coefficients on the original predictors.

Centering, Whitening, and Defining the Slices

Let \((X_i,Y_i)\), \(i=1,\ldots,n\), be observed independent copies of a pair \((X,Y)\), where \(X_i\in\mathbb R^p\) is the predictor vector and \(Y_i\) is a response. Write \(X_i=(X_{i1},\ldots,X_{ip})'\). Define the sample mean and the sample covariance using divisor \(n\):

\bar X=\frac{1}{n}\sum_{i=1}^n X_i,\qquad \widehat\Sigma_X=\frac{1}{n}\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)'.

The divisor convention matters for reproducing the calculations: here the whitened observations have sample covariance exactly \(I_p\), provided \(\widehat\Sigma_X\) is positive definite. If software uses divisor \(n-1\), its covariance and whitening factor differ by a scalar; be consistent throughout the calculation.

Definition: When \(\widehat\Sigma_X\) is positive definite, the centered, whitened predictor for observation \(i\) is \(Z_i=\widehat\Sigma_X^{-1/2}(X_i-\bar X)\). The matrix \(\widehat\Sigma_X^{-1/2}\) is the symmetric inverse square root: if \(\widehat\Sigma_X=Q\Lambda Q'\), then \(\widehat\Sigma_X^{-1/2}=Q\Lambda^{-1/2}Q'\).

Choose cut points in the response and use them to divide the observations into \(H\) nonempty, disjoint slices. The slices form a partition: each observation belongs to exactly one slice. Write \(I_h\) for the set of indices in slice \(h\), \(n_h=|I_h|\) for its count, and \(w_h=n_h/n\) for its weight. Thus \(\sum_{h=1}^H w_h=1\). The whitened slice mean is \(\bar Z_h=n_h^{-1}\sum_{i\in I_h}Z_i\).

The Sample SIR Matrix

The sample SIR matrix is the weighted sum of the outer products of the whitened slice means. It summarizes the variation among those means, giving each slice influence in proportion to its observed share of the sample.

Definition: For a fixed partition into \(H\) nonempty slices, the sample SIR matrix is

\widehat M_{\mathrm{SIR}} =\sum_{h=1}^H w_h\,\bar Z_h\bar Z_h'.

Its eigenvectors are estimated directions in whitened coordinates. If a leading eigenvector is \(\hat\eta\), the corresponding coefficient vector on the original predictors is \(\hat\beta=\widehat\Sigma_X^{-1/2}\hat\eta\): indeed, \(\hat\eta'Z_i=\hat\beta'(X_i-\bar X)\). Eigenvectors are conventionally normalized to length one in whitened coordinates; multiplying a coefficient vector by a nonzero scalar does not change the one-dimensional direction it represents.

Worked Calculations

Worked Example: Two Slices and a Rank-One Direction

Take four predictor observations in two dimensions: \(X_1=(1,0)'\), \(X_2=(0,1)'\), \(X_3=(-1,0)'\), and \(X_4=(0,-1)'\). Let the response assign observations 1 and 2 to slice 1, and observations 3 and 4 to slice 2.

The sample mean is \(\bar X=(0,0)'\). The covariance calculation is

\widehat\Sigma_X =\frac14 \left[ \begin{pmatrix}1&0\\0&0\end{pmatrix} +\begin{pmatrix}0&0\\0&1\end{pmatrix} +\begin{pmatrix}1&0\\0&0\end{pmatrix} +\begin{pmatrix}0&0\\0&1\end{pmatrix} \right] =\frac12 I_2.

Therefore \(\widehat\Sigma_X^{-1/2}=\sqrt2 I_2\), and the whitened observations are \((\sqrt2,0)'\), \((0,\sqrt2)'\), \((-\sqrt2,0)'\), and \((0,-\sqrt2)'\). Each slice has weight \(1/2\). Their means are \(\bar Z_1=(1/\sqrt2,1/\sqrt2)'\) and \(\bar Z_2=(-1/\sqrt2,-1/\sqrt2)'\).

Both slice means have the same outer product, so

\widehat M_{\mathrm{SIR}} =\frac12 \begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix} +\frac12 \begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix} = \begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix}.

Multiplying this matrix by \((1,1)'/\sqrt2\) gives \((1,1)'/\sqrt2\), so that vector has eigenvalue 1. Multiplying it by \((1,-1)'/\sqrt2\) gives \((0,0)'\), so the second eigenvalue is 0. The leading estimated direction is therefore proportional to \((1,1)'\) in these whitened coordinates.

Worked Example: Unequal Slice Weights

Use the same four whitened observations as in the preceding calculation, but assign \(Z_1\) alone to slice 1, \(Z_2\) alone to slice 2, and \(Z_3,Z_4\) to slice 3. The counts are \(1,1,2\), so the weights are \(1/4,1/4,1/2\), not \(1/3\) each.

The first two slice means are \((\sqrt2,0)'\) and \((0,\sqrt2)'\). The third slice mean is \((-\sqrt2/2,-\sqrt2/2)'\). Its outer product is \(\begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix}\).

The weighted contributions are consequently

\widehat M_{\mathrm{SIR}} =\frac14\begin{pmatrix}2&0\\0&0\end{pmatrix} +\frac14\begin{pmatrix}0&0\\0&2\end{pmatrix} +\frac12\begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix} = \begin{pmatrix}3/4&1/4\\1/4&3/4\end{pmatrix}.

For \(v_+=(1,1)'/\sqrt2\), multiplication gives \(\widehat M_{\mathrm{SIR}}v_+=v_+\), so its eigenvalue is 1. For \(v_-=(1,-1)'/\sqrt2\), multiplication gives \(\widehat M_{\mathrm{SIR}}v_-=(1/2)v_-\), so its eigenvalue is \(1/2\). The unequal slice sizes are reflected in the matrix through the weights; replacing them with equal weights would give a different calculation.

Worked Example: Returning a Direction to the Original Scale

Suppose the four raw predictor observations are \(X_i=\mu+D Z_i\), where \(\mu=(10,5)'\), \(D=\operatorname{diag}(2,1)\), and the \(Z_i\) are the four whitened observations from the first example. Use the same two-slice partition as there.

Since the \(Z_i\) have mean zero and covariance \(I_2\), the raw sample mean is \(\bar X=\mu\), and the raw sample covariance is \(\widehat\Sigma_X=DI_2D'=D^2=\operatorname{diag}(4,1)\). Thus its symmetric inverse square root is \(D^{-1}=\operatorname{diag}(1/2,1)\), as required to recover the stated \(Z_i\) from \(X_i-\bar X\). The whitened SIR matrix and its leading unit eigenvector are those already calculated: \(\widehat M_{\mathrm{SIR}}=\begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix}\) and \(\hat\eta=(1,1)'/\sqrt2\).

Transform the direction back:

\hat\beta=\widehat\Sigma_X^{-1/2}\hat\eta =\begin{pmatrix}1/2&0\\0&1\end{pmatrix} \frac{1}{\sqrt2}\begin{pmatrix}1\\1\end{pmatrix} =\frac{1}{\sqrt2}\begin{pmatrix}1/2\\1\end{pmatrix}.

The same direction can be written proportionally as \((1,2)'\). The coefficient vector \((1,1)'\) would instead be a direction in whitened coordinates; using it directly on the original, differently scaled predictors would omit the required back-transformation.

Why the Matrix Has These Properties

Result: For nonempty slices, \(\widehat M_{\mathrm{SIR}}\) is positive semidefinite and has rank at most \(\min(p,H-1)\).

To prove this, first note that the whitened observations are centered: \(\sum_{i=1}^n Z_i=\widehat\Sigma_X^{-1/2}\sum_i(X_i-\bar X)=0\). Partitioning this sum by slices gives \(\sum_h n_h\bar Z_h=0\), or equivalently \(\sum_h w_h\bar Z_h=0\). For any \(a\in\mathbb R^p\),

a'\widehat M_{\mathrm{SIR}}a =\sum_{h=1}^H w_h(a'\bar Z_h)^2\geq 0,

because every \(w_h\) is positive and every squared term is nonnegative. This proves positive semidefiniteness. Moreover, the \(H\) slice means satisfy the nontrivial weighted linear relation \(\sum_h w_h\bar Z_h=0\), so their span has dimension at most \(H-1\); it also has dimension at most \(p\). The range of \(\widehat M_{\mathrm{SIR}}\) is contained in that span, since each term in its defining sum maps any vector to a multiple of \(\bar Z_h\). Hence the rank bound follows. \(\square\)

The rank bound explains why the matrix cannot produce more than \(H-1\) nonzero eigenvalues from \(H\) slices. It is a property of this finite-sample construction, not a guarantee that every nonzero eigenvalue corresponds to a scientifically meaningful direction.

Result: Let \(C_X=\sum_{h=1}^H w_h(\bar X_h-\bar X)(\bar X_h-\bar X)'\), where \(\bar X_h=n_h^{-1}\sum_{i\in I_h}X_i\). Then \(\widehat M_{\mathrm{SIR}}=\widehat\Sigma_X^{-1/2}C_X\widehat\Sigma_X^{-1/2}\).

Indeed, linearity of the whitening transformation gives \(\bar Z_h=\widehat\Sigma_X^{-1/2}(\bar X_h-\bar X)\). Substituting this expression into the definition of the SIR matrix yields

\begin{aligned} \widehat M_{\mathrm{SIR}} &=\sum_{h=1}^H w_h \widehat\Sigma_X^{-1/2}(\bar X_h-\bar X)(\bar X_h-\bar X)' \widehat\Sigma_X^{-1/2}\\ &=\widehat\Sigma_X^{-1/2} \left[\sum_{h=1}^H w_h(\bar X_h-\bar X)(\bar X_h-\bar X)'\right] \widehat\Sigma_X^{-1/2} =\widehat\Sigma_X^{-1/2}C_X\widehat\Sigma_X^{-1/2}. \end{aligned}

Here \(\widehat\Sigma_X^{-1/2}\) is symmetric, which justifies the right-hand factor in the first line. This identity offers a useful check: the slice means must be centered about the overall sample mean before their weighted outer products are combined.

Common Mistakes and Practical Checks

Using equal weights when slice sizes differ. The definition uses \(w_h=n_h/n\). Equal weights are correct only when the slice counts are equal.
Forgetting the outer product. A slice mean is a vector; its contribution is \(\bar Z_h\bar Z_h'\), a \(p\times p\) matrix, not just the mean or its coordinatewise square.
Using the wrong centering. Center predictors by \(\bar X\), and use \(\bar X_h-\bar X\) when expressing slice means on the raw scale. Centering each slice independently would erase the between-slice information SIR uses.
Reading eigenvectors on the wrong scale. Eigenvectors of \(\widehat M_{\mathrm{SIR}}\) are in whitened coordinates. Multiply by \(\widehat\Sigma_X^{-1/2}\) to obtain coefficients on the original predictors.
Making an arithmetic claim without checking it. Verify a proposed eigenpair by multiplying the matrix by the vector and comparing the result with the claimed eigenvalue times that vector. For example, a zero result means eigenvalue zero, not one.

A reliable calculation records the sample covariance convention, slice counts and weights, slice means, the full matrix, and the eigenvector checks. If \(\widehat\Sigma_X\) is singular, the inverse square root used here does not exist; the calculation requires a separate treatment rather than silently applying this formula.

Check Your Understanding

Work through these questions using the definitions and calculations above.

Why does using divisor \(n\) in \(\widehat\Sigma_X\) make the covariance of the whitened observations equal to \(I_p\)?
Three slices contain \(4\), \(3\), and \(5\) observations. What weights belong in the sample SIR matrix?
For a slice mean \(m=(2,-1)'\) with weight \(1/4\), calculate its contribution to \(\widehat M_{\mathrm{SIR}}\).
Explain why four nonempty slices cannot produce more than three nonzero eigenvalues in the sample SIR matrix.
If \(\widehat\Sigma_X=\operatorname{diag}(9,1)\) and a whitened eigenvector is proportional to \((1,1)'\), what is the corresponding direction on the original predictor scale?