Diagnosing Linearity and Coverage in Inverse Regression

Why a Weak Direction Needs Diagnosis

A small estimated inverse-regression direction does not, by itself, identify what went wrong. It may reflect a violation of the predictor linearity condition used to justify an operator, an inverse-regression operator that does not capture the relevant feature of the response, or a response partition that has averaged away useful information. Sampling variability and unstable slice boundaries can add further noise. These explanations call for different checks: changing slice boundaries will not generally fix a structural blind spot, and a different operator does not establish that the predictor condition holds.

The goal is therefore not to select an operator by a single diagnostic. It is to compare the assumptions and signals behind the estimated directions. The definitions and results below make that comparison explicit for standardized predictors and inverse-regression operators.

Definition: Let $X$ be a $p$-dimensional predictor vector with mean $\mu$ and positive-definite covariance matrix $\Sigma$, and define the standardized predictor $Z=\Sigma^{-1/2}(X-\mu)$, so $E[Z]=0$ and $\operatorname{Cov}(Z)=I$. For a $p\times d$ matrix $B$ with orthonormal columns, the linearity condition along $B$ is $$ E[Z\mid B^\mathsf{T}Z]=BB^\mathsf{T}Z \quad \text{almost surely}. $$ Equivalently, the conditional mean of the component of $Z$ orthogonal to the columns of $B$ is zero given $B^\mathsf{T}Z$. The coverage condition for an inverse-regression operator means that the range of its population operator contains the central subspace, the smallest subspace sufficient for the conditional distribution of $Y$ given $X$.

The linearity condition holds, for example, when $Z$ has a spherically symmetric distribution and the relevant conditional expectations exist. Elliptical predictor distributions also give the corresponding linear conditional-mean property after standardization. A nonelliptical distribution is a reason to check the condition more carefully, not proof that it fails. Conversely, marginally normal-looking predictors do not establish a multivariate conditional-mean property.

Two Population Operators and What They Can Miss

Write $m(Y)=E[Z\mid Y]$ and $C(Y)=\operatorname{Cov}(Z\mid Y)$. Under finite second moments, the population SIR operator is $$ M_{\mathrm{SIR}}=E[m(Y)m(Y)^\mathsf{T}]. $$ With response slices $A_1,\ldots,A_H$, its sliced version replaces $m(Y)$ by $m_h=E[Z\mid Y\in A_h]$, giving $M_{\mathcal P}=\sum_{h=1}^H p_hm_hm_h^\mathsf{T}$, where $p_h=P(Y\in A_h)>0$. Thus its range is generated by the inverse conditional means represented by the chosen slices.

A standard covariance-based comparator is SAVE, whose population operator, for standardized $Z$, is $$ M_{\mathrm{SAVE}}=E[(I-C(Y))^2]. $$ Here $C(Y)$ is symmetric, so the squared matrix is $(I-C(Y))(I-C(Y))$. SAVE uses changes in conditional covariance rather than only changes in conditional means. These operators are summaries of different inverse-regression features; comparing them is informative precisely because their signals need not agree.

Theorem: Suppose $E[\|Z\|^2]<\infty$, the columns of $B$ are orthonormal, the conditional distribution of $Y$ given $Z$ depends on $Z$ only through $B^\mathsf{T}Z$, and the linearity condition along $B$ holds. Then $$ \operatorname{Range}(M_{\mathrm{SIR}})\subseteq \operatorname{span}(B). $$ This is a containment result, not a guarantee of coverage: SIR can still fail to recover every direction in the central subspace.

Proof. Conditional dependence only through $B^\mathsf{T}Z$ implies that $Z$ and $Y$ are conditionally independent given $B^\mathsf{T}Z$. By iterated expectation and the linearity condition, $$ E[Z\mid Y] =E\{E[Z\mid B^\mathsf{T}Z,Y]\mid Y\} =E\{E[Z\mid B^\mathsf{T}Z]\mid Y\} =BB^\mathsf{T}E[Z\mid Y]. $$ The last equality uses that $B$ is fixed and conditional expectation is linear. Thus $m(Y)$ lies in $\operatorname{span}(B)$ almost surely. For any vector $v$ orthogonal to that span, $v^\mathsf{T}m(Y)=0$, so $M_{\mathrm{SIR}}v=E[m(Y)\{m(Y)^\mathsf{T}v\}]=0$. Since $M_{\mathrm{SIR}}$ is symmetric, its range is orthogonal to its null space and is contained in $\operatorname{span}(B)$. $\square$

Check the Predictor Condition Along Candidate Scores

In practice, the full conditional-mean condition is not directly verifiable from a finite sample, particularly when $p$ is large. A targeted check asks whether components orthogonal to a candidate subspace have a systematic conditional mean given its scores. Choose a matrix $Q$ whose columns form an orthonormal basis for the orthogonal complement of $B$. Under linearity along $B$, the diagnostic relation is $$ E[Q^\mathsf{T}Z\mid B^\mathsf{T}Z]=0. $$ One can examine flexible fitted curves or smooth regressions of each component of $Q^\mathsf{T}Z$ on the candidate scores. Curvature or structured residual means are evidence against the condition along those scores. They are not a universal test of linearity in every possible direction, and a lack of visible curvature is not proof that the condition holds.

Predictor-distribution checks can help decide how seriously to investigate this issue. Useful checks include multivariate plots when dimension permits, projection plots along candidate scores, and comparisons of fitted conditional means with linear fits. Marginal skewness or a normality check on each predictor alone cannot settle the multivariate condition. Moreover, because $B$ is often estimated from the same data, it is prudent to assess the pattern on held-out data or to account for estimation when interpreting a diagnostic.

Worked Example: A Nonlinear Conditional Mean in the Predictors

Let $Z_1$ be uniform on $[-\sqrt{3},\sqrt{3}]$, and define $$ Z_2=\frac{Z_1^2-1}{\sqrt{4/5}}. $$ Take $Z=(Z_1,Z_2)^\mathsf{T}$ and response $Y=Z_1$. Check standardization and the linearity condition for the candidate direction $B=(1,0)^\mathsf{T}$.

For this uniform distribution, $E[Z_1]=0$, $E[Z_1^2]=1$, and $E[Z_1^4]=9/5$. Therefore $E[Z_2]=0$ and $$ \operatorname{Var}(Z_2)=\frac{E[(Z_1^2-1)^2]}{4/5} =\frac{9/5-2+1}{4/5}=1. $$ Symmetry gives $E[Z_1Z_2]=0$, so $\operatorname{Cov}(Z)=I$.

Since $B^\mathsf{T}Z=Z_1$, the required conditional mean under linearity would be $BB^\mathsf{T}Z=(Z_1,0)^\mathsf{T}$. But $Z_2$ is determined by $Z_1$, and hence $$ E[Z\mid Z_1]=\left(Z_1,\frac{Z_1^2-1}{\sqrt{4/5}}\right)^\mathsf{T}. $$ Its second component is not zero almost surely, so the condition fails. This gives a concrete target for a predictor diagnostic: along the score $Z_1$, the orthogonal component has a quadratic conditional mean. The example also shows why standardizing the covariance does not itself guarantee linearity.

Vary Slices, but Interpret the Change Carefully

Slicing approximates the response-conditioned mean by averages over response intervals. A coarse partition can conceal variation that a finer partition separates. There is a useful population fact when one partition refines another.

Theorem: Suppose a finite coarse partition of the response space is refined by a finite partition whose cells all have positive probability, and $E[\|Z\|^2]<\infty$. Then the span of the coarse sliced SIR means is contained in the span of the refined sliced means. Consequently, the rank of the coarse population sliced SIR operator cannot exceed the rank of the refined operator.

Proof. Let $A_h$ be a coarse cell, formed as the union of fine cells $A_j$ for $j\in J_h$. Write $p_j=P(Y\in A_j)$ and $m_j=E[Z\mid Y\in A_j]$. The law of total expectation gives $$ E[Z\mid Y\in A_h] =\sum_{j\in J_h}\frac{p_j}{\sum_{k\in J_h}p_k}m_j. $$ Thus each coarse mean is a linear combination of fine means, proving the span inclusion. For any collection of means $m_j$ with positive weights $p_j$, the matrix $\sum_jp_jm_jm_j^\mathsf{T}$ has range equal to their span: its null space consists exactly of vectors $v$ for which $\sum_jp_j(v^\mathsf{T}m_j)^2=0$, which, since each $p_j>0$, is equivalent to $v^\mathsf{T}m_j=0$ for every $j$. The rank claim follows. $\square$

This result concerns population spaces for nested partitions. It does not say that adding sample slices will improve estimated directions: smaller slices have fewer observations, and their estimated means can be noisy. Nor does it say that any two unrelated boundary choices must have nested spaces. Compare plausible, adequately populated partitions and look for directions that persist, rather than treating the partition with the largest estimated rank as automatically preferable.

Worked Example: A Slice Choice Can Create or Remove a SIR Signal

Let $Z_1\sim N(0,1)$, $Z_2\sim N(0,1)$ independently, and set $Y=Z_1$. If the entire response range is treated as one slice, its inverse mean is $E[Z]=0$, so the sliced SIR operator is the zero matrix.

Instead, use two slices $Y\leq 0$ and $Y>0$, each with probability $1/2$. By symmetry, $E[Z_1\mid Z_1\leq0]=-\sqrt{2/\pi}$ and $E[Z_1\mid Z_1>0]=\sqrt{2/\pi}$. Independence gives a zero conditional mean for $Z_2$ in both slices. Therefore the two sliced means are $(-\sqrt{2/\pi},0)^\mathsf{T}$ and $(\sqrt{2/\pi},0)^\mathsf{T}$, and $$ M_{\mathcal P} =\frac12\begin{pmatrix}2/\pi&0\\0&0\end{pmatrix} +\frac12\begin{pmatrix}2/\pi&0\\0&0\end{pmatrix} =\begin{pmatrix}2/\pi&0\\0&0\end{pmatrix}. $$ The direction $e_1=(1,0)^\mathsf{T}$ is now visible. The one-slice result was uninformative because it averaged over the entire response range, not because the population response lacked an inverse-mean signal.

Compare Operators to Investigate Coverage

A population operator covers a target subspace when its range contains that subspace. Even with a suitable predictor condition, SIR coverage can fail if inverse conditional means do not vary in every relevant direction. SAVE can provide a complementary signal because conditional covariance may vary even when conditional means do not. The next example isolates that distinction. It is a diagnostic comparison, not a claim that one operator is universally superior.

Worked Example: Symmetry Hides a Direction From SIR

Let $Z_1,Z_2$ be independent standard normal variables and let $Y=Z_1^2$. The conditional distribution of $Y$ depends only on $Z_1$, so the central subspace is $\operatorname{span}(e_1)$. For any response value $y>0$, $Z_1$ given $Y=y$ takes the two values $\sqrt y$ and $-\sqrt y$ with equal probability. Thus $E[Z_1\mid Y]=0$; independence also gives $E[Z_2\mid Y]=0$. Consequently $M_{\mathrm{SIR}}=0$, and no choice of slices based only on $Y$ can create a nonzero sliced inverse mean.

For SAVE, $C(Y)=\operatorname{Cov}(Z\mid Y)=\operatorname{diag}(Y,1)$: the conditional variance of $Z_1$ is $Y$, and that of $Z_2$ remains $1$. Hence $$ M_{\mathrm{SAVE}} =E\left[\begin{pmatrix}(1-Y)^2&0\\0&0\end{pmatrix}\right] =\begin{pmatrix}2&0\\0&0\end{pmatrix}. $$ Indeed, $Y=Z_1^2$ has $E[Y]=1$ and $E[Y^2]=3$, so $E[(1-Y)^2]=1-2+3=2$. SAVE therefore has the direction $e_1$, whereas SIR has no inverse-mean direction to estimate. The contrast points to an SIR coverage limitation in this symmetric model; altering response slice boundaries cannot fix it.

A Practical Diagnostic Sequence

Check the predictor geometry.
Inspect multivariate and projection patterns, then examine whether orthogonal predictor components have nonlinear conditional means given candidate scores. Treat these as evidence about linearity, not a definitive pass-or-fail verdict.

Check sensitivity to slicing.
Compare reasonable boundary choices and slice counts, including partitions with adequate observations per slice. Look for stable directions, and distinguish boundary sensitivity from changes caused by sampling noise.

Compare inverse-regression features.
Contrast mean-based SIR directions with covariance-based SAVE directions, considering the sizes and stability of their signals. Disagreement can reflect different features or coverage limitations; it does not by itself identify which estimate is correct.

Report the limitation explicitly.
State which predictor checks, partitions, and operators were compared, and whether the estimated direction persisted. Separate evidence about assumptions from uncertainty due to finite-sample estimation.

Common Mistake: Treating a Diagnostic as a Proof

Wrong reasoning: “The predictors look roughly normal, so the linearity condition holds; SIR has a weak direction, so there is no relevant reduction direction.” Both conclusions go too far. A visual or marginal distribution check cannot establish the needed conditional-mean property, and weak SIR can occur because inverse means cancel, as in the symmetric squared-response example. The correction is to use predictor checks as targeted evidence, vary reasonable slices to assess approximation sensitivity, and compare operators whose population signals depend on different inverse-regression features. No single comparison proves coverage, but together these checks help locate plausible explanations for a weak direction.

Key takeaway: A weak estimated direction is a clue to investigate, not a diagnosis. Linearity concerns the predictor distribution conditional on candidate scores; slicing controls how inverse means are summarized; coverage concerns whether an operator’s population range contains the target subspace. Keep these questions separate when interpreting results.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

For standardized $Z$, what conditional-mean relation expresses linearity along an orthonormal matrix $B$?
Why does the nested-partition result not guarantee that adding sample slices improves estimated directions?
In the model $Y=Z_1^2$ with independent standard normal predictors, why does changing response boundaries fail to produce a SIR inverse-mean signal?
What feature of the conditional distribution allows SAVE to detect a direction in that model?
Give one reason why apparent agreement between SIR and SAVE should not be treated as proof of coverage.