Sufficient Dimension Reduction Through Conditional Independence

Why Reduce Predictors?

In multivariate regression, a predictor vector $X\in\mathbb{R}^p$ may have many components, while the response $Y$ depends on only a smaller collection of combinations of them. A useful reduction keeps the information in $X$ that is relevant to the conditional distribution of $Y$, while replacing the original predictors with fewer coordinates.

The key is to define “relevant” through conditional distributions, not merely through a fitted mean or a correlation. A reduction is sufficient when, once its coordinates are known, the remaining information in $X$ tells us nothing further about $Y$.

Definition: Let $X\in\mathbb{R}^p$ and let $Y$ be a random response taking values in a standard Borel space. For a fixed matrix $B\in\mathbb{R}^{p\times d}$, define the reduced coordinates $Z=B^T X$. The reduction $B^T X$ is sufficient for the regression of $Y$ on $X$ if $$ Y\mathrel{\perp\!\!\!\perp} X\mid B^T X. $$ That is, conditional on $B^T X$, knowing $X$ gives no additional information about $Y$.

Because $Z=B^TX$ is a function of $X$, this conditional independence has a direct distributional interpretation. Under the usual existence of regular conditional distributions, sufficiency means that the conditional law of $Y$ given $X=x$ depends on $x$ only through $B^Tx$, up to sets of predictor values having probability zero.

Equivalent formulation: Write $K_X(\cdot\mid X)$ for a regular conditional distribution of $Y$ given $X$, and $K_Z(\cdot\mid Z)$ for one given $Z=B^TX$. Then $B^TX$ is sufficient exactly when $$ K_X(A\mid X)=K_Z(A\mid B^TX)\quad\text{almost surely} $$ for every measurable response set $A$. Thus sufficiency concerns the full conditional distribution, not just its mean.

Worked Examples of Sufficient Reductions

Worked Example: A Linear Gaussian Response

Suppose $X=(X_1,X_2,X_3)^T$ has any distribution, and $$ Y=2X_1-X_2+\varepsilon, $$ where $\varepsilon\sim N(0,1)$ is independent of $X$. Identify a one-dimensional sufficient reduction.

Take $B=(2,-1,0)^T$, so $Z=B^TX=2X_1-X_2$. Conditional on $X=x$, the response is $Z+\varepsilon$, and therefore $$ Y\mid X=x\sim N(2x_1-x_2,1). $$ Conditional on $Z=z$, the same model gives $Y=z+\varepsilon$, so $$ Y\mid Z=z\sim N(z,1). $$ These conditional distributions agree whenever $z=2x_1-x_2$. For example, $x=(1,3,7)^T$ gives $z=-1$ and conditional law $N(-1,1)$; any other predictor vector with $2x_1-x_2=-1$ gives that same law. Hence $Y$ and $X$ are conditionally independent given $B^TX$. The third predictor does not enter the response model, and the first two enter only through their indicated linear combination.

Worked Example: A Nonlinear Response Distribution

Let $X=(X_1,X_2,X_3)^T$, and suppose $Y$ is binary with $$ P(Y=1\mid X=x)=\frac{\exp\{(x_1+x_2)^2\}}{1+\exp\{(x_1+x_2)^2\}}. $$ Show that $Z=X_1+X_2$ is a sufficient reduction.

Choose $B=(1,1,0)^T$, giving $Z=B^TX=X_1+X_2$. For every $x$, $$ P(Y=1\mid X=x)=\frac{\exp(z^2)}{1+\exp(z^2)},\qquad z=x_1+x_2. $$ Since $Y$ is binary, this probability determines its entire conditional distribution: the probability of $Y=0$ is one minus the displayed quantity. Thus both conditional probabilities depend on $x$ only through $z$, proving sufficiency.

The dependence need not be linear in the reduced coordinate. For $z=0$, the success probability is $1/2$, whether $x=(0,0,5)^T$ or $x=(1,-1,2)^T$. For $z=2$, it is $e^4/(1+e^4)$, approximately $0.9820$. The reduction is one-dimensional even though the response probability is a nonlinear function of that coordinate.

Worked Example: Different Coordinates for the Same Predictor Subspace

Suppose $X=(X_1,X_2,X_3)^T$, and $$ Y=X_1^2+X_2^2+\varepsilon, $$ where $\varepsilon\sim N(0,1)$ is independent of $X$. Use the matrix $$ B=\begin{pmatrix}1&0\\0&1\\0&0\end{pmatrix}. $$ Then $Z=B^TX=(X_1,X_2)^T$. Given $X=x$, $Y$ has distribution $N(x_1^2+x_2^2,1)$; given $Z=z$, it has distribution $N(z_1^2+z_2^2,1)$. These agree at $z=(x_1,x_2)^T$, so the two reduced coordinates are sufficient.

Now set $$ C=\begin{pmatrix}1&1\\0&1\end{pmatrix},\qquad B'=BC. $$ The new reduced coordinates are $$ Z'=(B')^TX=C^TB^TX=(X_1,X_1+X_2)^T. $$ Since $C$ is invertible, $Z'$ recovers $Z$: $X_1=Z'_1$ and $X_2=Z'_2-Z'_1$. The conditional response mean can therefore be written as $$ X_1^2+X_2^2=(Z'_1)^2+(Z'_2-Z'_1)^2. $$ So $Z'$ is also sufficient. Its coordinates differ from those of $Z$, but the columns of $B$ and $B'$ span the same two-dimensional subspace of predictor directions.

Conditional Independence and a Full Distributional Criterion

For a precise verification, conditional independence can be stated using conditional expectations. Since $Z=B^TX$ is determined by $X$, $Y\mathrel{\perp\!\!\!\perp}X\mid Z$ means that for every measurable response set $A$, $$ E[\mathbf{1}_{\{Y\in A\}}\mid X]=E[\mathbf{1}_{\{Y\in A\}}\mid Z]\quad\text{almost surely}. $$ The right side is a function of the reduced coordinates alone. The following argument explains why this is equivalent to equality of the conditional response laws.

Result: Assume regular conditional distributions of $Y$ given $X$ and given $Z=B^TX$ exist. Then $Y\mathrel{\perp\!\!\!\perp}X\mid Z$ if and only if the two conditional distributions agree almost surely.

Proof. If conditional independence holds, then for every measurable $A$, $$ P(Y\in A\mid X,Z)=P(Y\in A\mid Z)\quad\text{almost surely}. $$ Because $Z$ is a function of $X$, conditioning on $(X,Z)$ is the same as conditioning on $X$. Thus $P(Y\in A\mid X)=P(Y\in A\mid Z)$ almost surely. These conditional probabilities are evaluations of the two regular conditional distributions at $A$. To conclude equality of the distributions, first take $A$ in a countable determining class for the standard Borel response space; the equalities hold simultaneously outside one probability-zero set. The uniqueness of probability measures agreeing on such a class gives equality on all measurable sets.

Conversely, if the conditional distributions agree almost surely, then their probabilities agree for every measurable $A$, so $P(Y\in A\mid X)=P(Y\in A\mid Z)$ almost surely. This is the conditional-independence criterion above, and hence $Y\mathrel{\perp\!\!\!\perp}X\mid Z$. $\square$

Model Structure That Guarantees Sufficiency

A common way to establish sufficiency is to express the response as a measurable function of the reduced coordinates and an error independent of the predictors. This allows nonlinear effects and non-Gaussian errors; the reduction is determined by which aspects of $X$ enter the response-generating mechanism.

Theorem: Suppose $Z=B^TX$, $\varepsilon$ is independent of $X$, and $Y=h(Z,\varepsilon)$ for a measurable function $h$. Then $Y\mathrel{\perp\!\!\!\perp}X\mid Z$.

Proof. Let $A$ be any measurable set of response values and define $$ q_A(z)=P\{h(z,\varepsilon)\in A\}. $$ This is measurable in $z$ under the stated measurability and standard Borel assumptions. Since $\varepsilon$ is independent of $X$, conditioning on $X$ leaves the distribution of $\varepsilon$ unchanged. Therefore $$ P(Y\in A\mid X)=P\{h(B^TX,\varepsilon)\in A\mid X\}=q_A(B^TX). $$ Conditioning instead on $Z$ gives $P(Y\in A\mid Z)=q_A(Z)$. As $Z=B^TX$, these conditional probabilities are equal almost surely for every $A$. By the conditional-independence criterion, $Y\mathrel{\perp\!\!\!\perp}X\mid Z$. $\square$

The condition is sufficient, not necessary: there are response models with dependence expressed through a conditional distribution without a convenient representation using an independent error. The conditional-distribution criterion remains the definition to check.

Coordinates and Subspaces Are Different Objects

The vector $B^TX$ is a random vector in $\mathbb{R}^d$: it contains the reduced coordinates for each observed predictor vector. The column space $\operatorname{span}(B)$, by contrast, is a $d$-dimensional-or-smaller subspace of $\mathbb{R}^p$. It describes predictor directions, not the coordinate values for an individual observation.

When $B$ has full column rank and $C\in\mathbb{R}^{d\times d}$ is invertible, $BC$ spans the same subspace as $B$. Its reduced coordinates are $C^TB^TX$. Because $C^T$ is invertible, these coordinates determine $B^TX$, and vice versa. Thus the two coordinate vectors contain exactly the same information, even though their numerical values and coordinate descriptions differ. A redundant, rank-deficient $B$ can also be replaced by a full-rank matrix for its column space without increasing the number of distinct directions it represents.

This distinction matters in reporting and interpretation. A basis matrix is one way to encode directions; its columns are not uniquely meaningful as individual directions. The reduction is about the information carried by the coordinates, while the associated subspace records the directions represented by the matrix.

Why Sufficiency Matters—and a Common Pitfall

A sufficient reduction can replace a high-dimensional predictor vector with a lower-dimensional argument for the entire conditional response distribution. This can simplify visualization, estimation, and interpretation without restricting the response to depend linearly on the original predictors. It also clarifies what a reduction does not claim: it does not, by itself, say that the displayed coordinates are unique, that the reduction has the smallest possible dimension, or that a particular method will find it.

Common mistake: “The conditional mean depends on $B^TX$, so $B^TX$ is sufficient.” This reasoning is incomplete. Two conditional distributions can have the same mean but different variances, tail behaviour, or probabilities of particular outcomes. For example, a model could have mean zero for every $X$ while its conditional variance changes with $X$; the mean would reveal no change, but the conditional distribution would still depend on $X$. The correction is to verify that the full conditional law of $Y$, not only one summary such as its mean, depends on $X$ through the proposed coordinates.

Key takeaway: A sufficient reduction $B^TX$ preserves the full conditional distribution of $Y$ given $X$. The coordinates $B^TX$ are random values; the column space of $B$ is the predictor subspace those coordinates represent.

Check Your Understanding

Use conditional response distributions, rather than only conditional means, in your reasoning.

State the conditional-independence definition of a sufficient reduction $B^TX$.
If $Y\mid X=x\sim N(x_1+x_2,\,x_1^2+1)$, is $Z=X_1+X_2$ sufficient? Explain using the full conditional distribution.
In the model $Y=h(B^TX,\varepsilon)$, what assumption about $\varepsilon$ ensures the stated sufficiency result?
For full-column-rank $B$ and invertible $C$, explain why $B^TX$ and $(BC)^TX$ contain the same information.
Why does dependence of $E[Y\mid X]$ on $X$ only through $B^TX$ not, by itself, establish sufficiency?