Basis Invariance and Identifiability of Reduced Directions

Why a Direction Is Not a Unique Coordinate Description

In sufficient dimension reduction, the goal is to replace a predictor vector \(X\in\mathbb{R}^p\) by a lower-dimensional vector without losing information about a response \(Y\). A matrix \(B\) can describe this reduction through the index \(B^T X\). But the columns of \(B\) are coordinates for a subspace, not necessarily uniquely defined directions: changing those coordinates by an invertible transformation leaves the information in the index unchanged.

This distinction matters when interpreting a fitted reduction. The central subspace is a population feature of the conditional distribution of \(Y\) given \(X\); a particular matrix of basis vectors is only one representation of that feature. The dimension and the subspace can be identifiable even when no individual basis vector is.

Definition: A matrix \(B\in\mathbb{R}^{p\times d}\) with full column rank defines a sufficient dimension reduction if \(Y\) is conditionally independent of \(X\) given \(B^T X\), written \(Y\perp X\mid B^T X\). The column space \(\operatorname{span}(B)\) is the subspace represented by \(B\). When the smallest such subspace exists and is unique, it is the central subspace, denoted \(\mathcal{S}_{Y\mid X}\); its dimension is the central-subspace dimension.

The conditional-independence condition says that, once \(B^T X\) is known, learning the full predictor vector \(X\) gives no further information about \(Y\). Existence and uniqueness of the central subspace are assumed here. The present question is how to describe and identify it when a matrix basis is not unique.

Invertible Reparameterizations Preserve the Reduction

Suppose \(A\in\mathbb{R}^{d\times d}\) is nonsingular. The transformed index based on \(BA\) is \((BA)^T X=A^T B^T X\). Since \(A^T\) is invertible, this transformation can be reversed exactly. Thus the two indices contain the same information.

Theorem: If \(B\in\mathbb{R}^{p\times d}\) has full column rank and \(A\in\mathbb{R}^{d\times d}\) is nonsingular, then \(Y\perp X\mid B^T X\) if and only if \(Y\perp X\mid (BA)^T X\). In addition, \(\operatorname{span}(BA)=\operatorname{span}(B)\).

Proof. Let \(Z=B^T X\) and \(W=A^T Z=(BA)^T X\). Because the maps \(z\mapsto A^Tz\) and \(w\mapsto (A^T)^{-1}w\) are measurable inverses, \(Z\) and \(W\) generate the same sigma-field: \(\sigma(Z)=\sigma(W)\). Conditional independence of \(Y\) and \(X\) given a random vector depends on the sigma-field generated by that vector. Hence conditioning on \(Z\) is equivalent to conditioning on \(W\), proving the conditional-independence claim in both directions. For the subspaces, every column of \(BA\) is a linear combination of columns of \(B\), so \(\operatorname{span}(BA)\subseteq\operatorname{span}(B)\). Since \(B=(BA)A^{-1}\), the reverse inclusion also holds. Therefore the column spaces are equal. \(\square\)

Worked Example: Changing Coordinates in a Two-Dimensional Reduction

Let \(X=(X_1,X_2,X_3)^T\), and suppose a sufficient reduction is represented by \(B=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}\). Choose \(A=\begin{pmatrix}2&1\\1&1\end{pmatrix}\). Its determinant is \(2(1)-1(1)=1\), so it is nonsingular.

The original reduced coordinates are \(u=B^T X=(X_1+X_3,\;X_2+X_3)^T\). The transformed basis is \(BA=\begin{pmatrix}2&1\\1&1\\3&2\end{pmatrix}\), so its coordinates are \(v=(BA)^T X=(2X_1+X_2+3X_3,\;X_1+X_2+2X_3)^T\).

In terms of \(u=(u_1,u_2)^T\), these are \(v_1=2u_1+u_2\) and \(v_2=u_1+u_2\). Solving gives \(u_1=v_1-v_2\) and \(u_2=-v_1+2v_2\). Thus neither coordinate system contains information absent from the other. The column space, and therefore the reduction, has not changed.

Exactly When Do Two Bases Describe the Same Subspace?

The invariance result has a converse: two full-rank \(p\times d\) matrices represent the same \(d\)-dimensional subspace precisely when one is obtained from the other by a nonsingular \(d\times d\) change of basis. This identifies the full ambiguity in the matrix representation.

Proposition: Let \(B_1,B_2\in\mathbb{R}^{p\times d}\) both have full column rank. Then \(\operatorname{span}(B_1)=\operatorname{span}(B_2)\) if and only if there is a nonsingular matrix \(A\in\mathbb{R}^{d\times d}\) such that \(B_2=B_1A\).

Proof. If \(B_2=B_1A\) for nonsingular \(A\), the theorem's column-space argument gives equal spans. Conversely, suppose the spans are equal. Each column of \(B_2\) belongs to \(\operatorname{span}(B_1)\), so there is a matrix \(A\) such that \(B_2=B_1A\). If \(A\) were singular, there would be a nonzero vector \(c\in\mathbb{R}^d\) with \(Ac=0\). Then \(B_2c=B_1Ac=0\), contradicting the full column rank of \(B_2\). Thus \(A\) is nonsingular. \(\square\)

This proposition means the basis matrices for one subspace form an equivalence class: \(B\) and \(BA\) represent the same object for every nonsingular \(A\). Identifying a particular matrix \(B\) without choosing extra conventions would confuse a coordinate choice with the subspace itself.

Worked Example: A One-Dimensional Direction Has No Preferred Sign or Scale

Consider \(B=(1,2)^T\). For any nonzero scalar \(a\), \(A=(a)\) is nonsingular and \(BA=(a,2a)^T\). In particular, the bases \((1,2)^T\), \((3,6)^T\), and \((-1,-2)^T\) all span the same line.

If \(u=B^T X=X_1+2X_2\), then the index from the basis \((-1,-2)^T\) is \(-u\). A conditional distribution expressed as a function of \(u\), say \(F_{Y\mid X}(y\mid X)=G(y,u)\), can instead be written as \(G(y,-v)\) when \(v=-u\). The conditional distribution is unchanged; only the argument used to label the reduced coordinate has changed.

Consequently, the sign or magnitude of a one-dimensional basis vector is not an identifiable feature of the subspace. A normalization, such as fixing its length and choosing a sign convention, can select one representative, but that convention does not add information about the underlying subspace.

Subspace Summaries That Do Not Depend on a Basis

A useful invariant summary is the orthogonal projection matrix onto the column space of a full-column-rank \(B\):

P_B=B(B^TB)^{-1}B^T.

The inverse exists because \(B\) has full column rank. This matrix projects vectors in \(\mathbb{R}^p\) onto \(\operatorname{span}(B)\). It does not change under a nonsingular reparameterization:

\begin{aligned} P_{BA} &=BA\bigl((BA)^TBA\bigr)^{-1}(BA)^T\\ &=BA\bigl(A^TB^TBA\bigr)^{-1}A^TB^T\\ &=BA\bigl(A^{-1}(B^TB)^{-1}(A^T)^{-1}\bigr)A^TB^T\\ &=B(B^TB)^{-1}B^T=P_B. \end{aligned}

In the third line, the inverse formula follows by multiplying \(A^T(B^TB)A\) by \(A^{-1}(B^TB)^{-1}(A^T)^{-1}\) and obtaining the identity matrix. A projection matrix is therefore a basis-free way to represent a subspace. Other comparisons can also be made at the subspace level, for example by comparing projection matrices or the principal angles between subspaces, rather than matching basis columns one by one.

Worked Example: Checking Whether Two Estimated Bases Match

Suppose two procedures return \(B_1=(1,0)^T\) and \(B_2=(0,1)^T\), each a \(2\times1\) full-rank matrix. Their projection matrices are \(P_{B_1}=\begin{pmatrix}1&0\\0&0\end{pmatrix}\) and \(P_{B_2}=\begin{pmatrix}0&0\\0&1\end{pmatrix}\). These matrices differ, so the estimated subspaces differ. No nonzero scalar \(a\) satisfies \(B_2=B_1a\), which gives the same conclusion directly.

By contrast, if the second procedure returns \(\widetilde B_2=(4,0)^T\), then \(P_{\widetilde B_2}=\begin{pmatrix}1&0\\0&0\end{pmatrix}=P_{B_1}\). The basis vectors differ, but their subspaces agree exactly. Comparing columns entry by entry would incorrectly treat a harmless rescaling as a different reduction.

What Is Identifiable, and What Is Only a Representation?

Under the assumed existence and uniqueness of the central subspace, its dimension \(d\) and its column space \(\mathcal{S}_{Y\mid X}\) are population features of the joint distribution of \((Y,X)\). Equivalently, a full-rank basis is identifiable only up to the transformations \(B\mapsto BA\) with nonsingular \(A\). The matrix coordinates, the ordering of basis vectors, their lengths, and (in one dimension) their signs are not individually determined by the subspace.

The reduced index itself is also coordinate-dependent: \(B^T X\) and \(A^TB^T X\) generally have different numerical values. What is invariant is the information they generate and the subspace they encode. A fitted conditional distribution written as a function of the reduced coordinates must change its coordinate representation along with the basis.

Identifiability is a population statement, not a guarantee that a finite sample will estimate the subspace accurately. Sampling variation, a weakly separated or poorly determined dimension, and the choice of estimation method can all affect estimates. It is therefore helpful to separate two questions: whether the population object is uniquely defined by the distribution, and how precisely a particular procedure can estimate it from observed data. In applications, report the estimated dimension and describe or compare the estimated subspace using basis-invariant summaries when possible.

Common Mistake: Interpreting Basis Columns as Unique Effects

A common error is to say that a fitted basis identifies “the first direction” and “the second direction” as unique effects because the returned matrix has two columns. That reasoning treats an arbitrary coordinate system as if it were part of the population target. Replacing \(B\) by \(BA\) can rotate, rescale, or mix its columns without changing the reduction.

The correction is to interpret the column space as the target. Individual columns can be discussed only relative to an explicitly stated normalization or coordinate convention. A further mistake is to infer that two reductions are equal merely because they have the same dimension: the worked comparison of the two coordinate axes shows that distinct one-dimensional subspaces can have the same dimension. Equality of dimension is necessary for equality of subspaces, but it is not sufficient.

Key takeaway: For a full-rank basis \(B\), every nonsingular transformation \(BA\) gives the same sufficient reduction. The identifiable object is the central subspace (and its dimension), not a particular choice of basis vectors.

Check Your Understanding

Use the definitions and results above to answer these questions.

Why does nonsingularity of \(A\) ensure that \(B^TX\) and \((BA)^TX\) contain the same information?
Two full-rank matrices have equal column spaces. What matrix relation must connect them, and why must the connecting matrix be nonsingular?
What feature of a reduction is represented by \(B(B^TB)^{-1}B^T\), and why is it unchanged by a change of basis?
Can two reductions of the same dimension represent different subspaces? Explain using two distinct lines in \(\mathbb{R}^2\).
Distinguish population identifiability of the central subspace from the accuracy of its estimate in a finite sample.