Signal Alignment And Interpolator Risk

Why Signal Direction Matters

Two regression signals can have the same coefficient norm and still lead to different prediction risks after interpolation. The reason is that interpolation does not generally recover every component of the signal: the minimum-norm solution keeps the part visible to the training design and discards the part in its nullspace. Prediction risk then weights the resulting error according to the covariance of a new feature vector. Thus the signal’s direction interacts with both the training geometry and the test-feature covariance.

We study this interaction in a noiseless, correctly specified linear model. The comparison is conditional on a fixed training design, so it isolates signal direction from variation across possible training samples. It also makes an important qualification precise: “high-variance direction” by itself does not imply either higher or lower interpolation risk.

Model and Risk Definition

The equality follows by writing the prediction error as \(x^T(\hat\beta-\beta^\ast)\) and using \(E[xx^T]=\Sigma\). Since the response has no additional noise, this risk is the squared prediction error relative to the true regression function. We do not include a separate irreducible-noise term.

Let the fixed training design be \(X\in\mathbb{R}^{n\times p}\), with full row rank and \(n<p\), and let \(y=X\beta^\ast\). Among all coefficient vectors that interpolate, the minimum-Euclidean-norm interpolator is $$ \hat\beta=X^T(XX^T)^{-1}y. $$ Full row rank ensures that \(XX^T\) is invertible. The row space of \(X\) is the subspace spanned by its rows; its orthogonal complement is the nullspace \(\mathcal{N}(X)=\{v:Xv=0\}\).

The Error Is the Signal’s Nullspace Component

Proof. Substitute \(y=X\beta^\ast\) into the minimum-norm formula:

The matrix \(P=X^T(XX^T)^{-1}X\) is symmetric, because \(XX^T\) and its inverse are symmetric. It is idempotent:

Its range is contained in the row space of \(X\), since \(Pv\) is a linear combination of the columns of \(X^T\). Conversely, if \(u=X^Ta\) is in that row space, then

Therefore \(P\) is the orthogonal projector onto the row space. The orthogonal complement of that space is \(\mathcal{N}(X)\), so \(I-P=P_{\mathcal{N}(X)}\). Hence \(\hat\beta-\beta^\ast=-(I-P)\beta^\ast\). Substitution into the risk definition gives the stated quadratic form; the minus sign cancels. \(\square\)

This result separates the two ingredients of risk. The training design determines which part of the signal is missed, through \(P_{\mathcal{N}(X)}\). The covariance \(\Sigma\) determines how costly that missed component is for prediction. A coefficient error of a given Euclidean size can therefore have very different prediction costs in different directions.

Reading Directional Risk in Eigen-Coordinates

Suppose the covariance has orthonormal eigenvectors \(u_1,\ldots,u_p\) and eigenvalues \(\lambda_1,\ldots,\lambda_p>0\), so that \(\Sigma u_j=\lambda_j u_j\). A high-variance eigendirection has a larger eigenvalue; a low-variance eigendirection has a smaller one. Write \(q=P_{\mathcal{N}(X)}\beta^\ast\) for the coefficient error up to sign, and \(q_k=u_k^Tq\). The risk is

In particular, if the signal is \(\beta^\ast=B u_j\), with norm \(|B|\), then

The same signal norm \(B^2\) is multiplied by a direction-dependent quantity. The covariance eigenvalue \(\lambda_j\) matters, but it is not the only relevant eigenvalue: the projection of \(u_j\) into the training nullspace can have components in several covariance eigendirections.

Worked Comparisons

A Useful Special Case and a Common Pitfall

When the training nullspace is one-dimensional, the comparison has a compact form. If it is spanned by a nonzero vector \(v\), then \(P_{\mathcal{N}(X)}=vv^T/(v^Tv)\). For a signal \(\beta^\ast=B u_j\), the risk is

This follows directly by projecting \(B u_j\) onto \(v\), then evaluating the covariance-weighted squared length. The shared factor \(v^T\Sigma v/(v^Tv)^2\) is the same for every direction in this fixed design. The directional comparison is then controlled by \((u_j^Tv)^2\): how strongly the signal direction overlaps the nullspace. With a higher-dimensional nullspace, the overlap becomes a vector of components and each is weighted by its covariance eigenvalue.

A common mistake is to argue that a signal aligned with a high-variance direction must have greater risk because prediction errors in that direction are more costly. That reasoning considers only the weighting by \(\Sigma\) and ignores whether the interpolator makes an error in that direction, and how large that error is. The correction is to calculate \(P_{\mathcal{N}(X)}\beta^\ast\) first and then evaluate its covariance-weighted squared length.

These calculations are conditional on the observed design. If the design is random, its own distribution changes the nullspace from sample to sample, and an unconditional risk comparison requires averaging over that randomness. The fixed-design examples establish a precise geometric point, not a universal ranking for all random-design models: equal signal norms do not guarantee equal risks, and high- versus low-variance alignment alone does not determine the ordering.