Label Noise And Exact-Fit Prediction

Why Exact Fit Does Not Mean Noise-Free Prediction

An interpolating estimator reproduces every training label exactly. If some labels contain random error, however, exact reproduction also means fitting that error. The resulting predictions can therefore vary with the training noise, even when the underlying regression function is fixed. How much this matters depends not just on the noise variance but also on the geometry of the estimator and the test covariates.

We study this effect for a fixed design and a linear estimator. The goal is to isolate the contribution of independent label noise to prediction risk, then calculate how that contribution changes when the label variances differ across observations.

Setup and Definition

Let \(X\) be an \(n\times p\) design matrix with full row rank, where \(p\geq n\). The training response vector follows the linear model

where \(\beta^*\in\mathbb{R}^p\) is fixed. Assume the training errors \(\varepsilon_1,\ldots,\varepsilon_n\) are independent, have mean zero, and have finite variances \(v_1,\ldots,v_n\). Thus \(\operatorname{Cov}(\varepsilon)=D=\operatorname{diag}(v_1,\ldots,v_n)\). The equal-variance case \(v_i=\sigma^2\) for all \(i\) is called homoskedastic; otherwise the errors are heteroskedastic.

A linear estimator has the form \(\hat\beta=By\), for a fixed \(p\times n\) matrix \(B\). Its fitted training values are \(X\hat\beta=XBy\).

For prediction, let \(x_0\in\mathbb{R}^p\) be an independent test covariate with mean zero and covariance matrix \(\Sigma=E[x_0x_0^T]\). We measure error against the clean regression function \(x_0^T\beta^*\), rather than against a noisy test response. For a vector \(u\), write \(\|u\|_\Sigma^2=u^T\Sigma u\). The conditional prediction risk averages over both training noise and the independent test covariate, with \(X\) held fixed.

The Exact Risk Decomposition

The estimator's error in coefficient space separates into a part that remains when the labels are noiseless and a part transmitted from the training errors:

The first term can be nonzero even when \(\varepsilon=0\). The second term is the effect of label noise. The next result gives the exact prediction-risk decomposition.

Proof. Set \(a=(BX-I_p)\beta^*\), a fixed vector conditional on \(X\). Then \(\hat\beta-\beta^*=a+B\varepsilon\). Conditional on \(\varepsilon\), the test-covariate expectation of the squared prediction error is

Taking expectation over \(\varepsilon\), the two cross terms vanish because \(E[\varepsilon]=0\). The remaining terms are

The trace identity \(\operatorname{tr}(B^T\Sigma BD)=\operatorname{tr}(\Sigma BDB^T)\) gives the stated expression. Setting \(\varepsilon=0\) leaves just \(a^T\Sigma a\), so the trace term is exactly the noise contribution. \(\square\)

Homoskedastic and Heteroskedastic Noise

Write \(b_i\) for column \(i\) of \(B\). Since \(BDB^T=\sum_{i=1}^n v_i b_i b_i^T\), the noise contribution has the equivalent form

Each observation's noise variance is multiplied by the test-prediction leverage of the corresponding column of the estimator. This makes the contrast between equal and unequal variances explicit.

Derivation. In the homoskedastic case, substitute \(D=\sigma^2I_n\) into the theorem's trace term and factor out \(\sigma^2\). In the heteroskedastic case, expand the diagonal matrix as \(D=\sum_i v_i e_i e_i^T\), where \(e_i\) is coordinate vector \(i\). Then \(Be_i=b_i\), so

This proves both formulas. \(\square\)

The result does not require normally distributed errors. Independence, zero means, and finite variances suffice for this second-moment calculation. If errors are correlated, their covariance matrix is not diagonal and the general trace formula still applies, but the separate-observation sum no longer describes all contributions.

Minimum-Euclidean-Norm Interpolation

A standard interpolator when \(p\geq n\) and \(X\) has full row rank is the minimum-Euclidean-norm solution. Its matrix is

Indeed, \(XB_{\min}=I_n\). For any response vector \(y\), \(B_{\min}y\) is orthogonal to every vector \(z\) satisfying \(Xz=0\), because \((B_{\min}y)^Tz=y^T(XX^T)^{-1}Xz=0\). Every other solution to \(X\beta=y\) differs from \(B_{\min}y\) by such a vector \(z\). The Pythagorean identity then gives \(\|B_{\min}y+z\|_2^2=\|B_{\min}y\|_2^2+\|z\|_2^2\), proving the minimum-norm claim.

Derivation. Substitute \(B_{\min}=X^T(XX^T)^{-1}\) and \(B_{\min}^T=(XX^T)^{-1}X\) into \(\operatorname{tr}(\Sigma BDB^T)\). The matrix \(XX^T\) is invertible because \(X\) has full row rank, so all factors are defined. This gives the displayed formula. In particular, for homoskedastic noise it becomes

Worked Calculations

Interpretation and Common Pitfalls

The trace formula separates two questions. The variances \(v_i\) describe how noisy the labels are; the quantities \(b_i^T\Sigma b_i\) describe how strongly the estimator converts each label into predictions in directions that matter for test covariates. A large variance need not dominate if its column has little test-prediction effect, while a moderate variance can matter substantially if its column is strongly amplified.

A common mistake is to reason, “The training residuals are zero, so the prediction error caused by training noise is zero.” The residuals are zero precisely because the estimator matches the noisy labels. In the model \(y=X\beta^*+\varepsilon\), the fitted coefficients contain \(B\varepsilon\); the theorem shows that its test-prediction contribution is \(\operatorname{tr}(\Sigma BDB^T)\), which is generally positive.

Another distinction is the target of prediction. The formulas above compare the fitted value with the clean conditional mean \(x_0^T\beta^*\). If the goal is instead to predict a future response \(Y_0=x_0^T\beta^*+\varepsilon_0\), and \(\varepsilon_0\) is independent, mean zero, and has variance \(\tau^2\), then response-prediction risk is the clean-function risk plus \(\tau^2\). That additional term is irreducible test-label noise; it is not amplification of the training labels.