Bias and Variance Across the Regularization Path

Why Follow Bias and Variance Along a Penalty Path?

A regularization path is the collection of fitted models obtained as a penalty strength changes. In high-dimensional genomic prediction, weak regularization can let a model respond strongly to sample-specific noise, while strong regularization can suppress genuine signal. The useful question is not simply whether coefficients become smaller as the penalty increases. It is how the resulting prediction error changes, and what parts of that error arise from systematic shrinkage versus sampling variability.

The prediction and variable-selection distinction matters here: a coefficient path is not itself a measure of predictive quality. A model can retain few features and predict well, or include many features and still generalize poorly. Bias and variance describe repeated-sample behavior of an estimator under a specified data-generating setup; they are not labels that can be read off a single fitted model without qualification.

Prediction Risk and Its Decomposition

Assume the conditional mean is \(E(Y_0\mid X_0)=X_0^T\beta\), with an intercept either included as an unpenalized term or removed by centering. Let \(Q=E(X_0X_0^T)\) be the second-moment matrix of future predictors. Suppose the future predictors are independent of the training sample and \(\hat{\beta}\) has finite second moments. The expected squared conditional-mean prediction error is

The first term is squared prediction bias, weighted by the future predictor distribution; the second is prediction variance. This follows by conditioning on \(\hat{\beta}\), writing \(\hat{\beta}-\beta=b+\{\hat{\beta}-E(\hat{\beta})\}\), and noting that the centered random part has expectation zero. The cross term therefore vanishes. If future predictors have mean zero and identity covariance, then \(Q=I\), and the expression reduces to squared coefficient bias plus the trace of the coefficient covariance. Identity covariance alone is not enough: a nonzero predictor mean contributes to \(Q\) and can change prediction risk.

If the future response is \(Y_0=X_0^T\beta+\epsilon_0\), where \(E(\epsilon_0\mid X_0)=0\), and the future error is independent of the fitted estimator, then response prediction risk adds the irreducible term \(E(\epsilon_0^2)\). That noise term does not decrease merely by changing the penalty. Throughout the numerical examples below, we focus on conditional-mean prediction risk unless response noise is explicitly mentioned.

A bias–variance decomposition describes expected error across hypothetical repetitions of the training process. It does not promise that the realized error on one particular sample will have the same components. Nor does it imply that every step toward more regularization decreases variance enough to offset increased bias.

Ridge Along a Fixed-Design Path

For a precise path calculation, consider the fixed-design linear model \(y=X\beta+\epsilon\), where \(X\) is an \(n\times p\) predictor matrix, \(E(\epsilon\mid X)=0\), and \(\operatorname{Cov}(\epsilon\mid X)=\sigma^2I\). The ridge estimator at \(\lambda\geq0\) minimizes residual sum of squares plus an \(L_2\) penalty:

For \(\lambda>0\), the inverse exists even when \(p>n\). At \(\lambda=0\), if \(p>n\), coefficients need not be unique; the minimum-norm least-squares solution is one conventional choice. The formulas that follow concern positive \(\lambda\), or a specified minimum-norm limit at zero.

Let \(d_j\) be the positive eigenvalues of \(X^TX\), and let \(v_j\) be corresponding orthonormal eigenvectors. In direction \(v_j\), ridge multiplies the least-squares component by \(d_j/(d_j+\lambda)\). This factor decreases from near one toward zero as \(\lambda\) grows. Directions with small \(d_j\) are shrunk more strongly at a given \(\lambda\), which is particularly relevant when predictors are highly correlated or the design has small singular values.

To verify the result, write \(X^Ty=X^TX\beta+X^T\epsilon\). Projecting the ridge estimator onto \(v_j\) gives

The second numerator has mean zero and variance \(\sigma^2d_j\), since \(Xv_j\) has squared length \(d_j\). Subtracting \(\beta_j\) yields the stated bias and variance. The squared bias increases with \(\lambda\), while variance decreases for \(\lambda\geq0\), since the derivative of \(d_j/(d_j+\lambda)^2\) is \(-2d_j/(d_j+\lambda)^3<0\). Their sum can decrease and then increase; its minimum in this single direction occurs at \(\lambda=\sigma^2/\beta_j^2\) when \(\beta_j\ne0\). Across directions, the best common \(\lambda\) depends on the whole signal, spectrum, and prediction target.

A Small High-Dimensional Example

The next example has \(p>n\), as is common in genomic prediction. It makes the effect of a path concrete while keeping the algebra transparent. Assume predictors have been put on a common scale and the intercept is handled separately. Let \(X\) be a fixed \(3\times5\) matrix of rank two with positive eigenvalues \(9\) and \(1\) for \(X^TX\). Let the corresponding unit eigenvectors be \(v_1\) and \(v_2\). The true coefficient vector is \(\beta=2v_1+v_2\), with zero components in the other three directions. Suppose the training errors have mean zero and covariance \(I\), so \(\sigma^2=1\), and future predictors have mean zero and covariance \(I_5\). Thus their second-moment matrix is \(I_5\), and expected conditional-mean prediction risk equals expected squared coefficient error.

In this example, \(y=X\beta+\epsilon\). The relevant risks are the sum of the two directions’ squared biases and variances. The three null directions contribute zero because the true coefficient there is zero and ridge returns zero components in those directions. For each \(\lambda>0\), use the ridge formulas above:

What the Path Can and Cannot Tell You

In the example, three training observations cannot identify five unrestricted coefficients: the design has a three-dimensional null space because its rank is two. Ridge makes the optimization solution unique for positive \(\lambda\), but uniqueness is not the same as learning every true coefficient. We set the true null-space components to zero, and the minimum-norm fit also sets its estimates there to zero. If the true signal had a nonzero component in a null direction, the observed training responses could not reveal it; ridge would incur bias in that direction. Prediction consequences would depend on whether future predictors vary in that direction, as represented by \(Q\).

For genomic applications, apparent complexity can come from many small coefficient changes rather than a single dramatic entry event. A rising penalty typically shrinks coefficients and often reduces estimator variance, but prediction risk need not fall monotonically. The path should therefore be interpreted together with the target population, predictor scaling, and prediction loss. The introductory material on prediction and variable selection distinguishes predictive performance from the identity of selected features; bias–variance analysis adds a repeated-sampling explanation for why those objectives can favor different parts of a path.

Diagnostics, Alternatives, and Common Pitfalls

A numerical path calculation is only as useful as the fit and prediction target behind it. Check that predictors were scaled consistently, the intercept was treated as intended, and the fitting algorithm converged. For a fixed penalty, inspect whether residual structure suggests a poor mean model, whether estimated performance is unusually sensitive to a few observations, and whether the coefficient path behaves consistently with the chosen penalty. Strong instability among highly correlated predictors may coexist with relatively stable predictions. If the future population has a different predictor distribution from the training set, training-design geometry alone may not describe prediction risk.

Ridge gives continuous shrinkage and has the explicit spectral bias and variance formulas above. Lasso uses an \(L_1\) penalty and can set coefficients exactly to zero; its path is not generally a simple fixed linear shrinkage of each eigen-direction, especially with correlated predictors. Elastic net combines \(L_1\) and \(L_2\) penalties, so it can both shrink and produce sparse fits. These are not interchangeable bias–variance formulas: the behavior depends on signal structure, correlation, scaling, and the estimator. In general, do not infer that lasso always has lower variance than ridge or that a sparse model must predict better.

A common mistake is to say, “The penalty is larger, so prediction error must be smaller because variance is smaller.” The correction is that increasing regularization generally trades reduced variance for increased shrinkage bias; only their combined, target-specific risk determines whether prediction improves. Another mistake is to equate a small coefficient norm with good prediction. The relevant metric weights coefficient error by \(Q=E(X_0X_0^T)\), and prediction of a future response also includes irreducible response noise.

• Underfitting warning: coefficients and fitted signal may be suppressed so strongly that prediction errors show systematic structure or the model misses real effects.
• Overfitting warning: a flexible fit may respond to idiosyncratic training noise, yielding unstable estimates or a large gap between training performance and performance on genuinely new data.
• Interpretation limit: a favorable prediction-risk estimate does not establish that the selected genes are causal, uniquely identified, or stable under resampling.

Bias and variance provide a conceptual account, not a direct decomposition that can usually be read from one dataset. Their numerical balance requires assumptions or repeated data, and it can differ across populations and loss functions. The path alone does not identify the best penalty; it only describes how fits change as regularization changes.