Shrinkage Is Not Variable Selection

Why Shrinking a Coefficient Is Not the Same as Removing a Predictor

A ridge model can make a predictor’s coefficient much smaller than its ordinary least-squares value. That reduction may make the fitted model more stable, but it does not, by itself, remove the predictor. For a finite positive penalty, ridge ordinarily leaves each nonconstant predictor in the model with a nonzero coefficient. Understanding that distinction matters when interpreting a fitted model: a small coefficient is not evidence that the fitting method selected the predictor out.

The qualification “ordinarily” is important. A ridge coefficient can be exactly zero for particular data configurations, for example when the relationships among predictors and the response produce an exact cancellation. Ridge does not generally force coefficients to zero as a selection rule. The discussion below uses the usual squared-error ridge objective with an unpenalized intercept, consistent with the earlier material on standardizing predictors and fitting ridge regression.

Ridge Coefficients and the Nonzero Property

Suppose the response is modeled as a linear function of \(p\) predictors. Let \(X\) be the design matrix with its predictor columns centered, let \(y\) be the centered response, and let \(\beta\) be the vector of predictor coefficients. Centering handles the intercept separately; the intercept is not included in the penalty. For a finite \(\lambda>0\), use the objective

This expression is a squared-error fitting objective plus a penalty on the squared coefficient magnitudes. It defines the ridge estimates as the values of \(\beta\) that minimize \(Q(\beta)\). The following equation applies to this objective and this penalty convention; software can use a differently scaled objective, in which case the numerical value assigned to \(\lambda\) may differ.

Differentiate with respect to \(\beta\) and set the gradient equal to zero:

Because \(X^\mathsf{T}X\) is positive semidefinite and \(\lambda>0\), the matrix \(X^\mathsf{T}X+\lambda I\) is positive definite and invertible. Thus the fitted coefficient vector is

This formula explains why ridge usually does not remove predictors. The penalty adds \(\lambda\) to the diagonal of the matrix being inverted; the estimating equation does not impose a requirement that any coordinate of \(\hat{\beta}_{\text{ridge}}\) equal zero. In contrast, a selection method must have a mechanism that can produce exact zeros.

Why Orthogonal Predictors Make the Point Clear

If the centered predictor columns are orthogonal, then \(X^\mathsf{T}X\) is diagonal. Write \(x_j\) for column \(j\) of \(X\). The ridge equation separates into one equation for each coefficient:

The denominator is positive for every \(\lambda>0\). Therefore, if \(x_j^\mathsf{T}y\ne 0\), the ridge coefficient is nonzero for every finite positive \(\lambda\). The coefficient is smaller in magnitude than the corresponding least-squares coefficient, which is \(x_j^\mathsf{T}y/\|x_j\|^2\), but it has not been removed. If \(x_j^\mathsf{T}y=0\), the coefficient is zero; the penalty has not created that zero by thresholding, because the unpenalized association in this orthogonal case was already zero.

With correlated predictors, the coefficient equations are coupled, so the simple one-predictor formula no longer applies. An exact zero can then arise through cancellation. The normal equation for coordinate \(j\) is

If \(\hat{\beta}_{j,\text{ridge}}=0\), the equation requires \(x_j^\mathsf{T}y=\sum_{k\ne j}x_j^\mathsf{T}x_k\hat{\beta}_{k,\text{ridge}}\). This can happen for a particular data set. It is an equality resulting from the data and the other fitted coefficients, not a general ridge rule that discards predictor \(j\).

An Exact Zero Can Occur Without Ridge Selecting a Variable

The usual nonzero pattern is not a theorem that every ridge coefficient must always be nonzero. Correlation among columns can make the coordinate equations depend on the other fitted coefficients. The following small fitted example verifies how an exact zero can occur even though ridge has no general zero-setting rule.

Reading Small Coefficients in a Fitted Model

A coefficient’s magnitude depends on the scale of its predictor. The earlier material on standardizing predictors addresses that comparability issue; it does not change the distinction between shrinkage and selection. Once the fitted coefficient scale is understood, report a small ridge coefficient as small, not as absent. The predictor still contributes to predictions whenever its coefficient is nonzero and its value differs from the model’s centering reference.

A finite positive penalty can make coefficients approach zero as the penalty grows, but an approach toward zero is not the same as reaching zero. In particular, do not round small coefficients to zero and then describe the fitting procedure as having removed the variables. Rounding is a reporting choice, not a property of the fitted ridge model.

Common Exam Traps

• Calling shrinkage selection. A coefficient smaller than its ordinary least-squares value has been shrunk; it has not been removed unless it is exactly zero.
• Claiming all ridge coefficients are guaranteed nonzero. That is too strong. Exact zeros can occur for particular response and predictor cross-products, especially when predictors are correlated.
• Interpreting a rounded value as the fitted value. If a displayed coefficient rounds to 0.000, inspect its unrounded value before claiming it is exactly zero.
• Including the intercept in the penalty without checking the model definition. The standard setup here leaves the intercept unpenalized and penalizes the predictor coefficients.

For a fitted model, the useful question is not simply whether a coefficient is “small.” Check whether it is exactly zero in the fitted output, understand the scale on which it is expressed, and distinguish the coefficient’s role in prediction from a formal decision to remove a variable. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.