Ridge Limitations and Common Misreadings

What Ridge Estimates Can—and Cannot—Tell You

Ridge regression is useful when a model has many predictors or predictors that carry overlapping information. Its penalty can stabilize coefficient estimates, but it does not make every interpretation safe. A ridge coefficient is affected by the penalty, the scale of its predictor, and the other predictors in the model. It is not automatically a measure of causal impact, and a small coefficient does not mean ridge has formally removed that predictor.

This tutorial focuses on these limits. The calculations use the usual squared-error ridge objective with an unpenalized intercept. For clarity, examples use centered responses and predictors, so the intercept is zero and the displayed objective has no intercept term. The penalty parameter is denoted by $\lambda\geq 0$.

Definition: For a centered response vector $y$ and centered predictor matrix $X$, the ridge estimate minimizes the residual sum of squares plus a penalty on the squared coefficients: $$ \widehat{\beta}_{\lambda} = \underset{\beta}{\operatorname{argmin}}\; \left\{(y-X\beta)^{T}(y-X\beta)+\lambda\beta^{T}\beta\right\}. $$ Here $\beta$ is the vector of predictor coefficients. The intercept, when included, is not penalized. For $\lambda>0$, the solution is $\widehat{\beta}_{\lambda}=(X^{T}X+\lambda I)^{-1}X^{T}y$. At $\lambda=0$, this inverse formula requires $X^{T}X$ to be invertible; otherwise the unpenalized least-squares coefficient vector need not be unique.

Shrinkage Is Not the Same as Selection

The ridge penalty discourages large coefficients by adding their squared magnitudes to the objective. For $\lambda>0$, this generally pulls estimates toward zero. But a ridge fit does not usually set coefficients exactly to zero. Thus, a predictor with a small ridge coefficient remains in the fitted model unless a separate rule is used to remove it.

A simple case makes this distinction precise. Suppose the predictor columns are mutually orthogonal, with each centered column $x_j$ having positive squared length $x_j^Tx_j$. The objective separates into one independent minimization for each coefficient. Differentiating with respect to $\beta_j$ gives

-2x_j^T(y-x_j\beta_j)+2\lambda\beta_j=0, \qquad \widehat{\beta}_{j,\lambda} = \frac{x_j^Ty}{x_j^Tx_j+\lambda}.

If $x_j^Ty\ne 0$, the numerator is nonzero, so the ridge estimate is nonzero for every finite $\lambda\geq 0$. When $\lambda>0$, its absolute value is smaller than the unpenalized estimate $\widehat{\beta}_{j,0}=x_j^Ty/(x_j^Tx_j)$. At $\lambda=0$, it is unchanged. This orthogonal-predictor result is a clear demonstration of shrinkage without selection; with correlated predictors the coefficients are coupled, but the penalty still does not generally produce exact zeros.

Worked Example: Shrinkage Without Removing Predictors

Consider four observations with centered response $y=(4,-4,3,-3)^T$ and two centered, orthogonal predictors $x_1=(1,-1,0,0)^T$ and $x_2=(0,0,1,-1)^T$. Fit ridge regression with $\lambda=2$. Compare the ridge coefficients with the unpenalized coefficients and check the fitted values.

Method. Use the orthogonal-predictor formula. First calculate the cross-products:

x_1^Tx_1=1^2+(-1)^2=2,\qquad x_2^Tx_2=1^2+(-1)^2=2,

x_1^Ty=(1)(4)+(-1)(-4)=8,\qquad x_2^Ty=(1)(3)+(-1)(-3)=6.

The unpenalized coefficients are $8/2=4$ and $6/2=3$. With $\lambda=2$, the ridge coefficients are

\widehat{\beta}_{1,2}=\frac{8}{2+2}=2,\qquad \widehat{\beta}_{2,2}=\frac{6}{2+2}=1.5.

Both coefficients are smaller in magnitude than their unpenalized counterparts, but neither is zero. The fitted values are

X\widehat{\beta}_{2} = 2x_1+1.5x_2 = (2,-2,1.5,-1.5)^T.

The residuals are $y-X\widehat{\beta}_{2}=(2,-2,1.5,-1.5)^T$. The calculation confirms that both predictors contribute to the fitted values. The common mistake this example avoids is calling a shrunken, nonzero coefficient “selected out.”

Why Predictor Scaling Matters

The ridge penalty acts on coefficient values, not directly on the predictor values. Therefore, the units used to record a predictor affect the penalty unless the predictors are scaled in a consistent way before fitting. Holding $\lambda$ fixed while changing a predictor’s units can change the fitted values. This is not a change in the underlying information; it is a change in how strongly the coefficient is penalized under that parameterization.

To see why, replace a predictor $x$ by $z=cx$, where $c\ne 0$. If $\beta$ is the coefficient on $x$ and $\gamma$ is the coefficient on $z$, equal predictions require $\beta x=\gamma z$, so $\beta=c\gamma$. The original penalty $\lambda\beta^2$ then becomes $\lambda c^2\gamma^2$. If the scaled model instead uses $\lambda\gamma^2$, it is applying a different penalty to the same predictions. The fitted model can therefore change.

Worked Example: Changing Units Changes the Ridge Fit

Suppose the centered data are $x=(-1,1)^T$ and $y=(-2,2)^T$, with one predictor and $\lambda=2$. Calculate the ridge fit using $x$, then replace the predictor with $z=10x$ and use the same $\lambda$. Compare fitted values, rounding only the final fitted values to two decimal places.

Original units. The needed cross-products are $x^Tx=2$ and $x^Ty=4$. Thus,

\widehat{\beta}_{2}=\frac{x^Ty}{x^Tx+2} =\frac{4}{2+2}=1, \qquad x\widehat{\beta}_{2}=(-1,1)^T.

Rescaled units. The new predictor is $z=(-10,10)^T$. Its cross-products are $z^Tz=100+100=200$ and $z^Ty=(-10)(-2)+(10)(2)=40$. Therefore,

\widehat{\gamma}_{2}=\frac{z^Ty}{z^Tz+2} =\frac{40}{202}=\frac{20}{101}.

The fitted values are $z\widehat{\gamma}_{2}=(-200/101,200/101)^T$, or $(-1.98,1.98)^T$ to two decimal places. They differ from the original fitted values $(-1,1)^T$. The reason is that the same numerical value of $\lambda$ does not represent the same penalty after changing units. To preserve the original objective under $z=10x$, the coefficient penalty in the $\gamma$ parameterization would be $\lambda(10\gamma)^2=200\gamma^2$, not $2\gamma^2$. The common mistake is to assume that rescaling a predictor leaves the ridge fit unchanged when $\lambda$ is held fixed.

A Shrunken Coefficient Is Not Automatically an Effect

A fitted coefficient describes the model’s relationship between a predictor and the response, conditional on the other predictors and on the chosen penalty and scaling. It does not, on its own, establish that changing the predictor would cause a change in the response. Causal interpretation requires a suitable study design and assumptions about how the data were generated, including whether relevant confounders have been addressed. Ridge regularization does not supply those assumptions.

Correlation among predictors also matters. When two predictors carry nearly identical information, the data may not reliably distinguish their individual contributions. The penalty can stabilize the fit, but it does not make the separate coefficients uniquely meaningful as effects. The following exact-duplicate example shows how ridge’s symmetry can determine a coefficient allocation that the data alone do not identify.

Worked Example: Duplicate Predictors and Coefficient Interpretation

Let $x_1=x_2=(-1,1)^T$, $y=(-2,2)^T$, and $\lambda=2$. Find the ridge coefficients and fitted values. Explain why the two coefficients should not be read as separately established effects.

Method. Write the residual sum of squares in terms of the coefficient sum $t=\beta_1+\beta_2$. Since the fitted values are $x_1t=(-t,t)^T$, the residual sum of squares is

$$ (-2+t)^2+(2-t)^2=2(2-t)^2. $$

For a fixed $t$, the penalty is smallest when the coefficients are equal. This follows from

\beta_1^2+\beta_2^2 = \frac{t^2}{2}+\frac{(\beta_1-\beta_2)^2}{2} \geq \frac{t^2}{2},

with equality at $\beta_1=\beta_2=t/2$. Thus the ridge objective at its minimum for a given $t$ is $2(2-t)^2+2(t^2/2)=2(2-t)^2+t^2$. Differentiating this expression gives $-4(2-t)+2t=0$, so $6t=8$ and $t=4/3$. Hence $\widehat{\beta}_1=\widehat{\beta}_2=2/3$.

The fitted values are $(-4/3,4/3)^T$. As a check, the residuals are $(-2/3,2/3)^T$, so the residual sum of squares is $8/9$; the penalty is $2[(2/3)^2+(2/3)^2]=16/9$; and the objective is $8/9+16/9=8/3$. The equal coefficient split reflects the symmetry of the ridge penalty. Because the predictor columns are identical, the observed data cannot distinguish their separate contributions. The ridge coefficients are useful for defining a stable penalized fit, but they do not establish two separate effects.

Common Misreadings to Avoid

Calling shrinkage selection. A coefficient that is smaller than its unpenalized counterpart has been shrunk; unless it is exactly zero, it has not been removed from the fitted model.
Claiming every positive penalty reduces a coefficient when $\lambda=0$ is allowed. In the orthogonal example, $8/(2+0)=4$, exactly the unpenalized coefficient. Reduction occurs for $\lambda>0$, not at zero.
Assuming units do not matter. A coefficient changes inversely with predictor units, and the penalty is applied to the coefficient. Scaling choices must therefore be considered when fitting and comparing ridge models.
Describing a coefficient as a causal effect without justification. A ridge coefficient is a penalized model estimate. It does not establish that intervening on a predictor would change the outcome.
Overinterpreting separate coefficients for overlapping predictors. When predictors are strongly related, their individual coefficient estimates can depend on the penalty and the other predictors in the model.

These limits do not make ridge unhelpful. Ridge can improve prediction stability, especially when predictors are correlated, but its purpose is not to certify variable importance or causality. Interpret the fitted model in light of the predictor scales, the penalty, the data design, and the predictive question being asked. These distinctions are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Key takeaway: Ridge penalizes coefficient magnitude; it does not generally perform automatic variable selection. Its coefficients depend on scaling and on other predictors, and they should not be interpreted as causal effects without additional justification.

Check Your Understanding

For each problem, consider what the ridge calculation does and does not establish.

An orthogonal centered predictor has $x^Tx=5$, $x^Ty=15$, and is fitted with $\lambda=5$. Find its unpenalized and ridge coefficients, and state whether ridge has removed it. (Answer: The unpenalized coefficient is $15/5=3$; the ridge coefficient is $15/(5+5)=1.5$, so it is shrunk but not removed.)
A centered one-predictor data set has $x^Tx=4$, $x^Ty=12$, and $\lambda=2$. Calculate the ridge coefficient, then calculate it at $\lambda=0$. (Answer: $12/(4+2)=2$ at $\lambda=2$, and $12/4=3$ at $\lambda=0$.)
For centered data, $x^Tx=2$, $x^Ty=6$, and $\lambda=2$. Replace $x$ by $z=4x$, keeping $\lambda=2$. Find the scaled coefficient and fitted values at the two observations where $x=(-1,1)^T$. (Answer: $z^Tz=32$, $z^Ty=24$, and $\widehat{\gamma}=24/(32+2)=12/17$; the fitted values are $(-48/17,48/17)^T$.)
Two identical centered predictors each equal $(-1,1)^T$, the response is $(-2,2)^T$, and $\lambda=2$. What ridge coefficients result, and what limitation should be noted? (Answer: The coefficients are $2/3$ and $2/3$, with fitted values $(-4/3,4/3)^T$; identical predictors do not allow their separate contributions to be distinguished from the data.)
A ridge model gives a small nonzero coefficient for a predictor. Is it correct to say that ridge selected the predictor out, or that changing the predictor causes the corresponding change in the response? (Answer: No. A nonzero coefficient means it remains in the fitted model; the coefficient alone establishes neither automatic removal nor a causal effect.)

What You'll Learn