Why Correlated Predictors Need Careful Comparison
When two predictors carry much of the same information, a fitted model may have difficulty assigning their shared contribution between them. Ridge and lasso both penalize coefficient size, but their penalties behave differently: ridge generally keeps both coefficients nonzero, while lasso can set one exactly to zero. With strong correlation, a small change in the response can affect how lasso divides the contribution between predictors.
The calculations below use centered predictors and a centered response, so the intercept is zero in the displayed fits. Each predictor is standardized to have mean square one. This scaling makes the penalty comparable across the two predictors. The examples use the following notation: \(G=X^\mathsf{T}X/n\) is the predictor cross-product matrix, and \(z=X^\mathsf{T}y/n\) is the vector of response cross-products. With two predictors, these are sufficient for the coefficient calculations.
The correlation \(\rho\) measures how similar the predictor columns are. If \(|\rho|\) is close to one, the columns are nearly redundant. This does not mean that their separate fitted coefficients must be close, or that a particular coefficient must move monotonically toward zero as the penalty changes.
Worked Example: Ridge Can Increase an Individual Coefficient
Worked Example: Ridge Can Increase an Individual Coefficient
Suppose \(\rho=0.90\) and \(z=(0.80,0.70)^\mathsf{T}\). Calculate the unpenalized least-squares fit and the ridge fit at \(\lambda=0.20\). Compare the coefficient changes, taking care not to assume that each individual coefficient is smaller under ridge.
Method. Least squares solves \(G\widehat\beta=z\). Ridge solves \((G+\lambda I)\widehat\beta^{\,R}=z\). Here \(G=\begin{pmatrix}1&0.90\\0.90&1\end{pmatrix}\). Its determinant is \(1-(0.90)^2=0.19\), so \(G^{-1}=(1/0.19)\begin{pmatrix}1&-0.90\\-0.90&1\end{pmatrix}\).
The unpenalized coefficients are \(\widehat\beta_1=(0.80-0.90(0.70))/0.19=0.17/0.19=0.8947\), and \(\widehat\beta_2=(0.70-0.90(0.80))/0.19=-0.02/0.19=-0.1053\).
For ridge, \(G+0.20I=\begin{pmatrix}1.20&0.90\\0.90&1.20\end{pmatrix}\). Its determinant is \(1.20^2-0.90^2=1.44-0.81=0.63\). Therefore, \(\widehat\beta^{\,R}_1=(1.20(0.80)-0.90(0.70))/0.63=(0.96-0.63)/0.63=0.5238\), and \(\widehat\beta^{\,R}_2=(1.20(0.70)-0.90(0.80))/0.63=(0.84-0.72)/0.63=0.1905\).
The first coefficient falls from \(0.8947\) to \(0.5238\), but the second rises from \(-0.1053\) to \(0.1905\). Its absolute value also rises. Ridge reduces the overall coefficient-vector length: the squared lengths are \(0.8947^2+(-0.1053)^2\approx0.8116\) and \(0.5238^2+0.1905^2\approx0.3107\). Thus the vector is smaller overall even though one component is larger.
Check. Multiplying the ridge matrix by the calculated coefficients gives first component \(1.20(0.5238)+0.90(0.1905)\approx0.8000\), and second component \(0.90(0.5238)+1.20(0.1905)\approx0.7000\), recovering \(z\). The trap is to interpret ridge shrinkage as a guarantee that every coordinate moves closer to zero. It is the coefficient vector, not each coordinate separately, that the penalty controls.
Why the Ridge Formula Holds
For the stated ridge objective, expand the squared error in \(\beta\), omitting the constant \(y^\mathsf{T}y/(2n)\). The terms that depend on \(\beta\) are \(\beta^\mathsf{T}G\beta/2-z^\mathsf{T}\beta+\lambda\beta^\mathsf{T}\beta/2\). Differentiating with respect to \(\beta\) gives \((G+\lambda I)\beta-z\). Setting this gradient to zero yields the ridge normal equations.
The inverse expression applies when \(G+\lambda I\) is invertible; for \(\lambda>0\), it is positive definite for any positive semidefinite \(G\). When \(\lambda=0\), the inverse requires \(G\) to be invertible. In the two-predictor standardized case with \(|\rho|<1\), the explicit ridge coefficients are
The terms involving the other predictor’s response cross-product explain why a component need not shrink individually. When the predictors are correlated, changing the penalty changes how the fitted signal is allocated between them. The formula also shows why the sign of a coefficient can change.
Lasso Optimality Conditions and Active Predictors
Unlike the differentiable ridge penalty, the absolute-value penalty has a corner at zero. A lasso fit is characterized by conditions on residual correlations. Let \(r=y-X\widehat\beta\). For a nonzero coefficient, the residual correlation with that predictor equals \(\lambda\) times the coefficient’s sign. For a zero coefficient, the absolute residual correlation cannot exceed \(\lambda\). These conditions both identify a candidate fit and provide a direct check of it.
To see why these conditions hold, consider changing one coefficient by a small amount. The derivative of the squared-error term in coordinate \(j\) is \(-x_j^\mathsf{T}r/n\). At a nonzero coefficient, the derivative of its penalty is \(\lambda\operatorname{sign}(\widehat\beta_j)\); the sum must be zero at a minimum. At zero, the absolute-value penalty has possible slopes from \(-\lambda\) to \(+\lambda\), so the squared-error derivative can be balanced only when the residual correlation lies within that interval.
Worked Example: Lasso Selects One of Two Correlated Predictors
Use the same \(G\) and \(z\) as in the ridge example, but fit lasso with \(\lambda=0.10\). Determine whether a fit with the first predictor active and the second set to zero satisfies the optimality conditions.
Method. If the second coefficient is zero and the first is positive, the active-coordinate condition gives \(\widehat\beta_1=z_1-\lambda=0.80-0.10=0.70\). Thus the proposed fit is \((0.70,0)^\mathsf{T}\), with positive first coefficient as assumed.
The first residual correlation is \(z_1-\widehat\beta_1=0.80-0.70=0.10\), exactly \(\lambda\), as required for a positive coefficient. The residual correlation for the inactive second predictor is \(z_2-\rho\widehat\beta_1=0.70-0.90(0.70)=0.70-0.63=0.07\). Since \(|0.07|\leq0.10\), the second coefficient can remain zero. The proposed fit satisfies both conditions.
Check. The active coefficient is positive, its residual correlation equals the penalty, and the inactive correlation is within the allowed interval. The lasso fit is therefore \((0.70,0)^\mathsf{T}\). The trap is to solve the ordinary least-squares equations and then simply set a small coefficient to zero: a lasso solution must satisfy the residual-correlation conditions after the remaining coefficients have been fitted.
Worked Example: A Small Response Change Alters Lasso Selection
Worked Example: A Small Response Change Alters Lasso Selection
Keep \(\rho=0.90\) and \(\lambda=0.10\), but suppose a small response perturbation changes the cross-product vector from \((0.80,0.70)^\mathsf{T}\) to \((0.80,0.731)^\mathsf{T}\). Find the lasso fit and compare its selected predictors with the preceding fit.
Method. First test the former active set, with predictor 1 active and predictor 2 zero. Its coefficient remains \(0.80-0.10=0.70\). The second predictor’s residual correlation is now \(0.731-0.90(0.70)=0.731-0.630=0.101\). Since \(0.101>0.100\), the inactive-coordinate condition fails. Predictor 2 must enter the active set.
Assume both coefficients are positive and solve the active-coordinate equations. They require \(G\widehat\beta=z-\lambda(1,1)^\mathsf{T}=(0.700,0.631)^\mathsf{T}\). The determinant of \(G\) is \(1-0.90^2=0.19\). Consequently, \(\widehat\beta_1=(0.700-0.90(0.631))/0.19=(0.700-0.5679)/0.19=0.1321/0.19=0.6953\), and \(\widehat\beta_2=(0.631-0.90(0.700))/0.19=(0.631-0.630)/0.19=0.001/0.19=0.0053\). Both coefficients are positive, so the assumed signs are consistent.
Check. The fitted cross-products are \(G\widehat\beta=(0.700,0.631)^\mathsf{T}\), so \(z-G\widehat\beta=(0.100,0.100)^\mathsf{T}\). Each active residual correlation equals \(\lambda\), as required. The second coefficient changes from exactly zero to approximately \(0.0053\), following a change of only \(0.031\) in the second response cross-product. This illustrates selection sensitivity near the lasso threshold; it does not imply that every small response change alters the selected set.
Interpreting the Comparison
In these examples, ridge retains both predictors, while lasso produces a sparse fit for the original response cross-products. That contrast is useful, but it is not a universal rule that lasso will always keep just one of a correlated pair. Depending on \(z\), \(\lambda\), and the signs of the associations, it may retain both, retain one, or assign a negative coefficient to one predictor. The optimality conditions, rather than a general story about which predictor “wins,” determine the answer.
A coefficient change and a prediction change are also different quantities. If two predictor columns are highly correlated, alternative coefficient allocations can produce similar fitted values. Conversely, similar-looking coefficient vectors do not guarantee identical predictions for every observation. Interpret coefficient selection in the context of the fitted model and the predictor relationships; do not treat a zero coefficient as proof that the associated variable has no relationship with the response.
The calculations here assume centered, equally scaled predictors and the stated penalty normalization. If variables are on different scales, the same numerical \(\lambda\) does not impose a comparable penalty on their original-scale coefficients. Also, if the predictor columns are exactly collinear, coefficients may not be uniquely identified by the unpenalized least-squares equations. Ridge with positive \(\lambda\) still has a unique coefficient vector, but lasso can have multiple coefficient solutions in some designs. For the examples above, \(|\rho|=0.90<1\), so the two predictor columns are not exactly collinear.
Check Your Understanding
For each problem, use the penalty conventions and standardized-predictor setup stated above. Check active-set signs and inactive-coordinate conditions where relevant.
- For ridge, let \(G=\begin{pmatrix}1&0.80\\0.80&1\end{pmatrix}\), \(z=(0.60,0.50)^\mathsf{T}\), and \(\lambda=0.20\). Calculate both coefficients. (Answer: \((0.4000,\,0.1500)^\mathsf{T}\).)
- For orthogonal predictors, let \(G=I\), \(z=(0.45,0.12)^\mathsf{T}\), and use lasso with \(\lambda=0.20\). State the fitted coefficients and selected predictor set. (Answer: \((0.25,\,0)^\mathsf{T}\); only predictor 1 is selected.)
- For lasso, let \(\rho=0.80\), \(z=(0.70,0.65)^\mathsf{T}\), and \(\lambda=0.10\). Test a fit with both coefficients positive, calculate it, and check that the signs are consistent. (Answer: \((0.4444,\,0.1944)^\mathsf{T}\); both coefficients are positive and selected.)
- For lasso, let \(\rho=0.90\), \(z=(0.80,0.74)^\mathsf{T}\), and \(\lambda=0.10\). Starting by testing predictor 1 alone, determine whether predictor 2 must enter and calculate the fit if both coefficients are positive. (Answer: predictor 2 must enter; \((0.6526,\,0.0526)^\mathsf{T}\).)
- With \(\rho=0.90\) and \(\lambda=0.10\), suppose \(z=(0.80,0.70)^\mathsf{T}\) changes to \((0.80,0.72)^\mathsf{T}\). Does the one-predictor lasso fit change its selected set? (Answer: no; the fit remains \((0.70,\,0)^\mathsf{T}\), since the inactive residual correlation is \(0.72-0.90(0.70)=0.09\leq0.10\).)