Tutorials › Exam SRM - Statistics for Risk Modeling › Ridge With Correlated Predictors

Home  /  Tutorials  /  Exam SRM / Ridge and Lasso Regression

Exam SRM · Ridge and Lasso Regression · Tutorial 220 of 926

Ridge With Correlated Predictors

Work through correlated-predictor examples to see why ridge usually keeps both variables while shrinking their coefficients together, and how lasso can behave differently.

Advanced 11 min read

What You'll Learn

  • Set up ridge and lasso objectives for centered predictors with an unpenalized intercept
  • Distinguish predictor-predictor correlations from predictor-response inner products
  • Solve ridge coefficients when two predictors are highly correlated
  • Check lasso coefficient patterns using its optimality conditions
  • Explain how ridge moderates unstable coefficient contrasts
  • Recognize when correlation makes a one-variable interpretation unreliable

Why Correlated Predictors Need Special Care

When two predictors contain nearly the same information, many combinations of their coefficients can produce similar fitted values. Ordinary least squares can then assign large, opposing coefficients to the two predictors, even when the overall predictions are not extreme. Ridge regression addresses this instability by penalizing the squared sizes of coefficients. It generally retains both correlated predictors and shrinks their coefficients jointly; it does not usually choose one and set the other exactly to zero.

Lasso uses an absolute-value penalty instead. In some correlated-predictor problems it retains one predictor and removes another, though this is not a universal outcome. The examples below focus on what the coefficient calculations show, rather than treating either method as an automatic way to identify which correlated variable is the “true” one.

Model and convention: Consider the linear model \(y=\beta_0\mathbf{1}+X\beta+\varepsilon\). Center the response and predictor columns, so the intercept estimate is handled separately and is not penalized. For ridge, use the objective \(\frac{1}{2}\|y-X\beta\|^2+\frac{\lambda}{2}\|\beta\|^2\), where \(\lambda\geq 0\). For lasso, use \(\frac{1}{2}\|y-X\beta\|^2+\lambda\|\beta\|_1\). The examples report centered, scaled predictors with Euclidean norm 1. These are optimization definitions; a probability distribution for \(\varepsilon\) is not needed to calculate the penalized estimates.

With two predictors, write \(G=X^\mathsf{T}X\) for their Gram matrix and \(c=X^\mathsf{T}y\) for the vector of predictor-response inner products. The entries of \(c\) are not generally Pearson correlations: a Pearson correlation also divides by the relevant vector norms, with a consistent centering convention. If the two predictor columns are centered and have norm 1, their Gram-matrix off-diagonal entry is their Pearson correlation.

Ridge With Two Correlated Predictors

For two unit-norm predictor columns with correlation \(\rho\), the ridge normal equations are

$$ \begin{pmatrix} 1+\lambda & \rho\\ \rho & 1+\lambda \end{pmatrix} \begin{pmatrix} \hat{\beta}_1\\ \hat{\beta}_2 \end{pmatrix} = \begin{pmatrix} c_1\\ c_2 \end{pmatrix}. $$

The off-diagonal terms matter: each coefficient equation involves both predictors. When \(|\rho|\) is close to 1, changing one coefficient can be partly offset by changing the other, so solving the equations as though the predictors were independent gives the wrong answer.

Worked Example: A Nearly Duplicate Predictor Is Retained

A hypothetical claim-severity analysis uses two centered, unit-norm predictors with correlation \(0.98\). Let \(x_1\) be a centered vector with norm 1, and let \(z\) be a centered, unit-norm vector orthogonal to \(x_1\). Define \(x_2=0.98x_1+\sqrt{1-0.98^2}\,z\) and \(y=x_1\). Thus the predictors are nearly duplicates, and the inner products are \(c_1=x_1^\mathsf{T}y=1\) and \(c_2=x_2^\mathsf{T}y=0.98\). Find the ridge coefficients for \(\lambda=0.1\), then find the lasso coefficients for \(\lambda=0.1\).

Method: solve ridge’s coupled equations, then verify a lasso candidate using its optimality conditions. For ridge, the matrix equation is

$$ \begin{pmatrix}1.1&0.98\\0.98&1.1\end{pmatrix} \begin{pmatrix}\hat{\beta}_1\\\hat{\beta}_2\end{pmatrix} = \begin{pmatrix}1\\0.98\end{pmatrix}. $$

The determinant is \(1.1^2-0.98^2=1.21-0.9604=0.2496\). Inverting the matrix gives

$$ \hat{\beta}_1=\frac{1.1(1)-0.98(0.98)}{0.2496} =\frac{1.1-0.9604}{0.2496} =\frac{0.1396}{0.2496} =0.559294\ldots, $$ $$ \hat{\beta}_2=\frac{1.1(0.98)-0.98(1)}{0.2496} =\frac{1.078-0.98}{0.2496} =\frac{0.098}{0.2496} =0.392628\ldots. $$

Rounded to four decimal places, the ridge coefficients are \((0.5593,0.3926)\). Both are positive and nonzero: ridge has shrunk and retained both predictors.

For lasso, consider \((\hat{\beta}_1,\hat{\beta}_2)=(0.9,0)\). Its residual is \(r=y-0.9x_1=0.1x_1\). Therefore \(x_1^\mathsf{T}r=0.1\) and \(x_2^\mathsf{T}r=0.98(0.1)=0.098\). For a positive active coefficient, lasso requires \(x_1^\mathsf{T}r=\lambda\); for an inactive coefficient it requires \(|x_2^\mathsf{T}r|\leq\lambda\). Here \(0.1=0.1\) and \(0.098\leq0.1\), so both conditions hold. Convexity of the lasso objective makes this candidate a minimizer. Lasso drops \(x_2\) in this example.

Check: ridge satisfies its equations: \(1.1(0.559294\ldots)+0.98(0.392628\ldots)=1\), and \(0.98(0.559294\ldots)+1.1(0.392628\ldots)=0.98\). The different coefficient patterns are not a contradiction: the two penalties have different forms, and the lasso optimality conditions allow an inactive coefficient when its residual inner product is within the penalty threshold. The common mistake to avoid is calling \(c_1\) and \(c_2\) predictor-response correlations without checking the response norm.

Why Ridge Shrinks the Coefficient Contrast

The coupled equations have a useful sum-and-difference form. Adding them gives the equation for the coefficient sum; subtracting them gives the equation for the coefficient difference:

$$ \hat{\beta}_1+\hat{\beta}_2 =\frac{c_1+c_2}{1+\rho+\lambda}, \qquad \hat{\beta}_1-\hat{\beta}_2 =\frac{c_1-c_2}{1-\rho+\lambda}. $$

This follows directly because the sum of the two equations has coefficient \(1+\rho+\lambda\) on each coefficient, while their difference has coefficient \(1-\rho+\lambda\) on the contrast. When \(\rho\) is close to 1, the unpenalized contrast denominator \(1-\rho\) is small. Adding \(\lambda\) increases that denominator and reduces the magnitude of the difference that a given \(c_1-c_2\) produces. This is one precise sense in which ridge stabilizes correlated-predictor coefficients.

The result does not mean ridge must assign positive coefficients, or equal coefficients, in every problem. The signs and sizes still depend on \(c\). Nor does it mean ridge identifies which predictor is more important: when predictors are highly correlated, their separate contributions may not be well determined by the data.

Worked Example: Ridge Moderates an Unstable Contrast

Suppose centered, unit-norm predictors have correlation \(0.98\), and a centered response has inner products \(c_1=1\) and \(c_2=0.9\). Find the ordinary least squares coefficients and the ridge coefficients for \(\lambda=0.5\). The stated \(c_j\) values are inner products, not Pearson correlations.

Method: solve the unpenalized equations, then solve the ridge equations. The Gram matrix is \(\begin{pmatrix}1&0.98\\0.98&1\end{pmatrix}\), whose determinant is \(1-0.98^2=0.0396\). Ordinary least squares therefore gives

$$ \hat{\beta}_{1,\mathrm{OLS}} =\frac{1(1)-0.98(0.9)}{0.0396} =\frac{0.118}{0.0396} =2.979797\ldots, $$ $$ \hat{\beta}_{2,\mathrm{OLS}} =\frac{1(0.9)-0.98(1)}{0.0396} =\frac{-0.08}{0.0396} =-2.020202\ldots. $$

The large coefficients of opposite signs arise even though the two inner products are both positive and close together. For ridge, the diagonal entries become \(1+0.5=1.5\), and the determinant is \(1.5^2-0.98^2=2.25-0.9604=1.2896\). Hence

$$ \hat{\beta}_{1,\mathrm{ridge}} =\frac{1.5(1)-0.98(0.9)}{1.2896} =\frac{0.618}{1.2896} =0.479218\ldots, $$ $$ \hat{\beta}_{2,\mathrm{ridge}} =\frac{1.5(0.9)-0.98(1)}{1.2896} =\frac{0.37}{1.2896} =0.286910\ldots. $$

Rounded to four decimal places, the ridge estimates are \((0.4792,0.2869)\), while the ordinary least squares estimates are \((2.9798,-2.0202)\). As a check using the sum-and-difference result, the ridge sum is \(1.9/(1+0.98+0.5)=1.9/2.48=0.766129\ldots\), and its difference is \(0.1/(1-0.98+0.5)=0.1/0.52=0.192307\ldots\). These give \((0.766129\ldots+0.192307\ldots)/2=0.479218\ldots\) and \((0.766129\ldots-0.192307\ldots)/2=0.286911\ldots\), consistent up to rounding.

Common mistake avoided: correlation does not justify treating the predictors as independent. Ignoring the off-diagonal \(0.98\) would discard the source of the instability and would not solve either model’s coefficient equations.

Lasso and Ridge Need Not Split Credit the Same Way

The lasso’s absolute-value penalty can produce exact zeros, as the first worked example showed. But it does not always select just one member of a correlated pair. When the response relates similarly to both predictors, lasso may retain both. Ridge instead penalizes both coefficients smoothly and, except in special cases, does not produce exact zeros from its squared penalty.

Worked Example: Similar Associations Can Keep Both Lasso Coefficients

Let \(x_1\) and \(x_2\) be centered, unit-norm predictors with correlation \(0.98\), and take \(y=(x_1+x_2)/1.98\). Then \(x_1^\mathsf{T}y=x_2^\mathsf{T}y=1\). Find the ridge and lasso coefficients using \(\lambda=0.1\) for each objective.

Method: use symmetry for ridge, then verify the lasso residual conditions. By symmetry, the ridge coefficients are equal; write each as \(b\). Either ridge equation becomes \(1.1b+0.98b=1\). Thus \(2.08b=1\), so \(b=1/2.08=0.480769\ldots\). The ridge estimates, rounded to four decimals, are \((0.4808,0.4808)\).

For lasso, symmetry suggests equal positive coefficients \(b\). The residual inner product for either predictor is \(1-(1+0.98)b\). Setting this equal to \(\lambda=0.1\) gives \(1-1.98b=0.1\), so \(b=0.9/1.98=0.454545\ldots\). Both coefficients are positive, and both residual inner products equal the required threshold \(0.1\), so the lasso conditions are satisfied. The lasso estimates, rounded to four decimals, are \((0.4545,0.4545)\).

Check: the ridge equation gives \(2.08(0.480769\ldots)=1\); the lasso residual inner product is \(1-1.98(0.454545\ldots)=0.1\) for each predictor. The example shows why “lasso always chooses one of two correlated predictors” is an incorrect rule. Its behavior depends on the response associations and the penalty.

Interpreting the Coefficients

For highly correlated predictors, the data may support a stable prediction while providing weak information about how the prediction should be divided between the predictors. Ridge controls coefficient size and often stabilizes that division. This can be useful when a model should retain a group of related measurements, but the resulting coefficients should not be interpreted as isolated causal effects.

The most important calculation habits are to keep the Gram-matrix off-diagonal terms, state whether quantities are inner products or correlations, and verify proposed lasso zeros against the residual conditions. A small error in any of these steps can lead to a false claim about which predictor is retained.

Key takeaway: With correlated predictors, ridge solves a coupled system and typically shrinks their coefficients without setting either exactly to zero. Lasso can set one coefficient to zero, but it can also retain both. Neither coefficient pattern alone establishes that one variable is uniquely responsible for the response.

Check Your Understanding

For each problem, use the stated ridge and lasso objective conventions and distinguish inner products from correlations.

  1. Two centered, unit-norm predictors have correlation \(0.9\), and \(c=(1,0.9)\). For ridge with \(\lambda=0.1\), write the two normal equations. (Answer: \(1.1\hat{\beta}_1+0.9\hat{\beta}_2=1\) and \(0.9\hat{\beta}_1+1.1\hat{\beta}_2=0.9\).)
  2. For the predictors in Question 1, take \(y=x_1\) and lasso \(\lambda=0.1\). Verify that \((0.9,0)\) satisfies the lasso conditions. (Answer: \(x_1^\mathsf{T}r=0.1=\lambda\) and \(|x_2^\mathsf{T}r|=0.09\leq\lambda\).)
  3. Two centered, unit-norm predictors have correlation \(0.8\), with \(c_1=c_2=1\). Find the ridge coefficient on each predictor when \(\lambda=0.2\). (Answer: \(1/(1+0.8+0.2)=0.5\), so the coefficients are \((0.5,0.5)\).)
  4. For \(\rho=0.95\), \(c_1=1\), \(c_2=0.9\), and ridge \(\lambda=0.25\), calculate the coefficient sum and difference using the sum-and-difference formulas. (Answer: sum \(=1.9/2.2=0.8636\), difference \(=0.1/0.30=0.3333\), to four decimal places.)
  5. Explain why a lasso solution with both coefficients positive is possible when two predictors are correlated. (Answer: if each predictor’s residual inner product equals \(\lambda\) for a positive coefficient, both active-coefficient conditions hold; correlation alone does not require either coefficient to be zero.)

Previous

Next

Back to Tutorials
Clinical Biostats