Ridge Fits With Correlated Predictors

Why Correlated Predictors Make Coefficients Unstable

When two predictors carry nearly the same information, a regression can often fit the observed responses well using many different allocations of coefficient size between them. Ordinary least squares (OLS) chooses the allocation that minimizes residual sum of squares, but small changes in the data can shift that allocation substantially. Ridge regression adds a penalty on coefficient size. For correlated predictors, this penalty often shares coefficient size more evenly and dampens changes in the direction that distinguishes the predictors from each other.

That coefficient stabilization does not necessarily mean a dramatic change in fitted values. If the predictors move together, changing their coefficients in opposite directions can have little effect on the combined prediction. This tutorial works through that distinction using a pair of strongly correlated predictors.

Model and Calculation Setup

Consider a linear regression with two predictors. The predictors are centered and equally scaled, and the intercept is not penalized. Equivalently, the calculations below can be performed after removing the intercept by centering the response and predictor columns. Write \(X\) for the two-column design matrix and \(y\) for the centered response vector.

Definition: For a fixed nonnegative tuning parameter \(\lambda\), the ridge coefficient estimate minimizes the residual sum of squares plus a squared-coefficient penalty:

\hat{\boldsymbol{\beta}}_{\lambda} =\underset{\boldsymbol{\beta}}{\operatorname{argmin}} \left\{(y-X\boldsymbol{\beta})^{\mathsf T}(y-X\boldsymbol{\beta}) +\lambda\boldsymbol{\beta}^{\mathsf T}\boldsymbol{\beta}\right\}.

Here \(\lambda=0\) gives OLS when the OLS coefficients are uniquely determined. Positive \(\lambda\) penalizes large coefficient vectors; the intercept, if included, is left outside this penalty. The predictor scaling matters because the penalty acts on coefficient magnitudes.

For the numerical examples, use the following predictor cross-products:

X^{\mathsf T}X= \begin{pmatrix} 10&9\\ 9&10 \end{pmatrix}.

Each predictor has squared length 10, and their correlation is \(9/10=0.9\). The high correlation means that the predictors have a strong common direction and a much weaker contrast direction. In coefficient space, the common direction is proportional to \((1,1)\): both coefficients change together. The contrast direction is proportional to \((1,-1)\): one coefficient rises as the other falls.

The matrix has eigenvalue 19 in the common direction and eigenvalue 1 in the contrast direction. Thus, the data distinguish changes in the contrast direction much less strongly. For response cross-products \(X^{\mathsf T}y=(q_1,q_2)^{\mathsf T}\), the ridge normal equations are

(X^{\mathsf T}X+\lambda I)\hat{\boldsymbol{\beta}}_{\lambda} =X^{\mathsf T}y.

In the common and contrast directions, the equation divides the corresponding response cross-product by \(19+\lambda\) and \(1+\lambda\), respectively. OLS divides by 19 and 1. Consequently, for a fixed positive \(\lambda\), ridge makes a proportionally larger change in the weak contrast direction. That is the key reason ridge can stabilize the allocation between correlated predictors.

Worked Example: A Shared Signal and Similar Coefficients

Suppose a hypothetical centered data set has the predictor cross-products above, \(X^{\mathsf T}y=(10,10)^{\mathsf T}\), and \(y^{\mathsf T}y=11\). Compare OLS with ridge at \(\lambda=1\). These summaries are compatible with an observed data set; the calculations use the cross-products needed for fitting.

For OLS, solve \(X^{\mathsf T}X\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=X^{\mathsf T}y\). Because the two entries on the right are equal, both coefficients are equal:

\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \begin{pmatrix}10&9\\9&10\end{pmatrix}^{-1} \begin{pmatrix}10\\10\end{pmatrix} = \begin{pmatrix}10/19\\10/19\end{pmatrix} \approx \begin{pmatrix}0.5263\\0.5263\end{pmatrix}.

For ridge, add 1 to each diagonal entry of the cross-product matrix:

\hat{\boldsymbol{\beta}}_{1} = \begin{pmatrix}11&9\\9&11\end{pmatrix}^{-1} \begin{pmatrix}10\\10\end{pmatrix} = \begin{pmatrix}0.5000\\0.5000\end{pmatrix}.

Both methods assign equal coefficients, as the response has equal cross-products with the predictors. Ridge shrinks each coefficient from 0.5263 to 0.5000. The fitted values are not identical: the OLS fitted component has squared length \(\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}^{\mathsf T}X^{\mathsf T}X\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=10.5263\), whereas the ridge fitted component has squared length \(\hat{\boldsymbol{\beta}}_{1}^{\mathsf T}X^{\mathsf T}X\hat{\boldsymbol{\beta}}_{1}=9.500\). The penalty changes the fit as well as the coefficients, but it does not favor one of these equally correlated predictors over the other in this symmetric example.

How Ridge Changes the Allocation

To see what happens when the response has slightly different cross-products with the two predictors, write a coefficient pair as \((a+b,a-b)\). Here \(a\) is the shared coefficient size, while \(b\) measures the difference between the coefficients. A positive \(b\) gives the first predictor the larger coefficient; a negative \(b\) gives the second predictor the larger one.

The predictor cross-products make the squared fitted length particularly easy to express in these coordinates:

\begin{pmatrix}a+b&a-b\end{pmatrix} \begin{pmatrix}10&9\\9&10\end{pmatrix} \begin{pmatrix}a+b\\a-b\end{pmatrix} =38a^2+2b^2.

The contrast coefficient \(b\) has a much smaller data-supported direction than the common coefficient \(a\): its associated eigenvalue is 1 rather than 19. Ridge adds \(\lambda\) to both eigenvalues. It therefore reduces the contrast response by the factor \(1/(1+\lambda)\), while reducing the common response by \(19/(19+\lambda)\). For positive \(\lambda\), the contrast factor is smaller.

Worked Example: Ridge Shrinks the Coefficient Difference

Now suppose the hypothetical response cross-products are \(X^{\mathsf T}y=(10.1,9.9)^{\mathsf T}\), with the same predictor cross-products and \(y^{\mathsf T}y=11\). The slight imbalance in response cross-products favors the first predictor. Compare OLS and ridge at \(\lambda=1\).

The common and contrast components of the cross-products are \(q_1+q_2=20\) and \(q_1-q_2=0.2\). OLS divides these components by the eigenvalues 19 and 1. Thus \(a=20/(2\cdot19)=0.5263\) and \(b=0.2/(2\cdot1)=0.1000\). The coefficients are:

\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \begin{pmatrix}a+b\\a-b\end{pmatrix} = \begin{pmatrix}0.6263\\0.4263\end{pmatrix}.

For ridge at \(\lambda=1\), the corresponding denominators are 20 and 2. Therefore \(a=20/(2\cdot20)=0.5000\) and \(b=0.2/(2\cdot2)=0.0500\), giving:

\hat{\boldsymbol{\beta}}_{1} = \begin{pmatrix}0.5500\\0.4500\end{pmatrix}.

The coefficient difference is 0.2000 under OLS and 0.1000 under ridge. Ridge has halved the difference, so the two correlated predictors receive more similar coefficients. It has not set either coefficient to zero. This is an important contrast with methods designed to produce zero coefficients; ridge shrinks coefficient values but generally retains predictors with nonzero estimates.

The fits remain close in this example. Their difference has coefficient changes \((0.0763,-0.0237)\); using the predictor cross-products, the squared length of the difference in fitted vectors is \(10(0.0763)^2+18(0.0763)(-0.0237)+10(-0.0237)^2\approx0.0313\). Thus the coefficient allocation changes visibly while the difference between the fitted vectors is modest. This happens because much of the coefficient change is in the weak contrast direction.

Coefficient Stability and Fitted-Value Stability

The following calculation isolates the stability issue. Keep the predictors fixed and compare two hypothetical response vectors with \(y^{\mathsf T}y=11\), but exchange their response cross-products: \(X^{\mathsf T}y_A=(10.1,9.9)^{\mathsf T}\) and \(X^{\mathsf T}y_B=(9.9,10.1)^{\mathsf T}\). This small reversal can occur when the observed response changes slightly. The equal response lengths are compatible with both sets of cross-products.

Worked Example: A Small Data Change, a Smaller Ridge Change

For response \(y_A\), the preceding calculation gives OLS coefficients \((0.6263,0.4263)\) and ridge coefficients \((0.5500,0.4500)\). Exchanging the cross-products reverses the contrast. The common component remains \(a=0.5263\) for OLS and \(a=0.5000\) for ridge, but the contrast component changes sign:

\hat{\boldsymbol{\beta}}_{\mathrm{OLS},B} = \begin{pmatrix}0.4263\\0.6263\end{pmatrix}, \qquad \hat{\boldsymbol{\beta}}_{1,B} = \begin{pmatrix}0.4500\\0.5500\end{pmatrix}.

From response \(y_A\) to \(y_B\), each OLS coefficient changes by 0.2000 in absolute value. Each ridge coefficient changes by 0.1000. For the fitted vectors, the OLS change is \(0.2(x_1-x_2)\), where \(x_1\) and \(x_2\) are the predictor columns. Since \((x_1-x_2)^{\mathsf T}(x_1-x_2)=10+10-2(9)=2\), the squared length of this fitted-vector change is \(0.2^2\cdot2=0.0800\). Under ridge, the fitted-vector change is \(0.1(x_1-x_2)\), with squared length \(0.1^2\cdot2=0.0200\).

In this comparison, ridge reduces both the coefficient changes and the fitted-vector change. More generally, correlated predictors can produce large coefficient changes with comparatively small prediction changes: opposing coefficient movements partly cancel when the predictors themselves are similar. Ridge dampens that unstable contrast, but a particular numerical example does not guarantee that every fitted value changes less under ridge than under OLS.

Why the Distinction Matters

A regression coefficient describes the fitted change associated with a predictor while holding the other predictors fixed. When two predictors are highly correlated, there may be limited information in the data about what happens when one changes while the other is held fixed. Their separate coefficient estimates can consequently be sensitive to small changes in the observed responses, even when the fitted combination of their effects is stable.

Ridge addresses this numerical and statistical difficulty by penalizing coefficient size, including size in the weak contrast direction. It can give more stable coefficient estimates and predictions across modest data perturbations, at the cost of shrinking coefficients and introducing bias. It does not establish which correlated predictor is scientifically important, and it does not turn an association into a causal effect. Coefficients depend on predictor scaling, the value of \(\lambda\), and the other predictors in the model.

Key takeaway: With strongly correlated predictors, ridge often reduces coefficient differences by shrinking the weak contrast direction. The individual coefficients can be unstable under OLS even when their combined fitted contribution is comparatively stable. Ridge changes the fit too, so assess coefficients and predictions as distinct quantities.

Common Mistake

Wrong reasoning: “The OLS coefficients change a lot when two correlated predictors are exchanged, so the fitted values must also change a lot.”
Correction: Coefficients are weights on predictor columns, not fitted values by themselves. When columns are similar, opposite changes in their weights can largely cancel in the prediction. Compare \(X\hat{\boldsymbol{\beta}}\), or an appropriate prediction measure, rather than inferring prediction instability from coefficient changes alone.
Another pitfall: “Ridge makes the two coefficients equal whenever predictors are correlated.” Equality arose in the symmetric example because the response cross-products were equal. In general ridge shrinks the contrast; it does not necessarily eliminate it.

Check Your Understanding

Use the distinction between shared and contrast directions to answer these questions.

For the predictor cross-product matrix in this tutorial, which coefficient direction has the smaller eigenvalue, and what does that imply about its support in the data?
With \(\lambda=1\), what happens to the common and contrast response components when solving the ridge equations?
In the example with response cross-products \((10.1,9.9)\), why does ridge make the coefficient estimates more similar?
Why can OLS coefficients change noticeably when predictors are correlated even if fitted values change only modestly?
Does ridge generally set one coefficient in a correlated pair exactly to zero? Explain.