Worked Example: Stabilizing Correlated Risk Factors

Why Correlated Risk Factors Can Be Difficult to Interpret

Suppose an insurer is modeling a hypothetical claim-cost measure using two risk factors that tend to move together. Least squares can fit the observed data, but when predictors are strongly correlated, it may be difficult to determine which factor should receive credit for the fitted relationship. Small changes in the data can produce substantial changes in the separate coefficients, even when predictions for similar profiles change little.

Ridge regression addresses this instability by adding a penalty on coefficient size. This tutorial compares the least-squares and ridge calculations for a small numerical example, examines what happens after a modest change in the data, and calculates fitted values for hypothetical risk profiles. The focus is on understanding the calculations and their implications, not on choosing a penalty through model validation.

Definition: For a response vector $y$, a centered predictor matrix $X$, and a nonnegative tuning parameter $\lambda$, ridge regression minimizes the residual sum of squares plus a penalty on the squared predictor coefficients: $$ \sum_{i=1}^{n}(y_i-\beta_0-x_{i1}\beta_1-\cdots-x_{ip}\beta_p)^2 +\lambda\sum_{j=1}^{p}\beta_j^2. $$ The intercept $\beta_0$ is not penalized. With centered predictors and a centered response, the predictor-coefficient estimate is $\hat{\boldsymbol{\beta}}_{\text{ridge}}=(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}y$, when the inverse exists. At $\lambda=0$, this is the least-squares solution when that solution is unique.

The numerical examples use two centered predictors with equal scaling. Their cross-product matrix is written $X^\mathsf{T}X$, and $X^\mathsf{T}y$ records each predictor’s cross-product with the centered response. The diagonal entries of $X^\mathsf{T}X$ are the predictors’ sums of squares; their off-diagonal entry is their cross-product. For two predictors with equal sums of squares, a large positive off-diagonal entry relative to the diagonal indicates strong positive correlation.

The displayed penalty convention is important: $\lambda$ is added to the diagonal of $X^\mathsf{T}X$. Some software and texts scale the residual sum of squares or penalty differently, so their numerical value of $\lambda$ need not match the one used here. In practice, the predictors’ scaling and the penalty convention must be consistent. For the calculations below, take the stated cross-products and $\lambda$ as given.

Worked Coefficients for Correlated Predictors

Worked Example: Comparing Least Squares and Ridge

A hypothetical pricing analyst fits a centered response using two centered risk factors. The supplied cross-products are $X^\mathsf{T}X=\begin{pmatrix}10&9\\9&10\end{pmatrix}$ and $X^\mathsf{T}y=\begin{pmatrix}10\\9\end{pmatrix}$. The predictors have correlation $9/10=0.9$. Calculate the least-squares coefficients and the ridge coefficients for $\lambda=1$. Then compare the predictions for a centered profile with predictor values $(1,1)$.

Method. Solve the least-squares normal equations for the first estimate. For ridge, add $\lambda=1$ to each diagonal entry of $X^\mathsf{T}X$, then solve the penalized normal equations.

For least squares, the determinant of the cross-product matrix is $10(10)-9(9)=100-81=19$. Its inverse is $\frac{1}{19}\begin{pmatrix}10&-9\\-9&10\end{pmatrix}$. Therefore, $$ \hat{\boldsymbol{\beta}}_{\text{LS}} =\frac{1}{19} \begin{pmatrix}10&-9\\-9&10\end{pmatrix} \begin{pmatrix}10\\9\end{pmatrix} =\frac{1}{19}\begin{pmatrix}100-81\\-90+90\end{pmatrix} =\begin{pmatrix}1\\0\end{pmatrix}. $$

For ridge, the penalized matrix and its determinant are $\begin{pmatrix}11&9\\9&11\end{pmatrix}$ and $11(11)-9(9)=121-81=40$. Thus, $$ \hat{\boldsymbol{\beta}}_{\text{ridge}} =\frac{1}{40} \begin{pmatrix}11&-9\\-9&11\end{pmatrix} \begin{pmatrix}10\\9\end{pmatrix} =\frac{1}{40}\begin{pmatrix}110-81\\-90+99\end{pmatrix} =\begin{pmatrix}0.725\\0.225\end{pmatrix}. $$

At the profile $(1,1)$, the fitted centered response from least squares is $1(1)+0(1)=1$. The ridge fitted centered response is $0.725(1)+0.225(1)=0.950$. The two estimates give similar predictions for this profile, even though they assign the least-squares effect entirely to the first factor and the ridge effect to both factors.

Check. Multiplying the ridge matrix by its estimate gives $11(0.725)+9(0.225)=7.975+2.025=10$ and $9(0.725)+11(0.225)=6.525+2.475=9$, the entries of $X^\mathsf{T}y$. The common mistake to avoid is treating the two coefficient estimates as if they described separate, unambiguous effects despite the predictors’ correlation.

Why Ridge Shrinks the Correlated Directions Differently

A short eigenvalue argument explains the coefficient change in the example. For a symmetric cross-product matrix, consider a direction $u$ satisfying $X^\mathsf{T}Xu=d u$, where $d$ is its eigenvalue. In that direction, the least-squares coefficient component is multiplied by $d/(d+\lambda)$ under ridge. This follows by applying the ridge operator to $u$:

(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}X u =(X^\mathsf{T}X+\lambda I)^{-1}d u =\frac{d}{d+\lambda}u.

The equality holds because $(X^\mathsf{T}X+\lambda I)u=(d+\lambda)u$. Thus, directions with smaller eigenvalues are shrunk more for the same positive $\lambda$. When two predictors are strongly correlated, the difference between their coefficients corresponds to a direction with relatively little information in the data.

In the example, the eigenvectors are $(1,1)$, the common direction, and $(1,-1)$, the contrast direction. Their eigenvalues are $19$ and $1$, respectively. With $\lambda=1$, the ridge multipliers are $19/20=0.95$ for the common direction and $1/2=0.5$ for the contrast direction. The least-squares estimate $(1,0)$ is half common direction and half contrast direction: $(1,0)=0.5(1,1)+0.5(1,-1)$. Shrinking these components gives $0.475(1,1)+0.25(1,-1)=(0.725,0.225)$, agreeing with the direct calculation.

This derivation explains why ridge can stabilize coefficients when predictors are correlated: it shrinks more strongly in directions that the observed predictors distinguish poorly. It does not make each coefficient a causal effect, nor does it establish that a particular $\lambda$ is appropriate. Those are separate questions.

How Sensitive Are the Separate Coefficients?

Worked Example: A Small Change in the Response Cross-Products

Keep the same two-predictor matrix $X^\mathsf{T}X=\begin{pmatrix}10&9\\9&10\end{pmatrix}$, but suppose a small change in the observed response produces $X^\mathsf{T}y=\begin{pmatrix}10\\9.2\end{pmatrix}$. Find the least-squares and ridge coefficients for $\lambda=1$, and compare them with the estimates from the first example.

Method. Reuse the inverse matrices already calculated, because $X^\mathsf{T}X$ has not changed. Only the response cross-products have changed.

For least squares, $$ \hat{\boldsymbol{\beta}}_{\text{LS}} =\frac{1}{19} \begin{pmatrix}10&-9\\-9&10\end{pmatrix} \begin{pmatrix}10\\9.2\end{pmatrix} =\frac{1}{19}\begin{pmatrix}100-82.8\\-90+92\end{pmatrix} =\begin{pmatrix}17.2/19\\2/19\end{pmatrix} \approx\begin{pmatrix}0.9053\\0.1053\end{pmatrix}. $$ The response cross-product for the second predictor changed by just $9.2-9=0.2$. Yet the second least-squares coefficient changed from $0$ to $2/19$, while the first changed from $1$ to $17.2/19$.

For ridge, the penalized matrix is unchanged, so $$ \hat{\boldsymbol{\beta}}_{\text{ridge}} =\frac{1}{40} \begin{pmatrix}11&-9\\-9&11\end{pmatrix} \begin{pmatrix}10\\9.2\end{pmatrix} =\frac{1}{40}\begin{pmatrix}110-82.8\\-90+101.2\end{pmatrix} =\begin{pmatrix}0.680\\0.280\end{pmatrix}. $$ Compared with the first ridge estimate $(0.725,0.225)$, the two coefficients change by $-0.045$ and $+0.055$. The coefficient difference changes from $0.725-0.225=0.500$ to $0.680-0.280=0.400$, whereas the least-squares difference changes from $1$ to $17.2/19-2/19=15.2/19=0.8$.

Check. Multiplication by the ridge matrix gives $11(0.680)+9(0.280)=10$ and $9(0.680)+11(0.280)=9.2$. Ridge has reduced the change in the contrast between coefficients in this calculation. It has not made the estimates identical or eliminated the need to examine the model’s predictive performance.

From Coefficients to Fitted Predictions

Coefficients are not the final output when the objective is prediction. For a profile $x=(x_1,x_2)$, a fitted value is $\hat{y}=\hat{\beta}_0+x_1\hat{\beta}_1+x_2\hat{\beta}_2$. If the response and predictors were centered before fitting, $\hat{\beta}_0$ is the response mean when the predictors’ means are zero. With a centered response, the centered fitted value is simply $x_1\hat{\beta}_1+x_2\hat{\beta}_2$. Ridge does not penalize the intercept.

Worked Example: Comparing Predictions Across Risk Profiles

A hypothetical analyst uses the first example’s coefficient estimates and has an unpenalized intercept of $100$ response units. Calculate predictions for three centered risk profiles: $A=(1,1)$, $B=(1,-1)$, and $C=(0,2)$, using both least squares and ridge.

Method. For each profile, substitute its two predictor values into the fitted-value equation. Use the least-squares estimate $(1,0)$ and the ridge estimate $(0.725,0.225)$, and add the intercept of $100$ in each case.

For profile $A=(1,1)$, the least-squares prediction is $100+1(1)+0(1)=101$. The ridge prediction is $100+0.725(1)+0.225(1)=100+0.950=100.950$.

For profile $B=(1,-1)$, the least-squares prediction is $100+1(1)+0(-1)=101$. The ridge prediction is $100+0.725(1)+0.225(-1)=100+0.725-0.225=100.500$.

For profile $C=(0,2)$, the least-squares prediction is $100+1(0)+0(2)=100$. The ridge prediction is $100+0.725(0)+0.225(2)=100+0.450=100.450$.

The predictions for $A$, where both factors rise together, are close: $101$ and $100.950$. The predictions for profiles that emphasize a contrast between the factors differ more. This is consistent with the eigenvalue explanation: the data contain less information about the contrast direction, so ridge shrinks that direction more.

Check. At $A$, the ridge predictor contribution is the sum of its coefficients, $0.725+0.225=0.950$. At $B$, it is their difference, $0.725-0.225=0.500$. The calculated predictions therefore match the common and contrast directions, respectively. The common mistake is to omit the intercept when reporting predictions on the original response scale.

Common Scaling Trap

The ridge calculation must use a consistent matrix and penalty convention. In the examples, the predictor cross-products have diagonal entries $10$, the cross-product is $9$, and $\lambda=1$ is added to the diagonal of that cross-product matrix. A tempting but incorrect shortcut is to replace $X^\mathsf{T}X$ with the correlation matrix $\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}$ while leaving $X^\mathsf{T}y=(10,9)^\mathsf{T}$ unchanged. These quantities are on different scales.

That incorrect calculation would produce $\begin{pmatrix}2&0.9\\0.9&2\end{pmatrix}^{-1}(10,9)^\mathsf{T}$. Its determinant is $4-0.81=3.19$, giving coefficients $((20-8.1)/3.19,(18-10)/3.19)\approx(3.7304,2.5078)$. These are not the ridge estimates for the stated data and penalty. The correction is to use the original cross-product matrix, producing $(0.725,0.225)$, or to transform the full problem consistently if using a different scaling. A change of predictor scale also changes the meaning of a penalty on squared coefficients.

Key takeaway: With strongly correlated predictors, least-squares coefficients can allocate a relationship sharply between factors, while ridge shrinks the weakly identified coefficient directions more heavily. Compare fitted predictions as well as coefficients, and keep predictor scaling, response cross-products, intercept treatment, and the ridge penalty convention consistent.

Check Your Understanding

For each problem, use the ridge convention in this tutorial: add $\lambda$ to the diagonal of $X^\mathsf{T}X$, and do not penalize the intercept.

Let $X^\mathsf{T}X=\begin{pmatrix}5&4\\4&5\end{pmatrix}$ and $X^\mathsf{T}y=(9,8)^\mathsf{T}$. Find the least-squares coefficients and the ridge coefficients for $\lambda=1$. (Answer: Least squares $(13/9,4/9)$; ridge $(1.10,0.60)$.)
For the matrix in Question 1, change the response cross-products to $(9,8.2)^\mathsf{T}$. Find the least-squares coefficients. (Answer: $(14.2/9,4/9)$, approximately $(1.5778,0.4444)$.)
A ridge model has intercept $50$, coefficients $(0.7,0.3)$, and a profile $(2,-1)$. Calculate the fitted value. (Answer: $51.10$.)
Why is it incorrect to combine a predictor correlation matrix with response cross-products computed from unscaled predictors? (Answer: The matrix and response cross-products are on inconsistent scales, so the resulting coefficients are not the solution to the stated ridge objective.)
Let $X^\mathsf{T}X=\begin{pmatrix}8&6\\6&8\end{pmatrix}$, $X^\mathsf{T}y=(14,10)^\mathsf{T}$, and $\lambda=2$. Find the least-squares and ridge predictions for profile $(1,1)$ with intercept $20$. (Answer: Least squares $20+12/7\approx21.7143$; ridge $21.50$.)

What You'll Learn