Tutorials › Exam PA - Predictive Analytics › Correlated Predictors Along Penalized Coefficient Paths

Home  /  Tutorials  /  Exam PA / Tuning and Regularized Model Use

Exam PA · Tuning and Regularized Model Use · Tutorial 472 of 966

Correlated Predictors Along Penalized Coefficient Paths

Compare how ridge, lasso, and elastic net allocate fitted signal among correlated, standardized predictors.

Advanced 12 min read

What You'll Learn

  • Explain why correlated predictors can have similar predictive value but different fitted coefficients.
  • Calculate ridge coefficients for a standardized pair of correlated predictors.
  • Use lasso optimality conditions to check whether a substitute predictor is excluded.
  • Calculate elastic-net coefficients when two correlated predictors are both active.
  • Interpret coefficient paths without assuming that each coefficient changes monotonically.
  • Identify when a selected predictor should not be treated as the uniquely important one.

Why Correlation Changes a Coefficient Path

When two predictors carry much the same information, a fitted model can often achieve similar predictions using either one or both. The coefficient path—the fitted coefficients as the penalty strength changes—then reflects not only the outcome signal but also how the penalty allocates that signal among substitutes. This is why a lasso fit may retain one of two nearly interchangeable predictors, while ridge regression tends to distribute the fitted effect across them.

The calculations below use a centered response and centered, standardized predictors. Standardization matters because otherwise different predictor scales change the effective penalty; the earlier material on predictor standardization explains this issue. The intercept is not penalized. To keep the algebra explicit, use the squared-error objective

$$ Q(\boldsymbol{\beta})= \frac{1}{2}\|y-X\boldsymbol{\beta}\|^2+ \lambda\left\{\alpha\sum_{j=1}^p|\beta_j| +\frac{1-\alpha}{2}\sum_{j=1}^p\beta_j^2\right\}. $$

Here \(X\) is the centered design matrix, \(y\) is the centered response, \(\boldsymbol{\beta}\) is the coefficient vector, \(\lambda\geq 0\) is the penalty strength, and \(0\leq\alpha\leq1\) determines the penalty type. With this scaling, \(\alpha=1\) gives lasso and \(\alpha=0\) gives ridge. Elastic net uses a value strictly between zero and one. A software package can use a different scaling of the loss or penalty, so numerical values of \(\lambda\) should be interpreted under the stated objective.

Definition: Two predictors are correlated when their observed values tend to vary together. In the two-predictor calculations below, each predictor has squared length 1, and their inner product is \(\rho\), so the Gram matrix is \(X^\top X=\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix}\). The vector \(z=X^\top y\) contains each predictor’s unpenalized association with the response. A large \(|\rho|\) means the predictors contain overlapping information; it does not, by itself, establish which predictor is more important.

Ridge Shares Signal Through the Correlation Structure

For ridge, \(\alpha=0\). Differentiating the squared-error-plus-quadratic-penalty objective and setting the gradient to zero gives

$$ (X^\top X+\lambda I)\widehat{\boldsymbol{\beta}}=X^\top y, \qquad \widehat{\boldsymbol{\beta}}=(X^\top X+\lambda I)^{-1}z. $$

The ridge penalty adds \(\lambda\) to the diagonal of the Gram matrix. For two standardized predictors this produces the system

$$ \begin{pmatrix}1+\lambda&\rho\\\rho&1+\lambda\end{pmatrix} \begin{pmatrix}\widehat\beta_1\\\widehat\beta_2\end{pmatrix} = \begin{pmatrix}z_1\\z_2\end{pmatrix}. $$

When the predictors are positively correlated and both have similar positive associations with the response, each coefficient contributes to a shared fitted signal. Ridge shrinks both coefficients, but does not generally set either coefficient exactly to zero at a finite penalty. The same logic applies with signs reversed when the predictors are negatively correlated.

Worked Example: Ridge Shares Signal Between Correlated Predictors

Suppose two standardized predictors have correlation \(\rho=0.8\), their response inner products are \(z_1=1.0\) and \(z_2=0.9\), and the ridge penalty is \(\lambda=0.5\). Calculate the coefficients and verify the system.

Method. For ridge, add \(\lambda=0.5\) to each diagonal of the Gram matrix. The coefficient system is

$$ \begin{pmatrix}1.5&0.8\\0.8&1.5\end{pmatrix} \begin{pmatrix}\widehat\beta_1\\\widehat\beta_2\end{pmatrix} = \begin{pmatrix}1.0\\0.9\end{pmatrix}. $$

The determinant is \(1.5(1.5)-0.8(0.8)=2.25-0.64=1.61\). Therefore

$$ \widehat\beta_1=\frac{1.5(1.0)-0.8(0.9)}{1.61} =\frac{1.50-0.72}{1.61} =\frac{0.78}{1.61}=0.4845, $$
$$ \widehat\beta_2=\frac{1.5(0.9)-0.8(1.0)}{1.61} =\frac{1.35-0.80}{1.61} =\frac{0.55}{1.61}=0.3416. $$

The coefficients are reported to four decimal places. Checking the unrounded values in the system gives \(1.5(0.484472)+0.8(0.341615)\approx1.0000\) and \(0.8(0.484472)+1.5(0.341615)\approx0.9000\). Both predictors receive positive coefficients, and together their fitted contribution is shared rather than assigned entirely to one predictor.

Common mistake avoided: treating the two predictors as independent and solving each coefficient separately ignores the off-diagonal correlation terms. Here the \(0.8\widehat\beta_j\) terms are part of both equations.

Why Lasso Can Keep One Substitute and Drop the Other

Lasso adds an absolute-value penalty. Unlike ridge, it can set coefficients exactly to zero. For the two-predictor objective with \(\alpha=1\), a nonzero positive coefficient must satisfy the usual derivative equation with a penalty subtraction of \(\lambda\). A coefficient proposed to be zero must instead satisfy an inequality: the remaining association between that predictor and the residual must be no greater than \(\lambda\) in absolute value. This inequality is the key check when deciding whether a one-predictor solution is valid.

Result: For a standardized two-predictor lasso fit, suppose a candidate has \(\widehat\beta_1>0\) and \(\widehat\beta_2=0\). It satisfies the lasso optimality conditions when \(\widehat\beta_1=z_1-\lambda>0\) and \(|z_2-\rho\widehat\beta_1|\leq\lambda\). The second condition checks whether predictor 2 has enough residual association to overcome the penalty.

To see why the condition holds, let the residual be \(r=y-X\boldsymbol{\beta}\). With \(\beta_2=0\), the derivative of the squared-error part with respect to \(\beta_1\) is \(-x_1^\top r\). For a positive \(\beta_1\), the derivative of the penalty is \(+\lambda\), so the active-coordinate equation is \(x_1^\top r=\lambda\). Since \(x_1^\top r=z_1-\beta_1\), this gives \(\beta_1=z_1-\lambda\). At a zero coefficient, the absolute-value penalty has a range of possible slopes from \(-\lambda\) to \(+\lambda\); optimality requires \(|x_2^\top r|\leq\lambda\). Here \(x_2^\top r=z_2-\rho\beta_1\), giving the stated inequality. If that inequality fails, the proposed zero coefficient is not optimal.

Worked Example: Checking Whether Lasso Selects One Substitute

Two standardized predictors have correlation \(\rho=0.95\), response inner products \(z_1=0.90\) and \(z_2=0.85\), and lasso penalty \(\lambda=0.20\). Determine whether a solution using predictor 1 alone satisfies the lasso conditions.

Method. If predictor 2 is excluded, the positive active coefficient is \(\widehat\beta_1=z_1-\lambda=0.90-0.20=0.70\). It is positive, as required. The residual association for predictor 2 is

$$ z_2-\rho\widehat\beta_1 =0.85-0.95(0.70) =0.85-0.665 =0.185. $$

Its absolute value is \(0.185\), which is less than or equal to \(\lambda=0.20\). Therefore the candidate \((\widehat\beta_1,\widehat\beta_2)=(0.70,0)\) satisfies both optimality conditions. The lasso keeps predictor 1 and excludes predictor 2 at this penalty strength.

The coefficient does not prove that predictor 1 is uniquely responsible for the response signal. The predictors are strongly correlated, so predictor 2 is a plausible substitute; here its residual association falls just below the threshold for inclusion. The selection is a property of this fitted optimization problem, not a general ranking of scientific or business importance.

Common mistake avoided: checking only whether \(z_2\) is large. The relevant quantity is its association with the residual after predictor 1 has entered, \(z_2-\rho\widehat\beta_1\), not its original association \(z_2\).

Elastic Net Combines the Two Behaviors

When \(0<\alpha<1\), elastic net combines the lasso’s absolute-value penalty with a quadratic penalty. For two predictors that are both positive and active, the optimality equations are

$$ \begin{pmatrix} 1+\lambda(1-\alpha)&\rho\\ \rho&1+\lambda(1-\alpha) \end{pmatrix} \begin{pmatrix}\widehat\beta_1\\\widehat\beta_2\end{pmatrix} = \begin{pmatrix}z_1-\lambda\alpha\\z_2-\lambda\alpha\end{pmatrix}. $$

This equation assumes both coefficients are positive. If either calculated coefficient is zero or negative, the assumed active signs are inconsistent, and the appropriate zero-coefficient condition must be checked instead. The quadratic part stabilizes the fit when predictors overlap; the absolute-value part can still produce exact zeros.

Worked Example: Solving an Elastic-Net Active Pair

Let \(\rho=0.5\), \(z_1=0.8\), \(z_2=0.6\), \(\lambda=0.4\), and \(\alpha=0.5\). Assume initially that both coefficients are positive. Calculate the elastic-net coefficients and check that the assumption holds.

Method. The absolute-value penalty contribution is \(\lambda\alpha=0.4(0.5)=0.2\), and the diagonal adjustment from the quadratic penalty is \(\lambda(1-\alpha)=0.4(0.5)=0.2\). Thus the diagonal entries are \(1+0.2=1.2\), and the adjusted right-hand side is \((0.8-0.2,\ 0.6-0.2)^\top=(0.6,\ 0.4)^\top\). The system is

$$ \begin{pmatrix}1.2&0.5\\0.5&1.2\end{pmatrix} \begin{pmatrix}\widehat\beta_1\\\widehat\beta_2\end{pmatrix} = \begin{pmatrix}0.6\\0.4\end{pmatrix}. $$

The determinant is \(1.2(1.2)-0.5(0.5)=1.44-0.25=1.19\). Inverting the \(2\times2\) system gives

$$ \widehat\beta_1=\frac{1.2(0.6)-0.5(0.4)}{1.19} =\frac{0.72-0.20}{1.19} =\frac{0.52}{1.19}=0.4370, $$
$$ \widehat\beta_2=\frac{1.2(0.4)-0.5(0.6)}{1.19} =\frac{0.48-0.30}{1.19} =\frac{0.18}{1.19}=0.1513. $$

Rounded to four decimal places, both coefficients are positive, so the assumed active signs are consistent. Substitution into the adjusted equations gives \(1.2(0.4370)+0.5(0.1513)\approx0.6000\) and \(0.5(0.4370)+1.2(0.1513)\approx0.4000\), using unrounded coefficients for the check.

Common mistake avoided: after subtracting \(\lambda\alpha\) from both entries of \(z\), use those adjusted values consistently when solving the system. The second coefficient comes from the second adjusted right-hand-side value in the inverse-matrix formula, not from reusing the first value.

Reading Paths Without Overinterpreting Selection

At \(\lambda=0\), the objective reduces to unpenalized least squares when that solution is identifiable. Increasing \(\lambda\) strengthens shrinkage. Ridge usually moves coefficients continuously toward zero without making them exactly zero at a finite \(\lambda\); lasso can change the active set as \(\lambda\) varies; elastic net can do both shrinkage and exact exclusion. For squared-error lasso and elastic net, a path is often examined over a sequence of penalty values. Individual coefficients do not have to decrease monotonically in absolute value: changes in the active predictors can alter how the remaining signal is allocated.

With highly correlated predictors, small differences in their observed associations may determine which lasso coefficient is nonzero. A coefficient path that shows one predictor entering while its near-substitute remains at zero should therefore be read as a penalized representation of the available signal—not as evidence that the excluded predictor has no relationship with the outcome. Ridge is often useful when the aim is stable prediction from overlapping information and retaining all predictors is acceptable. Lasso is useful when a sparse coefficient representation is desired, while recognizing that a selected member of a correlated set may depend on the data and tuning choice. Elastic net offers a compromise, but it does not guarantee identical coefficients or identical selection for correlated predictors.

1
Check the predictor scale and correlation.
Interpret the path in the context of standardized predictors and note whether predictors provide overlapping information.
2
Identify the penalty and its settings.
Distinguish ridge, lasso, and elastic net, and interpret \(\lambda\) under the loss scaling used by the fit.
3
Read the coefficients as a joint allocation.
For correlated predictors, a coefficient’s path depends on what the other predictors contribute; do not interpret each path as an isolated effect.
4
Separate prediction from attribution.
A model can predict adequately while allocating coefficients differently among substitutes. Avoid treating a single selected predictor as uniquely responsible solely because its coefficient is nonzero.

A common exam trap is to confuse a zero lasso coefficient with zero marginal association. A predictor can have a substantial \(z_j\) and still be excluded because another correlated predictor already accounts for much of its association with the response. Conversely, a predictor with a small apparent lead over its substitute may enter the lasso solution and receive the nonzero coefficient. The residual-association condition, rather than the original association alone, determines whether a candidate zero coefficient is compatible with the fit.

Key takeaway: Ridge typically shares signal among correlated predictors; lasso can choose one predictor from a set of substitutes; and elastic net combines shrinkage with possible sparsity. A coefficient path describes the fitted model’s allocation of signal under a particular penalty, not a definitive ranking of predictor importance.

Check Your Understanding

Use the stated standardized two-predictor setup and show the intermediate calculations before checking each answer.

  1. For ridge regression, let \(\rho=0.5\), \(z=(0.8,0.4)^\top\), and \(\lambda=0.5\). Calculate both coefficients. (Answer: \(\widehat\beta_1=0.5000,\ \widehat\beta_2=0.1000\))
  2. For lasso, let \(\rho=0.9\), \(z_1=0.75\), \(z_2=0.68\), and \(\lambda=0.20\). Check whether the solution \((0.55,0)\) satisfies the active and zero-coefficient conditions. (Answer: Yes; \(\widehat\beta_1=0.55\), and \(|0.68-0.9(0.55)|=0.185\leq0.20\))
  3. For elastic net, let \(\rho=0.4\), \(z=(0.9,0.7)^\top\), \(\lambda=0.5\), and \(\alpha=0.4\). Assuming both coefficients are positive, calculate them and check the sign assumption. (Answer: \(\widehat\beta_1=0.4641,\ \widehat\beta_2=0.2418\); both are positive)
  4. For ridge, let \(\rho=-0.5\), \(z=(0.6,-0.4)^\top\), and \(\lambda=0.5\). Calculate both coefficients and state why their signs are plausible. (Answer: \(\widehat\beta_1=0.3500,\ \widehat\beta_2=-0.1500\); the signs match the opposing response associations)
  5. For lasso, let \(\rho=0.95\), \(z=(0.62,0.60)^\top\), and \(\lambda=0.20\). First test a candidate with predictor 1 alone using \(\widehat\beta_1=z_1-\lambda\); if it fails, solve the two-positive-coefficient equations. (Answer: The one-predictor candidate fails because \(|0.60-0.95(0.42)|=0.201>0.20\); both coefficients are active, \(\widehat\beta_1\approx0.4103,\ \widehat\beta_2\approx0.0103\))

Previous

Next

Back to Tutorials
Clinical Biostats