Grouping Correlated Predictors

Why Correlated Predictors Raise a Grouping Question

When two predictors carry nearly the same information, a fitted model can often make similar predictions using either one, or using both. A lasso fit may retain one predictor and set the other’s coefficient to zero. The elastic net adds an L2 penalty to the L1 penalty; that L2 component tends to discourage a large difference between coefficients of highly correlated predictors.

This is a tendency, not an unconditional rule. The clearest coefficient-similarity result applies when the predictors are active in the model and their coefficients have the same sign. Predictor scaling also matters: correlations and coefficient comparisons are most interpretable when predictor columns have been centered and scaled to have the same length. The earlier material on predictor scaling establishes that convention for this discussion.

Definition: For centered response vector $y$ and centered, standardized predictor columns $x_1,\ldots,x_p$, the elastic net coefficient estimate minimizes $$ \frac{1}{2n}\left\|y-X\beta\right\|_2^2 +\lambda\alpha\sum_{j=1}^{p}|\beta_j| +\frac{\lambda(1-\alpha)}{2}\sum_{j=1}^{p}\beta_j^2, $$ where $n$ is the number of observations, $\lambda\geq 0$ is the overall penalty strength, and $0\leq\alpha\leq1$ controls the balance between L1 and L2 penalties. Write $\lambda_1=\lambda\alpha$ and $\lambda_2=\lambda(1-\alpha)$. The intercept, if included, is unpenalized.

The L1 component, $\lambda_1\sum_j|\beta_j|$, can set coefficients exactly to zero. The L2 component, $\lambda_2\sum_j\beta_j^2/2$, shrinks coefficients and makes large coefficient differences costly. When $\alpha=1$, the objective is lasso; when $\alpha=0$, it is ridge. For a genuine grouping contribution from the L2 term, $\lambda_2$ must be positive.

In the results below, standardization means each predictor column has mean zero and squared length $n$: $x_j'x_j=n$. Thus the sample correlation between two columns is $\rho_{jk}=x_j'x_k/n$, and $$ \left\|x_j-x_k\right\|_2^2=2n(1-\rho_{jk}). $$ This identity connects predictor similarity to the difference between their fitted coefficients.

Deriving the Grouping Bound

Let $r=y-X\hat\beta$ be the residual vector at an elastic net solution. The coordinatewise optimality condition is $$ \frac{x_j'r}{n}=\lambda_1 s_j+\lambda_2\hat\beta_j, $$ where $s_j=1$ if $\hat\beta_j>0$, $s_j=-1$ if $\hat\beta_j<0$, and $s_j$ can be any value in $[-1,1]$ if $\hat\beta_j=0$. This is the subgradient condition for the absolute-value penalty together with the ordinary derivative of the squared-coefficient penalty.

Suppose both coefficients are nonzero and have the same sign. Then $s_j=s_k$, so subtracting their optimality conditions cancels the L1 term. The remaining difference is driven by the difference between the predictor columns.

Subtract the two conditions.
For same-sign, nonzero coefficients, $\lambda_2(\hat\beta_j-\hat\beta_k)=(x_j-x_k)'r/n$.

Use standardization.
Since $\left\|x_j-x_k\right\|_2=\sqrt{2n(1-\rho_{jk})}$, divide by $n\lambda_2$ to bound the coefficient difference.

Theorem: If $\lambda_2>0$, predictors are centered and standardized, and $\hat\beta_j,\hat\beta_k$ are both nonzero with the same sign, then $$ \left|\hat\beta_j-\hat\beta_k\right| \leq \frac{\sqrt{2n(1-\rho_{jk})}\,\|r\|_2}{n\lambda_2}. $$ For fixed $n$, residual length, and $\lambda_2$, the bound approaches zero as $\rho_{jk}$ approaches one. The condition that the coefficients are nonzero and have the same sign is essential: if their signs differ, the L1 subgradient terms do not cancel.

This derivation explains the mechanism behind grouping. If two standardized columns are close, then $x_j-x_k$ is short. Their residual correlations are therefore close, and the L2 term relates those residual correlations to their coefficient difference. The result does not say that all highly correlated predictors must be selected, must have the same sign, or must receive identical coefficients.

Worked Example: Equal Coefficients in a Symmetric Elastic Net Fit

Suppose two centered, standardized predictors have correlation $0.80$, and their response cross-products satisfy $x_1'y/n=x_2'y/n=1$. Fit an elastic net with $\lambda=0.20$ and $\alpha=0.50$. Find the two coefficients, and verify the solution.

Method. Use the coordinate optimality conditions. The equal cross-products and symmetric predictor relationships make an equal, positive coefficient solution a natural candidate. The penalty components are $\lambda_1=0.20(0.50)=0.10$ and $\lambda_2=0.20(1-0.50)=0.10$.

Write the common candidate coefficient as $b$. If both coefficients equal $b$, the residual cross-product for either predictor is $1-(1+0.80)b=1-1.80b$. For a positive coefficient, the corresponding optimality condition is $$ 1-1.80b=0.10+0.10b. $$ Therefore $0.90=1.90b$, so $b=0.90/1.90=9/19\approx0.4737$.

Check. At $b=9/19$, the residual cross-product is $1-1.80(9/19)=0.1474$. The penalty side is $0.10+0.10(9/19)=0.1474$, so the coordinate condition holds for both predictors. The positive L2 component makes the objective strictly convex in the coefficients here, so the solution is unique; symmetry then confirms equal coefficients.

Mistake avoided. Correlation alone does not establish equal coefficients. Here equality follows from the symmetric cross-products as well as the symmetric predictor relationships.

What the Lasso Does Differently

The lasso has no L2 component, so the grouping bound above cannot be used: its denominator $\lambda_2$ would be zero. With highly correlated predictors, the lasso can select one while excluding another, especially when their associations with the response are similar but not identical. That does not mean it must always select exactly one. The outcome depends on the data, and identical predictors can even produce multiple lasso solutions.

The following comparison uses the same standardized two-predictor setup for both fits. The predictor correlation is $0.99$, and the response cross-products are $x_1'y/n=1$ and $x_2'y/n=0.99$. These values describe a hypothetical design, not observed industry data.

Worked Example: Lasso Selection Versus Elastic Net Grouping

For the design just specified, compare a lasso fit with $\lambda=0.10$ to an elastic net fit with $\lambda=0.10$ and $\alpha=0.50$. Show why the lasso can retain only the first predictor, then compute the elastic net coefficients.

Lasso calculation. Try $\hat\beta_1=0.90,\hat\beta_2=0$. With $\hat\beta_2=0$, the first predictor’s residual cross-product is $1-0.90=0.10$, equal to the lasso penalty $\lambda=0.10$ for a positive coefficient. The second predictor’s residual cross-product is $0.99-0.99(0.90)=0.099$. Since $|0.099|\leq0.10$, the zero-coefficient condition for the second predictor is satisfied. Thus the lasso solution is $(0.90,0)$.

Elastic net calculation. Now $\lambda_1=0.10(0.50)=0.05$ and $\lambda_2=0.05$. If both coefficients are positive, their optimality conditions give $$ \begin{pmatrix} 1.05 & 0.99\\ 0.99 & 1.05 \end{pmatrix} \begin{pmatrix}\hat\beta_1\\\hat\beta_2\end{pmatrix} = \begin{pmatrix}1-0.05\\0.99-0.05\end{pmatrix} = \begin{pmatrix}0.95\\0.94\end{pmatrix}. $$ The determinant is $1.05^2-0.99^2=1.1025-0.9801=0.1224$. Solving, $$ \hat\beta_1=\frac{1.05(0.95)-0.99(0.94)}{0.1224} =\frac{0.0669}{0.1224}=\frac{223}{408}\approx0.5466, $$ $$ \hat\beta_2=\frac{1.05(0.94)-0.99(0.95)}{0.1224} =\frac{0.0465}{0.1224}=\frac{155}{408}\approx0.3799. $$ Both are positive, as assumed. Their difference is $68/408=1/6\approx0.1667$, rather than the lasso’s difference of $0.90$.

Check. Substituting the elastic net coefficients into the two left sides gives $1.05(223/408)+0.99(155/408)=0.95$ and $0.99(223/408)+1.05(155/408)=0.94$, as required. The lasso condition for the excluded predictor also holds because $0.099\leq0.10$.

Mistake avoided. Do not infer that the lasso must select just one of any correlated pair. This example verifies a one-predictor lasso solution; it does not establish a universal selection rule.

Interpreting the Bound and Its Limits

The bound depends on more than correlation. It also depends on the residual length and the L2 penalty $\lambda_2=\lambda(1-\alpha)$. A larger $\lambda_2$ makes the bound tighter, while a longer residual vector makes it looser. The bound can be too loose to be informative even when the predictors are highly correlated. It is an upper bound under stated conditions, not a formula that predicts the actual coefficient difference.

Worked Example: Applying the Grouping Bound

Suppose $n=100$, two centered, standardized predictors have correlation $0.995$, and the fitted residual vector has length $\|r\|_2=8$. The elastic net uses $\lambda=0.50$, $\alpha=0.20$, and both estimated coefficients are nonzero and positive. Find the bound on their coefficient difference.

Method. First compute $\lambda_2=0.50(1-0.20)=0.40$. The distance between the predictor columns is $$ \|x_j-x_k\|_2=\sqrt{2(100)(1-0.995)} =\sqrt{200(0.005)}=\sqrt{1}=1. $$ The grouping bound is therefore $$ |\hat\beta_j-\hat\beta_k| \leq\frac{1(8)}{100(0.40)} =\frac{8}{40}=0.20. $$

Check. The predictors’ correlation is below one, so their column distance is positive; the L2 penalty is also positive, so the bound applies. Under the stated same-sign and nonzero conditions, the fitted coefficients can differ by no more than $0.20$.

Mistake avoided. Using $\lambda$ rather than $\lambda_2$ in the denominator would ignore the mixing parameter. Here $\lambda=0.50$, but the grouping component is $\lambda_2=0.40$.

An especially clear limiting case occurs when two standardized predictors are identical. Their fitted values depend on the sum of their coefficients. For a positive total coefficient, the L1 penalty is unchanged by splitting that total between the predictors, as long as both coefficients remain nonnegative. But the L2 penalty is minimized by splitting it equally: for a fixed sum $c$, $$ \beta_j^2+\beta_k^2 =\beta_j^2+(c-\beta_j)^2 =2\left(\beta_j-\frac{c}{2}\right)^2+\frac{c^2}{2}. $$ The expression is minimized at $\beta_j=\beta_k=c/2$. This explains why the L2 component favors an equal split for identical predictors, while a lasso objective can leave multiple ways to allocate the same total.

Practical Interpretation

Grouping can be useful when several predictors are plausible measurements of a shared underlying signal. An elastic net fit may retain and shrink a correlated set together rather than making the model depend on a single arbitrarily favored member. However, coefficient similarity is not evidence that the predictors have separate causal effects, nor does it guarantee equal predictive contribution in every data set.

Check that predictors were scaled comparably before interpreting coefficient comparisons.
Check whether the coefficients are both active and have the same sign before applying the bound.
Distinguish a tendency toward similar coefficients from a guarantee of equal coefficients.
Do not treat a lasso’s choice among correlated predictors as a stable ranking of their importance.
Remember that a positive L2 component is needed for the elastic net grouping bound.

These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: the exam-relevant reasoning is to identify which penalty is operating, state the conditions for the result, and avoid claiming more than the coefficient calculation supports.

Check Your Understanding

For each problem, use the stated standardization and penalty convention. Assume the coefficients are positive whenever a same-sign grouping bound is requested.

Two standardized predictors have $n=80$, correlation $0.98$, and fitted residual length $6$. An elastic net uses $\lambda=0.30$ and $\alpha=0.50$. Find the grouping bound on the coefficient difference. (Answer: $0.8944$)
Two standardized predictors have correlation $0.50$, and both response cross-products $x_j'y/n$ equal $0.80$. For $\lambda=0.20,\alpha=0.25$, find the common positive elastic net coefficient. (Answer: $5/11\approx0.4545$ for each predictor)
For a lasso with $\lambda=0.10$, let the standardized predictor correlation be $0.95$, with response cross-products $0.70$ and $0.68$. Check whether a solution with only the first coefficient active is valid; if not, solve for both positive coefficients. (Answer: one active coefficient is not valid; $\hat\beta_1\approx0.5026,\ \hat\beta_2\approx0.1026$)
Two predictors are identical and standardized, with $x_j'y/n=0.90$. For $\lambda=0.20,\alpha=0.50$, find the equal elastic net coefficients. (Answer: $8/21\approx0.3810$ for each predictor)
For the identical predictors in question 4, instead fit a lasso with $\lambda=0.20$. Describe the full set of nonnegative coefficient pairs that minimize the objective. (Answer: any pair $(\hat\beta_j,\hat\beta_k)$ with $\hat\beta_j+\hat\beta_k=0.70$)

What You'll Learn