A Worked Elastic Net Example With Correlated Predictors

Why Work Through a Correlated-Predictor Example?

When predictors are correlated, a fitted response pattern can often be represented by more than one combination of coefficients. The elastic net balances two kinds of penalty: an absolute-value penalty that can set coefficients exactly to zero, and a squared-coefficient penalty that shrinks coefficients while allowing correlated predictors to share weight. Calculating a small example makes that balance concrete.

This tutorial develops the calculations for two centered, standardized predictors. The examples focus on coefficients and their interpretation, not on selecting a penalty from validation data. The material is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For centered response $y$ and centered, standardized predictor columns $x_1,\ldots,x_p$, use the elastic net objective $$ Q(\beta)=\frac{1}{2n}\left\|y-\sum_{j=1}^{p}x_j\beta_j\right\|_2^2 +\lambda\left\{\alpha\sum_{j=1}^{p}|\beta_j|+ \frac{1-\alpha}{2}\sum_{j=1}^{p}\beta_j^2\right\}. $$ Here $n$ is the number of observations, $\lambda\geq 0$ is the overall penalty strength, and $0\leq\alpha\leq1$ is the mixture: $\alpha=1$ gives the lasso objective, while $\alpha=0$ gives the ridge objective. The intercept is omitted because the response and predictors are centered; in a model with an intercept, it is ordinarily not penalized.

For the two-predictor calculations below, standardization means $x_j^\mathsf{T}x_j/n=1$. Define $r=x_1^\mathsf{T}x_2/n$ as the predictor correlation and $c_j=x_j^\mathsf{T}y/n$ as the predictor–response association. These quantities summarize the parts of the data needed for the coefficient calculation. The objective and these definitions must use the same scaling; changing the loss scaling without changing the penalty changes the numerical meaning of $\lambda$.

Conditions for the Coefficients

Write $\tau=\lambda\alpha$ for the lasso component and $\gamma=\lambda(1-\alpha)$ for the ridge component. For two predictors, differentiating the squared-error part of the objective gives the matrix $G+\gamma I$, where $$ G= \begin{pmatrix} 1&r\\ r&1 \end{pmatrix}, \qquad c= \begin{pmatrix} c_1\\c_2 \end{pmatrix}. $$ If both coefficients are nonzero, let $s_j=1$ when $\beta_j>0$ and $s_j=-1$ when $\beta_j<0$. The active-coefficient equations are $$ \begin{pmatrix} 1+\gamma&r\\ r&1+\gamma \end{pmatrix} \begin{pmatrix}\beta_1\\\beta_2\end{pmatrix} = \begin{pmatrix}c_1-\tau s_1\\c_2-\tau s_2\end{pmatrix}. $$ A solution to these equations is a candidate: its coefficient signs must agree with the signs assumed in $s$. A zero coefficient instead has an inequality condition, so solving as though both coefficients were nonzero is not always valid.

Theorem: For a coefficient $\beta_j=0$, the elastic net optimality condition is $$ \left|c_j-\sum_{k\ne j}G_{jk}\beta_k\right|\leq\tau. $$ For a nonzero coefficient, the corresponding signed condition is $$ c_j-\sum_{k=1}^{p}G_{jk}\beta_k =\gamma\beta_j+\tau\,\operatorname{sign}(\beta_j). $$ These conditions apply to the stated standardized, centered objective. In particular, an inactive coefficient must be checked against the lasso threshold $\tau$, not against $\lambda$ or $\gamma$.

Why the Conditions Hold

The squared-error term has derivative $-x_j^\mathsf{T}(y-X\beta)/n$ with respect to $\beta_j$. The ridge part contributes $\gamma\beta_j$. At a nonzero coefficient, the derivative of $|\beta_j|$ is $\operatorname{sign}(\beta_j)$, so setting the total derivative to zero gives the signed condition above. At zero, the absolute-value function has subgradients between $-1$ and $1$. An optimum is possible exactly when the remaining derivative can be balanced by some value in that interval, which yields the inequality with threshold $\tau$. This is why a candidate zero coefficient needs a separate check.

Worked Calculations

Worked Example: Two Positively Correlated Predictors Share Weight

Suppose a hypothetical claim-cost model has centered, standardized predictors with $r=0.80$, and predictor–response associations $c_1=0.90$ and $c_2=0.85$. Calculate the elastic net coefficients when $\lambda=0.20$ and $\alpha=0.50$, then compare them with the coefficients at the same $\lambda$ for the lasso and ridge endpoints.

Method. Calculate $\tau=\lambda\alpha=0.20(0.50)=0.10$ and $\gamma=\lambda(1-\alpha)=0.20(0.50)=0.10$. Both coefficients are initially treated as positive, so the active equations have right-hand side $(0.90-0.10,\;0.85-0.10)^\mathsf{T}=(0.80,\;0.75)^\mathsf{T}$:

\begin{pmatrix}1.10&0.80\\0.80&1.10\end{pmatrix} \begin{pmatrix}\beta_1\\\beta_2\end{pmatrix} = \begin{pmatrix}0.80\\0.75\end{pmatrix}.

The determinant is $1.10(1.10)-0.80(0.80)=1.21-0.64=0.57$. Therefore $$ \beta_1=\frac{1.10(0.80)-0.80(0.75)}{0.57} =\frac{0.88-0.60}{0.57} =\frac{0.28}{0.57}=0.4912, $$ and $$ \beta_2=\frac{1.10(0.75)-0.80(0.80)}{0.57} =\frac{0.825-0.64}{0.57} =\frac{0.185}{0.57}=0.3246. $$ Both are positive, as assumed, so the signs are consistent and the active equations are valid.

At the lasso endpoint, $\alpha=1$, so $\tau=0.20$ and $\gamma=0$. The active right-hand side is $(0.70,0.65)^\mathsf{T}$, and the determinant is $1-0.80^2=0.36$. Thus $$ \beta_1=\frac{0.70-0.80(0.65)}{0.36} =\frac{0.18}{0.36}=0.5000,\qquad \beta_2=\frac{0.65-0.80(0.70)}{0.36} =\frac{0.09}{0.36}=0.2500. $$ At the ridge endpoint, $\alpha=0$, so $\tau=0$, $\gamma=0.20$, and the matrix diagonal is $1.20$. Its determinant is $1.20^2-0.80^2=1.44-0.64=0.80$, giving $$ \beta_1=\frac{1.20(0.90)-0.80(0.85)}{0.80} =\frac{1.08-0.68}{0.80}=0.5000,\qquad \beta_2=\frac{1.20(0.85)-0.80(0.90)}{0.80} =\frac{1.02-0.72}{0.80}=0.3750. $$ The elastic net estimates are between these endpoint results for the second coefficient in this example. Both predictors retain positive coefficients. The correlated predictors share the fitted signal rather than one being removed. The common mistake is to subtract the full $\lambda$ from each association for every mixture; the correct subtraction for a positive coefficient is $\tau=\lambda\alpha$. Coefficients are rounded to four decimals.

Worked Example: Checking Whether a Correlated Predictor Is Excluded

For a hypothetical severity model, let $r=0.90$, $c_1=0.80$, $c_2=0.68$, $\lambda=0.30$, and $\alpha=0.80$. Determine whether the solution can retain predictor 1 and set predictor 2 to zero.

Method. The penalty components are $\tau=0.30(0.80)=0.24$ and $\gamma=0.30(0.20)=0.06$. If only predictor 1 is active and positive, its coefficient is $$ \beta_1=\frac{c_1-\tau}{1+\gamma} =\frac{0.80-0.24}{1.06} =\frac{0.56}{1.06}=0.5283. $$ Now check the inactive condition for predictor 2. Its residual association after accounting for predictor 1 is $$ c_2-r\beta_1=0.68-0.90(0.5283) =0.68-0.4755=0.2045. $$ Since $|0.2045|\leq0.24$, the zero coefficient satisfies the required condition. The solution is $(\beta_1,\beta_2)=(0.5283,0)$, with coefficients rounded to four decimals.

The check matters because correlation changes predictor 2's association after predictor 1 is included. Comparing $c_2=0.68$ directly with the threshold $0.24$ would incorrectly suggest that predictor 2 must be active. The correct comparison uses $c_2-r\beta_1$. This is the common mistake avoided here.

Worked Example: A Nonzero Coefficient Can Be Negative

In a hypothetical renewal model, take $r=0.60$, $c_1=0.70$, $c_2=0.20$, $\lambda=0.25$, and $\alpha=0.40$. Find the coefficients, allowing the second coefficient to be negative.

Method. Here $\tau=0.25(0.40)=0.10$ and $\gamma=0.25(0.60)=0.15$. First test whether predictor 1 alone is sufficient. Its positive coefficient would be $$ \beta_1=\frac{0.70-0.10}{1.15}=\frac{0.60}{1.15}=0.5217. $$ The adjusted association for predictor 2 would then be $0.20-0.60(0.5217)=-0.1130$. Its absolute value, $0.1130$, exceeds $\tau=0.10$, so setting predictor 2 to zero does not satisfy the inactive condition. A negative coefficient is plausible because this adjusted association is negative.

Use signs $s_1=1$ and $s_2=-1$. The active equations become $$ \begin{pmatrix}1.15&0.60\\0.60&1.15\end{pmatrix} \begin{pmatrix}\beta_1\\\beta_2\end{pmatrix} = \begin{pmatrix}0.70-0.10\\0.20+0.10\end{pmatrix} = \begin{pmatrix}0.60\\0.30\end{pmatrix}. $$ The determinant is $1.15^2-0.60^2=1.3225-0.36=0.9625$. Hence $$ \beta_1=\frac{1.15(0.60)-0.60(0.30)}{0.9625} =\frac{0.69-0.18}{0.9625}=0.5299, $$ $$ \beta_2=\frac{1.15(0.30)-0.60(0.60)}{0.9625} =\frac{0.345-0.36}{0.9625}=-0.0156. $$ The resulting signs are positive and negative, matching the assumed signs, so the active solution is consistent. The small negative coefficient represents an adjusted association after accounting for the first predictor; it is not a statement of causation. The trap is to force all coefficients to be positive just because both original associations $c_1$ and $c_2$ are positive. Coefficients are rounded to four decimals.

Reading the Mixture and Avoiding Calculation Traps

Changing $\alpha$ at fixed $\lambda$ changes both components of the penalty: $\tau=\lambda\alpha$ and $\gamma=\lambda(1-\alpha)$. Increasing $\alpha$ increases the threshold that can set a coefficient to zero and reduces the ridge component. Decreasing $\alpha$ reduces that threshold and increases the ridge shrinkage. The actual coefficients depend on the predictor correlations and the response associations as well, so there is no general rule that every individual coefficient must move monotonically as the mixture changes.

In particular, correlated predictors can have similar fitted contributions even when their coefficient values change. A coefficient describes the fitted model conditional on its other predictors and on their scaling. It is not an isolated measure of a predictor's importance, and a selected predictor should not be given a causal interpretation from this calculation alone.

Confirm the convention.
Use the same loss scaling and predictor standardization that define $\lambda$, $\alpha$, $r$, and $c$.

Split the penalty.
Calculate $\tau=\lambda\alpha$ and $\gamma=\lambda(1-\alpha)$ before setting up coefficient equations.

Propose signs and active coefficients.
Use the signed active equations, then verify that the computed coefficients have the proposed signs.

Check every zero coefficient.
Compare its adjusted association with $\tau$. A zero coefficient is justified only if the absolute value is no greater than that threshold.

Common errors include using $c_j$ instead of the correlation-adjusted association for an inactive predictor, confusing the ridge component $\gamma$ with the lasso threshold $\tau$, and omitting the diagonal addition $\gamma$ in the active equations. Also, do not interpret coefficients on differently scaled predictors as directly comparable: standardization is part of the setup, not a cosmetic detail.

Key takeaway: For correlated predictors, elastic net coefficients come from a joint system of equations for active variables, together with a threshold inequality for each inactive variable. The mixture controls the balance between exact exclusion and smooth shrinkage; the predictor correlation determines how that balance is reflected in the fitted coefficients.

Check Your Understanding

For each question, use the centered, standardized two-predictor objective defined above. Round numerical coefficients to four decimals.

Let $r=0$, $c_1=0.70$, $c_2=0.40$, $\lambda=0.20$, and $\alpha=0.50$. Calculate both coefficients. (Answer: $\beta_1=0.5455,\ \beta_2=0.2727$.)
Let $r=0.50$, $c_1=0.80$, $c_2=0.65$, $\lambda=0.20$, and $\alpha=0.50$. Assuming both coefficients are positive, calculate them and verify their signs. (Answer: $\beta_1=0.5156,\ \beta_2=0.2656$, both positive.)
Let $r=0.80$, $c_1=0.70$, $c_2=0.50$, $\lambda=0.25$, and $\alpha=0.80$. Test the solution with predictor 1 active and predictor 2 zero. (Answer: $\beta_1=0.4762,\ \beta_2=0$; the inactive adjusted association is $0.1190$, within the threshold $0.2000$.)
Let $r=0.40$, $c_1=0.60$, $c_2=0.10$, $\lambda=0.20$, and $\alpha=0.50$. Is predictor 2 needed if predictor 1 is positive and active? (Answer: No; $\beta_1=0.4545,\ \beta_2=0$, and the absolute adjusted association $0.0818$ is below the threshold $0.1000$.)
For a ridge endpoint, let $r=0.70$, $c_1=0.80$, $c_2=0.65$, and $\lambda=0.20$. Calculate the coefficients and state whether the lasso threshold can set either coefficient to zero. (Answer: $\beta_1=0.5316,\ \beta_2=0.2316$; $\alpha=0$ gives $\tau=0$, so there is no positive lasso threshold.)

What You'll Learn