Elastic Net Penalized Least Squares Objective

Why the Elastic Net Objective Has Three Parts

A least-squares fit rewards coefficients that make fitted responses close to the observed responses. Penalized least squares adds a cost for coefficient size, so that fitting the observed data is not the only consideration. The elastic net combines two familiar costs: one based on absolute coefficient values and one based on squared coefficient values. Keeping these parts distinct is essential when calculating or interpreting the objective.

The objective is a criterion to be minimized over the regression coefficients. It does not, by itself, require a probability distribution for the responses. In particular, the definition below is an optimization rule; normality and other assumptions used for classical statistical inference are not needed to calculate its value.

Definition: Suppose there are $n$ observations, each with response $y_i$ and $p$ predictor values $x_{ij}$, for $i=1,\ldots,n$ and $j=1,\ldots,p$. For an affine linear prediction with intercept $\beta_0$ and slopes $\beta_1,\ldots,\beta_p$, define the residual for observation $i$ by $$ e_i=y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j. $$ Under the convention used here, the elastic net objective is $$ Q(\beta_0,\boldsymbol{\beta})= \sum_{i=1}^{n}e_i^2+ \lambda_1\sum_{j=1}^{p}|\beta_j|+ \lambda_2\sum_{j=1}^{p}\beta_j^2, $$ where $\boldsymbol{\beta}=(\beta_1,\ldots,\beta_p)$ and $\lambda_1,\lambda_2\geq 0$. The intercept is not included in either penalty.

The first term, $\sum e_i^2$, is the residual sum of squares (RSS). The second term is the L1 penalty, and the third is the L2 penalty. Each penalty sums over the slopes, not over the observations. With this convention, the objective is the RSS plus $\lambda_1$ times the sum of absolute slopes plus $\lambda_2$ times the sum of squared slopes.

This formula presumes a finite data set and a linear predictor evaluated at the observed predictor values. It defines a least-squares fitting criterion whether or not a stochastic model is assumed. If one additionally interprets the regression probabilistically, that is a separate modeling step: the objective alone does not assert independent errors, constant variance, or normally distributed errors.

What the Two Penalty Parameters Do

The parameter $\lambda_1$ sets the weight on the total absolute size of the slopes. Increasing it makes large absolute coefficients more costly relative to the RSS. The parameter $\lambda_2$ sets the weight on the sum of squared slopes, making larger magnitudes increasingly costly through their squares. Both are constrained to be nonnegative in this formulation. A negative value would reward coefficient size rather than penalize it.

The parameters do not have to be equal. If $\lambda_1=0$, the L1 term contributes nothing; if $\lambda_2=0$, the L2 term contributes nothing. If both are zero, the displayed objective is ordinary residual sum of squares. These statements describe terms in the objective, not a guarantee about which coefficients a minimizer will have.

The intercept is conventionally left unpenalized here so that the penalty applies to the slopes rather than shifting the overall fitted level. Some software or texts use a different convention, such as dividing the RSS by $n$, multiplying it by $1/2$, or including a factor of $1/2$ in the squared-coefficient penalty. Such formulas can represent related fitting criteria, but numerical penalty parameters are comparable only when the scaling convention is also the same.

Worked Example: Calculating Each Component of the Objective

A hypothetical pricing study fits a model with two slopes and an unpenalized intercept. At a proposed coefficient vector, the three residuals are $2,-1,$ and $0.5$, while the slopes are $1.5$ and $-2$. Let $\lambda_1=0.4$ and $\lambda_2=0.1$. Calculate the objective.

Method. Compute the RSS, L1 contribution, and L2 contribution separately, then add them. This prevents a penalty from being applied to residuals or an absolute value from being omitted.

RSS. Squaring each residual gives $2^2=4$, $(-1)^2=1$, and $0.5^2=0.25$. Therefore, $$ \operatorname{RSS}=4+1+0.25=5.25. $$

L1 contribution. The sum of the absolute slopes is $|1.5|+|-2|=1.5+2=3.5$. Thus the weighted L1 contribution is $0.4(3.5)=1.4$.

L2 contribution. The sum of the squared slopes is $1.5^2+(-2)^2=2.25+4=6.25$. Thus the weighted L2 contribution is $0.1(6.25)=0.625$.

Adding the three parts gives $$ Q=5.25+1.4+0.625=7.275. $$ The objective value is $7.275$. As a check, the penalty total is $2.025$, and $5.25+2.025=7.275$. The intercept, if there is one, does not enter either penalty under the stated convention.

Evaluating and Comparing Candidate Fits

A candidate coefficient vector can be evaluated directly from the objective; it is not necessary to solve for the minimizing vector first. For a fair comparison, use the same observations, residual definition, penalty parameters, and scaling convention for every candidate. A lower objective means a better value of this particular criterion, not necessarily better performance on new data.

Worked Example: Comparing Two Candidate Slopes

For a one-predictor fit with no intercept, the predictor values are $x=(1,-1)$, the responses are $y=(3,-1)$, and $\lambda_1=1,\lambda_2=0.5$. Compare the candidate slopes $\beta=1$ and $\beta=2$.

Method. For each candidate, calculate fitted values, residuals, RSS, and both penalties. A common mistake is to calculate the RSS correctly but then add the L2 penalty as $\lambda_2|\beta|$, rather than $\lambda_2\beta^2$.

For $\beta=1$, the fitted values are $(1,-1)$. The residuals, observed minus fitted, are $(3-1,-1-(-1))=(2,0)$, so the RSS is $2^2+0^2=4$. The L1 contribution is $1|1|=1$, and the L2 contribution is $0.5(1^2)=0.5$. Therefore, $$ Q(1)=4+1+0.5=5.5. $$

For $\beta=2$, the fitted values are $(2,-2)$. The residuals are $(3-2,-1-(-2))=(1,1)$, so the RSS is $1^2+1^2=2$. The L1 contribution is $1|2|=2$, and the L2 contribution is $0.5(2^2)=2$. Therefore, $$ Q(2)=2+2+2=6. $$

The candidate $\beta=1$ has the lower objective, $5.5<6$. The larger slope reduces the RSS by $2$, but increases the combined penalties by $2.5$, so its total objective is higher. This comparison identifies the better of these two candidates; it does not establish that $\beta=1$ is the global minimizer.

Why the Objective Can Be Minimized as a Single Criterion

The objective adds costs that are measured on different mathematical scales: squared residuals, absolute slopes, and squared slopes. The penalty parameters determine their relative weights. Once those weights and the data are fixed, every coefficient vector has one well-defined objective value, and minimization means choosing a vector with the smallest such value.

A useful justification for the form of the residual term is direct. Under the stated linear predictor, the fitted value for observation $i$ is $\hat y_i=\beta_0+\sum_jx_{ij}\beta_j$. Its prediction error in the observed data is $y_i-\hat y_i=e_i$. Least squares measures aggregate squared prediction error, so summing $e_i^2$ over the observations gives the RSS. The two additional terms are then explicit coefficient costs, weighted by their respective nonnegative parameters.

For fixed data, the RSS is a sum of squared affine functions of the coefficients, so it is convex. The absolute-value and squared-coefficient penalties are also convex, and nonnegative weighted sums of convex functions are convex. Consequently, this objective is convex. Convexity means that a local minimum is also a global minimum; it does not by itself guarantee that there is only one minimizing coefficient vector.

Worked Example: Finding a One-Predictor Minimizer

Suppose a hypothetical two-record data set has no intercept, predictor values $x=(1,-1)$, and responses $y=(2,0)$. Use $\lambda_1=0.5$ and $\lambda_2=0.25$. Find the minimizing slope for the stated objective.

Method. Write the objective as a function of the single slope and minimize separately on the regions $\beta<0$ and $\beta>0$, because the absolute-value term changes form at zero. This avoids differentiating $|\beta|$ as though it were a smooth function at zero.

The fitted values are $(\beta,-\beta)$, so the residuals are $(2-\beta,\beta)$. The RSS is $$ (2-\beta)^2+\beta^2=4-4\beta+2\beta^2. $$ The penalty contributions are $0.5|\beta|$ and $0.25\beta^2$. Thus $$ Q(\beta)=2.25\beta^2-4\beta+4+0.5|\beta|. $$

For $\beta>0$, this becomes $Q(\beta)=2.25\beta^2-3.5\beta+4$, whose derivative is $4.5\beta-3.5$. Setting this derivative to zero gives $\beta=3.5/4.5=7/9$, which is in the positive region. For $\beta<0$, the objective is $2.25\beta^2-4.5\beta+4$, with derivative $4.5\beta-4.5$. This derivative is negative for every $\beta<0$, so the objective decreases as $\beta$ increases toward zero; there is no minimum in the negative region. Since the objective is convex, the positive-region minimum is the global minimum.

At $\beta=7/9$, the residuals are $11/9$ and $7/9$. The RSS is $121/81+49/81=170/81$. The L1 contribution is $0.5(7/9)=7/18$, and the L2 contribution is $0.25(49/81)=49/324$. Therefore, $$ Q(7/9)=\frac{680}{324}+\frac{126}{324}+\frac{49}{324} =\frac{855}{324}=\frac{95}{36}\approx 2.6389. $$ The minimizing slope is $7/9$, and the objective value is approximately $2.6389$. The common mistake avoided here is treating the L1 term as differentiable at zero without checking its two regions.

Common Errors and Practical Interpretation

Confusing the two penalties. The L1 term uses $|\beta_j|$; the L2 term uses $\beta_j^2$. In general, neither can be substituted for the other.
Penalizing residuals instead of coefficients. Only the RSS uses residuals. The penalty sums use the fitted slopes.
Forgetting a weight. Each penalty sum must be multiplied by its own parameter before the terms are added.
Including the intercept without checking the convention. The formula here excludes it. If a different stated objective penalizes the intercept, follow that formula instead.
Mixing scaling conventions. An objective using RSS divided by $n$ will not generally have the same numerical value or the same penalty-parameter calibration as the unscaled RSS used here.

Coefficient units also matter: changing the units of a predictor changes the numerical slope associated with the same fitted relationship, which can change its penalty contribution. In applications, predictor scaling is therefore an important consideration, but the objective must still be evaluated using the coefficients and convention actually specified. A value of $Q$ is meaningful as a comparison only when the data and conventions are held fixed.

Key takeaway: Under the convention in this tutorial, calculate the elastic net objective as RSS plus $\lambda_1$ times the sum of absolute slopes plus $\lambda_2$ times the sum of squared slopes. State the convention, keep the intercept treatment clear, and evaluate all three parts separately.

Check Your Understanding

For each problem, use the unscaled RSS convention and leave any intercept unpenalized unless the problem says otherwise.

A proposed fit has residuals $(-1,2)$, slopes $1$ and $-3$, and $\lambda_1=0.2,\lambda_2=0.1$. Calculate its objective. (Answer: $6.8$)
A one-predictor model has no intercept, $x=(1,-1)$, $y=(3,-1)$, $\lambda_1=1,\lambda_2=0.5$. Compare $\beta=1$ and $\beta=2$, and identify the lower-objective candidate. (Answer: $Q(1)=5.5$, $Q(2)=6$; $\beta=1$ is lower.)
A two-record model has an unpenalized intercept of $1$, one slope $\beta=2$, and residuals $(1,1)$. With $\lambda_1=0.5,\lambda_2=0.25$, calculate the objective. (Answer: $4$)
A candidate fit has residuals $(0,1,-2)$, slopes $2$ and $-1$, and $\lambda_1=0.5,\lambda_2=0.2$. Calculate the RSS, each weighted penalty, and the objective. (Answer: RSS $=5$, L1 contribution $=1.5$, L2 contribution $=1$, objective $=7.5$.)
For a one-predictor fit with no intercept, let $x=(1,-1)$, $y=(2,0)$, and $\lambda_1=0.5,\lambda_2=0.25$. Write the objective as a function of $\beta$, and state its minimizing slope. (Answer: $Q(\beta)=2.25\beta^2-4\beta+4+0.5|\beta|$; the minimizing slope is $7/9$.)

What You'll Learn