Why The Intercept Is Usually Unpenalized

Why Separate the Intercept from the Slopes?

In a penalized regression, the slope coefficients are constrained by a penalty, while the intercept is usually left out of that penalty. This is not merely a software convention. The intercept locates the fitted relationship on the response scale; the penalty is intended to control the size of the predictor effects. Penalizing the intercept would make the fit depend on where zero happens to be on the response scale.

Centering the predictors and response makes this separation especially clear. With centered data, the intercept in the centered-coordinate model is zero at the optimum. That does not mean the original-scale intercept is zero: it must be recovered using the sample means and the fitted slopes.

Deriving the Intercept for Fixed Slopes

Suppose the slopes \(\beta_1,\ldots,\beta_p\) are fixed. The penalty is then fixed as well, and minimizing the elastic net objective over \(\beta_0\) is equivalent to minimizing the residual sum of squares over \(\beta_0\). Define \(\bar{x}_j=n^{-1}\sum_i x_{ij}\) and \(\bar{y}=n^{-1}\sum_i y_i\).

Differentiate the squared-error part with respect to the intercept. Setting that derivative to zero requires the residuals to sum to zero. Solving this condition gives the intercept that minimizes squared error for the chosen slopes.

This condition says that the average fitted value equals the average observed response: \(\overline{\widehat{y}}=\bar{y}\). The result holds for any fixed slopes when the loss is ordinary squared error and the intercept is unpenalized. The elastic net affects which slopes are chosen, but it does not change this conditional intercept calculation.

The same result can be seen by decomposing the residual sum of squares. Let \(x_{ij}^{c}=x_{ij}-\bar{x}_j\) and \(y_i^{c}=y_i-\bar{y}\). Then

The cross-term between the two parts is zero because both centered variables have sample mean zero. The second term is nonnegative and is minimized at \(\beta_0=\bar{y}-\sum_j\bar{x}_j\beta_j\). Thus, centering isolates the slope-fitting problem from the intercept calculation.

Centering Predictors and Response

When both predictors and response are centered, the model for the centered response has no intercept at its optimum:

The slope estimates can be obtained by applying the elastic net penalty to the centered predictors and fitting the centered response. The intercept in this centered-coordinate model is zero because the centered response and each centered predictor have mean zero. To express predictions in the original coordinates, use the fitted slopes in \(\widehat{\beta}_0=\bar{y}-\sum_j\bar{x}_j\widehat{\beta}_j\).

If predictors are centered but the response is not, the original-scale intercept is \(\bar{y}\). If the response is centered but predictors are not, the centered-response intercept is \(-\sum_j\bar{x}_j\widehat{\beta}_j\). These are coordinate-specific versions of the same intercept formula.

Restoring the Intercept after Standardization

Predictors are often centered and scaled before penalized fitting so their penalty treatment is on comparable scales. Suppose the fitted model uses \(z_j=(x_j-\bar{x}_j)/s_j\), where \(s_j>0\) is the scale used for predictor \(j\). If the response is also standardized as \(z_y=(y-\bar{y})/s_y\), suppose the fitted centered-coordinate model is \(\widehat{z}_y=\sum_j\widehat{\gamma}_jz_j\). Converting predictions back gives the original-scale slopes and intercept:

The formula for the slopes follows by substituting the definitions of \(z_y\) and \(z_j\) into the standardized model and collecting the coefficients of the original predictors. The intercept then follows from the sample-mean condition. If only the predictors are standardized and \(y\) remains in its original units, use \(\widehat{\beta}_j=\widehat{\gamma}_j/s_j\), followed by the same original-scale intercept formula.

Weighted Fits and the Meaning of “Mean”

The intercept principle extends to weighted squared-error fitting, provided the intercept remains unpenalized. For positive observation weights \(w_i\), minimize \(\frac{1}{2}\sum_i w_i(y_i-\beta_0-\sum_jx_{ij}\beta_j)^2\) plus a penalty on slopes only. Differentiating with respect to \(\beta_0\) gives a weighted residual sum of zero, so the relevant means are weighted means, not ordinary sample means:

Why the Intercept Is Usually Unpenalized

Leaving the intercept unpenalized has two practical consequences. First, it ensures the fitted values have the same sample mean as the response for ordinary squared-error fitting. Second, it preserves the interpretation of a change of response origin. If a constant \(c\) is added to every response, an unpenalized-intercept fit can add \(c\) to the intercept while leaving the slopes and their penalty unchanged. A penalty on the intercept would instead treat the shifted intercept as a different penalty cost.

An intercept penalty is not mathematically impossible, but it changes the model and removes these properties. The usual elastic net objective penalizes predictor coefficients, not the intercept. When reporting a fitted model, state whether coefficients are standardized and whether the reported intercept is in centered or original coordinates.