Why the Intercept Is Usually Not Penalized

Why Separate the Intercept From the Slopes?

Ridge and lasso regression shrink slope coefficients toward zero to control model complexity. The intercept has a different role: it sets the baseline level of the response. Penalizing it would shrink that baseline toward zero, even when zero is arbitrary or far from the range of the observed responses. Leaving the intercept unpenalized also gives a useful algebraic result: for any fixed slopes, its least-squares estimate makes the residuals average zero.

That result allows the intercept to be separated from slope estimation. Once the response and predictors are centered by subtracting their sample means, the remaining problem is to choose slopes using centered data and the chosen penalty. This applies to both ridge and lasso under the usual unweighted squared-error objectives.

Model and objectives: For observations \(i=1,\ldots,n\) and predictors \(j=1,\ldots,p\), consider the linear predictor \(b_0+\sum_{j=1}^{p}b_jx_{ij}\), where \(b_0\) is the intercept and \(b_j\) is the slope for predictor \(j\). In this tutorial, ridge minimizes \(\sum_{i=1}^{n}(y_i-b_0-\sum_{j=1}^{p}b_jx_{ij})^2+\lambda\sum_{j=1}^{p}b_j^2\), and lasso minimizes \(\sum_{i=1}^{n}(y_i-b_0-\sum_{j=1}^{p}b_jx_{ij})^2+\lambda\sum_{j=1}^{p}|b_j|\), with \(\lambda\geq 0\). In both objectives, the intercept is excluded from the penalty. These definitions use unweighted squared errors; a different loss function or observation weights can change the intercept calculation.

Deriving the Intercept for Fixed Slopes

Fix any proposed slope vector \(b_1,\ldots,b_p\). The penalty in either objective is then constant as a function of \(b_0\), because it contains only the slopes. Thus, conditional on those slopes, choosing the intercept is the same as minimizing the residual sum of squares.

RSS(b_0)=\sum_{i=1}^{n}\left(y_i-b_0-\sum_{j=1}^{p}b_jx_{ij}\right)^2.

Differentiate with respect to \(b_0\) and set the derivative equal to zero. The second derivative is \(2n>0\), so this stationary point is the unique minimum.

\frac{\partial RSS}{\partial b_0} =-2\sum_{i=1}^{n}\left(y_i-b_0-\sum_{j=1}^{p}b_jx_{ij}\right)=0.

Rearranging gives the conditional intercept estimate. Here \(\bar y=n^{-1}\sum_i y_i\) and \(\bar x_j=n^{-1}\sum_i x_{ij}\) denote the sample means.

\widehat b_0(b_1,\ldots,b_p) =\bar y-\sum_{j=1}^{p}b_j\bar x_j.

At this intercept, the residuals sum to zero. Indeed, summing \(y_i-\widehat b_0-\sum_j b_jx_{ij}\) over \(i\) yields \(n\bar y-n\widehat b_0-n\sum_jb_j\bar x_j=0\). This is a property of the least-squares intercept when it is unpenalized; it need not hold if the intercept is constrained or penalized.

Worked Example: Finding the Intercept for a Fixed Slope

Suppose two observations have \(x=(1,3)\) and \(y=(5,11)\). A fitting procedure has fixed the slope at \(b_1=2\). Find the intercept that minimizes squared error, then calculate the fitted values and residual sum of squares. The intercept is unpenalized, and the slope penalty, if any, is constant while choosing the intercept.

Method. For a fixed slope, minimize the residual sum of squares over the intercept. The adjusted responses are \(y_i-b_1x_i\): \(5-2(1)=3\) and \(11-2(3)=5\). Their mean is \((3+5)/2=4\), so the minimizing intercept is \(\widehat b_0=4\).

The fitted values are \(4+2(1)=6\) and \(4+2(3)=10\). The residuals, defined as observed minus fitted, are \(5-6=-1\) and \(11-10=1\). Therefore, \(RSS=(-1)^2+1^2=2\). The residuals sum to zero, as expected.

Check and common mistake. The adjusted responses average to 4, and the fitted values reproduce that intercept calculation. Using an intercept of 3 would give fitted values 5 and 9, residuals 0 and 2, and \(RSS=4\); those residuals do not sum to zero, so that intercept does not minimize squared error.

Centering Separates the Intercept From Slope Shrinkage

Define the centered response and predictors by \(y_i^c=y_i-\bar y\) and \(x_{ij}^c=x_{ij}-\bar x_j\). Substitute the minimizing intercept \(\widehat b_0=\bar y-\sum_j b_j\bar x_j\) into each residual:

y_i-\widehat b_0-\sum_{j=1}^{p}b_jx_{ij} =(y_i-\bar y)-\sum_{j=1}^{p}b_j(x_{ij}-\bar x_j) =y_i^c-\sum_{j=1}^{p}b_jx_{ij}^c.

Consequently, after minimizing over the intercept, the ridge problem is equivalent to minimizing the centered-data squared errors plus the ridge penalty on the slopes. The lasso problem has the same centered squared-error term and its absolute-value penalty on the slopes. Centering changes neither slope penalty: the penalty remains a function of the original \(b_j\) values.

\begin{aligned} \text{Ridge:}\quad& \min_{b_1,\ldots,b_p} \left\{\sum_{i=1}^{n}\left(y_i^c-\sum_{j=1}^{p}b_jx_{ij}^c\right)^2 +\lambda\sum_{j=1}^{p}b_j^2\right\},\\ \text{Lasso:}\quad& \min_{b_1,\ldots,b_p} \left\{\sum_{i=1}^{n}\left(y_i^c-\sum_{j=1}^{p}b_jx_{ij}^c\right)^2 +\lambda\sum_{j=1}^{p}|b_j|\right\}. \end{aligned}

Thus, one may center the response and each predictor, fit the penalized slope problem to those centered variables without an additional intercept, and then recover the intercept on the original scale as \(\widehat b_0=\bar y-\sum_j\widehat b_j\bar x_j\). If the centered-data fit included its own freely estimated intercept, its least-squares value would be zero for fixed slopes because both the centered response and predictors have mean zero.

Worked Example: Checking Centering With Two Predictors

For three observations, let \(x_1=(0,1,2)\), \(x_2=(1,0,2)\), and \(y=(2,4,6)\). Evaluate the fit at slopes \(b_1=1\), \(b_2=1\). Find the intercept, the fitted values, and the squared-error term. Then calculate the full ridge and lasso objectives when \(\lambda=0.5\).

Method. The means are \(\bar x_1=(0+1+2)/3=1\), \(\bar x_2=(1+0+2)/3=1\), and \(\bar y=(2+4+6)/3=4\). The intercept conditional on the slopes is \(\widehat b_0=4-(1)(1)-(1)(1)=2\).

The fitted values are \(2+0+1=3\), \(2+1+0=3\), and \(2+2+2=6\). The residuals are \(-1,1,0\), so \(RSS=1+1+0=2\).

For ridge, the penalty is \(0.5(1^2+1^2)=1\), giving objective value \(2+1=3\). For lasso, the penalty is \(0.5(|1|+|1|)=1\), also giving objective value \(3\).

Centered-data check. The centered predictors are \(x_1^c=(-1,0,1)\) and \(x_2^c=(0,-1,1)\), while \(y^c=(-2,0,2)\). The centered fitted values at these slopes are \((-1,-1,2)\), so the centered residuals are \((-1,1,0)\) and their squared sum is 2, matching the original-scale calculation. The common mistake is to center the data but forget to restore the original-scale intercept when reporting fitted values.

The intercept is tied to the chosen origin of the response. If the same constant \(c\) is added to every response, then \(\bar y\) increases by \(c\), but every centered response \(y_i-\bar y\) remains unchanged. With an unpenalized intercept, the fitted slopes from either centered-data objective are therefore unchanged, and the fitted intercept increases by \(c\). This is the natural response to changing the response's zero point.

Penalizing the intercept would instead add pressure to keep the baseline near zero. That pressure depends on the arbitrary response origin, unlike the slope penalty, which controls changes in the fitted response as predictors vary. Penalizing the intercept can also prevent residuals from averaging zero. It is not mathematically impossible to define such a model, but it is a different modeling choice and generally lacks the useful centering equivalence above.

Worked Example: How Penalizing the Intercept Changes the Baseline

Consider an intercept-only fit to \(y=(2,4)\). Compare the unpenalized least-squares intercept with an alternative objective that adds a ridge penalty \(2b_0^2\). Repeat the comparison after adding 10 to both responses.

Unpenalized fit. With no predictors, the minimizing intercept is the response mean: \((2+4)/2=3\). The residuals are \(-1,1\), and \(RSS=2\).

Penalized alternative. The objective is \((2-b_0)^2+(4-b_0)^2+2b_0^2\). Its derivative is \(2(b_0-2)+2(b_0-4)+4b_0=8b_0-12\). Setting it to zero gives \(b_0=12/8=1.5\), not the response mean. The residuals are \(0.5,2.5\), which do not sum to zero.

Now shift the responses to \((12,14)\). The unpenalized intercept becomes their mean, 13, and the slopes are irrelevant because there are no predictors. Under the penalized alternative, the objective derivative is \(2(b_0-12)+2(b_0-14)+4b_0=8b_0-52\), giving \(b_0=6.5\). If the earlier penalized estimate simply shifted by 10, it would be 11.5; instead, it moves only to 6.5 because it is pulled toward zero. This illustrates why an intercept penalty can make the baseline depend on the response origin.

Common mistake. Treating the intercept like every slope and adding it to the penalty changes the model. In the usual ridge and lasso objectives here, only slopes are penalized; the intercept is estimated separately.

Practical Checks and Exam Traps

The centering result is an exact algebraic equivalence for the stated objectives, not an approximation. It does not say that the slopes are unpenalized, nor does it specify which slopes ridge or lasso will select. Those are separate questions about the slope optimization.

Do not penalize the intercept by habit. Write the objective explicitly and check whether \(b_0\) appears in its penalty.
Do not use the response mean as the intercept when slopes are nonzero. The correct conditional estimate is \(\bar y-\sum_jb_j\bar x_j\).
Do not confuse centering with standardizing. Centering subtracts a mean. Standardizing also rescales a predictor, which can change how a common penalty affects coefficients.
Do not assume the result applies unchanged to every fitting criterion. The derivation uses unweighted squared errors and a penalty that does not involve the intercept. Other losses or weights require their own derivation.

Key takeaway: With an unpenalized intercept and squared-error loss, the intercept conditional on the slopes is \(\bar y-\sum_jb_j\bar x_j\). Substituting it into the objective centers the response and predictors, leaving ridge or lasso to penalize slopes alone.

Check Your Understanding

For each problem, use the unpenalized-intercept squared-error setup unless the question explicitly changes it.