The Ridge Objective Function

Why Ridge Regression Uses an Objective

Ordinary least squares chooses coefficients to make the observed responses as close as possible to the model’s predictions, measured by the sum of squared residuals. Ridge regression keeps that fit measure but adds a cost for large slope coefficients. The resulting objective makes the trade-off explicit: a coefficient set must fit the data well while also avoiding unnecessarily large coefficient values.

This tutorial considers a continuous response and a linear predictor with \(n\) observations and \(p\) predictors. The ridge objective can be evaluated for any proposed coefficient values; a statistical model is not needed just to do that arithmetic. To interpret the fit as a conventional Gaussian linear regression, suppose \(y_i=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}+\varepsilon_i\), where the errors are independent with mean zero and common variance. If a Gaussian likelihood interpretation is wanted, assume additionally that \(\varepsilon_i\) are normally distributed. Ridge’s squared-error objective can still be used without Gaussian errors, but for a non-Gaussian response, squared error may not be the appropriate model-fitting loss.

A proposed coefficient set has a fitted value \(\hat{y}_i=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}\) for each observation, and residual \(e_i=y_i-\hat{y}_i\). Thus the objective is the sum of the squared residuals plus \(\lambda\) times the sum of the squared slopes. The ridge fit is the coefficient set that minimizes this objective. Calculating \(J\) for one or several proposed coefficient sets does not, by itself, establish which values minimize it.

The intercept convention matters when calculating the penalty: \(\beta_0\) is omitted, but every included slope coefficient is squared and added. A model with no intercept has no \(\beta_0\) term. The objective above is one common scaling convention. Other conventions multiply the RSS and penalty by constants, so their numerical tuning parameter values need not be directly comparable.

Working Through the Objective

Keep the fit term and penalty term separate until both have been calculated. First use the proposed coefficients to obtain every fitted value and residual. Then square and add the residuals. Finally square and add the slope coefficients, multiply by \(\lambda\), and add that penalty to the RSS.

The tuning parameter \(\lambda\) determines how much the objective charges for large slopes. At \(\lambda=0\), the penalty vanishes, and minimizing this objective is the ordinary least-squares criterion. For \(\lambda>0\), a coefficient set with larger slopes pays a larger penalty, all else equal. Increasing \(\lambda\) makes that penalty more influential relative to RSS.

Do not interpret this statement as a guarantee that every individual slope must decrease whenever \(\lambda\) increases. The coefficients are chosen together, and relationships among predictors can make their individual changes complicated. The objective gives the trade-off; its minimizer depends on the data and all predictors.

Why a Penalty Represents a Constraint

The penalty can be understood as a way to express a preference for small coefficients when fitting the data. One related formulation is to minimize RSS subject to a bound on the sum of squared slopes. This provides a short justification for why adding a squared-coefficient term produces a fit-versus-size trade-off.

This derivation shows why the penalized objective is connected to limiting coefficient size. It does not mean that \(\lambda\) itself is the bound \(t\): \(\lambda\) is the multiplier on the penalty, while \(t\) specifies a constraint in the alternative formulation. The relationship between them depends on the data.

Common Exam Traps

• Adding rather than squaring coefficients. The ridge penalty uses \(\sum_j\beta_j^2\), not \(\sum_j\beta_j\) or \(\sum_j|\beta_j|\).
• Applying \(\lambda\) to the whole objective. In the convention here, \(J=\mathrm{RSS}+\lambda\sum_j\beta_j^2\). It is not \(\lambda(\mathrm{RSS}+\sum_j\beta_j^2)\).
• Penalizing the intercept. The displayed convention excludes \(\beta_0\). Follow the stated objective if a problem specifies a different convention.
• Calling a candidate score the minimum. A calculated \(J\) is the objective value at those coefficients. To claim that a coefficient set is the ridge fit, it must minimize \(J\) over the allowed coefficient values.
• Comparing tuning parameters across scaling conventions. Some formulations divide RSS by a sample-size factor or put a constant beside the penalty. Such changes affect the numerical meaning of \(\lambda\). Compare objectives only after writing down their exact conventions.