Why Derive the Ridge Normal Equations?
Ridge regression estimates coefficients by balancing fit to the observed responses against a penalty on coefficient size. The normal equations make that balance explicit: they show exactly how the penalty changes the least-squares system and give a closed-form estimator when the required inverse exists. Deriving the equations is also a useful check on matrix dimensions and on the scaling of the penalty parameter.
The objective’s scaling matters. Here, use the normalized convention \(J(\boldsymbol{\beta})=\frac{1}{2n}\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2+\frac{\lambda}{2}\|\boldsymbol{\beta}\|_2^2\). This convention is useful for stating the derivation clearly. An equivalent unnormalized objective uses a penalty of \(n\lambda\), not \(\lambda\), on the squared coefficient norm. The resulting estimator is the same when the penalty is translated consistently.
The dimensions are worth checking before differentiating. The residual vector \(\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\) is \(n\times1\), so its squared length is a scalar. The matrix \(\mathbf{X}^\top\mathbf{X}\) is \(p\times p\), and \(\mathbf{X}^\top\mathbf{y}\) is \(p\times1\). Thus the coefficient equation must be a \(p\times p\) matrix multiplying a \(p\times1\) vector, with a \(p\times1\) right-hand side.
Deriving the Ridge Normal Equations
Expand the squared residual term and use the symmetry of \(\mathbf{X}^\top\mathbf{X}\). The resulting expression is a quadratic function of \(\boldsymbol{\beta}\). Its gradient combines the contribution from the data-fit term with the contribution from the penalty.
\((\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^\top(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})=\mathbf{y}^\top\mathbf{y}-2\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{y}+\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\).
The gradient of the fit term is \(-\mathbf{X}^\top\mathbf{y}/n+\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}/n\). The gradient of \(\lambda\boldsymbol{\beta}^\top\boldsymbol{\beta}/2\) is \(\lambda\boldsymbol{\beta}\).
Multiplying the resulting equation by \(n\) gives \((\mathbf{X}^\top\mathbf{X}+n\lambda\mathbf{I}_p)\boldsymbol{\beta}=\mathbf{X}^\top\mathbf{y}\).
The derivative can also be justified directly from the quadratic expansion. For a symmetric matrix \(\mathbf{A}\), the gradient of \(\boldsymbol{\beta}^\top\mathbf{A}\boldsymbol{\beta}\) is \(2\mathbf{A}\boldsymbol{\beta}\); the gradient of \(\mathbf{c}^\top\boldsymbol{\beta}\) is \(\mathbf{c}\). Here \(\mathbf{A}=\mathbf{X}^\top\mathbf{X}\), which is symmetric, and the cross-term is \(-2\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{y}\). Applying these results to the expanded objective gives
At a minimizer the gradient is zero. Multiplying by \(n\), collecting terms in \(\boldsymbol{\beta}\), and introducing the \(p\times p\) identity matrix \(\mathbf{I}_p\) gives the normal equations:
This condition concerns the existence of the inverse in the formula; it does not assert that every choice of \(\lambda\) is appropriate for prediction. The tuning parameter is treated as fixed during this derivation. Selecting or comparing tuning parameters is a separate task.
Worked Examples
Worked Example: Solve a Two-Coefficient Ridge System
Suppose \(n=4\), \(p=2\), \(\mathbf{X}^\top\mathbf{X}=\begin{bmatrix}6&2\\2&5\end{bmatrix}\), \(\mathbf{X}^\top\mathbf{y}=\begin{bmatrix}8\\7\end{bmatrix}\), and \(\lambda=0.5\). Using the normalized objective defined above, calculate the ridge estimate and verify the normal equations.
Method. Form \(\mathbf{X}^\top\mathbf{X}+n\lambda\mathbf{I}_p\), then solve the two-by-two system. The penalty added to the diagonal is \(n\lambda=4(0.5)=2\), so
Its determinant is \(8(7)-2(2)=56-4=52\). The inverse formula for a two-by-two matrix gives
To check, multiply the system matrix by the estimate: its first component is \(8(21/26)+2(10/13)=168/26+40/26=8\), and its second is \(2(21/26)+7(10/13)=42/26+140/26=7\). These equal the components of \(\mathbf{X}^\top\mathbf{y}\). The estimate is approximately \((0.8077,0.7692)^\top\), rounded to four decimal places. The common mistake avoided is adding \(\lambda\mathbf{I}_p\) instead of \(n\lambda\mathbf{I}_p\) under the normalized objective.
Worked Example: Build the Equation From a Design Matrix
A hypothetical calibration study has three observations, two predictors, design matrix \(\mathbf{X}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\), and response vector \(\mathbf{y}=\begin{bmatrix}2\\1\\3\end{bmatrix}\). With \(\lambda=1/3\), calculate the ridge estimate. Show how the data summaries are formed and check the result.
Method. Here \(n=3\), \(p=2\), and \(n\lambda=1\). The cross-product matrix and response cross-product are
The coefficient system is therefore \(\begin{bmatrix}3&1\\1&3\end{bmatrix}\widehat{\boldsymbol{\beta}}_{\mathrm{ridge}}=\begin{bmatrix}5\\4\end{bmatrix}\). The determinant is \(3(3)-1(1)=8\), so
Check the system: its left side is \(\begin{bmatrix}3(11/8)+7/8\\11/8+3(7/8)\end{bmatrix}=\begin{bmatrix}40/8\\32/8\end{bmatrix}=\begin{bmatrix}5\\4\end{bmatrix}\), as required. Thus the estimate is \((1.375,0.875)^\top\). This calculation avoids treating rows as predictors: the design matrix has three rows because there are three observations, and its two columns correspond to the two coefficients.
Worked Example: Translate Between Penalty Conventions
For a hypothetical data set with \(n=6\), suppose \(\mathbf{X}^\top\mathbf{X}=\begin{bmatrix}10&2\\2&8\end{bmatrix}\), \(\mathbf{X}^\top\mathbf{y}=\begin{bmatrix}14\\10\end{bmatrix}\), and the normalized-objective tuning parameter is \(\lambda=1/6\). Find the estimate, then state the equivalent penalty coefficient for an unnormalized residual sum of squares.
Method. Since \(n\lambda=6(1/6)=1\), the matrix in the normal equations is \(\begin{bmatrix}11&2\\2&9\end{bmatrix}\). Its determinant is \(11(9)-2(2)=99-4=95\). Therefore
The estimate is approximately \((1.1158,0.8632)^\top\), rounded to four decimal places. In the equivalent unnormalized objective \(\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2+\kappa\|\boldsymbol{\beta}\|_2^2\), the penalty coefficient is \(\kappa=n\lambda=1\). Substitution verifies the first normal equation component: \(11(106/95)+2(82/95)=(1166+164)/95=14\). The second is \(2(106/95)+9(82/95)=(212+738)/95=10\). The common mistake avoided is carrying the same numerical penalty value from one objective scaling to the other without converting it.
Penalty Scaling and Common Exam Traps
A second common convention writes the objective without dividing the residual sum of squares by \(2n\). For clarity, let that objective be \(\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2/2+\kappa\|\boldsymbol{\beta}\|_2^2/2\). Its gradient is \(-\mathbf{X}^\top\mathbf{y}+\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}+\kappa\boldsymbol{\beta}\), so its normal equations use \(\mathbf{X}^\top\mathbf{X}+\kappa\mathbf{I}_p\). It matches the normalized objective when \(\kappa=n\lambda\).
For example, with the first worked example’s \(n=4\) and normalized \(\lambda=0.5\), the equivalent unnormalized penalty is \(\kappa=2\). The correct matrix is \(\begin{bmatrix}8&2\\2&7\end{bmatrix}\). Incorrectly adding only \(0.5\mathbf{I}_2\) gives \(\begin{bmatrix}6.5&2\\2&5.5\end{bmatrix}\), with determinant \(6.5(5.5)-4=31.75\). That incorrect system yields coefficients \(30/31.75=0.9449\) and \(29.5/31.75=0.9291\), rather than the correctly calculated \(0.8077\) and \(0.7692\). The issue is not matrix inversion; it is using a penalty from one convention in the normal equations for another.
| Objective convention | Matrix added to \(\mathbf{X}^\top\mathbf{X}\) | Penalty relationship |
|---|---|---|
| \(\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2/(2n)+\lambda\|\boldsymbol{\beta}\|_2^2/2\) | \(n\lambda\mathbf{I}_p\) | \(\kappa=n\lambda\) |
| \(\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2/2+\kappa\|\boldsymbol{\beta}\|_2^2/2\) | \(\kappa\mathbf{I}_p\) | \(\lambda=\kappa/n\) |
When solving by hand, keep the matrix order, dimensions, and scaling visible. In particular, \(\mathbf{I}_p\) must have one row and column per coefficient, and the penalty changes the diagonal of \(\mathbf{X}^\top\mathbf{X}\), not the right-hand side \(\mathbf{X}^\top\mathbf{y}\). A final multiplication check is often the quickest way to catch an arithmetic or transcription error.
Check Your Understanding
For each problem, use the normalized objective defined in this tutorial and check dimensions and penalty scaling.
- Let \(n=2\), \(p=1\), \(\mathbf{X}^\top\mathbf{X}=[4]\), \(\mathbf{X}^\top\mathbf{y}=[6]\), and \(\lambda=0.5\). What is the ridge estimate? (Answer: \(6/(4+2(0.5))=1.2\).)
- For \(n=3\), \(\mathbf{X}^\top\mathbf{X}=\begin{bmatrix}4&1\\1&3\end{bmatrix}\), \(\mathbf{X}^\top\mathbf{y}=\begin{bmatrix}5\\4\end{bmatrix}\), and \(\lambda=1/3\), state the matrix in the normal equations and solve for the estimate. (Answer: the matrix is \(\begin{bmatrix}5&1\\1&4\end{bmatrix}\), and the estimate is \(\begin{bmatrix}16/19\\15/19\end{bmatrix}\).)
- A data set has \(n=8\), and a software objective uses an unnormalized residual sum of squares with penalty coefficient \(\kappa=2\). What normalized-objective value of \(\lambda\) gives the same estimator? (Answer: \(\lambda=\kappa/n=2/8=0.25\).)
- For \(n=4\), \(\mathbf{X}^\top\mathbf{X}=\begin{bmatrix}5&1\\1&4\end{bmatrix}\), \(\mathbf{X}^\top\mathbf{y}=\begin{bmatrix}6\\5\end{bmatrix}\), and \(\lambda=0.25\), calculate the ridge estimate. (Answer: the system matrix is \(\begin{bmatrix}6&1\\1&5\end{bmatrix}\), with determinant \(29\), and the estimate is \(\begin{bmatrix}25/29\\24/29\end{bmatrix}\).)
- For \(\lambda=0\), what condition on \(\mathbf{X}\) ensures that the inverse-form estimator is defined? (Answer: \(\mathbf{X}\) must have full column rank \(p\), so that \(\mathbf{X}^\top\mathbf{X}\) is invertible.)