Why Penalize Least Squares

Why Least Squares Can Be Unstable

Least squares chooses coefficients to make the observed responses as close as possible to the fitted values. When predictors carry overlapping information, several very different coefficient combinations can produce almost the same fitted values. A small change in the data can then cause large changes in the estimated coefficients, even when the predictions barely change. The difficulty is especially clear when there are more predictors than observations: in that setting, least squares may not determine a unique coefficient vector at all.

A penalty offers a way to prefer some coefficient vectors over others. It discourages large coefficients, which can stabilize estimates, but the resulting estimates are generally biased toward zero. Penalization is therefore not a free improvement. Its purpose is to exchange some bias for potential gains in stability and prediction accuracy.

Model and assumptions: For the explanation below, use the fixed-predictor linear model \(Y=X\beta+\varepsilon\), where \(Y\) is the response vector, \(X\) is the design matrix, and \(\beta\) is the coefficient vector. Assume \(E(\varepsilon\mid X)=0\) and \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I\). For the ordinary least-squares formula involving an inverse, also assume the columns of \(X\) are linearly independent. These assumptions support the coefficient calculations; they do not make associations causal.

With linearly independent columns, ordinary least squares selects \(\hat{\beta}\) to minimize the residual sum of squares, \(\lVert Y-Xb\rVert^2\), over coefficient vectors \(b\). Its solution is \(\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\). Under the stated model, its conditional variance is \(\sigma^2(X^\mathsf{T}X)^{-1}\). The inverse matters: when columns of \(X\) are nearly dependent, \(X^\mathsf{T}X\) has a direction with very little information, and the corresponding coefficient estimates can have high variance.

Correlated Predictors: Unstable Coefficients, Similar Fits

Suppose two predictors tend to rise and fall together. The data then contain limited information about how to divide their shared association with the response between their two coefficients. One coefficient may increase while the other decreases, leaving the combined fitted values similar. This is a coefficient-stability problem, not necessarily a problem with the overall fit.

Worked Example: A Small Data Change With Correlated Predictors

Consider a hypothetical regression with two centered predictors. Each has a sum of squares of 100, and their correlation is 0.98. For these predictors, suppose the observed cross-products with the response are \(X^\mathsf{T}Y=(100.8,97.2)^\mathsf{T}\). Calculate the least-squares coefficients, then see what happens if those cross-products change by only \((1,-1)^\mathsf{T}\).

The predictor cross-product matrix is

X^\mathsf{T}X= \begin{pmatrix} 100 & 98\\ 98 & 100 \end{pmatrix}.

The first coefficient vector, \((1.4,-0.4)^\mathsf{T}\), solves the least-squares equations because

\begin{pmatrix} 100 & 98\\ 98 & 100 \end{pmatrix} \begin{pmatrix} 1.4\\ -0.4 \end{pmatrix} = \begin{pmatrix} 100.8\\ 97.2 \end{pmatrix}.

Now change the cross-products to \((101.8,96.2)^\mathsf{T}\). The resulting change in the coefficient vector is

\begin{pmatrix} 100 & 98\\ 98 & 100 \end{pmatrix}^{-1} \begin{pmatrix} 1\\ -1 \end{pmatrix} = \frac{1}{396} \begin{pmatrix} 100 & -98\\ -98 & 100 \end{pmatrix} \begin{pmatrix} 1\\ -1 \end{pmatrix} = \begin{pmatrix} 0.5\\ -0.5 \end{pmatrix}.

Thus the new estimates are \((1.9,-0.9)^\mathsf{T}\): a change of just one unit in each cross-product has shifted each coefficient by 0.5. The matrix inverse used here follows from its determinant, \(100(100)-98(98)=396\). The predictors’ high correlation makes the difference between their effects difficult to pin down. This calculation illustrates sensitivity; it does not establish that the underlying effects truly changed.

Why the Instability Occurs

The source of the sensitivity can be seen from the two-predictor cross-product matrix. If both predictors have sum of squares \(S\) and correlation \(r\), the matrix is \(S\begin{pmatrix}1&r\\r&1\end{pmatrix}\). Its directions \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) have eigenvalues \(S(1+r)\) and \(S(1-r)\), respectively. When \(r\) is close to 1, the second eigenvalue is small: changes in the data that distinguish the predictors are amplified when the matrix is inverted.

Here is the corresponding variance calculation. From the model assumptions, \(\operatorname{Var}(\hat{\beta}\mid X)=\sigma^2(X^\mathsf{T}X)^{-1}\). For a symmetric two-by-two matrix of this form, the inverse has diagonal entries \(1/[S(1-r^2)]\). Therefore, the conditional variance of either coefficient is \(\sigma^2/[S(1-r^2)]\). As \(r\) approaches 1, this variance increases. The fitted values can still be comparatively stable because the data constrain the predictors’ combined contribution more strongly than their separate coefficients.

Worked Example: Comparing Coefficient Variance at Two Correlations

Suppose two centered predictors each have sum of squares \(S=100\), and the error variance is \(\sigma^2=4\). Compare the conditional variance of either least-squares coefficient when their correlation is 0.50 and when it is 0.98.

The design is full rank in both cases because neither correlation is exactly 1, so the two-predictor variance formula applies. At \(r=0.50\),

\operatorname{Var}(\hat{\beta}_j\mid X) = \frac{4}{100(1-0.50^2)} = \frac{4}{75} \approx 0.05333.

At \(r=0.98\),

\operatorname{Var}(\hat{\beta}_j\mid X) = \frac{4}{100(1-0.98^2)} = \frac{4}{3.96} \approx 1.010.

Under these assumptions, the coefficient variance is about \(1.010/0.05333\approx18.94\) times as large at correlation 0.98. These are variances in squared coefficient units; the standard deviations are their square roots. The comparison isolates how the design’s correlation affects coefficient uncertainty, holding the predictor sums of squares and error variance fixed.

Predictors Outnumbering Observations

The same underlying issue becomes more severe when the model includes more predictor columns than observations. A design matrix with \(n\) rows and \(p\) columns has rank at most \(n\). If \(p>n\), its columns cannot all be linearly independent, so \(X^\mathsf{T}X\) cannot be inverted. Moreover, if one coefficient vector fits the responses exactly, adding any vector \(v\) satisfying \(Xv=0\) produces another coefficient vector with exactly the same fitted values. Least squares alone cannot choose among these alternatives.

Worked Example: More Predictors Than Observations

Consider a hypothetical data set with three observations and five predictors, whose predictor columns are \(x_1=(1,0,0)^\mathsf{T}\), \(x_2=(0,1,0)^\mathsf{T}\), \(x_3=(0,0,1)^\mathsf{T}\), \(x_4=x_1+x_2\), and \(x_5=x_2+x_3\). Let the response vector be \(Y=(2,3,4)^\mathsf{T}\). Show that there are distinct coefficient vectors with the same least-squares fit.

One exact fit is \(\beta=(2,3,4,0,0)^\mathsf{T}\), since \(X\beta=Y\). A different coefficient vector is \(\beta=(1,1,3,1,1)^\mathsf{T}\). Its fitted response is

X\beta = x_1+ x_2+3x_3+x_4+x_5 = (1,0,0)^\mathsf{T}+(0,1,0)^\mathsf{T} +3(0,0,1)^\mathsf{T}+(1,1,0)^\mathsf{T}+(0,1,1)^\mathsf{T} = \begin{pmatrix}2\\3\\4\end{pmatrix}.

Both vectors therefore have residual sum of squares zero, although their coefficients differ. In this example \(p=5>n=3\), and the predictor columns are dependent by construction. Ordinary least squares identifies the fitted response but not a unique coefficient vector. In a real analysis, this ambiguity makes it important to state how a model is chosen rather than treating one arbitrary least-squares solution as uniquely determined.

What a Penalty Adds

Penalized least squares adds a cost for coefficient size to the residual sum of squares. In general terms, it minimizes the sum of the residual sum of squares and a nonnegative penalty multiplied by a tuning amount. Ridge regression uses a penalty based on squared coefficients; lasso uses a penalty based on absolute coefficients. Both can discourage extreme coefficient values, but they do so differently. The key idea here is the reason for adding a penalty, not the detailed comparison of their solutions.

For a simple illustration, consider a model through the origin with one coefficient and objective

\sum_{i=1}^{n}(y_i-bx_i)^2+\lambda b^2, \qquad \lambda>0.

The second term makes large values of \(b\) less attractive. Minimizing this objective gives \(\hat{\beta}_{\lambda}=\sum_i x_i y_i/(\sum_i x_i^2+\lambda)\), rather than the ordinary least-squares estimate \(\hat{\beta}_{\text{LS}}=\sum_i x_i y_i/\sum_i x_i^2\). The denominator increase pulls the penalized estimate toward zero. That pull is the source of its bias when the true coefficient is not zero.

Worked Example: A Penalty Reduces Variance but Adds Bias

Suppose five hypothetical observations follow a through-origin model, with \(x=(1,1,1,1,2)^\mathsf{T}\) and observed \(y=(3,3,3,3,6)^\mathsf{T}\). Then \(\sum_i x_i^2=8\) and \(\sum_i x_i y_i=24\). Compare ordinary least squares with the squared-coefficient penalty above using \(\lambda=4\).

The ordinary least-squares estimate is

\hat{\beta}_{\text{LS}}=\frac{24}{8}=3.

The penalized estimate is

\hat{\beta}_{\lambda}=\frac{24}{8+4}=2.

To make the bias-variance trade explicit, suppose the true coefficient is \(\beta=2\) and the independent errors have mean zero and variance \(\sigma^2=10\). With fixed predictors, \(E(\hat{\beta}_{\text{LS}}\mid X)=2\) and \(\operatorname{Var}(\hat{\beta}_{\text{LS}}\mid X)=10/8=1.25\). For the penalized estimate,

E(\hat{\beta}_{\lambda}\mid X)=\frac{8}{12}\beta=\frac{4}{3}, \qquad \operatorname{Bias}(\hat{\beta}_{\lambda}\mid X)=\frac{4}{3}-2=-\frac{2}{3}, \qquad \operatorname{Var}(\hat{\beta}_{\lambda}\mid X)=\frac{10(8)}{12^2}=\frac{5}{9}.

Thus the penalty reduces conditional variance from 1.25 to about 0.556, while introducing bias of \(-0.667\). For squared error, the conditional mean squared error is variance plus squared bias: \(1.25\) for ordinary least squares and \(5/9+(-2/3)^2=1\) for the penalized estimate. In this specified model, the penalized estimate has lower mean squared error, even though it is biased. This is a repeated-sampling comparison under the assumed model, not a guarantee that penalization improves every estimate or data set.

Why the Trade Can Matter

If coefficient estimates are highly variable, reducing that variability can make predictions more dependable across new samples. This is particularly relevant when predictors are strongly correlated or numerous relative to the available observations. A penalty can also produce a unique estimate in settings where least squares does not identify a unique coefficient vector. But a stable estimate is not automatically an accurate one: excessive shrinkage can move coefficients too far from their true values, and a penalty can change which predictors appear important.

The overall goal is often prediction, not recovering a perfectly unbiased coefficient for every predictor. For prediction, a modest amount of bias may be worthwhile if it produces a large enough reduction in variance. The balance depends on the data and the analysis goal; the numerical value of a penalty cannot be judged in isolation. Predictor units also affect coefficient sizes, so the treatment of predictor scales must be considered when constructing a penalized model.

Common Mistakes

Assuming similar predictions mean stable coefficients. With correlated predictors, many coefficient combinations can give similar fitted values. Examine coefficient sensitivity as well as prediction performance.
Assuming a penalty removes uncertainty without a cost. Penalization generally introduces bias. It may reduce variance, but whether the total prediction error improves depends on the setting.
Assuming least squares is uniquely defined when predictors outnumber observations. If \(p>n\), the design cannot have full column rank. There may be multiple coefficient vectors with the same fit.
Interpreting a shrunken coefficient as proof of a smaller real effect. Shrinkage reflects the fitting procedure as well as the observed data; a penalized estimate is not, by itself, evidence of a causal effect.

Key takeaway: Least squares can have high coefficient variance when predictors are nearly redundant, and it may have no unique coefficient solution when predictors outnumber observations. A penalty can stabilize estimates by discouraging large coefficients, but it trades some bias for that potential stability. Whether the trade is useful depends on the data and the goal of the analysis.

Check Your Understanding

Try these questions without looking back.

Why can two highly correlated predictors have unstable individual coefficients even when their combined fitted values change little?
For predictors with equal sum of squares \(S\) and correlation \(r\), what happens to the conditional variance formula \(\sigma^2/[S(1-r^2)]\) as \(r\) approaches 1?
Explain why a design matrix with more columns than rows cannot have full column rank.
In the one-coefficient penalized example, why does the penalty generally introduce bias when the true coefficient is nonzero?
Give one reason a biased penalized estimate might still be useful, and one reason it might not be.

What You'll Learn