Ridge and Lasso for Manufacturing Quality Prediction

Why Penalize a Manufacturing Prediction Model?

A production line may record many measurements on each batch: temperature, pressure, cycle time, vibration, and several related sensor summaries. These measurements can be strongly correlated. A prediction model using all of them may fit the observed batches, yet small changes in the data can produce large changes in its individual regression coefficients. That makes the coefficients unstable and can also hurt predictions on new batches.

Ridge and lasso regression address this problem by adding a penalty to the fitting criterion. The penalty discourages large coefficients; lasso can additionally set some coefficients exactly to zero. A tuning value controls the strength of the penalty and is chosen to improve prediction on data not used to fit each candidate model. These methods are useful for prediction, but their penalized coefficients should not be treated as if they were ordinary least-squares estimates with ordinary standard errors and p-values.

Setup and Penalized Regression Objectives

Suppose there are \(n\) manufacturing batches, a quantitative outcome \(y_i\) measuring product quality for batch \(i\), and \(p\) quantitative predictors \(x_{ij}\), where \(j\) identifies a manufacturing measurement. The aim here is to predict quality from the measurements. Before fitting a penalized model, center the outcome and center and scale each predictor using the training data. Scaling is important because a penalty on coefficient size would otherwise depend on the units used: a measurement in micrometers and the same measurement expressed in millimeters could receive different effective penalties.

For clarity, the objectives below use centered \(y_i\), centered predictors \(z_{ij}\), and a training sample of size \(n\). The predictors have been scaled to a common scale, such as having mean squared value \(1\) in that training sample. The intercept is therefore zero in this centered representation. In the original measurement scale, the fitted intercept is recovered by undoing the centering.

At \(\lambda=0\), either objective reduces to the ordinary least-squares criterion. At positive \(\lambda\), the penalty trades some training fit for smaller coefficients. Ridge uses the sum of squared coefficients, so it generally retains all predictors with nonzero coefficients. Lasso uses the sum of absolute coefficients and can set some fitted coefficients exactly to zero. A zero lasso coefficient means that predictor is not used in that fitted linear prediction; it does not establish that the corresponding population relationship is absent.

The amount of shrinkage is governed by the penalty and the data together. In ridge regression, the coefficient vector's Euclidean norm does not increase as \(\lambda\) increases, but an individual coefficient need not decrease monotonically in absolute value. With correlated predictors, the penalty changes combinations of coefficients, so a particular coordinate can increase even while the overall vector shrinks. Lasso coefficients also depend jointly on the data and the penalty; changing \(\lambda\) can change which predictors have nonzero coefficients.

What Correlation Does to Ridge Coefficients

A short derivation makes the ridge behavior precise. Let \(Z\) be the centered, scaled predictor matrix, \(R=Z^\mathsf{T}Z/n\), and \(r=Z^\mathsf{T}y/n\). When the ridge solution is unique, differentiating its objective and setting the derivative equal to zero gives \((R+2\lambda I)\hat{\beta}_{\text{ridge}}=r\), under the penalty convention displayed above. Thus the solution is \((R+2\lambda I)^{-1}r\). The factor \(2\) follows from writing the penalty as \(\lambda\sum_j\beta_j^2\); alternative conventions can absorb that factor into \(\lambda\).

The matrix \(R\) is positive semidefinite. In an orthonormal eigenvector direction of \(R\) with eigenvalue \(d\geq 0\), the corresponding component of the ridge solution is the corresponding component of \(r\) divided by \(d+2\lambda\). As \(\lambda\) grows, each such component is reduced in magnitude. Consequently the squared Euclidean norm, which is the sum of the squared components in these orthogonal directions, cannot increase. But the ordinary predictor coefficients are mixtures of those eigenvector components, so an individual coefficient can increase in magnitude as those components change.

For example, take \(R=\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}\), whose eigenvectors are \((1,1)/\sqrt{2}\) and \((1,-1)/\sqrt{2}\), and let the components of \(r\) along those directions be \(0.95\) and \(-0.045\). The first coordinate of the ridge solution is about \(0.0354\) at \(\lambda=0\), but about \(0.1571\) at \(\lambda=1\) using the displayed penalty convention with \(2\lambda\). The important general result is the nonincreasing norm, not monotonic shrinkage of every individual coefficient.

How Lasso Can Set a Coefficient to Zero

Lasso's absolute-value penalty has a corner at zero, which permits exact zeros. The basic calculation is especially clear when two standardized predictors are uncorrelated, so their cross-products are zero. In that case each coefficient can be considered separately. If \(r_j=z_j^\mathsf{T}y/n\) is the predictor-outcome cross-product for predictor \(j\), minimizing its part of the lasso objective gives \(\hat{\beta}_j=\operatorname{sign}(r_j)(|r_j|-\lambda)_+\), where \((a)_+=\max(a,0)\). This is the soft-thresholding rule: the coefficient is moved toward zero by \(\lambda\), and is zero when \(|r_j|\leq\lambda\). With correlated predictors, lasso coefficients are coupled and this separate formula does not generally apply.

Selecting the Penalty by Cross-Validation

The tuning value \(\lambda\) should not be chosen just because it makes the training error small. Increasing \(\lambda\) typically raises training error while potentially improving prediction on new batches. Cross-validation estimates prediction error by repeatedly fitting a candidate model on part of the available data and evaluating it on held-out folds. A candidate's mean validation error is the average of its fold errors; the selected tuning value is the one with the smallest mean, or another value chosen by a stated rule.

All data-dependent preparation must be learned using only the training part of each fold. In particular, calculate predictor means and scales within the fold's fitting portion, then apply those values to its held-out portion. Calculating them once from the complete data would allow information from validation observations to enter the fitting process. If the intended use is prediction for future production, folds should also reflect that use: for example, batches from the same production run should not be split across fitting and validation portions when doing so would make validation unrealistically easy.

Interpretation, Assumptions, and a Common Pitfall

These methods are prediction tools. Their usefulness depends on having observations that represent the situations where predictions will be used, measuring predictors and quality comparably, and evaluating the model on data that were not used to choose its coefficients or tuning value. The linear predictor must also be a reasonable approximation for the intended prediction task. Large outliers, changes in production conditions, or dependence among batches can make validation performance unrepresentative unless the validation design reflects those features. Normal errors are not required to calculate ridge or lasso predictions, though error distributions matter for some forms of uncertainty assessment.

Cross-validation scores vary across possible fold assignments, and searching many tuning values can make the minimum observed score optimistic. A separate test set helps assess the performance of a fully selected model, provided it remains untouched until the final evaluation. A low validation error also does not establish that changing a manufacturing measurement would cause a quality change.

A frequent pitfall is to read a nonzero lasso coefficient as a statistically significant effect, or to compute ordinary least-squares standard errors and p-values after shrinkage and selection as though nothing had changed. Shrinkage deliberately biases coefficients toward zero, and lasso's data-dependent selection adds further complications. These coefficients can help form predictions, but ordinary regression inference does not automatically apply to them. Use appropriate inferential methods when the goal is estimation or testing rather than prediction.