Ridge and Lasso Regression in Biostatistics

Introduction

Clinical datasets often contain many candidate predictors: demographic variables, baseline laboratory measurements, disease characteristics, biomarkers, treatment indicators, concomitant medications, and longitudinal summaries. These predictors are frequently correlated with one another. A conventional ordinary least-squares (OLS) regression can still be fitted in this setting, but correlated predictors can produce unstable coefficient estimates and make prediction difficult when the number of candidate variables becomes large relative to the sample size.

Ridge regression and lasso regression are two penalized regression methods that address this problem by adding a penalty to the size of the regression coefficients. The penalty discourages unnecessarily large coefficients and can improve out-of-sample prediction.

The two methods differ in an important way. Ridge generally shrinks coefficients toward zero without setting them exactly to zero, whereas lasso can set some coefficients exactly to zero. Lasso therefore provides a form of automatic variable selection in addition to shrinkage.

Key idea: Penalization trades a small amount of bias for potentially much greater stability and predictive performance. The goal is usually not to reproduce the OLS coefficients, but to obtain a model that behaves better when applied to new patients.

Why Ordinary Least Squares Can Be Problematic

Consider a continuous clinical outcome \(Y\) and predictors \(X_1,\ldots,X_p\). The ordinary linear model is:

Y_i=\beta_0+\beta_1X_{i1}+\beta_2X_{i2}+\cdots+\beta_pX_{ip}+\varepsilon_i

OLS chooses the coefficients by minimizing the residual sum of squares:

\operatorname{RSS}(\boldsymbol{\beta}) = \sum_{i=1}^{n} \left( y_i-\beta_0-\sum_{j=1}^{p}\beta_jx_{ij} \right)^2

When predictors are highly correlated, many combinations of coefficients can produce similar fitted values. The fitted predictions may remain reasonable while the individual coefficients become unstable. A small change in the data can then produce a large change in the estimated coefficient vector.

This situation is common in biostatistics. For example, body weight, body-surface area, lean body mass, and several laboratory measurements may carry overlapping information. In biomarker studies, multiple markers may be strongly correlated because they participate in the same biological pathway.

Important distinction: Multicollinearity is primarily a problem of coefficient stability and interpretation. It does not necessarily mean that the overall fitted values or predictive accuracy are poor. Penalized regression is particularly attractive when prediction is an important goal.

The Basic Penalized Regression Idea

Ridge and lasso modify the OLS objective by adding a penalty for large coefficients. A general penalized objective can be written as:

\text{Loss}(\boldsymbol{\beta}) + \lambda\,P(\boldsymbol{\beta})

where \(\lambda\ge0\) controls the strength of penalization. When \(\lambda=0\), the penalty disappears and the method reduces to OLS for the corresponding linear model. As \(\lambda\) increases, the coefficients are pushed more strongly toward zero.

The intercept is normally not penalized. In practice, predictors are commonly standardized before fitting a penalized model so that the penalty treats coefficients on comparable scales.

Ridge Regression

Ridge regression uses an \(L_2\) penalty, meaning that it penalizes the squared coefficients:

\hat{\boldsymbol{\beta}}_{\text{ridge}} = \arg\min_{\beta_0,\boldsymbol{\beta}} \left[ \sum_{i=1}^{n} \left( y_i-\beta_0-\sum_{j=1}^{p}\beta_jx_{ij} \right)^2 + \lambda\sum_{j=1}^{p}\beta_j^2 \right]

The term \(\lambda\sum_j\beta_j^2\) becomes increasingly expensive as coefficients become large. The fitted model therefore tends to distribute information across correlated predictors rather than allowing individual coefficients to become extreme.

What Ridge Does to the Coefficients

For \(\lambda>0\), ridge shrinks the estimated coefficients toward zero. However, except in special cases, it does not make coefficients exactly zero.

Feature	Ridge behavior
Penalty	\(L_2\): sum of squared coefficients
Coefficient shrinkage	Yes
Exact zeros	Generally no
Variable selection	Not automatic
Correlated predictors	Often handled well by sharing/shrinking effects
Main strength	Stable prediction when many predictors contribute small amounts

Why Ridge Helps With Multicollinearity

Suppose two biomarkers are highly correlated. In OLS, their coefficients may be large and opposite in sign even when their combined contribution to prediction is modest. Ridge discourages this type of coefficient instability by making large coefficient values costly.

The result is a biased but typically more stable estimator. That trade-off can be valuable when the primary objective is prediction rather than unbiased estimation of every individual coefficient.

Lasso Regression

Lasso uses an \(L_1\) penalty, based on the absolute values of the coefficients:

\hat{\boldsymbol{\beta}}_{\text{lasso}} = \arg\min_{\beta_0,\boldsymbol{\beta}} \left[ \sum_{i=1}^{n} \left( y_i-\beta_0-\sum_{j=1}^{p}\beta_jx_{ij} \right)^2 + \lambda\sum_{j=1}^{p}|\beta_j| \right]

The absolute-value penalty has a distinctive consequence: some estimated coefficients can become exactly zero. Variables whose coefficients are zero do not contribute to the fitted lasso model.

Key distinction: Ridge is primarily a shrinkage method. Lasso is both a shrinkage method and a variable-selection method. This does not mean that a nonzero lasso coefficient is automatically a clinically important predictor; selection remains data-dependent and should be interpreted carefully.

Ridge Versus Lasso

Property	Ridge	Lasso
Penalty	\(L_2\)	\(L_1\)
Penalty term	\(\sum_j\beta_j^2\)	\(\sum_j\|\beta_j\|\)
Shrinks coefficients	Yes	Yes
Can set coefficients exactly to zero	Generally no	Yes
Automatic variable selection	No	Yes
Useful with many small effects	Often	Sometimes less so
Correlated predictor groups	Often retains them with shared shrinkage	May select one or a subset, depending on the data

The Elastic Net Connection

Ridge and lasso are endpoints of a broader family called the elastic net. Elastic net combines the \(L_1\) and \(L_2\) penalties:

\operatorname{Loss} + \lambda \left[ \alpha\sum_{j=1}^{p}|\beta_j| + (1-\alpha)\sum_{j=1}^{p}\beta_j^2 \right]

Here \(0\le\alpha\le1\). Roughly speaking, \(\alpha=1\) gives lasso and \(\alpha=0\) gives ridge, subject to the exact scaling convention used by the software.

Elastic net is particularly useful when there are groups of highly correlated predictors and the analyst wants both shrinkage and some degree of sparse variable selection.

What Does the Tuning Parameter \(\lambda\) Do?

The penalty parameter \(\lambda\) controls the bias-variance trade-off.

\(\lambda=0\): no penalty; the fitted model corresponds to ordinary least squares.

Small \(\lambda\): mild shrinkage; coefficients remain relatively close to the unpenalized estimates.

Moderate \(\lambda\): stronger shrinkage can reduce variance and improve generalization.

Large \(\lambda\): coefficients are strongly compressed toward zero and the model may underfit.

The optimal value of \(\lambda\) is therefore usually not chosen by looking at the training fit alone. A model with a smaller training error can still perform worse on new patients.

Cross-Validation for Choosing \(\lambda\)

A common approach is \(K\)-fold cross-validation. The dataset is divided into \(K\) folds. For each candidate value of \(\lambda\), the model is trained on \(K-1\) folds and evaluated on the remaining fold. The process is repeated until each fold has served as validation data.

The cross-validation error is then averaged across folds:

CV(\lambda) = \frac{1}{K} \sum_{k=1}^{K} L_k(\lambda)

where \(L_k(\lambda)\) is the validation loss for fold \(k\). The analyst commonly selects the value of \(\lambda\) with the lowest estimated cross-validation error.

The One-Standard-Error Rule

Many implementations also provide a more strongly penalized choice called \(\lambda_{\mathrm{1SE}}\). It is the largest penalty whose estimated cross-validation error is within one standard error of the minimum.

This choice often produces a simpler model with performance close to the cross-validation minimum. It can be attractive when parsimony matters, although the choice should be prespecified or justified rather than made opportunistically.

Practical rule: Do not select \(\lambda\) by choosing the value that makes the coefficients look most clinically plausible. Let the model-selection procedure operate on the prespecified prediction criterion, and then interpret the selected model in the context of the study.

Why Standardization Matters

Suppose one predictor is measured in kilograms and another in milligrams per deciliter. The numerical scales can differ substantially. Because ridge and lasso penalize coefficient magnitudes, failing to account for scale can cause the penalty to affect predictors unevenly.

A common preprocessing step is therefore to center and scale each predictor:

x_{ij}^{*} = \frac{x_{ij}-\bar{x}_j}{s_j}

Most modern implementations of penalized regression perform this standardization internally when requested. The analyst should nevertheless understand whether standardization is occurring and how coefficients are reported on the original scale.

Clinical Example: Predicting Change in Biomarker Level

Suppose a clinical study contains \(n=250\) participants and the objective is to predict change in a continuous biomarker from baseline measurements. Candidate predictors include age, body mass index, baseline biomarker value, renal function, hepatic function, inflammatory markers, treatment exposure, and several correlated laboratory biomarkers.

Predictor group	Example variables	Potential issue
Demographics	Age, sex, BMI	Different scales and modest correlations
Baseline disease measures	Baseline score, disease duration	Potentially correlated with laboratory values
Laboratory biomarkers	CRP, albumin, creatinine, enzymes	Correlation and redundant information
Treatment variables	Treatment group, exposure summary	Must preserve clinically important covariates

An OLS model may be useful as a reference model, but ridge or lasso can be useful when the analyst's main goal is predictive performance and the candidate predictor set is large relative to the sample size.

R Example: Ridge and Lasso with `glmnet`

The glmnet package is commonly used for ridge, lasso, and elastic-net models. For a continuous outcome, the default Gaussian family corresponds to penalized linear regression.

library(glmnet)

set.seed(2026)

# Example data
n <- 250
p <- 20

X <- matrix(rnorm(n * p), nrow = n, ncol = p)
colnames(X) <- paste0("biomarker_", 1:p)

y <- 1.5 * X[, 1] -
     1.0 * X[, 2] +
     0.7 * X[, 5] +
     rnorm(n, sd = 2)

# Ridge: alpha = 0
ridge_cv <- cv.glmnet(
  X, y,
  alpha = 0,
  nfolds = 10,
  standardize = TRUE
)

# Lasso: alpha = 1
lasso_cv <- cv.glmnet(
  X, y,
  alpha = 1,
  nfolds = 10,
  standardize = TRUE
)

plot(ridge_cv)
plot(lasso_cv)

ridge_cv$lambda.min
lasso_cv$lambda.min

The argument alpha=0 specifies ridge, while alpha=1 specifies lasso. Intermediate values of alpha produce elastic-net models.

Extracting the Selected Coefficients

After cross-validation, the fitted model can be evaluated at either lambda.min or lambda.1se. For example:

coef(lasso_cv, s = "lambda.min")
coef(lasso_cv, s = "lambda.1se")

coef(ridge_cv, s = "lambda.min")

For lasso, some entries may be exactly zero. Those variables are excluded from the corresponding fitted model. With ridge, most coefficients will remain nonzero, although they may be substantially smaller than their OLS counterparts.

Comparing OLS, Ridge, and Lasso

Model	Training fit	Coefficient behavior	Variable selection
OLS	Optimizes training RSS without penalty	Can be unstable with collinearity	No
Ridge	Slightly constrained by penalty	Shrinks coefficients toward zero	No
Lasso	Slightly constrained by penalty	Shrinks and can set coefficients to zero	Yes

The comparison should ultimately be based on the scientific objective. If the analyst needs a model containing a broad set of correlated predictors, ridge may be preferable. If a sparse predictor set is desirable, lasso may be attractive. If both behaviors are important, elastic net is a natural candidate.

Interpreting Ridge Coefficients

A ridge coefficient represents the fitted change in the outcome associated with a one-unit change in the predictor while holding the other model predictors fixed, but the coefficient is obtained under the shrinkage penalty.

This interpretation should not be treated as identical to an ordinary least-squares coefficient. In particular, the amount of shrinkage depends on \(\lambda\), and the fitted coefficient is generally biased toward zero.

Do not overinterpret shrinkage coefficients as causal effects. A penalized regression model is a statistical prediction model unless the study design and causal analysis support a causal interpretation. Variable selection by lasso does not by itself establish causality.

Interpreting Lasso Variable Selection

A zero lasso coefficient means that the selected penalized model assigns no coefficient to that predictor at the chosen value of \(\lambda\). It does not prove that the predictor has no biological, prognostic, or clinical relationship with the outcome.

This distinction is especially important with correlated biomarkers. If two variables contain nearly the same information, lasso may retain one and discard the other. Which variable survives can depend on the sample and the tuning parameter.

Therefore, lasso-selected variables should be considered model-selected predictors, not automatically definitive discoveries.

Correlated Predictors: A Practical Example

Imagine that five inflammatory biomarkers are strongly correlated. An OLS model may assign five unstable coefficients. Ridge tends to retain all five but shrink their coefficients. Lasso may retain only one or a subset.

Situation	Likely behavior
Five correlated biomarkers all contribute similar information	Ridge may retain all five with reduced coefficients
Only one biomarker adds substantial unique information	Lasso may retain that biomarker and set others to zero
Many correlated predictors and desire for sparse selection	Elastic net may be preferable

Prediction Versus Explanation

One of the most important decisions is whether the model is being developed for prediction or for scientific explanation.

Penalized regression is especially powerful for prediction because shrinking unstable coefficients can reduce variance and improve performance on new data. However, a model optimized for prediction is not automatically the best model for estimating a prespecified treatment effect or testing a scientific hypothesis.

Prediction: prioritize out-of-sample performance and validate the complete modeling pipeline.

Estimation: use an analysis designed around the parameter of interest, with appropriate inferential methods.

Variable discovery: treat selected variables as candidates for further scientific investigation rather than definitive causal findings.

Nested Cross-Validation and Honest Performance Assessment

If the same data are used both to select \(\lambda\) and to report predictive performance, the resulting performance estimate can be optimistic. This is a general model-selection problem.

When an unbiased estimate of the entire model-building procedure is important, nested cross-validation separates tuning from performance assessment. The inner loop selects \(\lambda\), while the outer loop evaluates the complete procedure on data that were not used for tuning.

Clinical prediction studies: When reporting model performance, describe every data-dependent step, including imputation, scaling, feature filtering, tuning, and variable selection. Ideally, operations that learn from the data are performed within the resampling procedure rather than before it.

Ridge and Lasso for Logistic Regression

Penalization is not restricted to continuous outcomes. For binary clinical outcomes, ridge and lasso can be applied to logistic regression.

\operatorname{logit}\{P(Y_i=1)\} = \beta_0+\sum_{j=1}^{p}\beta_jX_{ij}

The likelihood-based objective is then combined with an \(L_1\) or \(L_2\) penalty. In glmnet, this is specified with family="binomial".

# Penalized logistic regression

fit_logistic_lasso <- cv.glmnet(
  X, y_binary,
  family = "binomial",
  alpha = 1,
  nfolds = 10
)

fit_logistic_ridge <- cv.glmnet(
  X, y_binary,
  family = "binomial",
  alpha = 0,
  nfolds = 10
)

coef(fit_logistic_lasso, s = "lambda.min")

For a binary endpoint, prediction can be evaluated using measures appropriate to the modeling objective, such as log loss, area under the ROC curve, calibration, sensitivity, specificity, or clinically relevant decision metrics. The choice should be specified before evaluating the final model.

Ridge and Lasso for Survival Outcomes

Penalized methods can also be used with time-to-event outcomes. A Cox proportional hazards model can be penalized to stabilize estimation when there are many candidate covariates.

h(t\mid X) = h_0(t) \exp\left(\sum_{j=1}^{p}\beta_jX_j\right)

The same conceptual distinction remains: ridge shrinks coefficients, whereas lasso can set some coefficients to zero. For survival prediction, performance should be assessed using methods appropriate for censored outcomes rather than ordinary regression error.

# Penalized Cox regression

cox_lasso_cv <- cv.glmnet(
  X, survival_object,
  family = "cox",
  alpha = 1,
  nfolds = 10
)

cox_ridge_cv <- cv.glmnet(
  X, survival_object,
  family = "cox",
  alpha = 0,
  nfolds = 10
)

Missing Data and Penalized Regression

Penalization does not automatically solve missing-data problems. A complete analysis must define how missing predictor and outcome information is handled.

If multiple imputation is used, model selection and tuning should be incorporated into a strategy that respects the imputation structure. Simply filling in missing values once and treating the imputed dataset as if it were fully observed can understate uncertainty.

Important: Do not regard lasso as a missing-data method. Missingness, imputation, model selection, and validation are separate methodological issues that must be planned together.

Clinical Covariates That Should Not Be Dropped Casually

A common temptation is to run lasso over every available variable and interpret the resulting sparse model as the definitive clinical model. That approach can be inappropriate when certain covariates are required by the scientific question, study design, or prespecified analysis plan.

For example, a treatment indicator, key baseline prognostic factor, stratification factor, or clinically essential adjustment variable may need to remain in the model even if its penalized coefficient is small.

In such settings, the modeling strategy can distinguish between variables that must be included and variables that are candidates for penalized selection. The exact implementation should be chosen deliberately rather than assuming that every predictor must be subject to the same penalty.

How to Choose Between Ridge and Lasso

Primary concern	Reasonable starting point
Many predictors with small contributions	Ridge
Strong multicollinearity and prediction	Ridge
Need for a sparse model	Lasso
Automatic variable selection is important	Lasso
Groups of correlated predictors plus sparsity	Elastic net
Uncertainty about the appropriate penalty structure	Compare candidate methods using prespecified resampling and performance criteria

A Practical Modeling Workflow

Define the clinical objective. Decide whether the goal is prediction, estimation, variable screening, or another objective.

Define the outcome. Select the appropriate model family: Gaussian, binomial, Cox, or another supported likelihood.

Prespecify candidate predictors. Include clinically justified variables and distinguish mandatory covariates from selectable predictors.

Address missing data. Define an appropriate imputation or missing-data strategy before model tuning.

Standardize predictors. Ensure the penalty operates on comparable predictor scales.

Fit candidate penalties. Evaluate ridge, lasso, and potentially elastic net.

Select \(\lambda\). Use cross-validation or another prespecified tuning strategy.

Evaluate out-of-sample performance. Use an independent test set or nested resampling when appropriate.

Assess stability. Examine how selected variables and coefficients change across resamples.

Interpret cautiously. Distinguish predictive utility from causal or confirmatory inference.

Model Stability and Selection Frequency

A useful extension for lasso is to ask not only which variables were selected once, but how often they are selected across repeated resamples. A predictor that appears in 95% of bootstrap or repeated cross-validation samples provides a different degree of selection stability from one that appears in only 12%.

Selection frequency does not create a conventional p-value. It is a descriptive measure of model stability and should be reported alongside predictive performance and the modeling procedure.

Common Mistakes

Using lasso as a substitute for clinical reasoning. A data-driven selection procedure cannot determine which covariates are clinically essential.
Failing to standardize predictors. Unequal scales can distort the practical effect of the penalty.
Choosing \(\lambda\) from the training data alone. This can favor overly complex models and produce optimistic performance estimates.
Interpreting zeros as proof of no association. A zero lasso coefficient is a property of the selected penalized model.
Reporting only the selected variables. Report the tuning method, penalty, validation strategy, and performance.
Ignoring correlated predictors. Lasso selection can be unstable when predictors carry highly overlapping information.
Calling penalized coefficients causal effects. Shrinkage does not create causal identification.
Ignoring mandatory covariates. Important design or clinical adjustment variables should not be discarded simply because a sparse algorithm prefers a smaller model.
Using a single train/test split when data are limited. Resampling can provide a more informative assessment of model stability and performance.
Performing preprocessing before cross-validation. Data-dependent preprocessing should generally occur within the resampling workflow.

What Should Be Reported in a Statistical Analysis Plan?

For a clinical prediction model using ridge, lasso, or elastic net, the statistical documentation should make the modeling procedure reproducible.

Clinical prediction or modeling objective
Outcome definition and analysis population
Candidate predictor definitions
Variables that are mandatory regardless of penalized selection
Missing-data strategy
Predictor transformations and standardization
Penalty family: ridge, lasso, or elastic net
Value or selection procedure for \(\alpha\), when applicable
Candidate \(\lambda\) grid or tuning procedure
Cross-validation or nested-validation strategy
Primary performance metric
Approach for final model fitting
Internal or external validation strategy
Handling of correlated predictors
Assessment of model and selection stability
Software and package versions

Worked Example Summary

The central ideas can be summarized in one table.

Concept	Ridge	Lasso
Penalty	\(L_2\)	\(L_1\)
Coefficient shrinkage	Yes	Yes
Exact zero coefficients	Generally no	Yes
Variable selection	No	Yes
Multicollinearity	Often especially useful	Useful, but selection can be unstable with correlated predictors
Typical tuning method	Cross-validation	Cross-validation
Primary advantage	Stable shrinkage and prediction	Sparse models and variable selection

The Most Important Concept

Ridge and lasso are not simply alternative ways of calculating the same regression coefficient. They deliberately change the optimization problem by imposing a cost on coefficient magnitude.

Ridge uses an \(L_2\) penalty and generally keeps all predictors in the model, while shrinking their coefficients. Lasso uses an \(L_1\) penalty and can produce exactly zero coefficients, creating a sparse model.

The penalty strength \(\lambda\) is the central tuning parameter. It should usually be selected using a validation strategy rather than chosen because it produces a desired coefficient pattern.

Bottom line: Ridge and lasso are powerful tools for clinical prediction when the candidate predictor set is large, correlated, or otherwise prone to unstable estimation. Ridge is primarily a shrinkage method; lasso adds sparse variable selection. Cross-validation is commonly used to tune the penalty, and careful validation is essential because model selection itself can lead to optimistic performance estimates. In clinical research, penalized regression should complement—not replace—clinical reasoning, prespecified scientific objectives, appropriate handling of missing data, and rigorous validation.

References

Hoerl, A.E. & Kennard, R.W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267–288.
Zou, H. & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B, 67(2), 301–320.
Friedman, J., Hastie, T. & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1).
Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning. Springer.