Two Parameters, Two Different Decisions
When fitting a penalized regression with glmnet, alpha and lambda answer different questions. Alpha specifies the form of the penalty: setting alpha to zero gives ridge regression. Lambda specifies how strongly that penalty is applied. Treating the two parameters as interchangeable can lead to an incorrect model specification or an incorrect interpretation of the fitted model.
This distinction matters in practical modeling. If the intended method is ridge regression, alpha must remain zero; changing lambda then compares ridge fits with different amounts of shrinkage. By contrast, changing alpha changes the penalty itself. This tutorial focuses on that distinction and on how it appears in R.
The Penalized Objective
For Gaussian regression with \(n\) observations, response values \(y_i\), predictor vectors \(x_i\), intercept \(\beta_0\), and coefficient vector \(\boldsymbol{\beta}\), glmnet uses the following objective in its standard unweighted formulation:
The intercept is not included in either penalty term. When \(\alpha=0\), the L1 term disappears and the objective becomes
That is the ridge objective under this scaling convention. The factor \(1/2\) is part of the penalty convention used in the objective; it does not change the basic interpretation that lambda scales the squared-coefficient penalty. The displayed objective assumes Gaussian regression with equal observation weights. Other response families or weights change the loss part of the objective, but the distinction between alpha and lambda remains: alpha determines the penalty mixture, and lambda scales it.
family argument specifies the response model, such as Gaussian or binomial. Alpha does not select the response family.
Worked Example: Reading the Penalty Mixture
Worked Example: Reading the Penalty Mixture
Suppose a candidate coefficient vector is \(\boldsymbol{\beta}=(2,-1)\) and \(\lambda=3\). Calculate the penalty contribution for alpha equal to 0, 0.4, and 1. Interpret which setting is ridge regression.
Method. Use the penalty term \(\lambda[(1-\alpha)\|\boldsymbol{\beta}\|_2^2/2+\alpha\|\boldsymbol{\beta}\|_1]\). First calculate both norms: \(\|\boldsymbol{\beta}\|_2^2=2^2+(-1)^2=5\), and \(\|\boldsymbol{\beta}\|_1=|2|+|-1|=3\).
For \(\alpha=0\), the L1 part has weight zero. The penalty is \(3[(1-0)(5)/2+0(3)]=3(2.5)=7.5\). This is the ridge penalty.
For \(\alpha=0.4\), the penalty is \(3[(1-0.4)(5)/2+0.4(3)]=3[0.6(2.5)+1.2]=3(2.7)=8.1\). This setting mixes L2 and L1 penalties; it is elastic net, not ridge.
For \(\alpha=1\), the L2 part has weight zero. The penalty is \(3[0(5)/2+1(3)]=9\), which is the lasso penalty.
Check and trap avoided. The results use the same coefficient vector and lambda, so only alpha changes. A common mistake is to call every glmnet fit “ridge” because it uses squared-error loss. The penalty setting, not the loss alone, determines whether the fit is ridge.
Why Lambda Controls Shrinkage
For ridge regression, alpha is fixed at zero. The fitted coefficients minimize the residual loss plus \(\lambda\|\boldsymbol{\beta}\|_2^2/2\). Increasing lambda increases the cost assigned to nonzero coefficients relative to the residual loss. The optimizer therefore has more incentive to reduce coefficient magnitudes, although the exact fitted values depend on the data and the predictors.
A one-predictor calculation shows this relationship directly. Assume the response and predictor have been centered, the model has no intercept after centering, and the predictor satisfies \(\sum_{i=1}^{n}x_i^2/n=1\). Write \(c=\sum_{i=1}^{n}x_i y_i/n\). The ridge objective for the single coefficient \(\beta\) is
Differentiating with respect to \(\beta\) gives \(-c+\beta+\lambda\beta\). Setting this derivative to zero yields the fitted coefficient:
For this one-predictor setup, if \(c\) is fixed and positive, increasing lambda increases the denominator and decreases the fitted coefficient. This derivation relies on the centered, unit-scaled predictor and the stated Gaussian objective. It illustrates the shrinkage mechanism; it is not a general formula for every coefficient in a multiple-predictor fit.
Worked Example: Comparing Two Ridge Penalties
Consider a centered one-predictor dataset satisfying \(\sum x_i^2/n=1\) and \(\sum x_i y_i/n=1.2\). Use ridge regression and calculate the fitted coefficient at \(\lambda=0.5\) and \(\lambda=2\).
Method. The conditions match the one-predictor derivation, so \(c=1.2\) and \(\widehat{\beta}=c/(1+\lambda)\). Alpha is zero for both fits; only lambda changes.
At \(\lambda=0.5\), \(\widehat{\beta}=1.2/(1+0.5)=1.2/1.5=0.8\).
At \(\lambda=2\), \(\widehat{\beta}=1.2/(1+2)=1.2/3=0.4\).
The larger penalty gives the smaller coefficient, as expected. As a direct check, the unpenalized coefficient in this setup is \(c=1.2\); both ridge estimates have smaller magnitude, and \(0.4<0.8<1.2\). This calculation avoids the trap of assuming lambda changes the penalty type: both results are ridge fits because alpha remains zero.
Using Alpha and Lambda in R
For ridge regression with a Gaussian response, set alpha explicitly to zero. The predictor input should be a numeric matrix, and the response should be supplied as a vector. For example, with an already prepared matrix x_mat and response y, a fit can be requested as follows:
R: fit_ridge <- glmnet(x_mat, y, family = "gaussian", alpha = 0)
A glmnet fit normally contains estimates over a sequence of lambda values. To request a prediction at a particular value, provide that value through the s argument. For example:
R: predict(fit_ridge, newx = x_new, s = 0.2)
Here alpha was fixed when the model was fitted, while s = 0.2 identifies the lambda value at which to use the fitted model. The argument name s in this prediction call does not make it a second penalty-type setting; it supplies the desired lambda value. Exact numerical predictions require the fitted data and the new predictor values.
By default, glmnet standardizes predictors when fitting. Lambda values are meaningful in the context of the data scaling, response scale, and fitting setup used. When comparing candidate lambda values, keep these settings consistent. Do not interpret a particular numerical lambda as a universal amount of shrinkage across unrelated data sets.
Worked Example: Selecting Lambda While Keeping Ridge Fixed
A hypothetical analyst fits a ridge model for a continuous response using five-fold cross-validation. The average validation mean squared errors for five candidate lambda values are as follows. These values are hypothetical and are supplied to illustrate the selection rule.
| Lambda | Mean validation MSE |
|---|---|
| 0.01 | 1.42 |
| 0.10 | 1.18 |
| 0.50 | 1.11 |
| 1.00 | 1.16 |
| 2.00 | 1.31 |
Method. Keep alpha equal to zero for every candidate, then select the lambda with the smallest mean validation MSE. Comparing the table entries, \(1.11<1.16<1.18<1.31<1.42\), so the selected candidate is \(\lambda=0.50\).
One way to carry out this procedure in R is set.seed(41), followed by cv_ridge <- cv.glmnet(x_mat, y, family = "gaussian", alpha = 0, nfolds = 5, type.measure = "mse"). The selected value can be obtained with cv_ridge$lambda.min. In an actual analysis, that value is determined by the supplied data and the cross-validation results; it cannot be inferred from the code alone.
The interpretation is precise: cross-validation selected a lambda for a ridge model because alpha was fixed at zero. It did not compare ridge with lasso or elastic net. If the modeling task calls for comparing penalty types, fit candidates with different alpha values and compare them using consistent validation folds and an appropriate metric. The call to cv.glmnet with one alpha value does not search across alpha values.
Check and trap avoided. The minimum validation error selects the lambda in this table, not the alpha. Choosing the row with the largest lambda because it shrinks most would ignore the stated validation criterion.
Common Interpretation Traps
- Using a nonzero alpha and calling the fit ridge. Ridge requires alpha equal to zero. A value between zero and one mixes penalty types, and alpha equal to one gives lasso.
- Calling lambda the model-family setting. Lambda scales the penalty. In glmnet, the
familyargument determines the response family, such as Gaussian or binomial. - Assuming a larger lambda always improves prediction. It applies a stronger penalty, but predictive performance must be assessed using suitable validation data or cross-validation.
- Assuming cross-validation automatically chooses alpha. A call that fixes alpha at zero searches for a lambda within the ridge fit; it does not, by itself, compare penalty types.
- Comparing lambda values as if they had a universal scale. The practical effect of a numerical lambda depends on the modeling setup, including data scaling and response scale.
alpha = 0. Then choose or evaluate lambda values to control penalty strength. Keep the two decisions separate, and use validation performance—not the parameter names alone—to assess a model.
Check Your Understanding
For each problem, identify the penalty setting and show the calculation needed to support your answer.
- A glmnet fit uses Gaussian loss, alpha equal to zero, and lambda equal to 0.3. Is the penalty ridge, lasso, or elastic net? Which parameter would you change to apply a stronger ridge penalty? (Answer: Ridge; increase lambda while keeping alpha at zero.)
- For \(\boldsymbol{\beta}=(1,-2)\), lambda equal to 2, and alpha equal to 0, calculate the penalty contribution. (Answer: \(\|\boldsymbol{\beta}\|_2^2=1+4=5\), so the penalty is \(2(5/2)=5\).)
- A fitted model uses alpha equal to 0.25. Is it ridge regression? State which penalty terms are present. (Answer: No; it is elastic net, with both L2 and L1 terms present.)
- In the centered, unit-scaled one-predictor setup, \(c=0.9\). Calculate the ridge coefficient at lambda equal to 0.5 and lambda equal to 1. (Answer: \(0.9/1.5=0.6\) and \(0.9/2=0.45\), respectively.)
- A five-fold cross-validation run uses alpha equal to zero and reports minimum validation MSE at lambda equal to 0.8. What has been selected, and what has not been compared? (Answer: Lambda equal to 0.8 has been selected for a ridge fit; ridge has not been compared with lasso or elastic net.)