Tuning Alpha And Lambda Together

Why Tune Alpha and Lambda Together?

An elastic-net model has two tuning parameters with different roles. The mixing parameter $\alpha$ controls the balance between the lasso and ridge penalties; the penalty parameter $\lambda$ controls the overall strength of regularization. A value of $\alpha=1$ gives the lasso penalty, while $\alpha=0$ gives the ridge penalty. For intermediate values, the penalty combines both. The predictive performance of a particular $\alpha$ depends on which $\lambda$ accompanies it.

Consequently, comparing two alpha values at one arbitrary lambda can be misleading. One value may appear worse simply because that lambda is poorly suited to it. A more useful search evaluates candidate lambdas for each candidate alpha, then compares the best resampling performance found for each alpha. The selected result is a pair, not an alpha value considered in isolation.

Definition: A joint alpha-lambda search evaluates a set of candidate pairs $(\alpha,\lambda)$ using the same stated resampling criterion. The selected pair minimizes that criterion over the evaluated candidates. It is optimal only among the candidates and resampling procedure used; it is not guaranteed to be the best possible pair over all values.

Model and Selection Criterion

For a Gaussian response, a common elastic-net fit minimizes a scaled squared-error term plus a combined penalty. In the expression below, $y_i$ is the response, $x_i$ is the predictor vector, $\beta_0$ is the intercept, and $\beta$ is the vector of slopes. The intercept is ordinarily not penalized. The notation $\lVert\beta\rVert_1$ denotes the sum of the absolute slope coefficients, and $\lVert\beta\rVert_2^2$ denotes the sum of their squares.

\frac{1}{2n}\sum_{i=1}^{n}(y_i-\beta_0-x_i^\mathsf{T}\beta)^2 +\lambda\left(\alpha\lVert\beta\rVert_1+ \frac{1-\alpha}{2}\lVert\beta\rVert_2^2\right), \qquad 0\leq\alpha\leq1,\quad\lambda\geq0.

This objective describes the Gaussian case; it is not a universal loss formula for every response type. In particular, the meaning of a suitable tuning loss depends on the outcome and model family. The search in this tutorial uses cross-validated mean squared error for a continuous response.

Suppose the observations are divided into $K$ folds. For each fold, the model is fitted using the other $K-1$ folds and evaluated on the held-out fold. For a given candidate pair, the resampling criterion summarizes those held-out errors. With equally sized folds and fold mean squared errors $MSE_1,\ldots,MSE_K$, one possible criterion is their average:

CV(\alpha,\lambda)=\frac{1}{K}\sum_{k=1}^{K}MSE_k(\alpha,\lambda).

The criterion must be stated before choosing the pair. For example, the minimum cross-validated MSE and a one-standard-error selection rule do not necessarily select the same candidate. This tutorial uses the minimum of the specified cross-validation error as its selection rule.

Worked Example: Selecting a Pair From Fold Errors

A hypothetical continuous-outcome analysis compares three alpha values. Each candidate pair is evaluated on the same three equally sized folds. The listed values are fold mean squared errors. Select the pair with the smallest average fold error.

$\alpha$	$\lambda$	Fold 1 MSE	Fold 2 MSE	Fold 3 MSE
0.2	0.20	12	11	13
0.5	0.10	10	11	9
0.8	0.05	11	10	12

Method. Compute the stated criterion separately for each pair, then select its minimum. For $(0.2,0.20)$, the average is $(12+11+13)/3=36/3=12$. For $(0.5,0.10)$, it is $(10+11+9)/3=30/3=10$. For $(0.8,0.05)$, it is $(11+10+12)/3=33/3=11$.

The smallest average is $10$, so the selected pair is $(\alpha,\lambda)=(0.5,0.10)$. The check is $10<11<12$, consistent with the ordering of the three computed averages. The common mistake avoided here is selecting the pair from one favorable fold rather than from the prespecified resampling summary.

Why Reuse the Fold Assignments?

When alpha values are evaluated using different fold assignments, the comparison mixes two effects: differences between the models and differences between the particular training-validation splits. Reusing the same fold identifiers makes the comparison more controlled. Each candidate is tested on the same held-out observations in each fold, so a difficult or unusually easy fold affects the candidates in a more comparable way.

This does not make the estimated errors independent, remove sampling uncertainty, or guarantee that the selected pair generalizes best. It reduces avoidable variation in the comparison. Fold construction should still respect the data structure, and all data-dependent preprocessing must be learned within each training fold rather than from the full data set.

Key principle: Hold the resampling design fixed while comparing candidate alpha values. For each alpha, search a suitable lambda sequence, record the same stated cross-validation criterion, and select the pair according to that criterion.

Implementing the Search in R

In glmnet, cv.glmnet() tunes lambda for a supplied alpha. To search alpha as well, call it once for each candidate alpha and supply the same foldid each time. The candidate alpha values must lie between 0 and 1. For a Gaussian response, type.measure = "mse" requests mean squared error.

The following sketch assumes that x is a numeric predictor matrix and y is a continuous response. The fold identifiers have already been constructed so the folds are balanced. A fixed random seed makes the fold assignment reproducible; in an actual analysis, the assignment should also reflect any grouping or time structure in the observations.

Define folds once.
For example: set.seed(2026)
foldid <- sample(rep(1:5, length.out = nrow(x)))
This assigns every row to one of five folds. Reuse this same vector for all alpha values.

Run a lambda search for each alpha.
alphas <- c(0, 0.25, 0.5, 0.75, 1)
fits <- lapply(alphas, function(a) cv.glmnet(x, y, alpha = a, foldid = foldid, type.measure = "mse"))
Each call evaluates a lambda sequence for its supplied alpha.

Collect each minimum.
best <- do.call(rbind, lapply(seq_along(alphas), function(j) {
f <- fits[[j]]
c(alpha = alphas[j], lambda = f$lambda.min, cvm = min(f$cvm))
}))
Here lambda.min identifies the lambda with the smallest cross-validation error within that call, and cvm contains the cross-validation error curve.

Select and inspect the pair.
best <- as.data.frame(best)
best[which.min(best$cvm), ]
This selects the alpha-lambda pair with the smallest recorded minimum error among the searched alpha values.

The code uses the lambda sequence generated for each alpha by cv.glmnet(). That is often convenient: the relevant question is whether each alpha has an adequate search over lambda, not whether every alpha was evaluated at an identical list of penalty values. Inspect the error curves and the candidate ranges to confirm that the minimum is not at an inadequately explored endpoint.

A user can instead supply a common lambda sequence to each call when direct comparison at shared lambda values is important. That sequence must be appropriate for all alpha values being compared. If it does not cover a useful range for one alpha, that model may be disadvantaged. The common sequence should therefore be chosen deliberately, rather than imposed merely for visual symmetry.

Worked Example: Why a Shared Fold Assignment Matters

Suppose three equally sized validation folds are used to compare two candidate pairs. With shared folds, their fold MSE values are as follows:

Candidate pair	Fold 1 MSE	Fold 2 MSE	Fold 3 MSE
$(0.2,0.10)$	6	8	8
$(0.5,0.10)$	4	9	7

Method. Apply the same equal-fold average to both rows. For $(0.2,0.10)$, the average is $(6+8+8)/3=22/3=7.3333$ to four decimal places. For $(0.5,0.10)$, it is $(4+9+7)/3=20/3=6.6667$ to four decimal places. The second pair has the smaller criterion, so select $(0.5,0.10)$.

The calculation also shows why choosing a model from its best individual fold is unsound: the second pair has a fold error of 9, larger than either of the first pair's 8s, but its overall criterion is still smaller. The check is $20/3<22/3$. If each pair had been evaluated on different folds, this arithmetic comparison would no longer isolate model differences as cleanly.

Interpreting Alpha-Specific Lambda Results

The lambda selected for one alpha should not be treated as a universal penalty strength for every alpha. The penalty changes composition as alpha changes, and the candidate lambda sequences may also differ. Thus, a selected $\lambda$ of $0.10$ at $\alpha=0.5$ does not imply that $\lambda=0.10$ is the best choice at $\alpha=0.2$ or $\alpha=1$.

The comparison is between each alpha's best result under the agreed search procedure. A concise tuning summary should identify the selected alpha, its selected lambda, the minimized criterion, the fold design, and the candidate alpha values considered. It should not imply that the search covers every possible alpha or lambda.

Worked Example: Avoiding a Fixed-Lambda Comparison Trap

A hypothetical analysis reports these cross-validation results. The first two columns show MSE at the shared value $\lambda=0.10$; the final two columns show the minimum MSE found after searching lambda separately for each alpha.

$\alpha$	MSE at $\lambda=0.10$	Best lambda in its search	Minimum MSE
0.2	10.8	0.04	9.6
0.5	10.2	0.08	9.1
0.8	10.7	0.12	9.4

Method. The selection rule is to minimize the cross-validation MSE after searching lambda within each alpha. The relevant comparison is therefore $9.6$, $9.1$, and $9.4$, not the single-lambda values $10.8$, $10.2$, and $10.7$. The minimum is $9.1$, so select $(\alpha,\lambda)=(0.5,0.08)$.

The common mistake is to compare all alpha values only at $\lambda=0.10$, then report that alpha $0.5$ is best without checking whether the other alpha values perform better at their own suitable penalties. In this particular table, the same alpha happens to rank first under both comparisons, but the fixed-lambda comparison does not establish that the best alpha-lambda pair is $(0.5,0.10)$. The corrected result is $(0.5,0.08)$, with MSE $9.1$.

Common Mistakes and Practical Checks

Tuning alpha while holding lambda fixed. This can unfairly compare alpha values at a penalty that suits one better than another. Search lambda within each alpha before comparing their best results.
Using different folds for different alpha values. This adds avoidable split-to-split variation. Create the fold identifiers once and pass the same identifiers to every fit.
Assuming the smallest reported error is automatically reliable. A minimum is the winner among the candidates tested, not a guarantee of superior future performance. Report the resampling design and avoid overstating the result.
Ignoring the lambda range. If an apparent minimum occurs at an endpoint, check whether the sequence spans a sufficiently broad range. A grid that stops too early can miss a better candidate.
Comparing lambda values as if they had the same role across alpha. Lambda multiplies a penalty whose composition depends on alpha. Report the selected pair, not lambda alone.

A disciplined joint search is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it requires a candidate to distinguish model specification from tuning, interpret a resampling criterion correctly, and communicate the scope of a model-selection result.

Check Your Understanding

For each question, use the stated criterion and compare candidate pairs only as specified.

Two alpha values are evaluated on the same four equally sized folds. For $(\alpha,\lambda)=(0.25,0.16)$, the fold MSEs are 7, 9, 8, and 8. For $(0.75,0.06)$, they are 8, 8, 7, and 7. Which pair has the lower average fold MSE? (Answer: $(0.75,0.06)$, with MSE $7.5$ versus $8.0$.)
A search reports minimum cross-validation errors of 14.2 at alpha $0$, 13.8 at alpha $0.4$, and 14.0 at alpha $1$, each at its own selected lambda. Under a minimum-error rule, which alpha is selected, and what information is still needed to state the full selected model? (Answer: alpha $0.4$; the selected lambda must also be reported.)
One alpha is evaluated with fold identifiers $1,1,2,2,3,3$, and another with newly randomized identifiers. What should be changed to improve the comparison? (Answer: generate one foldid vector and reuse it for both alpha searches.)
At a shared lambda of $0.05$, alpha $0.2$ has MSE $6.1$ and alpha $0.8$ has MSE $6.4$. After separately searching lambda, their minimum errors are $5.7$ at lambda $0.02$ and $5.3$ at lambda $0.08$, respectively. Which pair is selected by minimum error, and why is the shared-lambda comparison insufficient? (Answer: $(0.8,0.08)$, because its separately optimized error $5.3$ is smaller; the shared-lambda comparison does not compare each alpha at its best searched lambda.)
A four-fold search for one candidate pair gives fold MSEs 5, 6, 7, and 10. A second pair gives 6, 7, 7, and 7. Under the equal-fold-average criterion, which pair wins, and what are the two averages? (Answer: the second pair wins; the averages are $7.0$ and $6.75$, respectively.)

What You'll Learn