Worked Elastic Net Tuning Example

From Predictor Scaling to an Evaluated Model

Elastic net tuning involves more than choosing a pair of tuning parameters. A defensible analysis also specifies how predictors are scaled, how candidate settings are compared, how the one-standard-error rule is applied, and how performance is finally assessed on data that played no part in choosing the model. This tutorial follows those steps in a hypothetical claim-cost modeling study. The response is a claim cost, and the predictors are recorded risk characteristics; no data below represent an actual insurer or portfolio.

The elastic net combines an absolute-value penalty, which can set coefficients exactly to zero, with a squared-coefficient penalty, which shrinks coefficients toward zero. Its tuning parameters are $\alpha$, controlling the penalty mixture, and $\lambda$, controlling the overall penalty strength. The numerical values of $\lambda$ are meaningful only alongside the stated objective-function convention and response scaling.

Definition: For $n$ observations and $p$ predictors, one elastic net convention minimizes $$ \frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}\beta_j z_{ij}\right)^2 +\lambda\left\{\alpha\sum_{j=1}^{p}|\beta_j|+ \frac{1-\alpha}{2}\sum_{j=1}^{p}\beta_j^2\right\}. $$ Here $z_{ij}$ is the standardized value of predictor $j$ for observation $i$, $\beta_0$ is an unpenalized intercept, $\beta_j$ are penalized slopes, $\lambda\geq 0$, and $0\leq\alpha\leq 1$. At $\alpha=1$, this is the lasso objective; at $\alpha=0$, it is the ridge objective. Other software conventions can scale the loss or penalty differently, so tuning values should be interpreted using the convention actually specified.

Standardize Predictors Using Training Data Only

For a training sample, standardize predictor $j$ by subtracting its training mean and dividing by its training sample standard deviation:

\bar{x}_j=\frac{1}{n}\sum_{i=1}^{n}x_{ij}, \qquad s_j=\sqrt{\frac{\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)^2}{n-1}}, \qquad z_{ij}=\frac{x_{ij}-\bar{x}_j}{s_j}.

The same training mean and standard deviation must be used to transform validation or test observations. In cross-validation, each fold has its own training portion: calculate its means and standard deviations from that portion, then apply them to its held-out fold. Calculating the scaling values from all observations before dividing into folds allows information from held-out predictors to influence the fitted process. The response is not standardized in the examples here; if a different analysis standardizes it, that choice must also be confined to each training portion and reversed appropriately for predictions.

Worked Example: Standardizing Two Predictors in a Training Fold

A training fold contains three hypothetical risks with predictor values $x_1=(1,2,3)$ and $x_2=(1,3,1)$. Use sample standard deviations. Find the standardized training values, and transform a held-out observation with $x_1=4$ and $x_2=3$.

Method. Calculate each predictor's mean and sample standard deviation from the three training observations only. Use those same quantities for the held-out observation.

For $x_1$, the mean is $\bar{x}_1=(1+2+3)/3=2$. The squared deviations sum to $(1-2)^2+(2-2)^2+(3-2)^2=2$, so $s_1=\sqrt{2/(3-1)}=1$. Its standardized training values are $-1,0,1$.

For $x_2$, the mean is $\bar{x}_2=(1+3+1)/3=5/3$. The squared deviations sum to $(1-5/3)^2+(3-5/3)^2+(1-5/3)^2=4/9+16/9+4/9=24/9=8/3$. Thus $s_2=\sqrt{(8/3)/2}=\sqrt{4/3}=2/\sqrt{3}$. The standardized values are $-1/\sqrt{3},\,2/\sqrt{3},\,-1/\sqrt{3}$.

For the held-out observation, $z_1=(4-2)/1=2$. For the second predictor, $z_2=(3-5/3)/(2/\sqrt{3})=(4/3)(\sqrt{3}/2)=2/\sqrt{3}$. Therefore its standardized pair is $(2,2/\sqrt{3})$. The calculation is checked by using the training-fold mean and standard deviation in both transformations; the held-out values do not enter either quantity.

Common mistake avoided. The squared deviations for $x_2$ sum to $8/3$, not $4/3$. Dividing the incorrect sum by two would give an incorrect standard deviation and therefore incorrect standardized values.

Standardization matters because the elastic net penalizes coefficient magnitudes, not predictor units. If one predictor is measured in dollars and another in thousands of dollars, their raw coefficients can differ greatly even when the predictors have similar predictive roles. Standardizing puts the predictors on a common scale before applying the penalty. A predictor with zero training-fold variance cannot be divided by $s_j$; it must be handled by a specified preprocessing rule, such as removing it from that fold's fit.

Why Scaling Changes the Penalty

The reason standardization is important can be seen by changing the units of a single predictor. Suppose $x'=c x$, where $c\neq 0$. To preserve the fitted contribution to the linear predictor, a raw-scale coefficient $b$ must become $b'=b/c$, because $b'x'= (b/c)(cx)=bx$. However, the coefficient penalty changes: $|b'|=|b|/|c|$ and $(b')^2=b^2/c^2$. Thus, without scaling, changing a predictor's units changes how strongly its coefficient is penalized, even though the predictions can be identical. Standardization removes this arbitrary units effect by expressing predictors in standard-deviation units before fitting.

After fitting with standardized predictors, coefficients can be reported on the original predictor scale. If the standardized-scale model is $\hat{y}=\beta_0^*+\sum_j\beta_j^*(x_j-\bar{x}_j)/s_j$, then the raw-scale slope for predictor $j$ is $\beta_j^*/s_j$, and the raw-scale intercept is $\beta_0^*-\sum_j\beta_j^*\bar{x}_j/s_j$. These conversions preserve fitted values; they do not change which model was selected.

Compare Candidate Settings by Cross-Validation

Cross-validation estimates how well each candidate tuning setting predicts observations not used to fit that fold's model. Here the score is mean squared error (MSE), the average of squared prediction errors. Smaller MSE is preferred. For a fair comparison, all candidate settings should be assessed on the same folds. The example uses three fold MSEs per setting, so all means and standard errors below are calculated from three values.

Worked Example: Choosing Alpha and Lambda With the One-Standard-Error Rule

A hypothetical claim-cost study compares five elastic net settings on the same three cross-validation folds. The table gives each fold's MSE and the mean number of nonzero slopes across the fold fits. Use the one-standard-error rule; among eligible settings, choose the one with the fewest average nonzero slopes, breaking any tie by choosing the larger $\lambda$.

Setting	$\alpha$	$\lambda$	Fold MSEs	Mean MSE	Mean nonzero slopes
A	0.5	0.2	10.0, 12.0, 11.0	11.000	4
B	0.5	0.6	10.8, 11.4, 11.1	11.100	3
C	1.0	0.4	10.5, 12.1, 11.2	11.267	5
D	0.5	1.0	11.2, 11.5, 11.0	11.233	2
E	0.0	0.8	11.1, 11.6, 11.3	11.333	6

Step 1: Find the minimum mean MSE. The five means are $11.000, 11.100, 11.267, 11.233,$ and $11.333$. The minimum is $11.000$, for setting A.

Step 2: Estimate the standard error for the best setting. Setting A has fold scores $10,12,11$, with mean $11$. Their squared deviations from the mean are $1,1,0$, summing to $2$. The sample variance is $2/(3-1)=1$, the sample standard deviation is $1$, and the estimated standard error of the mean is $1/\sqrt{3}=0.57735$.

Step 3: Form the one-standard-error threshold. Add this standard error to the minimum mean MSE: $11+0.57735=11.57735$. A candidate is eligible when its mean MSE is no greater than $11.57735$. Each listed mean is below that threshold, including setting E at $11.333$.

Step 4: Apply the stated simplicity rule. Setting D has the smallest mean number of nonzero slopes, 2, among the eligible candidates. Select $(\alpha,\lambda)=(0.5,1.0)$. The minimum-error setting A is not selected because the one-standard-error rule permits a simpler model whose mean error is within the threshold.

Method and mistake avoided. This uses the one-standard-error rule with the standard error calculated from the best setting's three fold scores. A common error is to call this a four-fold analysis or calculate a four-fold standard error when only three fold values were supplied. Here there are exactly three scores per candidate, so both the means and standard error use three folds.

The one-standard-error rule is a selection convention, not a claim that all eligible models have identical predictive performance. It first identifies the best mean score and then treats candidates within one estimated standard error of that score as acceptable for a simpler-model comparison. When $\alpha$ and $\lambda$ both vary, complexity is not always ordered by $\lambda$ alone: changing $\alpha$ can also change which coefficients are zero. State the complexity criterion before using it. In this example, the mean number of nonzero slopes is supplied as the criterion.

The standard error computed from fold scores is an estimate used by this rule; folds from one cross-validation partition are not generally independent replications. Consequently, the threshold should be understood as a practical model-selection rule, not as an exact confidence interval for future prediction error.

Refit the Selected Setting, Then Evaluate Once

After choosing the tuning setting, refit the model using the designated development data and the selected $(\alpha,\lambda)$. Recalculate preprocessing quantities from that development data, rather than carrying over the mean and standard deviation of a particular cross-validation fold. Keep the final test set out of standardization choices, tuning, model selection, and any revisions to the analysis. The test set is used only after those decisions are complete.

Worked Example: Calculating Test-Set MSE

After selecting setting D in the preceding example, a hypothetical model is refitted on the development data. Its fitted equation, using development-data standardization, is $\hat{y}=20+3z_1-2z_2$, where $y$ is measured in thousands of dollars. Four reserved test observations have the standardized predictor values and observed responses shown below. Calculate the test MSE and RMSE.

Observation	$z_1$	$z_2$	Observed $y$	Prediction $\hat{y}$	Residual $y-\hat{y}$	Squared residual
1	0	0	21	20	1	1
2	1	0	21	23	-2	4
3	0	1	18	18	0	0
4	-1	1	16	15	1	1

Step 1: Verify the predictions. For observation 1, $20+3(0)-2(0)=20$. For observation 2, $20+3(1)-2(0)=23$. For observation 3, $20+3(0)-2(1)=18$. For observation 4, $20+3(-1)-2(1)=15$. Subtracting each prediction from its observed response gives residuals $1,-2,0,1$.

Step 2: Average the squared residuals. The squared residuals sum to $1+4+0+1=6$. Therefore, test MSE is $6/4=1.5$ in squared thousands-of-dollars units. The test RMSE is $\sqrt{1.5}=1.2247$ thousand dollars, rounded to four decimal places.

Check. The squared-error total is $6$, and dividing by the four test observations gives $1.5$. Do not divide by $n-1$: test MSE is the average squared prediction error, not a sample variance estimate.

Define the fitting convention.
Specify the elastic net objective, the response scale, and the candidate $\alpha$ and $\lambda$ settings.

Preprocess within each fold.
Calculate predictor means and standard deviations from that fold's training portion and apply them to its held-out portion.

Compare matched-fold scores.
Calculate each setting's mean validation error using the same folds, then identify the smallest mean error and its estimated standard error.

Select and refit.
Apply the one-standard-error threshold and a stated simplicity criterion; refit the chosen setting using the development data and its own preprocessing values.

Use the test set once.
Transform test predictors using development-data scaling values, calculate predictions, and report the test metric without returning to model selection.

Interpretation and Exam Traps

A complete tuning result reports more than $\alpha$ and $\lambda$. It should identify the scoring metric, fold design, scaling procedure, minimum cross-validation score, one-standard-error threshold, selection rule, and final test metric. It should also distinguish the cross-validation score used to choose a setting from the test score used to assess the resulting modeling procedure. Choosing the best setting and reporting its same cross-validation minimum as an unbiased final performance estimate can be optimistic, because the settings were selected in part for favorable validation results.

In particular, a test set is not another validation fold to consult repeatedly. If its results lead to a new $\alpha$, a new $\lambda$, a different predictor transformation, or a revised model, then the test set has influenced model development and no longer provides an untouched final assessment. Report the test result as an evaluation of the already selected and refitted model.

Using full-data scaling before cross-validation: this lets held-out predictor information affect preprocessing. Scale separately within each fold.
Using a different fold split for each setting: this adds avoidable variation to candidate comparisons. Compare settings on matched folds.
Choosing the smallest $\lambda$ automatically: $\lambda$ alone does not determine complexity across different $\alpha$ values. Use an explicit criterion such as the number of nonzero slopes.
Confusing test MSE with a variance estimate: divide the sum of squared test errors by the number of test observations, not by one less than that number.
Reporting a test result after tuning on the test set: once test results guide choices, they are no longer an independent final evaluation.

Key takeaway: Standardize from training data only, compare tuning settings on common folds, apply the one-standard-error rule using a clearly stated simplicity criterion, and reserve the final test set for evaluation after the model and preprocessing are fixed.

Check Your Understanding

For each problem, give the requested result and check that the stated data support the calculation.

A training sample has predictor values $2,4,6$. Using the sample standard deviation, find its mean, standard deviation, and three standardized values. (Answer: Mean $4$, sample standard deviation $2$, standardized values $-1,0,1$.)
A candidate's three fold MSEs are $8,10,12$. Calculate its mean MSE and the estimated standard error of that mean using the sample standard deviation across folds. (Answer: Mean MSE $10$; standard error $2/\sqrt{3}=1.1547$.)
The minimum mean cross-validation MSE is $5.0$, with estimated standard error $0.4$. Candidate P has mean MSE $5.3$ and 4 average nonzero slopes; candidate Q has mean MSE $5.5$ and 2 average nonzero slopes; candidate R has mean MSE $5.6$ and 1 average nonzero slope. Apply the one-standard-error rule and choose the eligible candidate with the fewest average nonzero slopes. (Answer: Threshold $5.4$; P and Q are eligible, so choose Q.)
A final test set has four residuals $2,-1,0,1$. Find test MSE and test RMSE. (Answer: MSE $6/4=1.5$; RMSE $\sqrt{1.5}=1.2247$, in the corresponding response units.)
A model uses training mean $10$ and training sample standard deviation $2$ for a predictor. A held-out value is $14$. What standardized value should be supplied to the fitted model, and should the held-out value be included when recalculating those scaling quantities? (Answer: $z=(14-10)/2=2$; no, use the training mean and standard deviation.)

What You'll Learn