From Candidate Fits to a Defensible Choice
A regularized regression analysis is not complete when software returns coefficients. An actuarial analyst must decide what data can be used for tuning, compare penalty settings fairly, assess the selected procedure on data kept out of model development, and explain what the results do—and do not—support. These decisions matter because tuning can make a model look better on the same observations that guided its construction.
This workshop follows a hypothetical insurer estimating annual claim cost in thousands of dollars from policy characteristics. The response is continuous, predictors are numeric and complete for this example, and the objective is prediction rather than causal interpretation. The process also applies when predictors include factors or missing values, provided every data-dependent preprocessing step is repeated using only the relevant training portion.
Specify the Model and the Workflow
For a continuous response, suppose \(Y_i\) is the annual claim cost for policy \(i\), and \(x_{ij}\) is predictor \(j\) for that policy. The elastic-net fit minimizes a squared-error loss with a penalty:
Here \(n\) is the number of observations used for that fit, \(p\) is the number of predictors, \(\lambda\geq 0\) controls penalty strength, and \(0\leq\alpha\leq1\) controls the penalty mixture. The intercept \(\beta_0\) is not penalized. At \(\alpha=1\), the penalty is lasso; at \(\alpha=0\), it is ridge; intermediate values combine the two. This specification is for a Gaussian response with squared-error loss. Other response types require an appropriate model family and loss.
A larger \(\lambda\) generally shrinks coefficients more strongly. Lasso can set some coefficients exactly to zero; ridge generally shrinks coefficients without setting them exactly to zero. The optimal settings are not known in advance, so they must be chosen using training data and a validation procedure—not by looking for the best test-set score.
A leakage-safe order of work is:
Specify the response, prediction time, eligible predictors, and performance measure. Exclude identifiers and any information unavailable at prediction time.
Split observations before tuning. Do not use the test outcomes to select transformations, predictor sets, alpha, lambda, or a decision threshold.
Use the same folds to compare candidate alpha values. For each alpha, evaluate a lambda path and retain the fold-based results.
Fit the chosen specification using all training data, then calculate the prespecified test metric and report its limitations.
Preparing Predictors Without Leakage
Begin with one row per prediction unit, a correctly defined response, and predictors that would be available when a prediction is made. Remove duplicate rows and non-predictive identifiers. Check units, impossible values, factor levels, and the time ordering of each field. These are modeling decisions, not harmless formatting details: a post-outcome field can create excellent validation results while being unavailable in practice.
For this worked case, all predictors are complete numeric measurements, so there is no imputation to estimate. The columns are on different scales. The R package glmnet standardizes predictor columns by default when fitting; this is important because a coefficient penalty should not penalize a predictor differently merely because it is measured in dollars rather than thousands of dollars. Keep this behavior consistent during tuning and refitting. If predictors are centered, scaled, imputed, or otherwise transformed outside glmnet, estimate those quantities using only each fold's analysis portion, then apply them to that fold's assessment portion.
If a factor predictor is present, encode its levels consistently and define how new or rare levels will be handled. A data-derived imputation value, rare-level grouping rule, or feature-selection decision must be learned within the training portion of each resampling split. Creating a design matrix from all observations before cross-validation can let validation-fold information influence preprocessing. Even when this influence does not use the response, it can make the validation procedure less representative of the intended prediction process.
In R, first make a reproducible training/test split, then create the predictor matrix from the permitted predictors. For example, after defining a row index called train_id, use x_train <- as.matrix(dat[train_id, predictor_names]) and y_train <- dat$annual_cost[train_id]; create x_test and y_test from the remaining rows. Keep identifiers and the response out of predictor_names. This simple matrix setup is appropriate for the stated complete-numeric example; missing values and learned factor transformations require fold-wise preprocessing rather than a single preprocessing operation on all training rows.
Worked Example: Locking Down the Data Roles
A hypothetical portfolio has 1,000 policy records, a numeric response, and six complete numeric predictors. The analyst reserves 200 records as a test set and uses the other 800 for tuning. The target is prediction of next-year cost using information available at renewal.
Method. Split first, then tune only within the 800 training records. A reproducible R pattern is to set a seed, sample 200 row indices for the test set, and define the remaining 800 as train_id. The six predictor names are selected explicitly; a policy number and any field recorded after next-year cost is incurred are excluded.
Suppose the analyst wants five-fold cross-validation. Each of the 800 training records belongs to one assessment fold, and each fit is trained on the other four folds. The test set is not assigned to any of these folds. If an imputation median were needed, it would be computed on the four-fold analysis subset and applied to the omitted fold; it would not be computed once from all 800 rows before cross-validation.
Check. There are 5 folds of 160 assessment records each, since \(800/5=160\). Each fold fit uses 640 records, since \(800-160=640\). The 200 test records remain untouched during all five fits and all hyperparameter comparisons. The common mistake—tuning on all 1,000 records and then calling 200 of them a test set—would leak test information into the model-development decisions.
Tune Alpha and Lambda on Common Folds
For a practical grid, choose several alpha values spanning ridge, elastic net, and lasso, such as 0, 0.25, 0.5, 0.75, and 1. Use the same fold assignments for every alpha so that differences in average validation loss are less confounded by different random partitions. For each alpha, glmnet evaluates a sequence of lambda values. In R, a basic implementation uses cv.glmnet(x_train, y_train, alpha = a, foldid = fold_id, type.measure = "mse") for each value a. Set the random seed before creating fold_id and use the same vector in every call.
For Gaussian regression, type.measure = "mse" evaluates mean squared error. cv.glmnet reports the average cross-validation error over its lambda path, along with lambda.min, the lambda with the minimum estimated cross-validation error for that alpha, and lambda.1se, the largest lambda whose estimated error is within one standard error of the minimum for that alpha. Larger lambda on this path means stronger shrinkage.
To compare alpha values, record for each fit its alpha, selected lambda, cross-validation MSE, and the corresponding standard error. A simple rule is to select the alpha-lambda combination with the lowest cross-validation MSE. A more conservative one-standard-error rule requires care: the familiar lambda.1se is defined within a fixed alpha fit. If applying one common rule across the entire alpha-lambda grid, retain fold-level losses for all candidates, find the minimum mean loss and its standard error, and choose a prespecified simpler candidate whose mean loss is no more than that threshold. Do not assume that separately reported lambda.1se values automatically implement a global one-standard-error comparison across alpha values.
The one-standard-error rule is a selection convention, not proof that the chosen model is truly equivalent to the minimum-error model. It trades a small estimated performance difference for a more strongly regularized candidate. “Simpler” must also be defined for the task: it might mean fewer nonzero coefficients, stronger shrinkage, or a preference for ridge when correlated predictors make individual selections unstable.
Worked Example: Choosing a Candidate From Cross-Validation
Suppose five-fold tuning on the 800 training records produces these illustrative results at each alpha's minimum-error lambda. The values are hypothetical cross-validation summaries, not results from an actual insurer.
| Alpha | Selected lambda | Mean CV MSE | Estimated standard error |
|---|---|---|---|
| 0.00 | 0.18 | 118 | 8 |
| 0.50 | 0.12 | 112 | 8 |
| 1.00 | 0.08 | 115 | 9 |
Method. First apply the prespecified minimum-mean-CV-MSE rule. The smallest mean is 112, so the provisional choice is alpha \(=0.50\), lambda \(=0.12\). This is an elastic-net fit, not a lasso fit, because alpha is strictly between zero and one.
For a global one-standard-error comparison, the minimum mean is 112 and its standard error is 8, giving a threshold of \(112+8=120\). The three displayed candidates have means 118, 112, and 115, all no greater than 120. If a prespecified simplicity rule prefers ridge because individual variable selection is unstable, the alpha \(=0\) candidate is eligible under this threshold. The analyst may select it, but must state that this choice follows the simplicity rule rather than claiming it has the lowest estimated error.
The displayed rows alone do not show how many predictors each candidate retains or the fold-level errors at every lambda. Those must be checked before making a genuine global one-standard-error selection. A common trap is to call alpha \(=0.50\) “the one-SE model” just because its mean is minimum; that is the minimum-error choice, not a one-standard-error choice.
Held-Out Evaluation and Selection Stability
After selecting the procedure using training data, refit it on all training observations. In R, fit glmnet(x_train, y_train, alpha = chosen_alpha, lambda = chosen_lambda), then obtain predictions for x_test. For a continuous response, test RMSE is the square root of the average squared difference between actual and predicted outcomes:
Use this test result to assess the already selected model, not to search among candidates for the winner. If several candidates are displayed on test data, describe that comparison as descriptive; selecting the best one from the test results turns the test set into another validation set. Report the response scale and metric clearly. For example, if costs are in thousands of dollars, an RMSE of 2 means a typical error magnitude of about 2 thousand dollars, though RMSE is not literally the average absolute error.
Also examine whether selected variables are stable. Across resamples, lasso may retain different members of a group of correlated predictors, even when predictions remain similar. Record selection frequencies or coefficient variation, and distinguish instability in the identity of selected variables from instability in predictions. Selection frequency is descriptive evidence about sensitivity to the sampled data; it is not a probability that a variable is causally important or a guarantee of future selection.
Worked Example: Test RMSE and an Unstable Selection
Suppose the analyst's prespecified simplicity rule selects a ridge model from the previous comparison. On four hypothetical held-out policies, actual annual costs and predictions, all measured in thousands of dollars, are: actuals 20, 30, 40, 50; ridge predictions 22, 29, 42, 48; elastic-net predictions 21, 31, 38, 52; and lasso predictions 19, 34, 35, 55.
Method. Evaluate the selected ridge model with the test RMSE formula. Its residuals, actual minus predicted, are \(-2, 1, -2, 2\). Their squared values are \(4, 1, 4, 4\), with sum \(13\). Thus test MSE is \(13/4=3.25\), and test RMSE is \(\sqrt{3.25}=1.8028\) thousand dollars, rounded to four decimal places. In dollars, this is approximately $1,802.78 when rounded to the nearest cent.
For comparison only, the elastic-net residuals are \(-1,-1,2,-2\), with squared-error sum \(1+1+4+4=10\). Its test RMSE is \(\sqrt{10/4}=\sqrt{2.5}=1.5811\) thousand dollars, rounded to four decimal places. The lasso residuals are \(1,-4,5,-5\), with squared-error sum \(1+16+25+25=67\). Its test RMSE is \(\sqrt{67/4}=\sqrt{16.75}=4.0927\) thousand dollars, rounded to four decimal places.
The elastic-net score is smallest on these four test cases, but the analyst must not replace the prespecified ridge choice solely because of this test comparison. Four observations also provide a highly uncertain assessment. In separate training resamples, suppose lasso selects predictor A in 6 of 10 fits and its strongly correlated predictor B in 5 of 10 fits, with both selected in 3 fits. These counts show that the selected identity changes across resamples; they do not establish that A or B has a stable individual effect. Ridge can be a defensible deployment choice when predictive performance is close and stable individual selection is not required, but the final decision should reflect the intended use and the available validation evidence.
Check. The ridge squared errors sum to 13, so its test MSE is \(3.25\); squaring its reported RMSE gives approximately \(1.8028^2=3.2501\), the small discrepancy arising from rounding. The comparison avoids the trap of ranking models on the test set and then presenting that same set as an unbiased assessment of the winner.
Make the Deployment Choice Explicit
A deployment-ready choice is a documented procedure, not merely a coefficient vector. State the response and prediction time, included predictors, preprocessing rules, selected alpha and lambda, tuning method, final training data, and the reported test metric. Explain whether the selected model was chosen for minimum estimated error or by a stated simplicity rule. Preserve the factor-level handling and any transformations so that future predictions use the same design matrix definition.
Finally, communicate relevant limits. A held-out score estimates performance under conditions represented by the test data; it does not establish performance after a change in portfolio mix, data quality, or operating practice. Penalization can reduce sensitivity to noisy coefficients, but it does not correct leakage, omitted predictors, measurement error, or biased outcome recording. Predictions and selected variables should be monitored after deployment, and material changes in data or use may require renewed validation. This workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, apply the workflow and keep tuning decisions separate from final test assessment.
- A hypothetical cost model has 600 records. An analyst uses 480 for tuning and 120 for testing, but computes an imputation median from all 600 records before five-fold cross-validation. Identify the leakage and state how to correct it. (Answer: Compute the median separately from each fold's analysis data, apply it to that fold's assessment data, and keep the 120 test outcomes out of every tuning and preprocessing decision.)
- Two candidates have cross-validation MSEs 86 with standard error 5 and 89 with standard error 4. Using a global one-standard-error threshold based on the minimum candidate, which candidate is eligible if the rule is to choose the stronger penalty whenever eligible? (Answer: The threshold is \(86+5=91\); both candidates are eligible, so choose the stronger-penalty candidate if it is the one with MSE 89.)
- A chosen model predicts held-out costs of 12, 18, and 25 thousand dollars when actual values are 10, 20, and 22 thousand dollars. Calculate test MSE and RMSE in thousands, rounding RMSE to four decimals. (Answer: Residuals are \(-2,2,-3\); squared errors sum to \(4+4+9=17\), MSE is \(17/3=5.6667\), and RMSE is \(\sqrt{17/3}=2.3805\) thousand dollars.)
- Across 20 resamples, lasso selects a utilization indicator in 8 fits and a correlated mileage indicator in 10 fits, with both selected in 4 fits. What can be concluded about selection stability, and what cannot be concluded? (Answer: The identities are not consistently selected across resamples; these counts do not establish causal importance or imply that either variable has a stable individual effect.)
- A team compares eight candidates on its test set, chooses the lowest-RMSE candidate, and reports that candidate's score as final test performance. Is the reported score an independent assessment? State the repair. (Answer: No. The test set was used for selection; choose the model using training data and validation, then assess once on a new untouched test set.)