Why Reproducibility Requires More Than a Random Seed
Penalized-model tuning involves several choices that can change the selected fit: which observations are held out in each fold, which alpha and lambda values are tested, how predictors are prepared, and which loss is used to compare predictions. A random seed helps repeat a random operation, but it does not by itself document all those choices. Reproducible work makes the choices explicit and saves them with the results.
For a fixed dataset, a sound audit trail should let another analyst answer: which observations were in each fold; which model settings were compared; how was each fold’s training data transformed; and which loss produced the reported comparison? This is especially important when a small change in a fold assignment or preprocessing step could change the selected settings.
A seed is useful when folds are generated randomly, but the resulting fold vector is the direct record of the split. Saving only the seed can be insufficient: changes in the data row order, random-number generator, software version, or code can produce different assignments. Even with exactly the same folds, a changed lambda grid or loss function is a different tuning experiment.
Record the Resampling Plan Explicitly
For ordinary \(K\)-fold cross-validation, each observation is assigned to one of \(K\) folds. For each fold, a model is fitted using observations outside that fold and evaluated on the held-out observations. The fold assignment is commonly represented by an integer vector, one entry per row of the analysis data.
For example, this R code creates and then records an assignment for 12 rows. The seed makes the random assignment repeatable under the same relevant computational setup; the saved vector records what was actually used.
R sketch: set.seed(731); foldid <- sample(rep(seq_len(4), length.out = 12)); saveRDS(foldid, "foldid.rds").
When rows are independent and exchangeable, randomly assigning rows may be reasonable. It is not a universal rule. If observations are clustered by policyholder, repeated over time, or otherwise dependent, the split must respect that structure; otherwise information from a related observation can appear in both training and validation data. The resampling design should match the prediction task.
Worked Example: Checking and Saving a Fold Assignment
A hypothetical pricing team has nine records and intends to use three-fold cross-validation. It uses the explicit vector \(f=(1,2,3,1,2,3,1,2,3)\). Verify the assignment and describe the reproducibility record. Use the vector in row order; do not regenerate it from a seed.
Method. Count the observations assigned to each fold, then identify the information that must accompany the vector. This avoids treating a seed as though it were the fold assignment itself.
Fold 1 contains rows 1, 4, and 7, so its size is 3. Fold 2 contains rows 2, 5, and 8, also size 3. Fold 3 contains rows 3, 6, and 9, also size 3. The total is \(3+3+3=9\), so every row is assigned once and all three folds are represented.
The vector can be passed to cv.glmnet through its foldid argument. The audit record should save this exact vector alongside the row identifiers or a fixed row-order description. If the row order changes, the same vector labels different observations, so the vector alone is no longer enough to identify the split.
Check. The fold sizes sum to the data size, and each row has one label in \(\{1,2,3\}\). The assignment is fully specified; a seed is unnecessary for repeating this particular split. The common mistake avoided is reporting a seed while failing to preserve which rows it assigned to each fold.
Fix the Tuning Grid and the Evaluation Rule
In glmnet, alpha controls the mixture of ridge and lasso penalties, while lambda controls penalty strength. For a reproducible comparison, record the alpha values and lambda values actually fitted—not merely a statement that both parameters were tuned. Also record the family, the loss measure, observation weights if used, and relevant fitting options such as standardization.
For example, cv.glmnet accepts a supplied fold vector through foldid, a supplied lambda sequence through lambda, and a fixed alpha through alpha. Alpha is fixed for an individual call, so a grid of alpha values is commonly handled by making separate calls with the same folds and the intended lambda sequence. A lambda sequence should be recorded in the order supplied; for glmnet it is conventionally decreasing.
A comparison is interpretable only when candidate fits are assessed using a common resampling plan and a consistent loss definition. For a Gaussian response evaluated by mean squared error, let \(y_i\) be the observed outcome and \(\hat{y}_i^{(-k)}\) the prediction for observation \(i\) from a model fitted without fold \(k\). The observation-level cross-validation mean squared error is
This formula assumes that every observation is evaluated once and that the prediction for an observation comes from a fit that excluded its fold. Each squared error is counted once; dividing by \(n\) gives the average squared prediction error per observation. It is not the appropriate formula if the intended evaluation criterion is a different loss, such as deviance or absolute error.
A compact R workflow can loop over alpha while keeping the fold assignments and lambda sequence fixed. For example, define alpha_grid <- c(0, 0.5, 1) and a decreasing lambda_grid, then call cv.glmnet(x, y, alpha = a, lambda = lambda_grid, foldid = foldid, type.measure = "mse") for each \(a\). Save the exact vectors and the returned cross-validation objects. If weights are part of the analysis, save their values and row order as well.
Worked Example: Selecting From a Recorded Alpha-Lambda Grid
A hypothetical portfolio study compares three prespecified candidates using the same folds, Gaussian models, and mean squared error. The candidates and their fold-level mean squared errors are: \((\alpha,\lambda)=(0,0.20)\), with fold losses \(0.42, 0.38, 0.44\); \((0.5,0.10)\), with losses \(0.36, 0.40, 0.38\); and \((1,0.05)\), with losses \(0.34, 0.43, 0.41\). The folds have equal sizes. Find the mean fold loss for each candidate and select the lowest-loss candidate.
Method. Average the three fold losses for each candidate. Because the fold sizes are equal, the unweighted average of fold means is also the observation-level average of squared errors. This avoids comparing candidates using different splits or different loss definitions.
For \((0,0.20)\), the sum is \(0.42+0.38+0.44=1.24\). Its mean is \(1.24/3=0.413333\ldots\), which is \(0.4133\) to four decimal places.
For \((0.5,0.10)\), the sum is \(0.36+0.40+0.38=1.14\). Its mean is \(1.14/3=0.380000\), or \(0.3800\) to four decimal places.
For \((1,0.05)\), the sum is \(0.34+0.43+0.41=1.18\). Its mean is \(1.18/3=0.393333\ldots\), or \(0.3933\) to four decimal places.
The smallest mean is \(0.3800\), so among these three candidates the selected setting is \((\alpha,\lambda)=(0.5,0.10)\). The calculation does not establish that this candidate is best outside the tested grid. The audit record should retain all three candidates, their fold losses, and the common fold vector—not just the winning pair.
Check. The ranking is \(0.3800<0.3933<0.4133\). The selected candidate is the middle alpha value, not the candidate with the smallest single-fold loss. The common mistake avoided is selecting on one fold rather than on the prespecified cross-validation criterion.
Keep Preprocessing Inside the Resampling Process
A saved model matrix is not automatically a valid resampling input. If missing values are imputed, variables are transformed, or factor levels are encoded using information from all observations before splitting, held-out data have influenced the training process. The earlier material on leakage-free preprocessing addresses this issue; the reproducibility requirement is to save the fold-specific steps as well as the folds.
There is an important distinction between glmnet’s own predictor standardization and preprocessing performed before fitting. With standardize = TRUE, glmnet standardizes predictors as part of the fit; during cross-validation the fit for a fold uses its training data. But glmnet does not automatically make arbitrary imputation or custom feature-engineering procedures fold-specific when those procedures were already applied to the input matrix. Record which steps were performed externally and how each training fold’s version was applied to its held-out fold.
Choose a resampling design suited to the data structure, generate or define the fold vector, and save it in the analysis row order.
Estimate imputation values, transformations, and encoding rules from that fold’s training observations, then apply those rules to its held-out observations.
Use the specified alpha and lambda values, weights, family, and loss. Save fold-level results as well as the summary used to compare candidates.
Worked Example: Auditing a Fold-Specific Imputation
In a hypothetical four-record dataset, a continuous predictor has values \(2,4,6,100\). For one validation split, the first three records are training data and the record with value 100 is held out. Suppose missing values, if any, would be imputed using the training-fold mean. Compare the correct training-fold mean with a mean calculated from all four records, and explain which value belongs in the saved preprocessing record.
Method. Calculate each mean from its stated set of records. The held-out value is excluded when estimating a training-fold preprocessing quantity.
The training-fold mean is \((2+4+6)/3=12/3=4\). The all-data mean is \((2+4+6+100)/4=112/4=28\). The two values differ by \(28-4=24\).
For this fold, 4 is the appropriate mean to record for the training-fold imputation rule. Using 28 would allow the validation value 100 to influence preprocessing. Even if no value is missing in this particular small example, it demonstrates why the recipe must be estimated from training data rather than calculated once on the complete dataset.
Check. The saved record for this fold should identify the training rows and the imputation mean 4, together with the rule used to apply it to held-out data. The common mistake avoided is storing one globally calculated preprocessing value and reusing it in every fold.
What to Save and What Reproducibility Can Promise
A useful run record is more complete than a printed minimum cross-validation error. It should make it possible to reconstruct both the inputs to tuning and the steps that produced the reported comparison.
- Data identity and ordering: save a version or checksum of the analysis data, the outcome and predictor definitions, and the row identifiers in model-matrix order.
- Resampling: save the seed used to generate random folds, if applicable, and always save the resulting fold vector. Record the number and type of folds.
- Candidate fits: save the alpha and lambda vectors, family, loss measure, weights, standardization choice, and relevant glmnet settings.
- Preprocessing: record the recipe and the fold-specific quantities estimated from training data, including rules for applying transformations to held-out observations.
- Results and software: save fold-level losses, summary results, the selected settings, R and package versions, and any parallel-computing configuration.
Reproducibility has levels. Explicit folds and settings support a repeated analysis, but exact numerical identity can still depend on software versions, numerical libraries, or parallel execution. Therefore, report whether a repeated run matches at the level of fold assignments and settings, or also at the level of numerical results within stated precision. A changed assignment is a different resampling analysis even if it happens to select the same alpha and lambda.
Check Your Understanding
For each problem, state the reproducibility check or calculation that applies. Answers are provided at the end of each item.
- A dataset has 15 rows and a saved five-fold vector assigns three rows to each fold. What additional item is needed to identify which observations those labels refer to after data rows are reordered? (Answer: The row identifiers in the original model-matrix order, or an equivalent fixed row-order record.)
- A random seed generates a fold assignment, but the analyst does not save the resulting vector. Give one reason a later run might use different folds. (Answer: For example, changed row order, random-number generator behavior, or code can change the assignment even if a seed is recorded.)
- Three equal-sized folds have mean squared errors \(0.25, 0.31, 0.28\). Calculate their mean, to four decimal places. (Answer: \((0.25+0.31+0.28)/3=0.84/3=0.2800\).)
- A fold has training predictor values \(3,5,10\) and a held-out value \(40\). If the training-fold mean is used for imputation, what mean should be recorded, and what incorrect value results from using all four observations? (Answer: Training-fold mean \(=(3+5+10)/3=6\); all-data mean \(=(3+5+10+40)/4=14.5\).)
- A tuning report gives the winning alpha and lambda but omits the tested grid and fold-level losses. Name two pieces of information needed to make the comparison auditable. (Answer: Any two of the explicit fold assignments, the tested alpha and lambda values, the loss definition, the fold-level losses, or the preprocessing record.)