Why Fold Assignment Is Part of the Modeling Design
In \(K\)-fold cross-validation, observations are divided into \(K\) validation folds. For each fold, the model is fitted using the observations outside that fold and evaluated on the held-out observations. In penalized regression, the assignments determine which observations are held out when cv.glmnet evaluates a model. They therefore affect the estimated predictive performance, not just the mechanics of fitting.
Randomly assigning rows can be reasonable when rows are independent and similarly distributed. It can be misleading when one class is rare, when multiple rows belong to the same person or organization, or when the purpose is to predict future outcomes. The fold design should reflect the intended prediction task and the unit that is meant to generalize.
cv.glmnet, the argument foldid supplies these assignments: it must have one entry per row of the response and predictor data, and its entries identify the fold to which each observation belongs. For each fold, that fold is held out and the other folds form the training data.
For a binary response, outcome stratification means assigning observations so that each fold has a similar proportion of each outcome class. For grouped data, the assignment unit is the group rather than the row. For time-dependent prediction, validation observations must occur after the training observations. These are different design requirements; satisfying one does not automatically satisfy the others.
What a Fold Assignment Should Preserve
First identify the prediction target. If the goal is to predict outcomes for new, independent records, random folds may be suitable. If records from the same customer, clinic, employer, or device are related, a row-level split can put some records from a group in training and others from that same group in validation. The validation result may then reflect familiarity with that group rather than performance on a new group.
For binary outcomes, stratification is useful when events are uncommon and a fold might otherwise contain very few or no events. It can make fold-level performance comparisons less erratic and ensure that folds contain the classes needed for the chosen evaluation. It does not create additional events or remove uncertainty caused by a small number of events. If there are too few events for the chosen number of folds, consider fewer folds or a different validation design.
Fold sizes also matter. If the number of observations is not divisible by \(K\), some folds will be one observation larger than others. The pooled out-of-fold mean loss weights each observation equally, so larger folds contribute more to that mean.
Here \(F_k\) is the set of observations in fold \(k\), \(n_k\) is its size, and \(L_i\) is the loss for observation \(i\) when it is predicted by a model that did not train on it. The identity follows by partitioning the total sum of losses into fold-specific sums: \(\sum_i L_i=\sum_k\sum_{i\in F_k}L_i\). Dividing by \(n\) gives the weighted fold average shown above. A simple unweighted average of the fold means equals this pooled mean only when all folds have equal size.
Worked Example: Stratifying a Rare Binary Outcome
A hypothetical insurer has 24 independent applications and a binary outcome \(y\), where 1 indicates a claim within the observation period. Six applications have \(y=1\) and 18 have \(y=0\). The analyst wants \(K=3\) folds for a binomial penalized model. Construct a stratified assignment and check the counts.
Method. Assign the six events across folds first, then distribute the non-events. This preserves the observed event proportion in each fold when the counts divide evenly.
There are \(6/3=2\) events per fold and \(18/3=6\) non-events per fold. Thus each fold has \(2+6=8\) observations, and its event proportion is \(2/8=0.25\). The full sample proportion is \(6/24=0.25\), so the fold proportions match the sample proportion.
One way to create such assignments in R is to assign fold numbers separately within each outcome class. For a response vector named y, use a loop over the values 0 and 1; within each class, randomly distribute its row indices as evenly as possible among folds 1, 2, and 3. Store the assigned fold number for each row in an integer vector named foldid. Then inspect the result with a table of y by foldid and a table of foldid alone. The required checks are two events, six non-events, and eight total observations in each fold in this example.
Check. Across all three folds, the event count is \(3(2)=6\), the non-event count is \(3(6)=18\), and the total is \(3(8)=24\), matching the data. Supply the resulting vector as foldid = foldid in cv.glmnet.
Common mistake avoided. Randomly dividing all 24 rows without checking the class counts might leave one validation fold with no events. Stratification makes the intended balance explicit; it does not guarantee that every other modeling assumption is appropriate.
Grouped Observations: Keep Each Group Together
When the intended prediction target is a new group, all rows from a group must be assigned to the same fold. Otherwise, information associated with a group can appear in both training and validation data. For example, multiple claims from one policyholder may share characteristics not represented by the predictors. Splitting those claims across folds can overstate how well the model predicts claims from previously unseen policyholders.
Build the assignment at the group level and then map each group’s fold number back to its rows. Inspect both the number of observations and the outcome counts in each fold. Balancing the number of groups alone does not necessarily balance the number of rows or events, because group sizes and outcomes can differ.
Worked Example: Grouped Claim Records
A hypothetical study contains records from six employers. Each row is a claim, and the prediction target is claims from a new employer. The employer-level data are: A has 4 claims, of which 2 are events; B has 3 claims, of which 0 are events; C has 2 claims, of which 1 is an event; D has 4 claims, of which 0 are events; E has 3 claims, of which 2 are events; and F has 2 claims, of which 0 are events. Assign employers to three folds, keeping each employer intact.
Method. Assign employers, not individual claims, to folds. One valid assignment is fold 1: A and D; fold 2: B and E; fold 3: C and F.
Fold 1 has \(4+4=8\) claims and \(2+0=2\) events, so it has 6 non-events. Fold 2 has \(3+3=6\) claims and \(0+2=2\) events, so it has 4 non-events. Fold 3 has \(2+2=4\) claims and \(1+0=1\) event, so it has 3 non-events. The respective event proportions are \(2/8=0.25\), \(2/6\approx0.3333\), and \(1/4=0.25\).
The fold sizes differ, and the event proportions are not identical. That is acceptable if keeping employers intact is essential to the prediction target. Assigning one employer’s claims to different folds just to improve row-level balance would violate the group-level design.
In R, create a mapping from each of the six employer identifiers to its assigned fold, then use that mapping to form a row-level foldid. Check that each employer maps to exactly one fold; tabulate the number of rows and events by fold; and confirm that the totals are 18 claims and 5 events. With the assignment above, those totals are \(8+6+4=18\) and \(2+2+1=5\).
Common mistake avoided. Applying a stratified row-level split before accounting for employers can put the same employer in training and validation data. The correction is to preserve groups first, then inspect the balance that results.
Time-Ordered Data Need Time-Ordered Validation
For forecasting, a random \(K\)-fold assignment usually answers the wrong question. In an ordinary fold split, the training data for an early validation period can include later observations. A model evaluated this way may benefit from information that would not have been available at the forecast date.
A forward-chaining or rolling-origin design instead trains on earlier observations and validates on a later block. foldid in cv.glmnet labels held-out rows, but it does not make each training set consist only of earlier rows. Assigning consecutive time blocks different fold numbers and passing them to cv.glmnet still leaves future blocks in the training data for earlier held-out blocks. For a genuine forward-time design, fit the model separately on each permitted training window and predict its later validation window.
Worked Example: Validating a Monthly Forecast
A hypothetical portfolio has observations for 12 successive months. The goal is to predict later months using only information available earlier. Use three two-month validation windows: months 7–8, 9–10, and 11–12.
Method. Fit on months 1–6 and validate on 7–8; then fit on months 1–8 and validate on 9–10; finally fit on months 1–10 and validate on 11–12. In each fit, the validation months are excluded, and no later month is included in its training data.
Suppose the six resulting out-of-sample losses, in month order 7 through 12, are 0.10, 0.20, 0.30, 0.10, 0.20, and 0.30. Their total is \(0.10+0.20+0.30+0.10+0.20+0.30=1.20\). The pooled mean loss is \(1.20/6=0.20\). Each validation window has two months, so the three window means are \(0.15\), \(0.20\), and \(0.25\); their unweighted mean is also \(0.20\).
The first six months are used for initial training and do not receive out-of-sample predictions in this design. The estimate therefore summarizes performance for the six later validation months, not for all 12 months. A random fold assignment followed by ordinary cross-validation would not reproduce this forecasting setup.
Use Consistent Folds When Comparing Alpha Values
In elastic net, \(\alpha\) controls the mix of ridge and lasso penalties. When comparing candidate values of \(\alpha\), use the same fold assignments for each call to cv.glmnet. Otherwise, the comparison combines two sources of variation: the change in penalty mixture and the change in which observations were held out. Reusing foldid makes the validation partition common to the fits.
For example, for candidate values 0, 0.5, and 1, make three calls with the same predictor matrix, response, and foldid, changing only the alpha argument. If the data are grouped, the common assignments must already keep groups intact; if the target is forecasting, use a time-appropriate evaluation procedure rather than treating an ordinary foldid partition as forward validation.
A shared fold assignment controls the partition, but it does not by itself make every other aspect of a comparison identical. In particular, if each call constructs its own candidate penalty sequence, the candidate values need not be identical across calls. The purpose here is narrower: avoid attributing differences caused by different held-out observations to \(\alpha\).
Practical Fold Checks and Common Traps
- Check the length. There must be one fold assignment for each row of the response and predictor data, in the same row order.
- Check the labels and counts. Confirm that the intended number of folds is present and none is empty. A fold assignment with a missing or misaligned row can silently invalidate the design.
- Check outcome counts. For classification, tabulate the response by fold. If events are rare, check whether each validation fold has enough examples for the intended assessment.
- Check groups. Confirm that each group occurs in only one fold when the target is a new group.
- Check time direction. For forecasting, make sure every training window precedes its validation window.
- Reuse the assignment. When comparing \(\alpha\) values, pass the same fold vector to each fit.
Check Your Understanding
For each problem, identify the fold design that matches the stated prediction target and check the relevant counts or structure.
- A binary data set has 40 independent records, including 8 events, and uses four folds. If the events and non-events divide evenly, how many of each and how many total records should be in each stratified fold? (Answer: 2 events, 8 non-events, and 10 records per fold.)
- A study has 12 clinics with 5 records per clinic. The goal is prediction for a new clinic, using three folds. How many clinics and records should be in each fold if clinics are divided evenly, and what must be checked? (Answer: 4 clinics and 20 records per fold; check that no clinic appears in more than one fold.)
- A monthly forecasting study uses validation months 13–15, 16–18, and 19–21. What training windows avoid using future observations, and how many months receive validation predictions? (Answer: Train on months 1–12, 1–15, and 1–18, respectively; 9 months receive validation predictions.)
- A response has 7 events and 21 non-events. The analyst wants four stratified folds with equal class counts, if possible. What prevents exact equality, and what are the closest event counts across folds? (Answer: 7 events cannot be divided equally among 4 folds; the closest allocation is 2, 2, 2, and 1 events.)
- An analyst compares \(\alpha=0.2\) and \(\alpha=0.8\) using different random fold assignments. What should be changed to make the comparison use the same held-out observations? (Answer: Create one valid foldid vector and pass that identical vector to both cv.glmnet calls.)