From a Prediction Case to a Validation Design
A validation plan is not simply a choice of a percentage for a random split. It is a sequence of decisions about what future predictions matter, what observations are genuinely independent, and what information would be available when the model is deployed. A plan that ignores repeated customers or the passage of time can produce a precise score for the wrong prediction task.
Start with the deployment question. Is the model predicting outcomes for new applicants, additional records from customers already observed, or future periods for the same business? Those settings imply different ways to separate training observations from evaluation observations. The split should mimic the information boundary at deployment, while the evaluation unit should match the entity on which the prediction will be used.
The earlier material on cross-validation and dependence supplies the mechanics of folds. Here the emphasis is choosing and connecting those mechanics for a complete case. In particular, distinguish the development data, which may be used to compare or tune candidate models, from a final test set, which is reserved for one final assessment after the modeling choices are settled.
A Practical Decision Sequence
State who or what receives a prediction, the outcome being predicted, the prediction time, and the information available at that time.
Check for repeated records, shared groups, or time ordering. The row is not automatically the correct unit to split or score.
Use random or stratified partitions for exchangeable independent observations, group-aware partitions for unseen groups, and time-ordered partitions for future prediction.
Use resampling on development data to select the modeling procedure. Keep a final test set untouched until the procedure is locked, if sufficient data are available.
Name the performance measure, its evaluation unit, and how fold results are combined. Note how the validation estimate may differ from production performance.
A random split is defensible only when observations can reasonably be treated as exchangeable for the target task. Stratification can help preserve the representation of an important outcome category in each partition, but it does not correct dependence, time leakage, or an unsuitable target. Preprocessing and feature creation that learn from data must be fitted within each resampling training partition, then applied to its validation partition.
Why Fold Scores Are Combined This Way
Suppose \(K\)-fold cross-validation divides \(n\) independent evaluation units into validation folds. Let fold \(k\) contain \(n_k\) units, with \(L_k\) the mean loss on that fold. Each unit is predicted while excluded from the corresponding model fit. If the desired summary is the mean loss per unit across all out-of-fold predictions, then the pooled estimate is
To see why, write each fold mean as \(L_k=(1/n_k)\sum_{j\in V_k}\ell_j\), where \(V_k\) is its validation set and \(\ell_j\) is the loss for unit \(j\). Multiplying by \(n_k\) gives the sum of losses in that fold. The validation folds are disjoint and together contain all \(n\) evaluation units, so adding these sums and dividing by \(n\) gives the mean loss over all out-of-fold predictions. If every fold has the same size, this reduces to the ordinary average of fold means. If sizes differ, an unweighted average generally does not equal the pooled mean.
This quantity estimates the average loss of the modeling procedure when trained on approximately \((K-1)n/K\) units, subject to the resampling design and the assumption that the folds represent the intended prediction task. It is not automatically the exact future loss of a model refitted on all development data. For group-based or time-based designs, define the relevant unit and weighting explicitly: the arithmetic must match the question being estimated.
Worked Example: Stratified Folds With a Protected Test Set
A hypothetical lender has 600 independent applicant records, of which 60 result in a specified adverse outcome. The deployment task is to predict that outcome for new applicants drawn from the same applicant population. The team wants a final test set and five-fold cross-validation for model selection. Design the partitions, check their sizes, and combine the supplied fold losses.
1. Make a stratified development/test split. Use 80% of the records for development and 20% for the final test. The development set has \(600(0.80)=480\) applicants, and the test set has \(600(0.20)=120\). Preserve the 10% outcome rate: assign 48 adverse outcomes and 432 non-adverse outcomes to development, and 12 adverse outcomes and 108 non-adverse outcomes to test. The totals check: \(48+12=60\) adverse outcomes and \(432+108=540\) non-adverse outcomes.
2. Divide development data into five stratified folds. Each validation fold should contain \(480/5=96\) applicants. Distribute the 48 adverse outcomes as \(10,10,10,9,9\), and distribute the 432 non-adverse outcomes as \(86,86,86,87,87\), in that same fold order. The fold sizes are therefore \(10+86=96\), \(10+86=96\), \(10+86=96\), \(9+87=96\), and \(9+87=96\). The outcome totals also check: \(10+10+10+9+9=48\), and \(86+86+86+87+87=432\). Each cross-validation model is fitted on the other \(480-96=384\) applicants.
3. Use cross-validation for development, not for a final claim. Suppose a candidate model has fold mean log losses \(0.42, 0.46, 0.44, 0.50,\) and \(0.48\). Since the folds each contain 96 applicants, the pooled mean is
The loss is lower when predicted probabilities are better calibrated to observed binary outcomes, so the team may use this score when comparing candidate procedures. After choices are fixed, refit the selected procedure using all 480 development records and evaluate it once on the 120 test records. The test outcomes must not guide feature, threshold, or model choices.
Method and trap avoided. Stratification maintains similar outcome proportions while random partitioning is appropriate for this independent, same-population target. A frequent allocation error is to list positive and negative counts that each sum to their totals but do not produce the claimed fold sizes. Checking each position gives five folds of exactly 96 here; for example, pairing 10 positives with 87 negatives would incorrectly produce a 97-record fold.
When Records Belong to Groups
If the deployment task concerns new people, businesses, households, or other groups, all records from a group should remain on one side of each split. Otherwise a model can exploit group-specific information seen in training, while evaluation appears to concern unseen groups. This is leakage relative to the stated target even if no outcome column is directly copied.
The scoring unit also matters. If each person contributes many rows, treating every row as a separate independent evaluation unit gives more influence to people with more records and can understate uncertainty. A plan should state whether the operational objective weights records equally or groups equally. Group splitting controls overlap; the scoring rule controls whose performance receives weight.
Worked Example: Evaluating Predictions for New Customers
A hypothetical insurer has three annual records for each of 800 customers, for 2,400 records total. It will deploy a model for customers not represented in model development. Each record contains an outcome and predictors available at the start of that year. Design a validation plan and summarize five-fold development performance, weighting customers equally.
1. Split by customer, not record. Assign 80% of the 800 customers to development and 20% to final testing: \(800(0.80)=640\) development customers and \(800(0.20)=160\) test customers. Since each customer has three records, these partitions contain \(640(3)=1{,}920\) and \(160(3)=480\) records. No customer appears in both partitions.
2. Cross-validate by customer within development. Five folds contain \(640/5=128\) customers each, or \(128(3)=384\) records each. Each fold model is fitted using 512 customers, or \(512(3)=1{,}536\) records. Keep the three annual records for a customer together in every split. Any preprocessing estimated from records must also be estimated using only the training customers in that fold.
3. Give each customer equal influence in the score. For each customer, first calculate the mean loss across that customer's three records. Then average those customer-level means within a fold. Suppose the five fold scores are \(0.31, 0.29, 0.34, 0.30,\) and \(0.36\). Since every fold has 128 customers, the equal-customer pooled cross-validation loss is
4. Finish with a one-time test. Use cross-validation results to settle the modeling procedure, refit it on all 640 development customers, and then assess it on the 160 customers held out from every development step. The test result speaks to new-customer performance, not to adding a new year of experience for an already observed customer.
Method and trap avoided. Grouped partitioning matches the target of predicting for unseen customers. A random row split could place one customer's first year in training and another year in validation; the resulting score would not represent the stated deployment setting. Equal-customer scoring is appropriate here because the case asks for average performance per customer, not average performance per record.
When Deployment Moves Forward in Time
For a future forecasting task, validation must preserve chronology. At each validation origin, training data must precede the validation period, and predictors must be restricted to information available at that origin. A random split can train on later periods and validate on earlier ones, which reverses the information flow of deployment.
An expanding-window design retains all earlier observations as training data while moving the validation window forward. A rolling-window design uses only a fixed recent span for training. The choice depends on the production process: expanding windows can reflect a system that accumulates data, while rolling windows may better reflect a process expected to adapt to changing conditions. Neither design guarantees future stability; changes in population or operating practice remain limitations to report.
Worked Example: Six-Month-Ahead Forecast Validation
A hypothetical service provider has 60 consecutive monthly observations of claim volume, measured in thousands of dollars. It will issue a forecast for the next six months using only data available at the forecast date. The last six months must serve as a final test. Construct a temporal plan and calculate a pooled cross-validation RMSE from four pilot windows.
1. Reserve the final period. Keep months 55–60 untouched as the final test set. The development history is months 1–54. No statistic, transformation, or model choice may be informed by months 55–60 before the final procedure is fixed.
2. Use four expanding-window validation periods. Fit on months 1–30 and validate on 31–36; fit on 1–36 and validate on 37–42; fit on 1–42 and validate on 43–48; then fit on 1–48 and validate on 49–54. Each validation window is six months, and each training window contains only months earlier than its validation window. The design therefore evaluates four successive six-month forecast periods without using future observations to predict past ones.
3. Combine the pilot errors correctly. Suppose the four window mean squared errors are \(4,9,16,\) and \(25\), in squared thousands of dollars. Each window has six months, so the pooled mean squared error is
Taking the square root gives \(\operatorname{RMSE}_{\mathrm{CV}}=\sqrt{13.5}\approx3.6742\) thousand dollars. In dollars this is approximately \(\$3{,}674.23\), rounded to the nearest cent. The square root is applied after pooling squared errors; averaging window RMSEs would answer a different question.
4. Refit and test in order. After selecting the modeling procedure using only months 1–54 and the four rolling-origin windows, fit it on months 1–54 and evaluate forecasts for months 55–60 once. This plan estimates performance for six-month-ahead forecasting, but its cross-validation windows use training histories from 30 to 48 months, whereas the final model uses 54 months. That difference, along with possible temporal change, is a limitation to disclose.
Common Planning Errors
The strongest plan is one whose partitions can be defended from the prediction question, not one that merely produces a large number of folds or a reassuring score. Check each choice against the information available at deployment and the unit that receives a prediction.
- Choosing a split before defining the target. Decide first whether the goal is new records, new groups, or future periods. A single split method cannot represent all three.
- Confusing stratification with independence. Preserving class proportions does not prevent related records from crossing partitions, nor does it fix time leakage.
- Scoring the wrong unit. Row-level weighting and group-level weighting can produce different summaries. State which one matches the operational question.
- Using unequal fold scores as if the folds had equal size. For mean per-observation loss, weight fold means by their evaluation counts. If scoring groups, use group counts or directly aggregate group-level losses.
- Letting the final test influence modeling. Repeatedly changing a model after inspecting test performance turns the test into part of model selection. Keep it reserved for the agreed final evaluation.
- Claiming more than the design supports. Cross-validation estimates performance for its resampling procedure and training sizes. It cannot guarantee future performance under population shift or a changed deployment process.
Check Your Understanding
For each case, identify the main partitioning choice and check the arithmetic or evaluation unit before accepting the plan.
- A dataset has 500 independent applications, with 50 positive outcomes. A stratified 20% final test is planned. How many records and positive outcomes belong in the test if the overall event proportion is preserved? (Answer: 100 records, including 10 positive outcomes.)
- A five-fold design has validation fold sizes 40, 40, 40, 40, and 40. The fold mean losses are \(0.20, 0.30, 0.25, 0.35,\) and \(0.40\). What is the pooled mean loss? (Answer: 0.30.)
- There are 300 customers with four records each, and the task is to predict for unseen customers. In a 25% group-level test split, how many customers and records are in the test set? (Answer: 75 customers and 300 records.)
- A monthly forecasting dataset covers months 1–48, with months 43–48 reserved as a final test. Give a valid three-window expanding design using six-month validation horizons and no overlap with the final test. (Answer: train 1–24/validate 25–30; train 1–30/validate 31–36; train 1–36/validate 37–42; final test 43–48.)
- Four equal-sized customer-level validation folds have mean losses \(0.18, 0.22, 0.26,\) and \(0.30\). What is their pooled customer-level mean loss, and what is the relevant evaluation unit if deployment is to unseen customers? (Answer: 0.24; the customer is the evaluation unit.)