Why the Order of Model Development Matters
A model’s reported performance depends not only on its form but also on how it was developed. If information from an evaluation set influences preprocessing, candidate selection, or tuning, that set is no longer an independent check of the resulting procedure. A defensible sequence therefore assigns each observation a role before any data-dependent fitting or choice begins.
The aim is not to keep training data from the development process: training data are meant to fit the model and its preprocessing. Validation data are also part of development, because they help choose among candidate procedures. The test set has a different role. It is held aside until the development choices are complete, then used once to estimate how the locked procedure performs on new observations from the target setting.
A split is useful only if the roles remain distinct. For example, the test set is not a spare source of data for resolving an uncertain validation decision. Once test results influence a decision, those results have become part of development, and the resulting performance number no longer has the same status as an evaluation of a locked procedure.
A Leakage-Safe Development Sequence
The following order makes the boundary between development and final evaluation explicit. The split proportions depend on the setting; the essential point here is the order of operations, not a universal percentage.
Specify the outcome, prediction target, and performance measure before comparing candidates. Use a measure that reflects the intended prediction goal.
Assign training, validation, and test observations before fitting preprocessing or candidate models. Keep the test observations out of development.
Fit each candidate’s preprocessing and model on training data. Apply the fitted procedure to validation data, compare results, and make development choices using training and validation information.
Specify the selected predictors, preprocessing operations, tuning choices, and fitting rule. Do not use test results to revise them.
Apply the locked procedure to the test observations, calculate the prespecified performance measure, and report it as a final evaluation under the assumptions of the design.
“Before fitting” includes more than estimating model coefficients. Any operation that learns a value or choice from observations—such as a fitted transformation or a selected set of predictors—belongs inside development. In a complete workflow, the operation is fitted using training data, then the same fitted operation is applied to validation and test observations. The precise details of particular preprocessing methods depend on the method; the sequence does not.
| Data role | Permitted use in the sequence | What it must not become |
|---|---|---|
| Training | Fit candidate models and training-dependent preprocessing. | A substitute for an independent evaluation. |
| Validation | Compare candidates and make development choices. | A final untouched estimate after choices have depended on it. |
| Test | Evaluate the procedure after it has been locked. | A source of information for another round of selection or tuning. |
Worked Examples
Worked Example: Assigning Roles Before Any Fit
A hypothetical insurer has 1,000 completed policy records and wants to predict each record’s claim cost. The development plan calls for 60% training data, 20% validation data, and 20% test data. Determine the number of records in each role and describe the correct order of work.
Step 1: Calculate the partition sizes. The training count is \(1{,}000(0.60)=600\). The validation count is \(1{,}000(0.20)=200\), and the test count is also \(1{,}000(0.20)=200\). Check: \(600+200+200=1{,}000\).
Step 2: Make the assignments first. Assign the 1,000 records to the three roles before estimating a model or calculating any data-dependent preprocessing quantities. The assignment method should suit the prediction setting; this example assumes the records can be assigned so that the three sets represent the same target population.
Step 3: Develop without the test set. Use the 600 training records to fit candidate procedures. Use the 200 validation records to compare those procedures and settle development choices. Keep the remaining 200 records unavailable to those choices.
Step 4: Evaluate the locked procedure. After recording the selected procedure, apply it to the 200 test records and calculate the prespecified claim-cost performance measure. The count check confirms that all 1,000 records have exactly one assigned role.
Method and mistake avoided: This is a role-first partition followed by development and final evaluation. A tempting but incorrect order is to fit a model on all 1,000 records and then divide the records for evaluation; that would allow evaluation observations to have influenced fitting.
Worked Example: Applying Training-Fitted Standardization
In a hypothetical risk-prediction project, the training observations for a numeric predictor \(X\), measured in exposure units, are \(2, 4, 6\). The validation value is \(8\), and a test value is \(0\). The model-development plan standardizes this predictor using the training mean and the sample standard deviation. Calculate the standardized values and state how they are used.
Step 1: Calculate the training mean.
Step 2: Calculate the training sample standard deviation. The deviations from 4 are \(-2,0,2\), whose squared values are \(4,0,4\). Thus the sample variance is \((4+0+4)/(3-1)=4\), and the sample standard deviation is \(\sqrt{4}=2\).
Step 3: Standardize using those fixed training quantities. For a value \(x\), use \(z=(x-4)/2\). The training values become \((2-4)/2=-1\), \((4-4)/2=0\), and \((6-4)/2=1\). The validation value becomes \((8-4)/2=2\), and the test value becomes \((0-4)/2=-2\).
Step 4: Preserve the transformation. Use the training mean of 4 and standard deviation of 2 when transforming validation and test observations. Do not recalculate either quantity separately for those sets. The calculations are consistent with the training values being centered at 0 after standardization, while the validation and test values are transformed using the same rule.
Method and mistake avoided: Fit the transformation on training data and carry its fitted quantities forward. Recomputing a mean or standard deviation on validation or test data would make that set contribute to the procedure rather than serve only in its assigned role.
Worked Example: Selecting With Validation and Evaluating Once
A hypothetical analyst compares two procedures for predicting claim amounts. On four validation records, the actual values are \(100,200,300,400\). Procedure A predicts \(120,180,330,370\), and procedure B predicts \(90,240,270,460\). The prespecified measure is mean absolute error (MAE). After choosing a procedure, the analyst evaluates it on four untouched test records: actual values \(150,250,350,450\), with procedure A predictions \(140,270,330,430\). Select the procedure and calculate the final test MAE.
Step 1: Calculate validation absolute errors for A. The four absolute errors are \(|100-120|=20\), \(|200-180|=20\), \(|300-330|=30\), and \(|400-370|=30\). Their sum is \(20+20+30+30=100\), so \(\mathrm{MAE}_A=100/4=25\).
Step 2: Calculate validation absolute errors for B. The errors are \(|100-90|=10\), \(|200-240|=40\), \(|300-270|=30\), and \(|400-460|=60\). Their sum is \(10+40+30+60=140\), so \(\mathrm{MAE}_B=140/4=35\). Since 25 is less than 35, select A using validation data.
Step 3: Lock A, then calculate its test errors. The test errors are \(|150-140|=10\), \(|250-270|=20\), \(|350-330|=20\), and \(|450-430|=20\). The sum is \(10+20+20+20=70\), so the test MAE is \(70/4=17.5\).
Step 4: Interpret the result. The selected procedure is A, with a test MAE of 17.5 claim-cost units on these four hypothetical test records. The calculation is checked by averaging four errors totaling 70. Because this is a small illustrative test set, the result is an example of the procedure, not a claim of precise performance for a real insurer.
Method and mistake avoided: Compare candidates on validation data, then calculate the test metric only for the selected procedure. Do not choose whichever candidate has the smaller test MAE: that would make test performance part of the selection rule.
Why the Test Metric Has This Role
The test metric is an estimate of the locked procedure’s performance on new observations, subject to how the test observations were obtained. The basic justification can be stated for an average loss. Let \(A\) denote the locked procedure, let \(L_j(A)\) be its loss on test observation \(j\), and let the test set contain \(n_{\mathrm{test}}\) observations. Its average test loss is
Suppose the test observations are independent of the development data and representative of the target population, and suppose the loss has a finite expectation. Once training and validation data have fixed \(A\), each test loss has the same expected value, \(R(A)\), the target-population expected loss of that locked procedure. Linearity of expectation then gives
The average test loss is therefore unbiased for \(R(A)\) under these conditions. This result does not say that a particular test value equals the target risk, nor does it guarantee that the test data match a future population. It explains why the separation matters: if test results help select \(A\), then the procedure is no longer fixed independently of the test losses, and the displayed argument does not apply in the same way.
Common Mistakes and Practical Checks
- Splitting after fitting. If the model or a data-dependent transformation has already used all observations, a later test partition cannot undo that information flow. Partition first.
- Using validation as the final report card. Validation results inform choices, so they describe performance during development. Reserve the test set for the final evaluation.
- Changing the procedure after seeing test results. A new model, predictor set, preprocessing choice, or tuning decision based on test outcomes makes the test set part of development. A fresh independent evaluation would be needed for a new final assessment.
- Reporting a metric without naming its role. State whether the number is a training, validation, or test result. The same numerical metric has a different interpretation depending on how the data were used.
A practical audit is to write down the target, the partition assignments, the prespecified metric, and the final procedure before examining test performance. Keep a record of which choices used training and validation information. If a test result triggers a change, do not present the resulting test score as though it evaluated a procedure that had been locked in advance.
Check Your Understanding
For each situation, follow the stated roles and keep the final evaluation separate from development.
- A hypothetical dataset has 800 observations. A plan assigns 65% to training, 20% to validation, and 15% to test. How many observations belong in each set? (Answer: 520 training, 160 validation, and 120 test observations.)
- A predictor’s training values are \(5,7,9,11\). Standardize using the training mean and sample standard deviation. What is the standardized value of a validation observation with value 13? Round to three decimals. (Answer: Training mean \(8\), sample standard deviation \(\sqrt{20/3}\approx2.582\), and standardized validation value \(5/\sqrt{20/3}\approx1.936\).)
- Two procedures have validation absolute-error totals of 30 and 27, each over three observations. Select by validation MAE. The selected procedure has four test absolute errors of \(6,9,12,13\). What are the selected procedure and test MAE? (Answer: Select the second procedure; its test MAE is \(40/4=10\).)
- An analyst compares three procedures on validation data, chooses one, sees its test result, then changes a predictor based on that result and reports the changed procedure’s score on the same test set. Is that score a final evaluation of a locked procedure? (Answer: No. The test result influenced development, so the test set has become part of the selection process.)
- A locked procedure has test losses \(2,4,6,8,10\) on five independent test observations from the target population. What is the average test loss, and what population quantity is it intended to estimate under the stated sampling conditions? (Answer: The average is \(30/5=6\); it estimates the locked procedure’s expected target-population loss.)