Why a Validation Winner Can Look Too Good
A time-series analyst may compare several forecasting specifications on the same validation period and select the one with the smallest forecast error. That is a reasonable use of validation data, but the winning score is not an impartial estimate of future performance: it helped determine which model won. Even if the candidates have identical underlying forecast accuracy, random variation can make one score unusually low, and a search tends to find that unusually low score.
This effect is called data snooping when the same observations are repeatedly examined to choose among alternatives. The search may involve formal model specifications, such as different orders or transformations, or many informal decisions made after inspecting results. The more opportunities the analyst gives the data to influence the choice, the less credible it is to treat the best observed validation score as an unbiased forecast of future error.
The key issue is not that validation data were used for selection; that is their purpose. The issue is interpreting the best validation result as if no selection took place. A separate, untouched test period provides a final assessment after the selection decisions are fixed. For a time series, the periods must respect time order: the test observations come after the data used for training and validation.
Why Taking the Minimum Creates Optimism
Suppose candidate \(j\) has a true forecast risk \(R_j\), meaning its expected loss under the intended forecasting conditions. Its observed validation loss can be represented as \(R_j+\varepsilon_j\), where \(\varepsilon_j\) is the sampling fluctuation in that estimate. Selecting the smallest observed loss selects both for a potentially good model and for a potentially favorable fluctuation.
A short derivation makes the mechanism clear. Assume, only for this illustration, that two candidates have the same true risk \(R\), and their validation losses are \(L_1=R+\varepsilon_1\) and \(L_2=R+\varepsilon_2\). Suppose the errors are independent and each has a normal distribution with mean zero and variance \(\sigma^2\). The selected loss is the smaller of \(L_1\) and \(L_2\). Using \(\min(a,b)=(a+b-|a-b|)/2\), we obtain:
The difference \(\varepsilon_1-\varepsilon_2\) is normal with mean zero and variance \(2\sigma^2\). For a mean-zero normal variable with standard deviation \(\tau\), the expected absolute value is \(\tau\sqrt{2/\pi}\). Thus \(E[|\varepsilon_1-\varepsilon_2|]=2\sigma/\sqrt{\pi}\), giving:
So, under these illustrative assumptions, the average of the winning validation scores is below the common true risk. This is not a correction formula to apply to arbitrary model searches: real candidate scores may be correlated, have different risks, or fail the normal approximation. It establishes the selection mechanism. With more candidates there are more chances for an unusually favorable fluctuation, although the amount of optimism depends on how related the candidates and their errors are.
Worked Example: Interpreting the Lowest Validation Loss
A hypothetical forecaster compares two candidate procedures. Assume both have true mean absolute error \(R=12\) units, and each observed validation mean absolute error is normally distributed with standard deviation \(1.5\) units. For this calculation, assume their validation errors are independent. What is the expected minimum of the two validation scores?
Method. Apply the two-candidate result derived above. The expected minimum is \(R-\sigma/\sqrt{\pi}\), with \(R=12\) and \(\sigma=1.5\).
First, \(\sqrt{\pi}\approx 1.77245\). Then \(1.5/1.77245\approx 0.8463\). Therefore:
The candidates’ common true mean absolute error is 12 units, but their expected winning validation score is about 11.1537 units. The gap, about 0.8463 units, is selection optimism under the stated assumptions. It does not mean that every observed winning score will be this far below 12.
Check. The expected minimum is less than 12, as it must be: the minimum of two scores cannot exceed their average, and the two scores each have expectation 12. The calculation avoids the mistake of treating the lowest observed score as an unbiased estimate merely because the individual scores are unbiased before selection.
A Chronological Selection and Assessment Protocol
A practical design separates model development from final assessment. The validation period is available for comparing candidates; the test period is not. In time-series work, preserve chronology rather than randomly assigning observations to these sets. Random splitting can allow information from later times to influence evaluation at earlier times and may fail to represent the forecasting task.
Specify the forecast origin, horizon, target, and loss measure before comparing candidates. For example, decide whether the comparison uses one-step-ahead mean absolute error or a fixed multi-step forecast loss.
Keep its outcomes out of candidate construction, model choice, and repeated informal inspection. Record the intended test evaluation rule in advance.
Estimate candidate specifications using training information and compare their forecasts on the validation period. Choose a procedure using a stated criterion, while recognizing that its best validation score is selected and potentially optimistic.
Fix the specification, transformations, forecast horizon, loss measure, and any other choices that would otherwise be adjusted after seeing the test outcomes.
Apply the frozen procedure to the later test period and report its forecast errors using the prespecified measure. Do not use that result to pick a new winner and still call the same period an untouched test set.
In a common final-refit approach, the chosen specification is re-estimated using all observations available before the test period, including the earlier validation observations. That is appropriate when it matches the procedure intended for use: the test outcomes remain unavailable during fitting and selection. The test then assesses the resulting procedure on later observations. If the forecasting task instead requires a model that is never refitted, the assessment design should match that requirement.
A single test period is still a finite sample. Its score has sampling variation and may not represent every future period or every relevant regime. It is a final assessment under the conditions represented by that test period, not proof of a universal error rate. Earlier material on forecast validation and forecast error measures supplies useful designs and scoring rules; the additional discipline here is to prevent the final evaluation from becoming another selection opportunity.
Worked Example: Choosing One Specification From a Shared Validation Period
A hypothetical analyst compares four forecasting specifications for a monthly benefit index. All four are evaluated on the same validation months using mean absolute error in index points. Their validation scores are 5.2, 4.8, 5.0, and 4.9. A later, untouched test period has already been reserved. Which candidate is selected, and what can be concluded from its validation score?
Method. Select the candidate with the smallest prespecified validation loss. Compare the four values directly: \(5.2\), \(4.8\), \(5.0\), and \(4.9\). The minimum is \(4.8\), belonging to candidate 2.
The correct selection is therefore candidate 2. The analyst can report that its validation mean absolute error was 4.8 index points and that it was the best of these four candidates on that period. The analyst should not conclude that its true future mean absolute error is 4.8. The same validation outcomes determined the winner, and trying four candidates created opportunities for a favorable fluctuation.
The next step is to freeze candidate 2 and the assessment choices, fit it using all permitted pretest data if that is the intended procedure, and evaluate its forecasts on the reserved test period. The other candidates’ test scores should not be used to select a different winner if the purpose is to preserve an untouched final assessment.
Check. Candidate 2 is the unique minimum, since \(4.8<4.9<5.0<5.2\). This comparison uses only the validation data for selection and does not claim that the observed minimum equals future risk.
Worked Final Assessment on a Test Period
A test score should be computed from forecasts produced without using the corresponding test outcomes. Consider a hypothetical monthly index with the selected specification fixed before five test observations are revealed. Let \(e_t=y_t-\hat{y}_t\) be the forecast error, where \(y_t\) is the observed value and \(\hat{y}_t\) is the forecast. The mean absolute error is the mean of \(|e_t|\); the root mean squared error is the square root of the mean of \(e_t^2\).
Worked Example: Computing the Frozen Procedure’s Test Error
After selecting and freezing one forecasting procedure, an analyst produces forecasts for five later months. The observed values are 104, 98, 101, 107, and 102; the corresponding forecasts are 102, 100, 100, 105, and 104. Calculate the test mean absolute error and root mean squared error.
Method. Subtract each forecast from its observation, take absolute values for mean absolute error, and square the signed errors for mean squared error.
| Month | Observed \(y_t\) | Forecast \(\hat{y}_t\) | Error \(e_t\) | \(|e_t|\) | \(e_t^2\) |
|---|---|---|---|---|---|
| 1 | 104 | 102 | 2 | 2 | 4 |
| 2 | 98 | 100 | -2 | 2 | 4 |
| 3 | 101 | 100 | 1 | 1 | 1 |
| 4 | 107 | 105 | 2 | 2 | 4 |
| 5 | 102 | 104 | -2 | 2 | 4 |
The absolute errors sum to \(2+2+1+2+2=9\), so the mean absolute error is \(9/5=1.8\) index points. The squared errors sum to \(4+4+1+4+4=17\), so the mean squared error is \(17/5=3.4\) squared index points. Taking the square root gives a root mean squared error of \(\sqrt{3.4}\approx1.8439\) index points, rounded to four decimal places.
Check. There are five errors, and the absolute and squared totals include each one. The root mean squared error is greater than the mean absolute error here, \(1.8439>1.8\), consistent with the greater penalty that squaring gives to larger errors. These scores assess the frozen procedure on these five test observations; they do not license another round of selection on the same observations.
When the Test Period Is No Longer a Test
Repeated use can turn a test set into validation data even without a formal table of candidate models. For example, an analyst might inspect the test score, change a transformation, alter a forecast horizon, and inspect the score again. Each change allows the test outcomes to influence the final procedure. The resulting lowest test score has the same selection problem as a repeatedly used validation score.
- Keep a record of the search. Count meaningful specification changes and repeated comparisons, including choices made after looking at validation results.
- Separate selection from assessment. Use validation data to choose; reserve test outcomes for the frozen final evaluation.
- Do not hide unsuccessful searches. Reporting only the winning score omits how many alternatives were tried and can exaggerate how persuasive the winner appears.
- If test results change the procedure, acknowledge the change. The test period has now contributed to selection. A fresh later test period or a suitable resampling design is needed for a new final assessment.
- Match the assessment to deployment. Keep the forecast origin, horizon, and loss definition relevant to the intended use, and preserve the temporal ordering of observations.
A larger search is not automatically invalid: flexible searches can be useful when the selection process is evaluated honestly. The important distinction is between the score used to choose a procedure and the independent evidence used to assess that choice. These principles are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, distinguish selection from final assessment and show the calculation needed for the requested answer.
- A forecaster evaluates three specifications on one validation period and obtains mean absolute errors of 6.1, 5.8, and 5.9 units. Which specification wins, and what is the limitation of its reported score? (Answer: The second specification wins with validation MAE 5.8; because it was selected as the minimum among three scores on the same period, 5.8 may be optimistic and is not an untouched estimate of future error.)
- Two equally accurate candidates have validation scores with independent normal errors, each with mean zero and standard deviation 3. Their common true risk is 20 units. Using the two-candidate result, find the expected minimum validation score. (Answer: \(20-3/\sqrt{\pi}\approx18.3074\) units.)
- A frozen model produces test-period observations 20, 22, 19, and 23, with forecasts 21, 20, 20, and 25. Calculate the test mean absolute error. (Answer: Absolute errors are 1, 2, 1, and 2; MAE \(=6/4=1.5\) units.)
- A monthly series has training observations through month 72, validation observations in months 73–84, and a reserved test period in months 85–96. The analyst chooses a model after comparing validation scores, refits the chosen specification through month 84, and then forecasts months 85–96 without inspecting their outcomes beforehand. Is the test chronology preserved? (Answer: Yes. The training and validation information precedes the test period, and refitting through month 84 does not use test outcomes.)
- An analyst evaluates a frozen procedure on a test period, dislikes its score, changes the specification, and reports the new score on the same period as the final test result. What is the problem, and what is needed for a fresh final assessment? (Answer: The test outcomes influenced specification selection, so the period now functions as validation data; a new untouched later test period or a suitable resampling design is needed for a fresh assessment.)