Why Leakage Matters in Forecast Selection
A time-series forecast is made using information available at a particular time. A model can appear accurate in a retrospective analysis yet fail in operation if its features, preprocessing, or selection decisions used information that would not have been available then. This is leakage: information crosses the intended boundary between the data available for fitting or selection and the future outcomes being forecast.
Leakage can be subtle. A predictor may be calculated with a centered moving average that includes a future observation. A transformation may be estimated from the entire series before the data are split. Or a validation period may be consulted repeatedly while the analyst adjusts the model, so it no longer serves as an independent assessment. In each case, the process being evaluated differs from the process that could actually have been used at the forecast date.
The relevant boundary is not simply the date on which a file was downloaded. It is the information that would have been accessible when the forecast was made. For example, a calendar date may be known in advance, while a revised economic indicator or a later-reported claim count may not be. A valid evaluation must reflect the actual availability of each input.
Future Information Hidden in Features
Features for a time-series model must be constructed with their forecast origin in mind. A lagged value such as \(y_{t-1}\) is available when forecasting \(y_t\). A centered average at time \(t\), however, often uses observations on both sides of \(t\), including \(y_{t+1}\). Such an average may be useful for describing a completed historical series, but it is not a valid predictor at time \(t\).
Worked Example: Detecting a Future Value in a Moving Average
A hypothetical service provider records weekly demand as \(y_1=8\), \(y_2=10\), and \(y_3=40\). At the end of week 2, an analyst wants a forecast for week 3. The analyst creates a three-observation centered average labelled for week 2, then uses it as a predictor. Determine whether the feature is available at the forecast origin and give a valid alternative using only observed demand.
Method. Check every observation used in the feature against the information available at the end of week 2. At that origin, \(y_1=8\) and \(y_2=10\) are known, but \(y_3=40\) is not.
The centered average is \[ \frac{y_1+y_2+y_3}{3} =\frac{8+10+40}{3} =\frac{58}{3} =19.3333\ldots. \] It uses \(y_3\), the outcome being forecast. Therefore, it leaks future information and cannot be used as a week-2 predictor.
A valid trailing average based on the two observed weeks is \[ \frac{y_1+y_2}{2} =\frac{8+10}{2} =9. \] This feature uses only information available at the origin. Check: both values in the trailing average are dated no later than week 2. The common mistake is to treat a feature's label, “week 2,” as proof that all its inputs were known at the end of week 2; the calculation, not the label, determines availability.
The same check applies to lags, rolling summaries, seasonal adjustments, and imputed values. If an earlier missing observation is interpolated using a later observation, that reconstructed value may not be available at the earlier forecast origin. A feature-generation rule should be applied as it would have been at each origin, rather than calculated once using the completed series.
Chronological Splits and Training-Only Transformations
For forecasting, training, validation, and test periods should respect time order. A random split can place later observations in the training set while earlier observations are used for validation. The resulting score answers a different question from forecasting forward from the past: the fitted model has had access to future-period data.
Preprocessing is part of the fitted procedure. Means and standard deviations for scaling, imputation values, feature-selection decisions, and estimated transformation parameters should be learned from the training data available at an origin. Apply those fitted quantities to the subsequent validation period. For rolling or expanding evaluation, refit preprocessing at each origin using only the data then available.
Worked Example: Fitting a Scaler Without Looking Ahead
A hypothetical forecasting model uses a predictor \(x\). Its training values are \(2,4,6\), and its validation-period value is \(8\). The analyst plans to standardize \(x\) by subtracting its mean and dividing by its sample standard deviation. Calculate the validation value using training-only estimates, then compare it with the value obtained by incorrectly fitting the scaler to all four observations.
Method. Estimate the transformation from the training period first. The training mean is \[ \bar{x}_{\mathrm{train}}=\frac{2+4+6}{3}=4. \] The training sample variance is \[ s_{\mathrm{train}}^2 =\frac{(2-4)^2+(4-4)^2+(6-4)^2}{3-1} =\frac{4+0+4}{2}=4, \] so \(s_{\mathrm{train}}=2\). The validation value standardized using training estimates is \[ z_{\mathrm{valid}}=\frac{8-4}{2}=2. \]
If the analyst instead fits the scaler to all four values, the full-sample mean is \[ \bar{x}_{\mathrm{all}}=\frac{2+4+6+8}{4}=5, \] and the full-sample sample variance and standard deviation are \[ s_{\mathrm{all}}^2 =\frac{(2-5)^2+(4-5)^2+(6-5)^2+(8-5)^2}{4-1} =\frac{9+1+1+9}{3} =\frac{20}{3}, \qquad s_{\mathrm{all}}=\sqrt{\frac{20}{3}}\approx2.5820. \] The validation value then becomes \[ z_{\mathrm{valid,all}} =\frac{8-5}{\sqrt{20/3}} \approx1.1619. \]
The two values differ because the validation observation affected the second transformation. For an evaluation of a procedure that would estimate its scaler from past data, the correct value is \(2\), not approximately \(1.1619\). Check: the training mean and standard deviation use only \(2,4,6\). The common mistake is to assume that a transformation is harmless because it does not use the validation outcome; using validation-period inputs can still change the evaluated procedure, especially for regularized models or other methods sensitive to scaling. If future covariates are genuinely known in advance and the deployed method is explicitly designed to use them, that is a different forecasting setup and must be specified consistently.
A chronological design makes the intended information boundary visible. For a single holdout, choose a cutoff: observations up to the cutoff are used for fitting and any model-selection decisions, and subsequent observations are reserved for validation or testing. For repeated forecast evaluation, expanding-window origins use all observations available up to each origin; rolling-window origins use a fixed-length recent history. In either case, each forecast must be produced before using its target observation in fitting or preprocessing.
Why the Information Boundary Is Necessary
The validity of a forecast depends on the information on which it is based. At origin \(t\), a permissible forecasting rule has the form \[ \widehat{y}_{t+h\mid t}=g(\mathcal{F}_t), \] where \(g\) includes feature construction, preprocessing, model fitting, and the forecast calculation. If any part of \(g\) uses \(y_{t+h}\), or another unavailable future value, then the computed forecast is not a function of \(\mathcal{F}_t\). It could not have been issued at time \(t\).
This gives a direct reason for chronological evaluation: the validation target must come after the information used to produce its forecast. A low error from a procedure that has already seen the target does not estimate the performance of a genuine forecast. It measures performance under an information advantage that will not exist in use.
Specify the forecast origin, horizon, target, and which future covariates, if any, are truly known at that origin.
Put observations used for fitting before those used for validation, and keep a later test period separate if a final assessment is required.
Estimate transformations, feature rules, and model parameters from the data available at each forecast origin.
Save each forecast, then compare it with the outcome using the chosen error measure.
Use validation for planned model selection, not as a repeatedly consulted substitute for a final independent assessment.
Repeated Use of a Validation Period
A validation period is intended to compare candidate procedures under data not used to fit them. But if an analyst repeatedly inspects its results, changes features or model settings in response, and scores the revised candidates on that same period, the period has influenced the final choice. Its score is no longer a fresh assessment of the selected procedure. The concern is not that every repeated look must produce a better score; it is that the reported score is being used both to guide choices and to assess the result of those choices.
A disciplined workflow specifies the comparison in advance as far as practical, records the validation results, and avoids presenting the best score from an adaptively revised sequence as an independent estimate of future performance. When the selection process has used validation extensively, an untouched later test period can provide a final assessment. That test period must not then be repeatedly consulted to make further choices.
Worked Example: When Validation Becomes Part of Model Selection
A hypothetical equipment operator compares weekly demand forecasts for three validation weeks. The observed demands are \(100,110,120\). Candidate A forecasts \(98,108,118\), while candidate B forecasts \(95,112,125\). The analyst selects the candidate with lower mean absolute error, then modifies it after inspecting the validation residuals. The modified forecasts are \(100,110,115\). Calculate each relevant validation MAE and assess whether the modified model's validation score is an independent final assessment.
Method. For forecasts \(\hat{y}_j\) and actual values \(y_j\), mean absolute error is the average of the absolute errors.
Candidate A has absolute errors \[ |100-98|=2,\qquad |110-108|=2,\qquad |120-118|=2, \] so its MAE is \((2+2+2)/3=2\).
Candidate B has absolute errors \[ |100-95|=5,\qquad |110-112|=2,\qquad |120-125|=5, \] so its MAE is \((5+2+5)/3=4\). The initial comparison therefore selects A.
The modified forecasts have absolute errors \[ |100-100|=0,\qquad |110-110|=0,\qquad |120-115|=5. \] Their MAE is \[ \frac{0+0+5}{3}=\frac{5}{3}\approx1.6667. \] The arithmetic is correct, but the score is not an independent final assessment: the analyst changed the model after inspecting those same three validation errors. The validation period has become part of the selection process. The common mistake is to report \(1.6667\) as if the modified model had been specified before seeing validation data. A later, untouched test period would be needed for a separate final assessment.
Common Leakage Traps
- Randomly splitting a sequence. Later observations can enter the training set while earlier dates are used for validation. Use date-ordered splits for forward forecasting.
- Calculating features once on the complete series. Centered windows, future-informed imputations, or full-series seasonal adjustments may carry later observations into historical predictors. Rebuild them using information available at each origin.
- Preprocessing before splitting. A full-sample mean, standard deviation, or imputation estimate can use validation or test-period data. Fit the preprocessing on the permitted training data and apply it forward.
- Updating choices after every validation result. Repeated changes based on one validation period make its score part of the selection process. Do not treat that score as an untouched assessment.
- Using revised data without checking availability. Historical values can be revised after their original release. Use the version available at the forecast origin when that distinction matters to the application.
Check Your Understanding
For each problem, identify the permitted information or calculate the requested quantity.
- At the end of day 2, demand is \(y_1=6\), \(y_2=12\), and \(y_3=30\), with \(y_3\) not yet observed. An analyst uses the centered mean of these three values to forecast day 3. What is that mean, and why is it invalid at the stated origin? (Answer: \(16\); it uses the unavailable day-3 outcome.)
- A predictor has training values \(2,4,6\) and validation value \(8\). Using a training mean and sample standard deviation, what is the standardized validation value? (Answer: \(2\).)
- For actual values \(50,60,70\), a candidate forecasts \(48,63,68\). What is its validation MAE? (Answer: \((2+3+2)/3=7/3\approx2.3333\).)
- An analyst randomly assigns dates 1, 3, and 5 to training and dates 2 and 4 to validation, then claims to assess forecasts made forward in time. State the problem and a valid basic split. (Answer: training includes dates later than validation dates; use an earlier training block followed by a later validation block, such as dates 1–3 for training and 4–5 for validation.)
- A model is chosen after comparing settings on weeks 21–24, then modified twice in response to its errors on those same weeks. What role can weeks 21–24 still play, and what is needed for an independent final score? (Answer: they can guide selection, but not provide an untouched final assessment; evaluate the finalized procedure on a later test period not used for further choices.)