Why Leakage Is a Forecasting Problem
A forecasting procedure is evaluated by asking what it could have predicted using the information available at a specified origin. If any step uses observations from after that origin, the resulting score no longer describes a forecast that could have been made in practice. This can happen before a model is fitted: a transformation, a choice of model order, or a tuning decision can all use future observations indirectly.
For example, calculating a mean and standard deviation from the entire series before splitting it into training and validation periods allows later values to affect the scale of earlier observations. Similarly, choosing an ARIMA order after inspecting the complete series uses information from the validation period, even if the final model is fitted only to the training observations. The forecast errors may then look better than a genuinely prospective evaluation would justify.
The rule is about information availability, not simply where a software split is placed. A validation set can be withheld from the final model fit yet still leak into the analysis if it was used to estimate a transformation or decide which model to report. Conversely, using an earlier observation to form a lagged predictor at a later origin is valid when that observation would be available then.
Which Steps Must Be Repeated at Each Origin?
A forecast pipeline can contain several fitted steps. Examples include estimating a centering or scaling constant, estimating a Box-Cox transformation parameter, estimating seasonal effects, selecting differencing orders, identifying candidate ARIMA orders, and choosing a tuning parameter. If a step learns anything from the observed series, estimate or select it using only the data available at the relevant origin.
Some operations do not estimate parameters from the full series. A fixed log transformation, for example, can be applied to each available observation without looking ahead, provided its use is specified in advance and the data are positive. A seasonal difference \(y_t-y_{t-s}\), for a prespecified period \(s\), also uses only observations available at \(t\). But estimating \(s\), deciding whether to difference, or choosing a transformation by examining later observations is a data-dependent choice and must be made using training data only.
| Operation | Leakage risk | Leakage-safe practice |
|---|---|---|
| Centering or scaling | Estimating constants from the full series includes validation values | Estimate constants from the training prefix; apply them unchanged to its forecast target |
| Seasonal adjustment | Estimating seasonal effects using the full series lets future seasons affect the past | Estimate effects using only the available prefix and apply the fitted adjustment to forecasts |
| Model-order selection | ACF, PACF, or candidate comparisons computed on the full series use future observations | Identify and compare candidates within the training data at each evaluation origin |
| Tuning | Choosing the best parameter from the final test errors makes the test data part of model selection | Tune on historical inner rolling-origin folds; reserve the outer test period for evaluation |
A time-series pipeline must also respect the forecast horizon. At origin \(t\), a multi-step forecast for \(t+h\) cannot use actual observations \(y_{t+1},\ldots,y_{t+h-1}\) as predictors. In a rolling evaluation, those observations can become available at later origins, but only after the forecast made at the earlier origin has been recorded.
Worked Example: Scaling Without Using the Validation Target
Worked Example: Scaling Without Using the Validation Target
A hypothetical monthly claim-index series has training observations \(2,4,6\), followed by a validation observation of \(10\). The analyst wants to center the series by subtracting its mean. Calculate the correctly centered validation value, show the value obtained by the leaking procedure, and explain the difference.
Method. Estimate the centering constant from the training observations, then apply that same constant to the validation observation. The training mean is
Therefore, the centered validation value is \(10-4=6\). The training values themselves would be centered to \(-2,0,2\), using the same training mean.
Leaking calculation. If the validation value is included when estimating the mean, the mean becomes
That procedure gives a centered validation value of \(10-5.5=4.5\). The validation observation has affected the transformation applied to itself, so \(4.5\) is not the result of a transformation that could have been fitted at the training origin.
Check. The leakage-safe centered training values sum to \((-2)+0+2=0\), as required when centering by their mean. The value \(6\), not \(4.5\), is the validation value under the training-fitted transformation. The common mistake is to scale the full series first and split it afterward.
Rolling-Origin Tuning and the Outer Test
At an inner validation origin \(t\), fit the complete candidate pipeline using \(y_1,\ldots,y_t\), generate the forecast, and compare it with the subsequently observed target. Repeat at later origins. These historical forecast errors can guide a tuning decision at a later outer origin, because all of those targets are then in the past. The key is that each inner forecast must itself be produced without using its target or later observations.
For a final evaluation, define an outer training period and an outer test period. Use rolling-origin folds entirely within the outer training period to choose transformations, model orders, and tuning parameters. Then refit the chosen procedure using the full outer training period and forecast the outer test period. Do not change the procedure after inspecting outer test errors if those errors are meant to estimate performance on unseen data.
Worked Example: Comparing Forecast Rules at Historical Origins
Worked Example: Comparing Forecast Rules at Historical Origins
A hypothetical quarterly portfolio index has observed values \(y_1=10,\ y_2=12,\ y_3=11,\ y_4=15,\ y_5=14\). At origins \(t=2,3,4\), compare two one-step forecast rules using squared error. Rule A is \(\hat y_{t+1|t}=y_t-0.2(y_t-y_{t-1})\). Rule B forecasts the mean of all observations available at the origin. Use forecast minus actual as the signed error, select the rule with lower mean squared error, and then state the Rule A forecast at origin \(t=5\).
Method. At each origin, calculate each forecast from the available prefix only. The three validation targets are \(y_3=11,\ y_4=15,\ y_5=14\).
At \(t=2\), Rule A gives \(12-0.2(12-10)=12-0.4=11.6\). Its error is \(11.6-11=0.6\), so its squared error is \(0.6^2=0.36\). Rule B gives \((10+12)/2=11\), with error \(11-11=0\) and squared error \(0\).
At \(t=3\), Rule A gives \(11-0.2(11-12)=11-0.2(-1)=11.2\). Its error is \(11.2-15=-3.8\), and its squared error is \((-3.8)^2=14.44\). Rule B gives \((10+12+11)/3=33/3=11\). Its error is \(11-15=-4\), and its squared error is \((-4)^2=16\).
At \(t=4\), Rule A gives \(15-0.2(15-11)=15-0.8=14.2\). Its error is \(14.2-14=0.2\), and its squared error is \(0.2^2=0.04\). Rule B gives \((10+12+11+15)/4=48/4=12\). Its error is \(12-14=-2\), and its squared error is \((-2)^2=4\).
Rule A's mean squared error is \((0.36+14.44+0.04)/3=14.84/3=4.9467\), to four decimal places. Rule B's mean squared error is \((0+16+4)/3=20/3=6.6667\), to four decimal places. On these inner folds, Rule A has the lower mean squared error.
At outer origin \(t=5\), Rule A uses the two latest available values: \(14-0.2(14-15)=14-0.2(-1)=14.2\). This is the forecast for \(y_6\); no value of \(y_6\) is needed to produce it. Check: the first Rule A error is positive, since \(11.6-11=0.6\), and its squared error is \(0.36\). The common mistake is to reverse that subtraction or to use the target \(y_6\) before it is observed.
Why the Origin Rule Works
The rolling-origin procedure follows directly from the definition of a forecast. Let \(\mathcal F_t\) denote the information available through origin \(t\). A forecast made at \(t\), including its preprocessing and tuning choices, must be a function of \(\mathcal F_t\). If a transformation parameter is estimated from \(y_{t+1}\), then the forecast procedure depends on information outside \(\mathcal F_t\), and it is not a valid forecast at that origin.
For example, suppose a transformation uses a fitted centering constant \(c\), and a model forecast on the transformed scale is \(\hat z_{t+1|t}\). To make the forecast, \(c\) must be estimated from the training prefix, and the original-scale forecast must be obtained by reversing the same transformation using that fitted \(c\). If instead \(c\) is calculated using the validation target, then both the transformed training data and the reverse transformation may depend on the target. This violates the information condition even if the model fit itself excludes that target.
List the observations and external predictors genuinely available at that time.
Estimate transformation parameters and select candidate specifications using only the available observations.
Do not insert later actual values into the forecast calculation, including for intermediate steps of a multi-step forecast.
At the next origin, add the newly observed value and repeat any steps that the procedure specifies should be refitted.
Leakage Traps in Practice
Leakage can be subtle because a transformation may appear to be only a convenient data-preparation step. A centered moving average uses observations on both sides of a time point, so it cannot be calculated prospectively at that point. Seasonal effects estimated from the complete series can likewise use later cycles to adjust earlier values. If seasonal adjustment is part of the forecast procedure, estimate its seasonal factors from the available prefix and carry out the forecast with those factors.
Differencing requires a more precise distinction. With a fixed lag \(s\), the seasonal difference \(y_t-y_{t-s}\) at time \(t\) uses no observation after \(t\). It is not leakage to calculate that difference when both observations are available. Leakage can arise if a differencing order is chosen after examining the validation period, if a transformation is fitted to the complete differenced series, or if a multi-step forecast uses future actual values to create later differences.
- Using the full series for preprocessing: refit learned transformations within each training prefix.
- Selecting after seeing validation errors: use inner rolling folds for selection, and preserve a separate outer test for evaluation.
- Letting a later origin rewrite an earlier forecast: record each forecast as it would have been issued at its own origin.
- Assuming a chronological split alone is sufficient: audit every learned step, not only the final model fit.
These controls matter because leakage can make a forecasting procedure appear more accurate than it will be in use. It can also change which model is selected, not merely the reported score. For an actuarial application such as forecasting a hypothetical monthly benefit count, the relevant procedure is the one that could have been run using the data available when the forecast was required. That discipline is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Worked Example: Seasonal Differencing at a Forecast Origin
A hypothetical monthly exposure index has values \(y_1=8,\ y_2=10,\ y_3=9,\ y_4=13\), and the prespecified seasonal lag is \(s=2\). At origin \(t=4\), calculate the available seasonal differences and their mean. Then explain what happens if the next observation is \(y_5=12\).
Method. For lag 2, the difference at time \(j\) is \(d_j=y_j-y_{j-2}\), and it is defined when \(j\geq 3\). At origin 4, the available values are \(d_3=y_3-y_1=9-8=1\) and \(d_4=y_4-y_2=13-10=3\). Their mean is \((1+3)/2=2\).
After \(y_5=12\) is observed, the origin can advance to 5. The new difference is \(d_5=y_5-y_3=12-9=3\), so the updated mean is \((1+3+3)/3=7/3=2.3333\), to four decimal places. At origin 4, \(d_5\) was unavailable and must not be included in the mean used for preprocessing or model fitting.
Check. The first two differences use only observations through time 4. The third difference becomes available only after \(y_5\) is observed. The common mistake is not seasonal differencing itself; it is letting the later difference affect the earlier-origin analysis.
Check Your Understanding
For each problem, apply the information available at the stated origin and check the final answer.
- A training prefix is \(3,5,7\), and the next observation is \(11\). If you center using the training mean, what is the centered next observation? (Answer: The training mean is \(5\), so the centered value is \(11-5=6\).)
- A series has \(y_1=20,\ y_2=23,\ y_3=21,\ y_4=26\), and the seasonal lag \(s=2\) is fixed in advance. At origin 4, calculate the available seasonal differences and their mean. (Answer: The differences are \(y_3-y_1=1\) and \(y_4-y_2=3\), with mean \(2\).)
- At two rolling origins, forecasts are \(8.5\) and \(10.2\), while the corresponding actual values are \(9.0\) and \(9.2\). Using forecast minus actual, calculate the two squared errors and their mean. (Answer: The errors are \(-0.5\) and \(1.0\); the squared errors are \(0.25\) and \(1.00\), so the mean is \(0.625\).)
- A min-max transformation is estimated from training values \(2,4,8\), and the next value is \(10\). Calculate the transformed next value using the training minimum and maximum, then state the value produced if the full series is incorrectly used to set the range. (Answer: Training range \(=8-2=6\), so \((10-2)/6=4/3\); using the full-series maximum 10 gives \((10-2)/(10-2)=1\), which leaks the validation value into the range.)
- An analyst uses origins 3, 4, and 5 to tune an ARIMA candidate, then evaluates it on observations 6 through 8. At origin 5, may the analyst use the observed value at time 4 when refitting the chosen pipeline? May the analyst use the observed value at time 6 to revise the origin-5 forecast? (Answer: Yes, time 4 is available at origin 5; no, time 6 is not available when the origin-5 forecast is issued. Revising the procedure using test outcomes would also mean the outer test was used for selection.)