Blocked Validation Versus Random Folds

Why the Validation Split Must Match the Forecasting Task

A validation score estimates performance on a particular prediction task. For a time series, that task is often to forecast future observations using information available at the forecast origin. A randomly shuffled split can change the task: a validation observation may have nearby observations on both sides in the training data, including observations that would not yet exist when a real forecast is made.

This distinction matters when observations are autocorrelated. If adjacent values tend to be similar, a training value just after a held-out time can reveal information about that held-out value. Random folds can then make predictions look more accurate than forecasts made strictly forward in time. This is a risk, not a guarantee: the size and direction of the difference depend on the series, model, and evaluation design.

The target must be clear before choosing a split. Forecasting next month’s value from data available this month is not the same task as estimating a missing value inside a completed historical series. The latter is an interpolation task, for which observations after the missing time may legitimately be available. Validation should reproduce the information set and time direction of the intended use.

How Autocorrelation Makes Random Folds Optimistic

Consider a stationary Gaussian AR(1) process, centered to have mean zero:

where the innovations are independent over time. Assume the parameters are known, so parameter-estimation uncertainty is not part of this comparison. If the task is to forecast \(Y_t\) using observations only through time \(t-1\), the one-step forecast is \(\rho Y_{t-1}\). Its forecast error is \(\varepsilon_t\), so its mean squared error is \(\sigma_\varepsilon^2\).

In contrast, suppose a held-out \(Y_t\) has both neighbors \(Y_{t-1}\) and \(Y_{t+1}\) available. Under this Gaussian model, the conditional distribution of \(Y_t\) given both neighbors has variance \(\sigma_\varepsilon^2/(1+\rho^2)\). The next derivation shows why.

Derivation: Conditioning on Both Neighbors

Given \(Y_{t-1}=a\), the density contribution involving \(Y_t=y\) from the transition into time \(t\) is proportional to \(\exp[-(y-\rho a)^2/(2\sigma_\varepsilon^2)]\). Given \(Y_{t+1}=b\), the transition out of time \(t\) contributes \(\exp[-(b-\rho y)^2/(2\sigma_\varepsilon^2)]\). Multiplying these terms gives the conditional density as a function of \(y\):

To identify its variance, collect the terms involving \(y\) in the numerator:

Completing the square in \(y\) gives a quadratic term \((1+\rho^2)\left(y-\frac{\rho(a+b)}{1+\rho^2}\right)^2\), plus terms that do not depend on \(y\). Comparing the resulting exponential with a normal density shows that

Because \(1+\rho^2>1\) whenever \(\rho\ne0\), the conditional variance using both neighbors is smaller than the one-step forecast error variance. Random folds can make this two-sided information available for some held-out times. A forecasting evaluation should not use that advantage when the future neighbor would be unknown at the forecast origin.

Building a Chronological Validation Design

A blocked design keeps the evaluation period later than the data used to fit the corresponding forecast. One simple design reserves the final contiguous portion of a series as validation data. Another uses several chronological forecast origins: fit through an origin, predict a later block, then move the origin forward. The earlier material on rolling-origin evaluation covers that repeated-origin setup; the key point here is that time order is preserved, unlike in randomly shuffled folds.

When scoring multiple forecast errors, pool the squared errors before taking the square root if the desired summary is overall RMSE. For errors \(e_1,\ldots,e_n\), RMSE is \(\sqrt{\sum_{j=1}^n e_j^2/n}\). The training and validation periods, forecast horizon, and scoring calculation should be reported clearly. If different folds have different sizes, averaging their RMSEs gives each fold equal weight, which is generally not the same as pooling all squared errors.

When Random Folds Are Appropriate—and When They Are Not

Random folds are not inherently invalid. They can be appropriate when observations are independent and exchangeable for the prediction task, or when the goal is to predict a randomly omitted point within a completed series and both past and future information are legitimately available. But a time index alone does not make a dataset safe for random folds: dependence between observations, the forecast horizon, and the information available at prediction time all matter.

For a future-forecasting task, random folds can understate error in several related ways. A held-out value may have a nearby future value in the training set; adjacent rows in a lagged dataset may share observations; and the validation set may mix easy interior periods with the genuinely difficult end-of-series forecast period. The resulting score estimates performance under that shuffled arrangement, not necessarily performance at a future forecast origin.

Common Exam Trap

A frequent mistake is to regard randomly shuffled cross-validation as a neutral substitute for chronological validation simply because every observation is used for both training and validation across folds. That argument ignores what information is available at the forecast origin. Randomly assigning times can put later, correlated observations into training for a held-out earlier time. A low score may then measure interpolation or a mixed-time prediction task rather than future forecasting.

The correction is to state the prediction target first, then enforce its information boundary in the split. For a forecast of time \(t\) using information available through \(t-1\), training observations after \(t-1\) must not help produce that validation forecast. Conversely, if the actual task is to fill a historical gap using data on both sides, a two-sided method may be suitable—but it should be described as interpolation rather than future forecasting.