Blocked Validation for Dependent Observations

Why the Split Must Match the Prediction Task

When observations are ordered in time, a validation split is part of the prediction problem, not just a way to divide a data set. A model trained on observations both before and after a held-out time interval is being tested on an interval that is missing from the training record. A model trained only on observations before a validation interval is being tested on future data. These procedures answer different questions.

Dependence makes this distinction especially important. Nearby observations may carry information about one another, so a validation score can be strongly affected by whether training data include observations close to, or later than, the validation times. Dependence also affects the uncertainty of an average score: many nearby losses need not provide as much information as the same number of independent losses.

Definition: Let $Z_t=(X_t,Y_t)$ denote the observation at ordered index $t$, and let $\ell(Y_t,\hat f(X_t))$ be the loss of a prediction. A blocked validation split holds out one or more contiguous intervals of indices. A forward-chaining split uses observations up to an origin $o$ for training and evaluates predictions on later indices, such as $o+1,\ldots,o+h$. The split is prospective only when the training data contain no observations later than the validation observations.

Which Risk Does a Split Target?

A risk has to specify how the predictor is trained and what kind of observation it must predict. For a training set $D$, let $\mathcal A(D)$ be the fitted prediction rule. Given a time origin $o$ and forecast horizon $h$, a forward-chaining score can be written as

S_{o,h} = \frac{1}{h}\sum_{j=1}^{h} \ell\!\left(Y_{o+j},\mathcal A(Z_1,\ldots,Z_o)(X_{o+j})\right).

Its unconditional target is the expected loss for that training protocol and those future indices, $E[S_{o,h}]$. This is an average over the joint distribution of the training history and the future observations. The conditional forecast risk is different: it is the expectation of future loss conditional on the observed training history. The realized validation score is random and is not itself that conditional expectation.

A blocked split that trains on observations on both sides of a held-out interval targets prediction for a missing interval given the retained data, including any later observations used in fitting. It does not, in general, estimate the risk of forecasting from the past alone. If the intended application is to predict the next week from the preceding weeks, a split that trains on the week after the held-out week does not reproduce that information set.

Theorem: Suppose $(Z_t)_{t\in\mathbb Z}$ is strictly stationary, the same measurable prediction rule $g$ is used at every validation index, and $E|\ell(Y_t,g(X_t))|<\infty$. For any fixed, nonempty finite set $V$ of indices, $$ E\!\left[\frac{1}{|V|}\sum_{t\in V}\ell(Y_t,g(X_t))\right] = E[\ell(Y_1,g(X_1))]. $$ Strict stationarity of the pair process is the relevant condition; stationarity of $Y_t$ alone is not sufficient.

Proof. Strict stationarity of $(Z_t)$ implies that every $Z_t=(X_t,Y_t)$ has the same marginal distribution as $Z_1$. Since $g$ and $\ell$ are fixed measurable functions, $\ell(Y_t,g(X_t))$ has the same distribution as $\ell(Y_1,g(X_1))$. Integrability makes their expectations finite and equal. By linearity of expectation,

E\!\left[\frac{1}{|V|}\sum_{t\in V}\ell(Y_t,g(X_t))\right] = \frac{1}{|V|}\sum_{t\in V}E[\ell(Y_t,g(X_t))] = E[\ell(Y_1,g(X_1))].

No independence among the validation losses is needed for this expectation identity. But the theorem concerns a fixed rule $g$; it does not, by itself, establish that a score from a data-dependent fitted rule targets a particular forecast risk.

For a fitted rule, a corresponding comparison is available when the full process is strictly stationary and the fitting procedure is the same measurable function of a fixed-length consecutive training segment at every origin. Shifting the origin then shifts the training segment and validation observations together. Strict stationarity gives the same joint distribution for the shifted inputs, so the score has the same expectation at each such origin. The target remains tied to that training length, fitting procedure, and forecast horizon; changing any of them can change the risk.

Worked Scores and Their Interpretation

Worked Example: A Two-Step Forward-Chaining Score

Suppose a procedure is fitted using observations through time $o$ and predicts $Y_{o+1}$ and $Y_{o+2}$. The observed values are 5 and 4, while its predictions are 3 and $8/3$, respectively. Use squared-error loss.

At the first validation time, the residual is $5-3=2$, so the squared loss is $2^2=4$. At the second time, the residual is $4-8/3=4/3$, so the squared loss is $(4/3)^2=16/9$. Thus the average score is

\frac{4+16/9}{2} = \frac{36/9+16/9}{2} = \frac{52/9}{2} = \frac{26}{9} \approx 2.889.

This is the realized average loss for this origin and horizon. Under an appropriate stationarity and fixed-protocol setup, averaging comparable origins can estimate the corresponding expected forward risk. The two losses may nevertheless be dependent, so the calculation alone does not justify an uncertainty estimate that treats them as independent.

Worked Example: A Held-Out Interior Block

Suppose a time series has $Y_1,\ldots,Y_6=(1,2,4,8,16,32)$. Hold out indices 3 and 4, and train using indices 1, 2, 5, and 6. For illustration, use the rule that linearly interpolates between the retained endpoints at indices 2 and 5. This rule predicts $Y_3$ by

2+\frac{3-2}{5-2}(16-2) = 2+\frac{14}{3} = \frac{20}{3},

and predicts $Y_4$ by

2+\frac{4-2}{5-2}(16-2) = 2+\frac{28}{3} = \frac{34}{3}.

The squared errors are $(4-20/3)^2=(-8/3)^2=64/9$ and $(8-34/3)^2=(-10/3)^2=100/9$. The average held-out-block score is therefore

\frac{64/9+100/9}{2} = \frac{164}{18} = \frac{82}{9} \approx 9.111.

The predictor uses $Y_5$, which occurs after the held-out block. This score describes the specified interpolation task, not a forecast made using only observations available at indices 1 and 2. The distinction comes from the information available to the fitting rule, not from the fact that the validation indices form a contiguous block.

Worked Example: Dependence Changes the Forecast Risk

Let $Y_t$ be a stationary Gaussian AR(1) process,

Y_t=0.8Y_{t-1}+\varepsilon_t, \qquad \varepsilon_t\sim N(0,0.36),

where the innovations are independent over time. Its stationary variance is $0.36/(1-0.8^2)=1$. For squared-error loss, the one-step predictor $0.8Y_t$ has error $Y_{t+1}-0.8Y_t=\varepsilon_{t+1}$, so its expected loss is $E[\varepsilon_{t+1}^2]=0.36$. The predictor that always forecasts zero instead has expected loss $E[Y_{t+1}^2]=1$. Thus the dependence in the series makes past values informative for future prediction, and the two predictors target different risks.

A forward-chaining score evaluates the predictor using the past-value information actually available at each origin. Randomly withholding individual times can leave nearby observations in the training set and create a different prediction task. It does not follow that every random split must have a particular numerical bias; the point is that its target need not be this one-step-ahead forecast risk.

Designing Blocks, Origins, and Gaps

A useful design starts from the intended deployment question. If the task is to predict a fixed number $h$ of future observations after a history of size $m$, use origins with at least $m$ available training observations and validate on the next $h$ observations. If the operational training window has fixed length, use that same window at each origin rather than an expanding window. These choices determine the risk being estimated.

For forward-chaining validation, a gap can be inserted between the end of training and the start of validation. A gap is appropriate when deployment includes a delay, when variables are only available after a lag, or when a short-range dependence or information-leakage concern calls for separation. A gap changes the target: it assesses prediction after that separation, not immediate next-step prediction. It also does not guarantee independence between training and validation; dependence can persist across longer lags.

For blocked validation with observations on both sides of a held-out interval, the block length should represent the missing stretches to which the model will be applied. A gap around the held-out block can reduce reliance on observations very close to the boundary. But the resulting score still concerns prediction of an interior interval using the remaining data, unless the training rule is constrained to use only past data.

Result: If $L_t=\ell(Y_t,g(X_t))$ is a stationary loss process with finite second moments, then for a fixed set of $q$ validation indices, $$ \operatorname{Var}\!\left(\frac{1}{q}\sum_{t\in V}L_t\right) = \frac{1}{q^2}\sum_{s\in V}\sum_{t\in V}\operatorname{Cov}(L_s,L_t). $$ For consecutive indices $V=\{1,\ldots,q\}$, this becomes $$ \operatorname{Var}\!\left(\frac{1}{q}\sum_{t=1}^{q}L_t\right) = \frac{1}{q^2}\left\{q\gamma_0+ 2\sum_{k=1}^{q-1}(q-k)\gamma_k\right\}, $$ where $\gamma_k=\operatorname{Cov}(L_t,L_{t+k})$.

Derivation. By definition, the variance of a sum is the sum of all pairwise covariances. Scaling the sum by $1/q$ scales its variance by $1/q^2$, giving the first identity. For consecutive indices, stationarity makes the covariance of $L_s$ and $L_t$ equal to $\gamma_{|t-s|}$. There are $q$ terms with lag zero; for each lag $k$ from 1 to $q-1$, there are $q-k$ pairs in each direction. Pairing those terms gives the second identity. In general, the off-diagonal covariances do not vanish, so an independent-loss variance calculation is not justified merely because the score is an average.

Common Mistakes and Practical Checks

Calling every contiguous holdout a forecasting split. A blocked split may train on observations later than the validation block. Check the training indices, not just the shape of the held-out set.
Assuming dependence automatically biases the average score. Dependence affects the score's joint distribution and uncertainty. Under the theorem's conditions, the expectation identity still holds for a fixed predictor and stationary pairs.
Using stationarity of the response alone. If predictors $X_t$ change their marginal distribution over time, the loss can change even when $Y_t$ is stationary. State assumptions about the full observation process, or directly about the loss process.
Interpreting an unconditional score as conditional risk. A validation score is a realized quantity. Conditional risk averages over future outcomes given the observed history; it is not generally equal to that one realized score.
Ignoring the protocol's training size and horizon. An expanding history and a fixed rolling window, or a one-step and a multi-step forecast, need not have the same risk.

Blocked and forward-chaining validation are useful when their information sets and horizons reproduce the intended use of a learning procedure. State the split, training rule, loss, and target risk together. Then interpret the average score as evidence about that particular target, while accounting for dependence when assessing its uncertainty.

Check Your Understanding

For each question, identify the information set and risk being considered.

A model is trained on indices 1–20 and evaluated on indices 21–25. What kind of prediction task does this split represent?
A model is trained on indices 1–4 and 8–12, then evaluated on indices 5–7. Why is this not a prospective forecast from index 4?
What additional assumption about the data process supports equal expected scores for a fixed rule at different validation times?
In a forward-chaining design, what target changes if a gap is inserted between training and validation?
Why can an average validation score have the correct expected value while an uncertainty calculation based on independent losses is still wrong?