Why Time Order Must Be Preserved
In ordinary cross-validation, observations are divided into folds so that each observation can be assessed after a model is fitted on other observations. For time-ordered data, that is not enough: the model must be fitted using information that would have been available before the assessment period. A random allocation can put a later observation in the training data for a model assessed on an earlier date, producing an evaluation that does not represent forecasting into the future.
Time-series cross-validation repeats a chronological training-and-assessment design at several forecast origins. At each origin, fit the specified modeling procedure on the permitted past data, then assess its predictions on a later window. The assessment window is a set of later observations reserved for that fold’s evaluation; it is not used to fit that fold’s model.
The order should reflect the time at which information is available, not merely the order in which rows happen to appear in a file. In a forecasting application, also check when each predictor and outcome becomes observable. For example, a predictor recorded after the prediction date cannot be treated as available at that date simply because it is present in the data set.
Specify the Fold Design Before Counting
Number the observations in chronological order from \(1\) through \(n\). Let \(h\) be the number of observations in each assessment window, and let \(s\) be the number of observations by which the origin advances from one fold to the next. In a common design, \(s=h\), so assessment windows are consecutive and non-overlapping. If \(s<h\), assessment windows overlap; if \(s>h\), there are gaps between them.
For an expanding-window design, let \(m\) be the initial training length. Fold \(k\) uses observations \(1\) through \(m+(k-1)s\) for training, followed by the next \(h\) observations for assessment. The fold is complete only if its last assessment observation is no later than \(n\). Thus the largest possible fold number is determined by
Equivalently, the number of complete folds is the largest integer \(K\) satisfying this inequality. If \(m+h \leq n\), it can be calculated as
For a fixed-width design, let \(w\) be the number of training observations. The first training window is \(1\) through \(w\), and the first assessment window is \(w+1\) through \(w+h\). Advancing by \(s\) moves both windows forward: fold \(k\) trains on observations \((k-1)s+1\) through \(w+(k-1)s\), then assesses observations \(w+(k-1)s+1\) through \(w+(k-1)s+h\). A complete fold must satisfy
Consequently, when \(w+h \leq n\), the number of complete fixed-width folds is \(\left\lfloor (n-w-h)/s \right\rfloor+1\). The expression is the same as for expanding windows after replacing the initial training length \(m\) with the fixed training width \(w\). The training observations themselves differ: in an expanding design the first observation remains in every training set, whereas in a fixed-width design it eventually leaves.
Why the Chronological Constraint Is Necessary
A fold’s assessment error is intended to describe how the procedure performs when predicting observations later than the data used to fit it. If an assessment observation at time \(t\) is evaluated using a model trained on an observation from a later time \(t'>t\), then the evaluation has used future information relative to \(t\). That is not the same prediction task.
The fold construction enforces the required ordering directly. In an expanding design, the last training index in fold \(k\) is \(m+(k-1)s\). The first assessment index is \(m+(k-1)s+1\), which is strictly later. In a fixed-width design, the last training index is \(w+(k-1)s\), and the first assessment index is \(w+(k-1)s+1\), also strictly later. The formula for the last assessment index then checks whether the whole window fits within the observed series.
Worked Example: Counting Every Complete Fixed-Width Fold
A hypothetical monthly claims-monitoring series contains 18 consecutive months. Use a fixed-width training window of 10 months, assess the following 2 months, and advance the training window by 2 months per fold. List every complete fold and identify any unused observations.
Method. Set \(n=18\), \(w=10\), \(h=2\), and \(s=2\). The fold-\(k\) assessment ends at \(w+(k-1)s+h\). A fold is complete if this is at most 18. The maximum fold number is
Now list the four folds. Fold 1 trains on months 1–10 and assesses months 11–12. Fold 2 shifts both windows by 2 months: train on months 3–12 and assess months 13–14. Fold 3 trains on months 5–14 and assesses months 15–16. Fold 4 trains on months 7–16 and assesses months 17–18.
Check. Each training window has 10 months, each assessment window has 2 months, and the assessment period follows the training period. The fourth fold ends at month 18, so it is complete. There are no observations left unused after the fourth assessment window. A fifth fold would assess months 19–20 and is not available.
Common mistake avoided: Stopping after the third fold because the fourth training window starts at month 7 rather than at month 9. Advancing by 2 months moves the training window to months 7–16; its next assessment window, months 17–18, is still inside the 18-month series.
Expanding and Fixed-Width Histories
The choice between these designs changes what past information is available at each forecast origin. An expanding window uses all observations from the start of the series through the current origin. A fixed-width window uses only the most recent \(w\) training observations. Neither is automatically preferable: the choice should reflect the prediction setting and the process that generated the data.
| Design | Training history as time advances | Useful when | Consideration |
|---|---|---|---|
| Expanding window | Grows by \(s\) observations per fold | Older observations remain relevant and accumulating data is part of the intended procedure | Older observations may become less representative if the process changes |
| Fixed-width window | Stays at \(w\) observations while moving forward | The intended procedure should emphasize a recent history of a set length | Each model uses fewer observations than an expanding window might use at the same origin |
These are choices about the training history, not changes to the chronological rule. In either design, training data must precede the assessment window. The training and assessment lengths should also be chosen to match the intended application: a one-month assessment window represents a different forecasting horizon from a six-month window.
Worked Example: Constructing Expanding and Fixed-Width Folds
A hypothetical quarterly portfolio data set contains 20 quarters. Compare an expanding design with initial training length \(m=8\) to a fixed-width design with \(w=8\). For both designs, use a 2-quarter assessment window and advance by \(s=2\) quarters. Give the folds and check that the two designs use the same number of folds.
Expanding design. Here \(n=20\), \(m=8\), \(h=2\), and \(s=2\), so
The six folds are: train 1–8, assess 9–10; train 1–10, assess 11–12; train 1–12, assess 13–14; train 1–14, assess 15–16; train 1–16, assess 17–18; and train 1–18, assess 19–20. The training history grows by 2 quarters each time.
Fixed-width design. With \(w=8\), the count is also \(\left\lfloor (20-8-2)/2 \right\rfloor+1=6\). The folds are: train 1–8, assess 9–10; train 3–10, assess 11–12; train 5–12, assess 13–14; train 7–14, assess 15–16; train 9–16, assess 17–18; and train 11–18, assess 19–20.
Check. In both designs, there are six assessment windows that cover quarters 9–20 with no gaps or overlap. At the final origin, the expanding model uses 18 training quarters, while the fixed-width model uses 8. The designs share assessment dates but do not provide the model with the same training data.
Assessment Windows and Summary Scores
After fitting the procedure separately in each fold, calculate the chosen performance measure from predictions for that fold’s assessment observations. If all assessment windows are disjoint and have equal length, the mean of the fold-level mean squared errors equals the pooled mean squared error across those assessment observations. For unequal window lengths, a simple mean of fold-level mean squared errors gives each fold equal weight, whereas pooling gives each observation equal weight.
Even when assessment windows overlap, fold-level scores can still describe performance at different forecast origins. But an observation appearing in multiple assessment windows contributes more than once if all predictions are pooled. The summary then describes the collection of forecast-origin predictions, not a set of unique observations with equal weight. State the design and the aggregation rule so the reported score can be interpreted correctly.
Worked Example: Averaging Fold Errors Versus Pooling Residuals
A hypothetical forecasting procedure is assessed in two disjoint windows, each containing two observations. Its residuals, defined as actual value minus predicted value, are \(1,-1\) in the first window and \(2,0\) in the second. Calculate each fold’s root mean squared error (RMSE), the unweighted mean of the two fold RMSEs, and the pooled RMSE across all four residuals.
Method. For residuals \(e_1,\ldots,e_q\), RMSE is the square root of the mean squared residual: \(\sqrt{\sum_{j=1}^{q}e_j^2/q}\).
For fold 1, the squared residuals are \(1^2=1\) and \((-1)^2=1\). Its mean squared error is \((1+1)/2=1\), so its RMSE is \(\sqrt{1}=1\). For fold 2, the squared residuals are \(2^2=4\) and \(0^2=0\). Its mean squared error is \((4+0)/2=2\), so its RMSE is \(\sqrt{2}\), approximately \(1.4142\).
The unweighted mean of the two fold RMSEs is
For the pooled calculation, add all four squared residuals: \(1+1+4+0=6\). Divide by four to obtain pooled mean squared error \(6/4=1.5\). Thus the pooled RMSE is \(\sqrt{1.5}\approx 1.2247\).
Check. The two summaries differ because taking a square root within each fold and then averaging is not the same operation as averaging squared residuals across observations and taking one square root. The unweighted fold mean is about 1.2071; the pooled RMSE is about 1.2247. The common error is to label these two different calculations as the same overall RMSE.
Practical Design Checks
Before using chronological folds, verify that the design matches the task being evaluated. A useful review is to write down the forecast origin, the training period, and the assessment period for every fold. Then check the following:
- Order: every training observation is available before its fold’s assessment period.
- Horizon: the assessment-window length matches the intended prediction horizon.
- Completeness: each listed assessment window ends at or before the last observed time.
- Overlap: identify whether consecutive assessment windows share observations or leave gaps.
- Information availability: predictors and outcomes used for training are available at the stated forecast origin.
- Training history: make clear whether older observations remain in the training set or are dropped as the window advances.
A common trap is to count folds using the number of forecast origins without checking the last assessment index. For example, a proposed origin may fit inside the series while its full assessment window extends beyond the observed data. Such a fold is incomplete and cannot be scored as specified. Check the end of the assessment window, not just the start of the training window.
Check Your Understanding
For each design, identify complete folds and check the final assessment index.
- A series has 27 monthly observations. An expanding design starts with 15 training months, assesses 3 months, and advances by 3 months. How many complete folds are there, and what are their training and assessment ranges? (Answer: 4 folds: train 1–15, assess 16–18; train 1–18, assess 19–21; train 1–21, assess 22–24; train 1–24, assess 25–27.)
- A series has 23 observations. Use a fixed training width of 9, an assessment length of 2, and a step of 3. List all complete folds. (Answer: 5 folds: train 1–9, assess 10–11; train 4–12, assess 13–14; train 7–15, assess 16–17; train 10–18, assess 19–20; train 13–21, assess 22–23.)
- An expanding design has \(n=16\), \(m=8\), \(h=3\), and \(s=2\). How many complete folds are possible, and which observation is unused? (Answer: 3 folds: train 1–8, assess 9–11; train 1–10, assess 11–13; train 1–12, assess 13–15; observation 16 is unused.)
- A fixed-width design has \(n=21\), \(w=7\), \(h=3\), and \(s=4\). How many complete folds are possible, and what are the last two? (Answer: 4 folds; the last two are train 9–15, assess 16–18 and train 13–19, assess 20–22. Correction: the second listed window ends at 22 and is not complete, so only 3 folds are complete; the last two complete folds are train 5–11, assess 12–14 and train 9–15, assess 16–18.)
- Two disjoint assessment folds each contain 2 observations. Their squared residuals are \(1,9\) and \(4,4\). Find the unweighted mean fold RMSE and the pooled RMSE, rounding both to 4 decimal places. (Answer: Fold RMSEs are \(\sqrt{5}\) and \(\sqrt{4}=2\), so their mean is \((\sqrt{5}+2)/2=2.1180\); pooled RMSE is \(\sqrt{(1+9+4+4)/4}=\sqrt{4.5}=2.1213\).)