Choosing a Time-Series Holdout Period

Why the Holdout Design Matters

A time-series model is used to forecast from information available at a particular point in time. Evaluating it therefore requires more than setting aside some observations: the evaluation must preserve the order in which information becomes available. If observations from later dates are used to train a model that is assessed on earlier dates, the resulting evaluation does not reproduce the forecasting task.

A well-designed holdout specifies three related but distinct choices: how much history is available for fitting, how far ahead each forecast reaches, and when forecasts are issued. These choices should reflect the intended use of the model. For example, a model used to forecast the next month should be assessed at a one-month horizon, even if a longer holdout is useful for checking performance across different dates.

Definition: Let \(y_1,\ldots,y_N\) be an ordered series, with larger indices representing later times. A forecast origin \(t\) is the time through which observations are available when a forecast is issued. A forecast with horizon \(h\) at origin \(t\) predicts \(y_{t+h}\). The forecast horizon \(H\) is the largest lead time being evaluated. The training span is the historical portion of the series used to fit the model at an origin.

For a single terminal holdout, choose a cutoff \(T\). Fit using observations through \(T\), and reserve the later observations for assessment. If the forecast horizon is \(H\), the targets for a forecast issued at \(T\) are \(y_{T+1},\ldots,y_{T+H}\). For a terminal holdout containing \(M\) observations, those observations are \(y_{N-M+1},\ldots,y_N\). The holdout length \(M\) and forecast horizon \(H\) need not be equal: a longer holdout can contain targets for more than one forecast origin.

\text{training at origin }t \subseteq \{y_1,\ldots,y_t\}, \qquad \text{forecast targets at horizon }H: y_{t+1},\ldots,y_{t+H}.

The information restriction is central: a forecast made at \(t\) must not use observations dated after \(t\) to estimate its model. If a fixed number \(L\) of observations is used, the training indices are \(t-L+1,\ldots,t\), provided that these observations exist. Alternatively, all observations available through \(t\) may be used. The choice of training span affects what historical information the model can use; it should be made to reflect the intended forecasting procedure rather than selected because it makes the holdout results look favorable.

Choosing the Holdout and Forecast Origins

Start by describing the operational forecasting task in time units. State how frequently forecasts are issued, what dates are forecast, and how far ahead the forecasts must reach. A horizon of three monthly observations means three months ahead, not three years or three observations at an unspecified frequency. Then choose origins and a chronological set of target observations that permit the intended forecasts to be evaluated.

A useful design asks whether the reserved observations answer the question of interest. If the model will be used for one-step-ahead forecasts, a test of only long-horizon forecasts may not be relevant. Conversely, a model needed for a four-quarter projection should not be judged only by one-quarter-ahead predictions. The holdout should also contain enough observations to make the assessment informative, while leaving enough history to fit the model. These are practical design constraints, not a universal formula for the “correct” holdout size.

Seasonality can influence the choice. If a series has an annual seasonal cycle, a holdout covering a full year lets the assessment include each part of that cycle. But this is not automatically best for every task. When the operational target is the next quarter, a shorter holdout may more directly represent that task; the result then provides evidence for the particular seasonal portion of the year being forecast, not every season. State this limitation rather than implying that a short test block establishes performance in all seasons.

Worked Example: Matching a Monthly Holdout to a Quarterly Forecast

A hypothetical insurer has 120 consecutive monthly observations of a claim count. The series has a known annual seasonal pattern, and the intended use is to forecast the next three months once a year. Design a single terminal holdout that assesses this forecast task, and determine the amount of training history and the target months.

Method. Translate the intended forecast into the observation frequency. Three months ahead means \(H=3\). Use the last three observations as a terminal holdout, so the cutoff is \(T=120-3=117\).

The training observations are \(y_1,\ldots,y_{117}\), giving a training span of 117 months. Since the series is monthly and the seasonal cycle is 12 months, this history contains \(117/12=9.75\) annual cycles. The holdout targets are \(y_{118},y_{119},y_{120}\). If index 1 is January, these are October, November, and December.

This design directly assesses a three-month forecast issued after month 117. It does not test forecast accuracy for all seasonal phases: the targets fall in the final quarter of the annual cycle. A full-year holdout would cover all 12 calendar months, but that would assess more than the stated three-month forecast from this single origin. The common mistake is to equate “annual seasonality” with “the holdout must always be one full year.” The correction is to select the horizon to match the task, then describe what seasonal coverage that holdout provides.

When several forecast origins are used, specify each origin and the targets forecast from it. Every training set must respect its own origin: a forecast at \(t\) can use information available through \(t\), but not later values. If forecast horizons overlap, some observed outcomes can be evaluated more than once. That may be appropriate for the intended evaluation, but the repeated targets are not separate observations of independent evidence.

Worked Example: Setting Origins Within a Chronological Holdout

A hypothetical utility has 64 quarterly observations. It wants to assess forecasts for the next four quarters at two distinct dates, using a terminal holdout. Specify the cutoff, forecast origins, training spans, and target quarters.

Method. Each forecast must have horizon \(H=4\). To obtain two consecutive, nonoverlapping four-quarter target blocks, use origins \(t=56\) and \(t=60\), with the terminal holdout \(y_{57},\ldots,y_{64}\). The holdout has \(64-56=8\) observations.

At origin 56, the available training observations are \(y_1,\ldots,y_{56}\), and the targets are \(y_{57},y_{58},y_{59},y_{60}\). At origin 60, the available training observations are \(y_1,\ldots,y_{60}\), and the targets are \(y_{61},y_{62},y_{63},y_{64}\). Thus each forecast covers four quarters and the two target blocks do not overlap.

The first forecast uses 56 quarters of history, or \(56/4=14\) years. The second uses 60 quarters, or \(60/4=15\) years. The two forecasts share much of their training history, but every forecast target occurs after its own origin. A common mistake is to use the same cutoff for both forecasts while still claiming to issue the second forecast later; the correction is to state the later origin and the information available then. This example specifies an evaluation schedule without asserting that these two origins are enough to establish stable performance.

Why Random Shuffling Breaks the Forecasting Task

In ordinary shuffled train-test splits, observations are randomly assigned to training and test sets without regard to their time indices. Such a split can put a future observation in training while an earlier observation is in the test set. This reverses the information flow that defines forecasting. In a genuine forecast at time \(t\), observations after \(t\) are not yet available.

The key condition is not that training and test indices must be numerically adjacent. It is that every training observation used for a forecast at \(t\) must be available by \(t\), while every target being forecast lies after \(t\). A random split can violate this condition many times. When the series has temporal dependence or changing patterns, access to future observations can make the evaluation look better than a real forecast would be. Even when observations are independent and identically distributed, a shuffled split does not reproduce the chronology of deployment; its relevance depends on the question being asked.

Justification: A forecast made at origin \(t\) is a prediction based on the information available by \(t\). If its fitted model uses a training observation \(y_j\) with \(j>t\), then that observation was not available when the forecast would have been issued. Therefore the evaluation uses information outside the forecast’s information set. A chronological split prevents this particular violation by restricting training observations to indices no later than each origin and reserving later indices as forecast targets.

Worked Example: Counting Future-Information Pairs in a Shuffled Split

A shuffled split of a hypothetical 12-week series places indices \(3,5,7,9\) in the test set and indices \(2,4,6,8,10,11,12\) in the training set. Count the training-test pairs in which a test observation occurs before a training observation, and explain why those pairs indicate a chronology problem.

Method. For each test index \(j\), count every training index \(k\) satisfying \(k>j\). Each such pair \((j,k)\) places a later training observation alongside an earlier test observation.

For test index 3, the later training indices are \(4,6,8,10,11,12\), giving 6 pairs. For test index 5, they are \(6,8,10,11,12\), giving 5 pairs. For test index 7, they are \(8,10,11,12\), giving 4 pairs. For test index 9, they are \(10,11,12\), giving 3 pairs. The total is \(6+5+4+3=18\) future-information pairs.

For a check, the training set has seven indices, and the counts of training indices later than each test index are \(6,5,4,3\), respectively; their sum is 18. These pairs show why random assignment can mix later and earlier dates. They are not a count of 18 separate forecasts, nor do they quantify a change in forecast error. The common mistake is to count only training indices after 9 for every test observation. The correction is to count later training indices separately for each test index, including those between the test date and 9.

Design Checks and Common Traps

Before comparing models using a holdout, write down the evaluation design in a way another analyst could reproduce. At minimum, record the observation frequency, cutoff or forecast origins, training span at each origin, forecast horizon, and exact target indices. If there are seasonal cycles, state whether the held-out targets cover a full cycle or only part of one.

Confusing holdout length with forecast horizon. A 12-observation holdout can support several forecasts with shorter horizons; one forecast issued before the entire block may instead have a 12-step horizon. State both quantities.
Ignoring the forecast origin. “Forecast the next four observations” is incomplete if the date at which those observations become the future is not specified.
Using a shuffled split by habit. Random assignment can put future observations into training. For a chronological forecasting task, reserve later observations and keep them after each relevant origin.
Choosing a test block that does not match the task. A model intended for short-horizon forecasts should not be assessed only at a distant horizon without explaining why that is relevant.
Overstating what a short holdout shows. A test block from one season or one unusual period may not represent other periods. Describe the coverage and its limits.

The holdout design is part of the forecasting question, not a mechanical afterthought. A chronological evaluation makes the information available at forecast time explicit and prevents a misleading reversal of time. The right design is the one that represents the intended use while leaving enough history to fit the model and enough held-out targets to examine the forecast task.

Check Your Understanding

For each problem, identify the chronological design and check that every target follows its forecast origin.

A hypothetical monthly series has 90 observations. A forecast is issued after observation 84 for the next three months. State the cutoff, training span, holdout targets, and holdout length. (Answer: \(T=84\); 84 training observations; targets \(y_{85},y_{86},y_{87}\); holdout length 3.)
A quarterly series has 52 observations and an annual seasonal cycle. A single forecast is issued after quarter 48 for the next four quarters. How many years of training history are available, and does the target block cover one annual cycle? (Answer: 48 quarters \(=12\) years; targets \(y_{49}\) through \(y_{52}\), which cover four quarters, or one annual cycle.)
A weekly series has 200 observations. The final 12 weeks are held out, and a forecast is issued at origin 188 for a six-week horizon. Give the target indices and determine how many held-out observations remain beyond that forecast’s targets. (Answer: targets \(y_{189}\) through \(y_{194}\); 6 held-out observations, \(y_{195}\) through \(y_{200}\), remain beyond those targets.)
A shuffled split has test indices 2, 6, and 8, and training indices 1, 3, 5, 7, 9. Count pairs in which a test index is earlier than a training index. (Answer: \(4+2+1=7\) pairs: for test 2, training indices 3, 5, 7, 9; for test 6, 7, 9; for test 8, 9.)
A hypothetical monthly model will be used to forecast the next two months at origins 70 and 72. Give the two target blocks and state whether they overlap. (Answer: at origin 70, targets \(y_{71},y_{72}\); at origin 72, targets \(y_{73},y_{74}\); the blocks do not overlap.)

What You'll Learn