Why the Order of Observations Matters
A random holdout is not always a credible test of future performance. If a model will be fitted using information available today and used to predict next month, an evaluation that trains on some later observations and tests on earlier ones does not reproduce that use. It can also make the evaluation look better than a true future test if patterns change over time.
A time-ordered split addresses this mismatch by using earlier observations for training and later observations for testing. Chronology alone, however, is not enough. A valid partition must also respect what was known at the model-fitting date: features must be available when a prediction is made, and a training outcome must already be observed when the model is fitted.
The cutoff \(c\) represents a real operational decision: the date and time at which the model is fitted. The test period begins after that cutoff. For a single fixed-model evaluation, the model is fitted once using eligible training data; outcomes from the test period are not used to update that model during the test period. This design evaluates a particular future period, rather than every possible future environment.
Constructing a Time-Ordered Holdout
A sound partition starts by defining the prediction task precisely. Specify the unit being predicted, the prediction time, the target horizon, when the target becomes known, and the date at which the model would be fitted. Then choose a time cutoff that separates the historical training period from the later evaluation period.
Identify the timestamp at which each prediction is intended to be made. Order observations by that timestamp, not by an unrelated row number.
Choose the date and time when the model is assumed to be fitted. The test period must occur after this point.
Include an earlier observation in training only if its target would have been known by the fitting cutoff. Earlier prediction times do not guarantee that the associated outcomes have matured.
Assign all observations at a tied timestamp to the same side of the boundary. Use a calendar-time cutoff rather than an arbitrary row split when several records share a date.
Generate predictions for the test records using the fitted model and calculate the chosen performance measure from their outcomes once those outcomes are available.
A chronological split need not allocate an exact percentage of rows to each partition. If the data are irregular, a cutoff by date can give an unexpected number of test records. That is not a reason to move individual records across the time boundary. The design should reflect the intended prediction setting; the resulting sample sizes should then be reported and assessed.
Worked Example: Delayed Outcomes in a Monthly Forecast
A hypothetical subscription service wants to predict, at each month-end, whether a customer will cancel during the next three months. A model is to be fitted at the end of December 2023. The data contain one prediction record for each month from January 2023 through April 2024. Assume a record's cancellation outcome becomes available only at the end of the third month after its prediction month. Determine which records can train the model and which belong to the future test period.
Method. Compare each prediction month with the fitting date, then check whether its three-month target is known by December 31, 2023. The test period consists of prediction months after the fitting date.
Training eligibility. The January prediction has an outcome available at the end of April 2023; February's outcome is available at the end of May; and so on. September's outcome is available at the end of December 2023, so it is known by the fitting cutoff. January through September therefore provide 9 eligible training records.
October's outcome is not available until the end of January 2024, November's until the end of February 2024, and December's until the end of March 2024. These three records are earlier than the future test period, but their outcomes were not known when the model was fitted. They cannot be used as labelled training records at that cutoff.
Test period. January, February, March, and April 2024 are all prediction months after December 31, 2023. They form the 4-record test set. Their target outcomes become available later, at the end of April through the end of July 2024, respectively.
Check. The 9 training records have observed outcomes by the fitting date; the 4 test records have later prediction times. The 3 intermediate records are excluded from this particular fit because their labels are not yet available. The common mistake avoided here is to include all 12 months of 2023 in training merely because their prediction dates precede 2024.
Worked Metric: A Chronological Split Versus a Random Split
The following example shows why partition design can change the meaning of a reported error. Consider hypothetical weekly counts of new service incidents. At the end of week 7, an analyst fits a simple forecasting rule: predict the average weekly count in the training data for every week in the test period. The ten observed counts are shown below.
| Week | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Incident count | 4 | 5 | 3 | 6 | 7 | 4 | 8 | 5 | 9 | 6 |
Worked Example: Comparing Future and Random Holdouts
Method. First make a chronological split using weeks 1–7 for training and weeks 8–10 for testing. Compute the training-average forecast and the test mean absolute error (MAE), where MAE is the average of the absolute differences between predicted and observed counts. Then compare it with a random split whose test weeks are 2, 8, and 10.
Chronological training average. The first seven counts sum to \(4+5+3+6+7+4+8=37\). The training average, and hence the fixed prediction for each test week, is \(37/7=5.2857\) incidents per week (rounded to four decimals).
The test counts for weeks 8–10 are 5, 9, and 6. Their absolute errors are \(|5-5.2857|=0.2857\), \(|9-5.2857|=3.7143\), and \(|6-5.2857|=0.7143\), using the unrounded training average in the calculations. The MAE is
Random-split comparison. If weeks 2, 8, and 10 are randomly selected as test observations, the remaining training counts sum to \(57-(5+5+6)=41\), where 57 is the sum of all ten counts. The random-split training average is \(41/7=5.8571\). Its test MAE is
The random-split error is smaller, but its test set includes week 2 while its training set includes later weeks, including week 9. It does not represent fitting at the end of week 7 and predicting future weeks. The correct choice for that stated task is the chronological MAE of 1.5714 incidents per week. The lower random-split value is not a valid substitute for that future-period evaluation.
Common mistake and correction. The wrong approach is to treat the random test MAE of 0.6190 as the estimated error for future weeks. Its arithmetic is correct for its selected rows, but its time ordering is wrong for the prediction task. Keep all training times before the later evaluation period, then calculate the test metric on that period.
A Short Justification for the Future-Period Estimate
A time-ordered test set is useful because it puts predictions and outcomes in the sequence implied by deployment. Its interpretation still depends on the relationship between the test period and the future period of interest. To see the basic argument, suppose a fitted model is held fixed, and let \(L_j\) be its loss on test observation \(j\). If test losses and losses in the intended future period have the same expected value, then the average test loss estimates that expected future loss.
The last equality requires the test observations to have the same expected loss as the future observations being represented. For example, it may be reasonable when the relevant data-generating conditions remain stable. It is not guaranteed merely because the dates are ordered. If risks, customer mix, recording practices, or operating conditions change, the test result describes performance in that observed period and may not carry over to a different future period. Dependence between observations can also affect the variability of the average, even when its expectation is appropriate.
Implementing and Checking the Split in R
In R, convert date columns to date or date-time objects before comparing them. A row-index split can accidentally divide records with the same timestamp, so define the boundary using the actual prediction time. Also create an outcome-availability date if targets are delayed.
Worked Example: Building a Cutoff in Base R
A hypothetical repair network has eight prediction records. Each record predicts whether a repair will require a second visit. The prediction time and the date the outcome becomes known are listed below. The model is fitted at the end of March 1, 2024; test predictions begin on March 2. Identify the training, test, and excluded records, and show a base R filtering approach.
| Record | Prediction date | Outcome available |
|---|---|---|
| A | Jan 5 | Feb 5 |
| B | Jan 5 | Feb 5 |
| C | Jan 20 | Feb 20 |
| D | Feb 1 | Mar 1 |
| E | Feb 1 | Mar 1 |
| F | Feb 18 | Mar 18 |
| G | Mar 2 | Apr 2 |
| H | Mar 2 | Apr 2 |
Step 1: Set the boundary. The model-fitting date is March 1, and the test period starts March 2. Records G and H have later prediction times, so both belong in the test set. They share a date and remain together.
Step 2: Check eligible training records. Records A through E have prediction dates before March 2 and outcomes available no later than March 1. All five can train the model. Record F has a prediction date before the test period, but its outcome is not available until March 18. It is not eligible for training at the March 1 fit.
Step 3: Apply the rule in R. Suppose a data frame named dat contains columns named id, prediction_date, and outcome_available. Convert the date columns with dat$prediction_date <- as.Date(dat$prediction_date) and dat$outcome_available <- as.Date(dat$outcome_available). Set cutoff <- as.Date("2024-03-01") and test_start <- as.Date("2024-03-02"). Then define training rows using dat$prediction_date < test_start & dat$outcome_available <= cutoff, and test rows using dat$prediction_date >= test_start. Use dat$id[training_rows] and dat$id[test_rows] to inspect the assigned records.
Step 4: Verify the result. The training IDs are A, B, C, D, and E, for 5 records. The test IDs are G and H, for 2 records. F is excluded from both partitions because it is an earlier prediction whose outcome was not available at the fitting cutoff. The counts reconcile: \(5+2+1=8\), the total number of records.
This filtering rule deliberately makes the boundary explicit. The common mistake is to put every record dated before March 2 into training, which would incorrectly include F. A second mistake is to split by row position and place one of the March 2 records in training while the other is in testing.
Practical Checks and Limits
Before accepting a time-ordered holdout, verify that the split corresponds to a clear operational question. In particular, document the fitting cutoff, the start and end of the test period, the target horizon, and any delay before outcomes are known. Confirm that the feature values used for each prediction could have been observed at that prediction time. A timestamp alone does not establish that a value was available then.
- Use time, not row order. Data may be sorted incorrectly, contain multiple records per date, or be arranged by another field. Sort or filter using the intended prediction timestamp.
- Do not divide tied timestamps. If all records with a date belong to the same decision period, keep them on one side of the cutoff.
- Do not ignore label delay. A prediction may be historical while its target is still unresolved at the model-fitting date.
- Do not overgeneralize one test period. A single later holdout describes performance in that period. Changes in the population or environment can limit its relevance to other periods.
Time-ordered splits are especially important when the intended use is prediction into the future. Their value is not that they guarantee a representative test set; rather, they prevent the evaluation from silently using a time arrangement that the real prediction process could not use. This treatment of prediction time and information availability is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each situation, determine the appropriate assignment or calculation from the information given.
- A model is fitted on June 30. Prediction records are dated June 15, June 30, July 1, and July 2. Outcomes for the first two are known by June 30, and the latter two outcomes are observed in August. If test predictions start July 1, how many records are in training and how many are in testing? (Answer: 2 training records and 2 test records.)
- A weekly model is fitted after week 8. Each weekly target is available at the end of that same week. Data cover 14 weeks, and the test period starts at week 9. How many observations are in each partition? (Answer: 8 training observations and 6 test observations.)
- A model is fitted at the end of April. A record has a March 10 prediction date, but its target will not be known until May 5. Can that record be used as a labelled training observation for this fit? (Answer: No; its outcome is not available by the April 30 fitting cutoff.)
- A fixed forecast model has training counts 2, 4, and 6, and is tested on later counts 5 and 7. Using the training average as the prediction for both test weeks, calculate the test MAE. (Answer: The training average is 4; the absolute errors are 1 and 3, so MAE is 2.)
- Two records have the same prediction timestamp, and the chosen cutoff places that timestamp in the test period. A row-index split would assign one record to training and one to testing. What should the time-ordered split do? (Answer: Assign both records to the test period so the tied timestamp is not divided.)