Data Latency And Historical Availability

Why Data Latency Matters

A record can describe something that happened weeks ago and still be unavailable when a prediction is made. A claim may occur on one date, be reported on another, and reach the prediction system later still. A historical analysis that uses the claim simply because its event date is earlier than the prediction date may use information that was not yet accessible in practice.

The key distinction is between when a fact happened and when a particular value could be used. This tutorial makes that distinction precise and applies it to historical prediction examples. The aim is not merely to order records by date, but to recreate the information set the prediction process could actually have used at each index time.

Definition: The event time (or event date) is when the underlying event occurred or the measured activity took place. The availability time is the earliest time the particular recorded value was accessible to the system making the prediction. Availability may depend on source publication, transmission, ingestion, and feature processing. A value is usable at prediction time $t$ only if its availability time is no later than $t$, subject to the system’s documented cutoff convention.

Defining Availability at a Prediction Time

Suppose a prediction is made at index time $t$, and a candidate predictor is built from records $j=1,\ldots,n$. For each record $j$, let $E_j$ denote its event time and $A_j$ the time its value became available to the prediction process. The event time may determine whether the record belongs in a measurement window; the availability time determines whether the record could have been used at the index time. These are separate tests.

Assume that timestamps are accurate, use a consistent time zone, and represent the system that would have made the prediction. If the system permits values available exactly at $t$, a record passes the availability test when $A_j \le t$. If the production process freezes its inputs before $t$, use that earlier cutoff instead. The rule must match the actual operational process; date-only fields may be insufficient to resolve same-day ordering.

\text{Record }j\text{ is available at index time }t \quad\Longleftrightarrow\quad A_j \le t.

For a feature based on an event window $W(t)$, both conditions are required: the record’s event time must lie in the defined window, and its value must have been available by the prediction cutoff. For example, “claims occurring in the 30 days before the index date” concerns event time, while “claims recorded in the system by the index time” concerns availability. A record may pass one test and fail the other.

Key takeaway: Use event time to define what a feature measures. Use availability time to determine what information could have been used. An old event is not automatically an available predictor.

How a Derived Feature Becomes Available

A feature assembled from multiple sources cannot be available before all required inputs are available and the feature has been computed. Suppose a feature requires inputs with availability times $A_1,\ldots,A_k$, and the documented processing takes $L$ units of time after the last input arrives. Under the assumptions that every input is required and the computation starts as soon as the last input is ready, its earliest availability time is:

A_{\text{feature}} = \max(A_1,\ldots,A_k)+L.

This follows because the input with the latest availability time is the first point at which all required inputs are present. Before that point, at least one required value is missing, so the feature cannot be computed. Once that final input arrives, the specified processing time $L$ must elapse before the computed feature is ready. If a system runs the feature only on a scheduled batch, use the actual batch completion time rather than assuming immediate processing.

This formula concerns operational availability, not the age of the events summarized by the feature. A feature can summarize events from an earlier period and still become available only after a delayed input arrives.

Worked Examples

Worked Example: Delayed Claims at a Monthly Prediction Date

A hypothetical insurer creates a risk score at 00:00 on September 1, 2025. One predictor is the number of claims with event dates in August that had reached the warehouse by the score cutoff. The records below include both the date reported to the insurer and the time ingested into the warehouse. Determine the predictor value.

Claim A occurred August 15, was reported August 29, and was ingested August 30. Claim B occurred August 25, was reported September 2, and was ingested September 2. Claim C occurred August 30, was reported August 31, and was ingested September 1 at 06:00.

Method. First check the event window: all three claims occurred during August. Then check warehouse availability at the index time, September 1 at 00:00.

Claim A was ingested August 30, before the cutoff, so it is available. Claim B was not reported or ingested until September 2, so it is unavailable. Claim C was ingested at 06:00 on September 1, six hours after the cutoff, so it is also unavailable.

The feature value is therefore $1$ claim. Once B and C are eventually ingested, a current extract might show all $3$ claims for August. That later total does not describe the information available to the September 1 prediction.

Check. The selected claim occurred in August and was in the warehouse by the cutoff; neither excluded claim passes the availability test. The common mistake avoided is counting every event with an August event date, regardless of when its record arrived.

Worked Example: A Revised External Data Release

A hypothetical underwriting model is scored at noon on April 10, 2025. Its feature is the latest published monthly unemployment rate, using the most recent reference month whose release was available by the score time. A January rate of 4.1% was published February 20. The original February rate of 4.2% was published March 20. The March rate of 4.0% was first published April 18. On April 18, the February rate was also revised to 4.0%. What value should the April 10 prediction use?

Method. List the releases and revisions that were available by April 10, then choose the latest reference month among them. The January value was available February 20. The original February value was available March 20. The March value and the February revision were both published April 18, after the prediction.

Thus February is the latest available reference month, and its available vintage on April 10 is the original value, 4.2%. The value for this prediction is 4.2%. The March value cannot be selected just because its reference month is before April 10: its publication date is April 18. Nor can the revised February value be substituted, since that version was also released after the cutoff.

Check. The chosen value is tied to both a reference month and a release version. The common mistake avoided is using a current data table that has replaced the original February value with its later revision.

Worked Example: A Rolling Transaction Feature With Processing Delay

A hypothetical account model is scored at 09:00 on June 30, 2025. Its feature is the total dollar amount of transactions with event times in the seven-day interval from June 23 at 09:00, inclusive, to June 30 at 09:00, exclusive. Transactions must also have reached the feature store by 09:00 on June 30. The store’s recorded availability time includes ingestion and processing. Determine the feature total from these records:

June 25 at 10:00, $40, available June 25 at 10:10.
June 29 at 20:00, $70, available June 30 at 09:05.
June 30 at 08:00, $30, available June 30 at 08:10.
June 24 at 15:00, $50, available June 30 at 09:10.
June 22 at 12:00, $90, available June 22 at 12:10.

Method. Apply the event-window test and the availability test separately to each transaction. The first transaction is in the interval and available by 09:00, so include $40. The second is in the interval but becomes available at 09:05, so exclude $70. The third is in the interval and available by 08:10, so include $30. The fourth is in the interval but is not available until 09:10, so exclude $50. The fifth is available but occurred before June 23 at 09:00, so exclude $90.

Add only the qualifying amounts: $40+30=70$. The feature total is $70.00. The event-window boundary is exclusive at June 30 at 09:00, and none of the listed transactions occurs exactly at that boundary. The common mistake avoided is treating a transaction’s event time as proof that its value had reached the feature store.

Reconstructing Historical Availability

A historical prediction should use the value that was available at the historical index time, not simply the value visible in a present-day extract. Sources that publish revisions, overwrite records, or load old events late require a way to identify what was known when. A useful record design preserves the event date, the source’s publication or report time, the system’s ingestion time, and—when applicable—the value’s version or revision time.

A practical reconstruction proceeds in a fixed order:

Fix the prediction cutoff.
Write down the index timestamp and the operational cutoff, including the time zone and whether values arriving exactly at the cutoff are eligible.

Define the event-time rule.
Specify the period or event condition the feature measures, including whether each boundary is inclusive or exclusive.

Select eligible versions.
Keep only values or record versions available by the cutoff. If a value was revised later, use the earlier version only when its timestamp and value can be established.

Apply feature processing.
Use the documented computation schedule and inputs. If a required input or completed feature was not available by the cutoff, do not treat it as usable then.

If an extract contains only event dates and current values, with no reliable arrival or revision history, it may be impossible to establish the historical value. Do not silently assume that event date equals availability date. Record the limitation and seek an archived snapshot, system log, or source vintage that resolves it. When exact timestamps are absent, a conservative rule may be necessary; state that rule rather than implying precision the data do not support.

Common Mistakes and Why They Matter

The most frequent error is ordering data only by event date. The following calculation shows why it fails. Suppose a prediction is made at 10:00, and a record describes an event from the previous day but arrives at 10:15. The wrong calculation treats “previous day is before prediction day” as sufficient and includes the record. The corrected availability test is $A_j \le t$: here, 10:15 is later than 10:00, so the record is excluded. Its event date can still place it in a historical window, but it was not usable at the prediction cutoff.

Other errors include using publication time while ignoring later system ingestion, using a revised value in place of the version available at the time, and ignoring the time of day when a prediction and data arrival occur on the same date. Each can make a historical feature set differ from what a real prediction process could have received. The practical safeguard is to specify both the event-time rule and the availability-time rule for every time-dependent feature.

Key takeaway: A defensible historical feature is defined by its measurement window, its data version, and its operational availability cutoff. If any of these cannot be established, the uncertainty should be documented rather than hidden by a date-only assumption.

Check Your Understanding

For each problem, apply the event-time and availability-time rules separately. Answers are given at the end of each question.

A prediction is made at 08:00 on March 12. Record A describes an event on March 10 and became available at 08:15 on March 12. Record B describes an event on March 11 and became available at 17:00 on March 11. Which record is available at prediction time? (Answer: Record B only.)
A monthly metric uses the latest reference month with a release available by noon on November 5, 2025. The September value, 71%, was published October 20. The October value, 74%, is published November 12. The September value is revised to 69% on November 20. What value should be used on November 5? (Answer: 71%, the September value available on October 20.)
A feature counts events from May 1 through May 31, inclusive, that were in the data system by 00:00 on June 3. The records are: May 3, $12, available May 4; May 18, $25, available June 3 at 00:00; May 31, $30, available June 3 at 00:01; and April 30, $40, available May 1. Under an availability rule that includes values available exactly at the cutoff, what is the feature total? (Answer: $37, from the $12 and $25 records.)
A derived feature requires sources A and B. A becomes available at 10:00 on January 8, B at 10:07, and the computation takes three minutes after both inputs are ready. Is the feature available for a prediction at 10:09? When is it first available under these assumptions? (Answer: No; it first becomes available at 10:10.)
A prediction is made at 09:00 on July 15. For a specified event window, three records qualify by event time. Their values became available at 08:45, 09:00, and 09:02 on July 15. The cutoff includes values available exactly at 09:00. How many records can be used? (Answer: Two records.)

What You'll Learn