Why Predictor Timing Matters
A renewal model is useful only if its inputs resemble the information available when a renewal decision is made. A predictor can be strongly associated with nonrenewal and still be invalid: it may have been recorded after the offer, after the customer accepted, or even as a direct consequence of the renewal outcome. Such a feature lets a model learn from the future rather than from information available to the decision-maker.
The relevant timestamp is not automatically the policy expiration date, the date a data extract was created, or the date the model is eventually run. It is the prediction decision time: the point at which the intended user needs the prediction. For example, if the model supports an offer issued on a particular morning, the predictors must be restricted to information available when that offer decision is made. If the intended decision occurs earlier, the cutoff must be earlier too.
A practical audit distinguishes three times. The event time is when something happened, such as a payment or inspection. The availability time is when the information reached the system or user who would make the decision. The prediction time is the cutoff for the intended prediction. An event can predate the cutoff yet still be unavailable at the cutoff if it was reported or loaded later.
The Point-in-Time Test
For each candidate field, establish what the field means, how it was produced, and which timestamp controls whether it was known. Then test the value against the prediction cutoff. For transaction aggregates, test every included record, not just the date on which the aggregate was calculated. A database query run today can return historical rows that were not available to a decision-maker at the historical prediction time.
State the business action the prediction supports and the exact time at which that action is taken. Fix the cutoff before reviewing model performance.
Identify whether it is a direct observation, an aggregate, a status field, or a value calculated from later records. Find the relevant event and availability timestamps.
Include only values recorded and accessible by the cutoff, and apply any defined historical lookback window. For revised records, use the version that existed at the cutoff.
Exclude fields that reveal whether the customer renewed or that are generated only after renewal, even if they appear in a historical extract.
Confirm that production scoring and historical feature construction apply the same cutoff and do not join future transactions or post-outcome fields.
The key principle can be expressed using the information set \(\mathcal{I}(t_0)\), meaning all information actually available to the decision-maker by time \(t_0\). A valid predictor \(X_j(t_0)\) must be determined by that information set. This is a timing condition, not a test of whether the field is statistically useful. A feature can be predictive, accurately measured later, and still fail the condition.
Worked Applications
Worked Example: Auditing a Renewal Offer Snapshot
A hypothetical insurer generates a renewal offer at 9:02 a.m. on April 1, 2026. A retention prediction is requested at 9:05 a.m. to support follow-up decisions. The candidate fields are shown below. Classify each field as eligible or ineligible for this prediction.
| Candidate field | Relevant timing or source |
|---|---|
| Prior-term payment history | Transactions through March 31; loaded by 8:00 a.m. April 1 |
| Inspection finding | Inspection occurred March 28; finding entered April 2 |
| Offered renewal premium | Offer generated and displayed at 9:02 a.m. April 1 |
| Acceptance status | Customer accepts at 2:00 p.m. April 1 |
| First payment on renewed term | Payment occurs April 3 |
Step 1: Fix the cutoff. The prediction cutoff is 9:05 a.m. on April 1. The offer has already been generated, so information displayed at 9:02 a.m. can be considered. If the prediction were instead needed to determine the offer itself, that same field would not yet be available.
Step 2: Test prior-term payment history. Its transactions end March 31 and the information was loaded by 8:00 a.m. on April 1. Both the events and their records are available before 9:05 a.m.; retain the field, subject to any separately specified lookback definition.
Step 3: Test the inspection finding. The inspection event occurred before the cutoff, but the finding was not entered until April 2. At 9:05 a.m. on April 1, the decision-maker could not use that finding. Exclude it. Using the inspection date alone would incorrectly treat later information as known.
Step 4: Test the offered premium. The offer was generated and displayed at 9:02 a.m., three minutes before the cutoff. It is eligible for this stated prediction decision. Eligibility depends on the specified timing; it is not a general claim that the offer premium is available for every renewal prediction.
Step 5: Test acceptance and first payment. The acceptance occurs at 2:00 p.m., and the first payment occurs April 3. Both are after the cutoff and are consequences or indicators of the subsequent renewal process. Exclude both.
Conclusion and check: Retain the prior-term payment history and offered renewal premium. Exclude the inspection finding, acceptance status, and first payment. The tempting but wrong calculation is to retain the inspection finding because March 28 precedes April 1; the correction is to require that the information itself was available by 9:05 a.m.
Worked Example: Filtering a Transaction Lookback
A hypothetical renewal prediction is made at 10:00 a.m. on June 15, 2026. The candidate predictor is the number of transactions in the preceding 90 days. Define the lookback as the interval from 10:00 a.m. on March 17 up to, but not including, the prediction time. Count a transaction only if its event falls in that interval and its record was available by the cutoff.
| Transaction | Event time | Available time |
|---|---|---|
| Endorsement | March 20 | March 21 |
| Payment | April 1 | April 1 |
| Payment reversal | May 2 | June 16 |
| New quote request | June 15, 10:03 a.m. | June 15, 10:03 a.m. |
| Cancellation | June 20 | June 20 |
Step 1: Apply the event-time window. The endorsement, payment, and reversal occurred after March 17 and before June 15 at 10:00 a.m. The quote request occurred three minutes after the cutoff, and the cancellation occurred later still. The latter two events fail the window test.
Step 2: Apply the availability-time test. The endorsement and payment were available by June 15 at 10:00 a.m., so both qualify. The reversal was not available until June 16. Although its event falls inside the lookback window, it was not known at prediction time and must not be counted.
Step 3: Compute the feature. The eligible records are one endorsement and one payment:
The correct value is 2. A tempting but wrong query might count every record whose event date falls in the 90-day interval and then add the two later events, producing \(3+2=5\). That result includes the late-loaded reversal and events after the cutoff. The correction is to apply both tests record by record; the audited count remains 2. The check is that neither the late reversal nor either post-cutoff event contributes to the historical feature.
Worked Example: Detecting an Outcome-Derived Field
A hypothetical analyst evaluates a renewal model at the time renewal offers are being considered. A data extract contains four policies and a field called “next-term written premium.” The field is populated only after a customer accepts and the next term is written. The observed extract is:
| Policy | Renewed by the defined deadline? | Next-term written premium |
|---|---|---|
| A | Yes | $1,200 |
| B | Yes | $900 |
| C | No | $0 |
| D | No | $0 |
Step 1: Check availability. At offer time, “next-term written premium” does not yet exist for a customer who has not accepted and had the term written. It is therefore unavailable at the intended prediction time.
Step 2: Show the misleading result. Suppose the analyst classifies a policy as renewed whenever this field is greater than zero. Policies A and B are classified as renewed; C and D are classified as not renewed. Compared with the observed outcomes, there are 2 correct renewed classifications and 2 correct nonrenewed classifications: \(2+2=4\) correct out of 4, or \(4/4=100\%\) accuracy.
Step 3: Interpret the check correctly. The perfect result does not demonstrate useful prediction. The field is populated as part of the post-acceptance process and directly reveals the outcome in these data. It cannot be used to predict that outcome at offer time.
Step 4: Correct the feature definition. Remove next-term written premium from the offer-time predictors. If the intended prediction occurs after an offer is generated, a quoted premium known at that later cutoff may be eligible, but it is not interchangeable with premium written after acceptance. Rebuild historical predictors from records available at each policy’s actual prediction cutoff, then evaluate only those predictors. This avoids treating information created by the outcome as information that preceded it.
Common Leakage Patterns and Controls
The same timing error can enter through many different fields. Names alone are not a reliable guide: a field called “current status” might be a valid pre-decision status or a status updated after renewal. The audit must follow the field’s definition and production history.
| Leakage pattern | Why it fails | Practical control |
|---|---|---|
| Post-renewal fields, such as written premium or a new-term policy status | They are created after the acceptance or renewal process has advanced. | Use only a field demonstrably available at the prediction cutoff; exclude post-outcome values. |
| Transactions after the prediction date | They describe future activity, not information available when the prediction was made. | Apply an explicit cutoff to transaction event times. |
| Late-reported events | The event may predate the cutoff, but the decision-maker did not yet know it. | Check the record’s availability or ingestion time as well as event time. |
| Outcome-derived fields or flags | They encode the response or a direct consequence of it. | Trace the field’s calculation and exclude fields that depend on the later renewal outcome. |
| Historical aggregates recalculated with later data | A present-day total can silently include transactions unavailable at the historical prediction date. | Reconstruct each aggregate as of its own cutoff, rather than using a current total. |
One common mistake is to define a predictor correctly but build it from a table that has since been updated. For example, “number of prior transactions” sounds like a historical measure, but a current policy table may include transactions posted after the historical prediction. The feature’s name does not make it point-in-time valid. The underlying records and their availability timestamps must support the value used.
A reliable audit records, for every predictor, its business definition, source table, event-time rule, availability-time rule, lookback interval if any, and treatment of corrections or revisions. When the historical system cannot establish what was available at the cutoff, the field cannot be assumed safe merely because its event date is earlier. Exclude it or obtain a defensible historical snapshot before using it.
Check Your Understanding
For each problem, use the stated prediction cutoff and distinguish event time from information availability.
- A prediction is made at 4:00 p.m. on September 8. A payment occurred September 7 and was loaded at 3:00 p.m. on September 8. A claim occurred September 6 but was first reported September 9. Which information can be used at the cutoff? (Answer: The payment can be used; the claim information cannot, because it was reported after the cutoff.)
- A 30-day transaction count uses the interval from 9:00 a.m. on October 1 to 9:00 a.m. on October 31, excluding the cutoff. Of four events, three occurred inside the interval; only two of those three were available by 9:00 a.m. on October 31. The fourth event occurred at 9:05 a.m. on October 31. What is the valid count? (Answer: 2.)
- A renewal model is scored at 8:00 a.m. before an offer is generated at 8:30 a.m. The offered premium is available after 8:30 a.m. Is it valid for the 8:00 a.m. prediction? (Answer: No; it is not available at the prediction cutoff.)
- A data extract contains a “renewed policy number” field populated only after acceptance. It is present in the historical training table and is strongly associated with the renewal outcome. Should it be retained for a prediction made before acceptance? (Answer: No; it is a post-outcome field unavailable at the decision time.)
- A transaction occurred inside the defined lookback window, but the record was loaded two days after the historical prediction cutoff. A later extract includes it. What should the historical feature do with that transaction? (Answer: Exclude it from that prediction’s feature value because it was not available by the cutoff.)