Why the Prediction Date Matters
A prediction is useful only if it can be made with information available when the prediction is required. A variable may be strongly associated with an outcome and still be unsuitable as a predictor if the event it describes occurs later. Including that variable can make a retrospective analysis look successful while answering a different question from the one an organization needs to answer.
For example, suppose an insurer wants to identify policies likely to lapse during the 60 days after a renewal offer. A customer’s response to a retention call made 10 days after the offer may help explain whether the policy eventually lapses. But that response is not known when the renewal offer is sent. A model using it does not estimate the risk that could have been estimated on the offer date.
A dataset may contain the eventual outcome and many events observed during the follow-up period. Their presence in a completed dataset does not make them available at the prediction index. The relevant question is not simply whether a value is recorded in the data; it is whether the information represented by that value was available at the time the prediction would have been made.
A prediction task should specify both the index and the outcome window. For instance: “At the date a renewal offer is issued, predict whether the policy will lapse within the next 60 days.” The issue date is the index; the following 60 days define the outcome window. Events during that window occur too late to be predictors for this task, even if they happen before the outcome window ends.
How to Screen for Post-Index Events
For every candidate predictor, identify the event or activity that gives the variable its meaning. Then ask when that event occurs relative to the index. A column called “contact status” is not enough to answer this question: it might describe a contact completed before the index, a contact scheduled for the future, or a contact made after the index. The underlying event date matters.
Name the population, the prediction index, the outcome, and the period during which the outcome is measured.
Identify what action, observation, or change the value represents. Do not rely only on the column label.
If the event occurs after the index, the variable cannot be used for a prediction made at that index.
Exclude the variable from the original prediction, or define a new prediction task at a later index where the event is already known.
Post-index events include customer actions, service decisions, medical visits, claim developments, and operational responses that happen after the prediction date. Some may occur well before the outcome is finalized; that does not make them valid for a prediction made earlier. A later event can contain useful information about the eventual outcome precisely because the process is already unfolding.
Worked Example: Renewal-Offer Date and a Later Customer Response
A hypothetical insurer wants to predict, on the day a renewal offer is issued, which policies will lapse within 60 days. In a completed dataset of 12 policies, 5 lapse and 7 renew. An existing pre-index risk flag identifies 3 of the 5 policies that lapse and 2 of the 7 that renew. A separate “accepted retention call” field records whether the customer accepted a call made on day 10. In these data, 4 renewing customers accepted a call; none of the customers who lapsed did. Compare simple classification rules using each field, and decide which field is valid at the renewal-offer date.
Method. First count correct classifications under each stated rule. Then compare when each field becomes knowable with the prediction index.
For the pre-index risk flag, use the rule “flagged predicts lapse; not flagged predicts renewal.” Of the 5 policies that lapse, 3 are flagged, so there are 3 correct lapse predictions and 2 missed lapses. Of the 7 that renew, 2 are flagged and incorrectly predicted to lapse, while the other 5 are correctly predicted to renew. Thus the rule makes \(3+5=8\) correct classifications out of 12:
For the later call-response field, consider the rule “accepted predicts renewal; did not accept predicts lapse.” The 4 accepting customers all renewed, giving 4 correct renewal predictions. The remaining 8 customers did not accept: 5 lapsed and are correctly classified, while 3 renewed and are incorrectly classified. The rule therefore makes \(4+5=9\) correct classifications out of 12:
The 75.00% result does not show that the call-response field is a valid predictor for the specified task. The call happens on day 10, after the renewal-offer date. The field was unavailable at the prediction index. It could be considered for a different task that makes a prediction on day 10, provided the prediction target and eligible population are defined for that later task. The common mistake is to choose the field with the higher retrospective accuracy without checking when its event occurs.
Why Leakage Can Make Results Look Better
The basic reason is that a model evaluated with information from after the index is allowed to use facts that the intended prediction process does not yet possess. To state this precisely, let \(X\) denote information available at the prediction index, let \(Z\) denote information from a post-index event, and let \(Y\) be the outcome. A model using \(X\) must make its prediction from \(X\); a model using both \(X\) and \(Z\) has access to a larger information set.
For an ideal classifier under zero-one loss, the best achievable error using an information set is the probability of assigning the less likely class after observing that information. Adding information cannot increase this ideal minimum error because a classifier with \(X\) and \(Z\) can always ignore \(Z\) and use the rule it would have used with \(X\) alone. Therefore, the ideal minimum error with both variables is no greater than the ideal minimum error with \(X\) alone.
This inequality explains why a later variable can improve apparent prediction. It does not establish that using the variable is appropriate: the comparison gives the \(X,Z\) classifier information unavailable at the original index. Nor does the inequality guarantee that a particular model fitted to a finite dataset will show better measured performance; sampling variation and model fitting matter. The central problem remains the same: an evaluation using post-index information does not represent the intended prediction-time task.
Worked Example: Claim Escalation After the Initial Report
A hypothetical insurer wants to predict, at the initial report of each claim, whether the ultimate claim cost will exceed \$10,000. Among 10 completed claims, 6 exceed \$10,000 and 4 do not. A later “escalated repair” event occurs on 5 claims: 5 of those claims exceed \$10,000, while 1 of the 5 claims without escalation also exceeds \$10,000. Evaluate the rule “escalation predicts cost over \$10,000; no escalation predicts cost at or below \$10,000.” Compare it with a rule that predicts cost over \$10,000 for every claim.
Method. Count correct classifications against the stated cost outcome, then check whether escalation was known at the initial-report index.
For the escalation rule, the 5 escalated claims are all over \$10,000, so they produce 5 correct predictions. Of the 5 claims without escalation, 4 are at or below \$10,000 and are correctly predicted by the rule; the remaining 1 is over \$10,000 and is missed. The total is \(5+4=9\) correct out of 10:
The rule that predicts “over \$10,000” for every claim is correct only for the 6 claims that actually exceed the threshold. Its accuracy is:
Escalation appears to improve accuracy by 30 percentage points in this small illustration. But escalation occurs after the initial claim report, so it cannot support a prediction made at that report date. It may be relevant to a separately specified prediction made after escalation status is known. The correction is not to ignore the observed association; it is to avoid presenting it as prediction-time performance for the earlier task.
Worked Example: A Later Safety Alert
A hypothetical fleet operator wants to predict, when a driver starts a 12-month monitoring period, whether the driver will have an incident during that period. In a completed group of 12 drivers, 4 have an incident and 8 do not. During the monitoring period, a safety alert is issued for 3 drivers who later have an incident and 1 driver who has no incident. Consider the rule “alert predicts incident; no alert predicts no incident.”
Method. Count correct and incorrect classifications, then assess whether the alert existed at the start of monitoring.
The 3 drivers with both an alert and an incident are correctly classified. The 1 alerted driver without an incident is incorrectly classified. Of the 8 drivers with no alert, 1 has an incident and is missed, while 7 have no incident and are correctly classified. The rule makes \(3+7=10\) correct predictions out of 12:
A rule that predicts no incident for every driver is correct for the 8 drivers without an incident:
The alert rule looks more accurate in these data, but the alert is generated during the monitoring period, not at its start. It cannot be used to make the start-of-period prediction. A prediction made after an alert is issued would be a different task with a different index. The common mistake is to treat a variable as valid merely because it is recorded before the final outcome is known. The relevant comparison is with the prediction index, not just with the date the outcome is finalized.
Choosing Between Exclusion and a Later Prediction
When a candidate variable describes a post-index event, there are two defensible options. For the original task, exclude it from the predictors. If the event is valuable for a later operational decision, define a new task whose index occurs after the event. That new specification must make clear which individuals are eligible at the later index and what future outcome will then be predicted. It should not be described as a prediction made at the original date.
A useful review question is: Could the intended user have known this value, in its relevant form, at the moment the prediction was due? If not, the variable is not available for that prediction, even if it is highly associated with the target. This reasoning is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it ties data design to the actual timing and purpose of a predictive analysis.
Check Your Understanding
For each problem, compare the event date with the stated prediction index before judging the predictor.
- A hypothetical insurer predicts, on the date a policy begins, whether a customer will make a claim during the next year. A roadside-assistance call occurs four months after the policy begins. Is the call a valid predictor for the stated task? (Answer: No; the call is a post-index event.)
- At an initial claim report, an analyst predicts whether ultimate cost exceeds \$5,000. In 8 claims, 3 exceed \$5,000 and 5 do not. A post-index review flag is present for all 3 high-cost claims and for 1 low-cost claim. The rule “flag predicts high cost; no flag predicts low cost” is correct for 3 high-cost and 4 low-cost claims. What is its accuracy, and can it be used for the initial-report prediction? (Answer: \(7/8=87.50\%\); no, the flag is post-index.)
- A hospital predicts 30-day readmission at discharge. A follow-up appointment occurs after discharge but before day 30. Does the appointment become a valid predictor just because it precedes some readmissions? (Answer: No; it occurs after the discharge index.)
- A hypothetical service team predicts at day 0 whether a customer will cancel within 30 days. A later retention offer is accepted by 6 customers; all 6 remain customers. Is acceptance available for the day-0 prediction, and what change could make it relevant to a valid task? (Answer: No; define a separate prediction at the time acceptance is known, with a future outcome measured from that later index.)
- A model is intended to predict, at policy renewal, whether a policy will lapse within 45 days. A field records whether a customer replied to a message sent on day 5. State the appropriate treatment of the field for this task and explain the timing reason. (Answer: Exclude it; the reply event occurs after the renewal index, even if it is recorded before the 45-day outcome window ends.)