Detecting Target Leakage in Candidate Predictors

Why Predictor Timing Matters

A model can score extremely well for the wrong reason. A candidate predictor may encode the target directly, or may record an event that occurs only after the prediction is meant to be made. Such a variable can make validation performance look impressive even though it cannot support a real prediction at the required time.

For an actuarial use case, begin by specifying the prediction time: the date or event at which the model’s prediction is required. Then ask what information would actually be available on that date. A field’s presence in a data table does not establish that it was available at prediction time. This timing question is central to detecting target leakage.

Definition: Target leakage occurs when a candidate predictor contains information about the target that would not legitimately be available for the intended prediction. It includes fields that directly encode the target and fields that record, summarize or reveal events occurring after prediction time. A predictor is not leakage merely because it is strongly associated with the target; its meaning, construction and availability must be examined.

The distinction is operational. A prior-year claim count might be available when pricing a renewal and may be genuinely informative. A claim’s final paid amount might be highly informative about its ultimate severity, but it is not available when an early severity estimate is needed. The same field can therefore be acceptable for one prediction task and leakage for another, depending on the prediction time and target definition.

A Practical Audit for Suspect Predictors

For each unusually strong candidate, investigate what the field means and how it was created, rather than relying only on its name. A compact audit can organize the diagnosis:

Fix the target and prediction time.
State what outcome is being predicted, over what period, and when the prediction must be made. For example, distinguish ultimate claim severity from paid severity as of a specified valuation date.

Trace the field’s definition.
Find out whether the field is calculated from the target, from a later version of the same record, or from an event after the prediction time. Review data dictionaries, source systems and calculation logic.

Check availability for each scored record.
Use event and update timestamps where possible. A field may be populated eventually but blank or unknown at the intended prediction time.

Re-evaluate without the suspect information.
Exclude the field, refit or rescore the model appropriately, and report the resulting validation metric. The difference indicates how much the apparent performance depended on that information.

A random train/validation split does not automatically solve leakage. If a field is calculated from the target for every record, both partitions can contain the same target-field relationship. The validation set then measures how well the model exploits that relationship, not how well it predicts using information available in practice.

Why the validation score can be misleading: Suppose \(Z_i=Y_i\) for each record \(i\), where \(Y_i\) is the target and \(Z_i\) is a candidate predictor. A model that predicts \(\hat{Y}_i=Z_i\) has zero squared error on any records for which this equality holds. Splitting records at random does not change \(Z_i=Y_i\) in the validation partition. The score is excellent because the predictor reveals the answer, not because the model has learned a usable forecast.

Worked Examples

Worked Example: An Ultimate-Payment Field in a Severity Model

A hypothetical insurer wants predictions of ultimate claim severity at the initial case review. There are eight claims: four are assigned to training and four to validation. The training severities are \(80, 100, 120,\) and \(140\) thousand dollars. The validation severities are \(90, 110, 130,\) and \(150\) thousand dollars. A candidate predictor, “ultimate indemnity paid,” equals the final target for every claim. Compare a constant-mean baseline with the apparent result using this candidate field. Use RMSE in thousands of dollars.

Method and baseline. With squared-error loss, the constant prediction that minimizes training squared error is the training mean:

\bar{y}_{\text{train}}=\frac{80+100+120+140}{4} =\frac{440}{4}=110.

The baseline predicts \(110\) for each of the four validation claims. Its errors, defined as observed minus predicted, are \(90-110=-20\), \(110-110=0\), \(130-110=20\), and \(150-110=40\). The squared errors are \(400,0,400,\) and \(1{,}600\). Thus:

\operatorname{RMSE} =\sqrt{\frac{400+0+400+1{,}600}{4}} =\sqrt{600} \approx 24.495.

The baseline validation RMSE is therefore about \(24.495\) thousand dollars. If “ultimate indemnity paid” is used as the prediction, every prediction equals its target, so all four errors are zero and the RMSE is \(0\).

Check and diagnosis. The validation set has four records, and the squared errors sum to \(2{,}400\); dividing by four gives \(600\), whose square root is approximately \(24.495\). The zero RMSE is arithmetically correct but operationally invalid: the field is the outcome itself and is unavailable at initial review. The method is to audit the field’s definition and timing, then exclude it. The common mistake is to interpret a lower validation RMSE as a genuine improvement without asking whether the predictor could exist at prediction time.

Worked Example: A Post-Investigation Flag in a Binary Model

A hypothetical model predicts whether a claim will ultimately be confirmed as fraudulent within 90 days. A validation set has four claims, with observed outcomes \(y=(1,0,1,0)\), where 1 means confirmed fraud. A candidate model uses “investigation outcome recorded,” which is populated after the 90-day investigation and is strongly tied to the target. Its validation probabilities are \((0.9,0.1,0.8,0.2)\). A model excluding that field assigns probability \(0.6\) to each claim. Compare their Brier scores and classification accuracy using a probability threshold of \(0.5\).

Method. For binary outcomes, the Brier score is the mean squared difference between the predicted probability and the observed 0-or-1 outcome. For the model using the investigation field, the squared errors are:

(0.9-1)^2=0.01,\quad (0.1-0)^2=0.01,\quad (0.8-1)^2=0.04,\quad (0.2-0)^2=0.04.

The sum is \(0.10\), so its Brier score is \(0.10/4=0.0250\). For the model without that field, the squared errors are \(0.16, 0.36, 0.16,\) and \(0.36\). Their sum is \(1.04\), giving a Brier score of \(1.04/4=0.2600\).

At the \(0.5\) threshold, the investigation-field model predicts classes \((1,0,1,0)\), all correct: accuracy is \(4/4=1.0000\). The other model predicts \((1,1,1,1)\): it correctly classifies the two positive outcomes, but incorrectly classifies both negative outcomes. Its accuracy is \(2/4=0.5000\).

Check and diagnosis. The probabilities produce the stated sums of squared errors, and the class predictions follow directly from comparing each probability with \(0.5\). The first model’s apparent advantage depends on a field recorded after the investigation, so it cannot be used for a prediction made before that investigation concludes. The common mistake is to call this a legitimate gain because both models were scored on the same validation records; a matched comparison cannot make unavailable information available.

Worked Example: Distinguishing a Valid Lagged Feature From Leakage

A hypothetical renewal team needs a severity estimate at the start of a policy year for claims arising during that year. It considers two fields: “claims in the prior policy year,” available at renewal, and “payments posted by year-end,” which includes payments made after the prediction date. For four validation claims, the observed annual severities are \(40,80,120,\) and \(160\) thousand dollars. A model using prior-year information predicts \(80,80,120,\) and \(120\). A comparison model also uses year-end payments and predicts \(40,80,120,\) and \(160\). Calculate each validation RMSE and identify the appropriate candidate for the stated task.

Method. For the model using the prior-year information, the errors are \(40-80=-40\), \(80-80=0\), \(120-120=0\), and \(160-120=40\). The squared errors are \(1{,}600,0,0,\) and \(1{,}600\). Therefore:

\operatorname{RMSE}_{\text{prior}} =\sqrt{\frac{1{,}600+0+0+1{,}600}{4}} =\sqrt{800} \approx 28.284

thousand dollars. The second model has four zero errors, so its RMSE is \(0\). Its predictions appear perfect, but year-end payments are recorded after renewal and can incorporate information about the claims being predicted.

Check and diagnosis. The first model’s squared errors sum to \(3{,}200\), and \(3{,}200/4=800\); the second model’s errors are all zero. For the stated prediction task, the prior-year field is eligible because it is available at renewal, subject to confirming its recorded values really were available then. The year-end field is leakage for this task. A field’s validity depends on timing, not simply on whether it is associated with severity or present in the extracted data.

Deriving the Role of Timing

The reason to exclude unavailable information can be stated in terms of the prediction task. Let \(X(t_0)\) denote the information available at prediction time \(t_0\), and let \(Y\) be the future outcome of interest. A deployable prediction must be a function of \(X(t_0)\), because those are the inputs available when the decision is made. If a field \(Z\) is only created after \(t_0\), a prediction that uses \(Z\) is not a function of the information set \(X(t_0)\).

This follows directly from the order of events: at \(t_0\), the analyst can calculate a prediction only from observed inputs available by \(t_0\). A later event cannot be included in that calculation unless it was already known, which would contradict the assumption that it occurs after \(t_0\). A validation score using \(Z\) therefore evaluates a different information set from the one available in deployment. The issue is not repaired by a more complex algorithm or a different metric.

Common Exam Traps

Wrong calculation: treating a perfect score as proof of a useful model. In the first example, one might report “RMSE is zero, so the ultimate-payment model is best.” The arithmetic is right, but the conclusion is wrong. The correction is to recognize that ultimate payment is the target itself and report the performance of a model using only information available at initial review.
Confusing association with leakage. A strong prior-year predictor is not leakage merely because it is predictive. Check whether it existed at prediction time and whether it was derived from the target period.
Checking column names but not provenance. A field called “status” might be a harmless status as of renewal or a final status assigned after the outcome. Inspect its definition and update history.
Assuming random partitioning prevents leakage. A target-derived field can reveal the outcome in both training and validation data. The validation score can remain excellent while the proposed prediction cannot be made in practice.
Using a field that is only partially available without checking the scoring population. If some records have the field at prediction time and others do not, evaluate the intended availability rule explicitly. Do not silently use later-populated values for records that would lack them at deployment.

Key takeaway: A candidate predictor is acceptable only if its meaning, construction and availability fit the stated prediction time. When a striking performance gain depends on a field that reveals the target or arrives later, exclude that information and reassess performance. This diagnostic approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, assess the field or calculate the requested validation measure before checking the answer.

A model predicts claim occurrence during the next policy year at renewal. “Claims reported during the next policy year” is among its candidate predictors. Is this field eligible at renewal? (Answer: No; it records events in the target period, after the prediction time.)
Three validation targets, in thousands of dollars, are \(20,30,50\), and predictions are \(25,35,40\). Calculate RMSE, rounding to two decimals. (Answer: \(\sqrt{(25+25+100)/3}=\sqrt{50}\approx 7.07\) thousand dollars.)
For outcomes \(y=(1,0,0)\), a model predicts probabilities \((0.7,0.2,0.4)\). Calculate its Brier score to four decimals. (Answer: \((0.09+0.04+0.16)/3=0.0967\).)
A renewal model uses the number of claims in the preceding policy year. The count is finalized and available at renewal, while the target is claim cost during the new policy year. Is the count automatically leakage? (Answer: No; it is a lagged, available predictor, provided the field truly excludes claims from the target period.)
A model predicting whether a claim will be referred for investigation uses a “referral completed” field that is recorded only after the referral decision. Its random-split validation accuracy is 98%. What is the appropriate diagnosis? (Answer: The field is post-decision information and is leakage for predicting referral; the 98% accuracy does not establish deployable performance.)

What You'll Learn