Why Outcome-Derived Predictors Are Dangerous
A model is useful only if its predictors contain information that could genuinely be used to make the intended prediction. A field can look informative because it is calculated from the very outcome the model is asked to predict. This is leakage from an outcome-derived predictor: the model receives the answer, or a direct encoding of it, as an input.
This issue can arise in insurance data when the target is a claims total, claim count, or claim-occurrence indicator. A claims-system field such as “total paid during the outcome period” might appear to be a strong predictor of the target “paid claims during the outcome period.” It is not a useful predictor: it is the target itself, or a quantity calculated from the same underlying claim records. A flag such as “any claim in period” can create the same problem when the target is whether a claim occurred in that period.
The key question is not whether a field has a strong association with the target. It is how the field was produced. A legitimate predictor may be highly associated with the outcome. A leaked predictor is disqualified because its value contains the outcome by construction.
For example, if \(Y_i\) is the total amount paid on policy \(i\) during a defined year, then a field calculated as the sum of paid amounts on that policy in that same year is outcome-derived. If \(Y_i\) is an indicator equal to one when at least one claim occurs during a period, a field defined as “claim occurred during that period” directly encodes the target.
By contrast, a claim count from a completed period before the prediction is a historical measurement, not a direct copy of the current target. It may still require careful data design, but it is not outcome-derived merely because it also involves claims. The period and source records matter.
Trace the Field Back to Its Construction
A column name is not enough to establish whether a predictor is safe. “Claim count” could refer to a prior year, the target year, or all claims in a file that includes the target period. A reliable audit states the target definition and then traces each suspicious field to its underlying records and calculation.
State what is measured, for which unit, and over which outcome period. For example: “total paid claims per policy during the stated coverage year.”
Find the source fields, claim records, aggregation rule, and time period used to calculate it. Do not rely on a short label or a data dictionary heading alone.
Ask whether the field is the target, a component of it, a transformation of it, or an event flag that says whether the target occurred.
Remove the field from the modeling inputs, then evaluate the model using only predictors appropriate to the intended prediction task.
A predictor does not need to equal the target numerically to be contaminated. A threshold flag, count, band, or other transformation can reveal the outcome even if it discards some detail. The relevant question is whether the predictor is constructed from the target or its defining records, not whether the predictor has the same units.
Worked Example: A Paid-Claim Flag Repeats the Target
In a hypothetical portfolio, the target \(Y_i\) equals 1 if a policy has any paid claim during a specified outcome year, and 0 otherwise. The modeling file contains four policies. The claims totals and a field called paid_claim_flag are shown below. A classifier predicts a claim when the flag is at least 0.5. Determine whether the field is valid and calculate the resulting apparent accuracy.
| Policy | Paid total in outcome year | Target \(Y_i\) | paid_claim_flag |
|---|---|---|---|
| A | $0 | 0 | 0 |
| B | $0 | 0 | 0 |
| C | $120 | 1 | 1 |
| D | $0 | 0 | 0 |
Method. Compare the field’s construction with the target definition, then calculate the classification results. The target is one exactly when the outcome-year paid total is positive. The flag has the same rule and the same values: \(\text{paid\_claim\_flag}_i=I(\text{paid total}_i>0)=Y_i\), where \(I(\cdot)\) is 1 when its condition is true and 0 otherwise.
The rule “predict a claim when the flag is at least 0.5” therefore predicts 0, 0, 1, 0. There is one true positive, three true negatives, no false positives, and no false negatives. The apparent accuracy is \((1+3)/4=1.0000\), or 100%.
Decision and check. The field is invalid for predicting this target because it directly encodes it. The arithmetic is internally consistent: all four predictions match all four targets. That perfect score does not show that a usable model has been built; it shows that the input contains the answer. If a simple reference rule predicted no claim for every policy in this small data set, its accuracy would be \(3/4=0.7500\), but that comparison does not make the leaked model valid.
Common mistake avoided: treating a perfect confusion table as evidence of model quality without checking how the flag was generated. The matching values are precisely the reason the field must be rejected.
Why Leakage Makes Performance Look Better Than It Is
Evaluation is meaningful only when the inputs used to produce predictions correspond to what would be available for the intended prediction. Let \(g\) be a prediction rule and let \(L\) be a loss function that assigns a penalty to a prediction error. If a leaked field is available in the evaluation data, the measured loss is calculated using that field. But if the field is created from the target records, it cannot serve as an input for a genuine prediction of that same target.
The simplest derivation shows why a direct copy can produce zero error. Suppose a numeric target is \(Y_i\), and a predictor is \(X_i=Y_i\). Consider the rule \(g(X_i)=X_i\), which predicts the target by returning the predictor. Its squared error for observation \(i\) is:
Every observation has zero squared error, so the root mean squared error (RMSE), the square root of the mean squared error, is also zero. The calculation is correct for those supplied inputs, but the result does not describe prediction from information available without the target. The evaluation has answered the wrong operational question.
The same logic applies to a binary outcome and an event flag equal to that outcome: every classification can be correct because the flag provides the label. Transformations can be less obvious but still revealing. If a target is a paid-amount total, then a band such as “positive paid amount” may disclose whether the target is zero, even though it does not disclose the exact amount.
Worked Example: A Claims Total Produces Zero RMSE
A hypothetical insurer defines the target \(Y_i\) as each policy’s total paid claims during a particular year. The file also includes paid_total_year, calculated by summing the same policy’s paid-claim records for that same year. For four policies, both columns contain $0, $80, $0, and $220, respectively. A proposed rule predicts the target by copying paid_total_year. Calculate the RMSE and assess whether it is a valid predictive result.
Method. First compare the target and predictor definitions. Both are sums of the same paid-claim records in the same year, so \(X_i=Y_i\) for all four policies. Then calculate the residuals:
The squared residuals are \(0,0,0,0\). Their mean is \((0+0+0+0)/4=0\), and the RMSE is \(\sqrt{0}=0\) dollars.
Decision and check. The zero RMSE is arithmetically correct but invalid evidence of predictive performance. The predictor is the target reconstructed from the same records. Its use answers, “What is the total when the total is already in the input?” rather than the intended prediction question.
Common mistake avoided: reporting zero RMSE as an exceptional forecasting result. The correct action is to remove the duplicated total and evaluate again with permissible predictors.
Event Flags and Indirect Encodings
Event flags deserve particular attention because they often look like ordinary administrative variables. A flag can be a direct encoding when its definition restates the target event, such as “any paid claim in the outcome year” when the target is “any paid claim in the outcome year.” It can also be a derived indicator, such as a positive-total flag when the target is a nonnegative paid total. In the latter case, the flag does not reveal the size of a positive total, but it does reveal whether the target is zero.
An event flag is not automatically leakage simply because it concerns claims. Its meaning must be compared with the target. A flag from a different, completed historical period might be a potential predictor; a flag generated from the target’s defining event records is outcome-derived. If the documentation does not make the period or construction clear, the field should be treated as unresolved until its source is verified, rather than assumed to be safe.
Worked Example: Auditing a Claims Feature List
A hypothetical analyst is modeling whether a policy has at least one claim during a specified outcome year. The target equals 1 for a policy with one or more claims in that year. The candidate fields are: (1) claim_count_outcome_year, the number of claim records in that year; (2) has_claim_record_outcome_year, set to 1 when at least one such record exists; (3) prior_year_claim_count, the number of claims in the completed year before the target year; and (4) paid_total_positive, set to 1 when the paid total in the outcome year is greater than zero. Classify the fields for this target.
Method. Apply the construction test to each field. The target is an occurrence indicator, so a field that counts the target-year records identifies the target whenever the count is positive. Thus, claim_count_outcome_year is outcome-derived: \(Y_i=I(\text{claim\_count\_outcome\_year}_i>0)\). The has_claim_record_outcome_year flag is defined by exactly the same event as \(Y_i\), so it is a direct encoding. The paid_total_positive flag is also derived from target-year claim outcomes; under the stated definitions, it reveals whether a positive paid claim occurred. It is therefore contaminated for this target.
The prior_year_claim_count is constructed from the completed year before the target year. It is not a direct calculation from the target-year outcome records, so it is not outcome-derived on the information given. It may be considered as a historical candidate after confirming its definition, but this classification alone does not establish that it is suitable for every modeling purpose.
Decision and check. Exclude fields (1), (2), and (4) as outcome-derived for this target; retain field (3) only as a candidate historical predictor subject to the rest of the data specification. This classification follows the record periods and construction rules, not the field names alone.
Common mistake avoided: keeping a target-period claim count because it is numeric while removing only the explicitly named target flag. The count is still a deterministic route to the target’s occurrence indicator.
Common Mistakes in Leakage Audits
A frequent error is to treat a derived field as harmless because it is not literally named after the target. For example, if the target is total paid claims, using “number of paid claims” may not reveal the exact total, but it can still be constructed from the target-period claim records and may encode part of the outcome. The analyst must inspect the field’s lineage and ask which outcome records contributed to it.
Another trap is to calculate a performance metric first and investigate the inputs only after seeing an unusually strong result. A perfect score should prompt an audit, but a less dramatic score does not establish that the predictors are clean. Leakage can be partial: a flag may disclose whether the target is zero, or a component may reveal some of the target’s value.
The wrong calculation below illustrates the central error. Suppose \(Y_i\) is the paid total and \(X_i\) is a field calculated from the same paid records. Reporting the resulting RMSE as evidence of forecasting skill silently treats \(X_i\) as an independent input. The correction is to establish that \(X_i\) is not a duplicate or outcome-derived field before interpreting the metric.
For an actuarial analyst, the practical consequence is important: a model with contaminated inputs may appear useful in a development exercise yet fail as soon as it is asked to predict an outcome that has not been recorded. The remedy is not to explain away the metric or adjust a threshold. It is to correct the predictor set and then reassess performance. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, decide whether the field is outcome-derived for the stated target and check the calculation where requested.
- A hypothetical target is 1 if a policy has any paid claim during a specified year. Across four policies, paid totals are $0, $45, $0, and $90. A field is defined as 1 when that same year’s paid total is positive. What is the field’s accuracy as a classifier of the target, and is it a valid predictor? (Answer: Accuracy is 1.0000; it is not valid because it directly encodes the target.)
- A target is the number of claims recorded for each policy during a particular year. The target values are 0, 2, 1, and 0, and a candidate field is calculated by counting those same year’s claim records. If a rule copies the field as its prediction, what is the RMSE and why? (Answer: RMSE is 0 claims; the field is the target count calculated from the same records.)
- A target is whether any claim occurred during the outcome period. A field called paid_event_flag is 0, 0, 1, 0 for four policies, and the target has the same values. A threshold of 0.5 is used to predict a claim. What are the apparent accuracy and the data-design conclusion? (Answer: Accuracy is 1.0000; the flag is a direct encoding and must be excluded.)
- A target is the paid-claim total for a stated year. Candidate A is a positive-total flag derived from that year’s paid records; Candidate B is the claim count from the completed year before it. Which candidate is outcome-derived for this target, based on the information given? (Answer: Candidate A is outcome-derived; Candidate B is not derived from the target-year records on the information given.)
- A target equals 1 when the paid total during the outcome year exceeds $100. For four policies, totals are $0, $80, $140, and $250. A candidate field is defined as 1 when that same total exceeds $100. A model predicts 1 when the field equals 1. Calculate its accuracy and state the appropriate action. (Answer: Accuracy is 1.0000; exclude the field because it is a direct threshold encoding of the target.)