Why Identifiers and Codes Need Scrutiny
A record number may look like a harmless label, and an administrative code may look like a useful summary of a case. Either can become a predictor in a model. But a strong association with the outcome does not, by itself, show that the field contains information that will be available and meaningful when predictions are made. A number might be assigned after an event, or a code might record a workflow decision that depends on the outcome or on who was selected for further review.
The practical question is not just whether a field is associated with the target. It is what process created the field, when it was created, and whether that process will be the same when the model is used. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Identifiers are often essential for joining tables, removing duplicate records, or tracing an observation back to its source. Those uses do not make them suitable as model inputs. Similarly, an administrative code may be useful operationally while still being inappropriate as a predictor. Treating a record number as a number, or a code as a stable category, can let a model learn the data-collection process rather than a relationship that will persist in use.
Trace the Field Back to Its Creation
For each candidate identifier or code, establish its purpose and assignment rule. Ask who or what creates it, when it is assigned, whether it can be revised, and whether assignment depends on information recorded after the prediction point. Also ask whether the code has the same meaning across time, business units, and data sources.
- Timing: Was the field known at the point when the prediction would be made?
- Assignment: Is it generated mechanically, chosen by a person, or assigned in response to a developing outcome?
- Meaning: Does the value represent a stable concept, or only an internal workflow step?
- Selection: Does the field indicate which records were investigated, completed, retained, or otherwise observed?
- Transport: Will the same assignment rule and code map apply to future records and the intended population?
A field that fails one of these checks is not automatically useless. It may be suitable for a narrowly defined task if it is available at prediction time and its assignment process is expected to persist. But its apparent predictive value needs interpretation before it is used. An identifier should normally remain a key, not a predictor; an administrative code needs a defensible, prospective meaning.
Worked Example: Claim Numbers Assigned at Closure
A hypothetical insurer is considering a model to flag new claims for early review. The target is whether a claim will require a large reserve increase within 60 days. The data team finds that claims with larger record numbers have higher target rates. In a sample of 20 claims, IDs 5001–5010 include 1 target event, while IDs 5011–5020 include 7. Assess the apparent signal.
Method. Calculate the outcome rate in each ID group, then compare the field’s creation time with the intended prediction time.
For IDs 5001–5010, the rate is \(1/10=0.10\), or 10%. For IDs 5011–5020, the rate is \(7/10=0.70\), or 70%. The observed rate difference is \(0.70-0.10=0.60\), or 60 percentage points. Across all 20 claims, there are \(1+7=8\) events, so the overall rate is \(8/20=0.40\), or 40%.
The difference is large in this small sample, but the assignment rule is decisive: the claims administrator confirms that these IDs are generated when the file is closed, after the 60-day development period. The ID is therefore unavailable when early review decisions must be made. Its ordering may also reflect the order of file completion, which can itself be affected by case development.
Conclusion. Keep the ID for record retrieval and joining, but exclude it from the early-review predictors. The computed rates describe this extract; they do not establish a usable prospective relationship. Common mistake avoided: interpreting a strong difference in observed rates as proof that a field is available and predictive at the intended decision point.
How Workflow Can Create an Association
A code can be associated with an outcome even when it does not represent a stable characteristic of the underlying case. For example, a code may be assigned because a case was escalated, reviewed, or routed through a particular workflow. Those steps may be related to outcome development, or to which outcomes are eventually recorded.
The following identity makes the mechanism explicit. Let \(Y\) be a binary outcome, \(C\) a code, and \(Z\) a workflow stage. Assuming the stages in \(Z\) form a complete, non-overlapping set, the law of total probability gives:
If, within each workflow stage, the code itself adds no outcome information, then \(P(Y=1\mid Z=z,C=c)=P(Y=1\mid Z=z)\). Substitution gives:
Thus, codes can have different observed outcome rates simply because they occur in different mixes of workflow stages. The formula does not prove that a code is invalid: a workflow difference may sometimes be available and useful for a defined prediction task. It does show why the observed association alone cannot establish that the code represents stable case risk.
Worked Example: A Review Code and Selective Observation
A hypothetical benefits administrator studies whether an application contains a confirmed documentation error. Code A means an application was routed for intensive review; code B means routine processing. In the source population, code A appears on 20 applications: 4 have an error and 16 do not. Code B appears on 80 applications: 8 have an error and 72 do not. However, the analysis extract includes only applications with a completed review. For A, it contains 4 error cases and 6 non-error cases. For B, it contains 4 error cases and 36 non-error cases. Calculate the source and extract rates, then interpret the difference.
Method. Compute the outcome rate separately in the source population and in the extract. Then compare the number retained in each outcome group to identify how observation differs.
In the source population, code A has rate \(4/(4+16)=4/20=0.20\), or 20%. Code B has rate \(8/(8+72)=8/80=0.10\), or 10%.
In the completed-review extract, code A has rate \(4/(4+6)=4/10=0.40\), or 40%. Code B has rate \(4/(4+36)=4/40=0.10\), or 10%.
For code A, the extract retains \(4/4=100\%\) of error cases and \(6/16=37.5\%\) of non-error cases. For code B, it retains \(4/8=50\%\) of error cases and \(36/72=50\%\) of non-error cases. The extract therefore changes the outcome composition for A but not B. The A-versus-B rate difference is 10 percentage points in the source population \((20\%-10\%)\), but 30 percentage points in the extract \((40\%-10\%)\).
Conclusion. In this hypothetical example, the code is tied to both review routing and differential completion of review. The extract’s rates do not represent the source rates, so treating the code’s apparent association as a stable risk relationship would be misleading. Common mistake avoided: computing rates only among completed records and assuming those rates describe every application.
When a Code May Be a Defensible Predictor
Not every administrative code is a leak or an artifact. A category assigned before the prediction point can be a defensible predictor when its meaning is clear and the assignment rule is expected to remain consistent. For instance, a product category fixed when a policy is issued may contain relevant information for a later prediction. The key is to use the category’s meaning, not merely its stored label, and to verify that future records use the same mapping.
Worked Example: A Product Code Whose Labels Change
A hypothetical insurer predicts whether a policy will have a service request during the next year. At policy issue in the training data, code 1 means Basic and code 2 means Plus. Among 40 Basic policies, 4 have a request; among 40 Plus policies, 12 have a request. Before a later data extract is prepared, the administrator reverses the numeric labels: code 1 now means Plus and code 2 means Basic. Determine the training rates and explain how to use the field.
Method. Calculate rates by semantic product category, then distinguish those categories from their temporary numeric labels.
For Basic, the training rate is \(4/40=0.10\), or 10%. For Plus, it is \(12/40=0.30\), or 30%. The overall training rate is \((4+12)/(40+40)=16/80=0.20\), or 20%.
The higher rate belongs to Plus, not intrinsically to the number 2. In the later extract, code 1 means Plus, so reading the old code map would incorrectly assign the 30% training rate to Basic. The appropriate field is the product category after applying the correct, dated code map. If the map cannot be established reliably, the raw code should not be treated as though its meaning were stable.
Conclusion. Product category is available at issue and has a coherent meaning in this scenario, so it may be considered as a predictor subject to checking that the mapping is correct. The raw numeric code is not a stable quantity. Common mistake avoided: assuming that code values have a permanent order or meaning just because they are stored as numbers.
Practical Decision Rule
A compact review can prevent many proxy problems. For an identifier or administrative code, record the field’s purpose, assignment time, assignment rule, and mapping history. Compare these facts with the prediction point and intended population. If the field is unavailable then, depends on later outcomes, or has unstable meaning, exclude it or reconstruct a valid pre-prediction version. Retain identifiers separately when needed for data management.
A code can still reflect a workflow that is genuinely part of the operating environment. In that case, document what it represents and consider whether the same workflow will apply when predictions are used. If codes reveal review intensity or observation status, distinguish that process from the outcome risk of the full target population. A predictive relationship that relies on a temporary process may disappear when procedures, staffing, or code definitions change.
The central distinction is between association in the assembled data and information that is available, interpretable, and stable at the intended prediction point. Rates and other summaries can reveal suspicious patterns, but resolving them requires understanding how the records were produced.
Check Your Understanding
For each situation, distinguish the observed association from what can defensibly be used as a predictor.
- A hypothetical repair dataset has 12 failures among 30 cases with a high-numbered work-order range and 3 failures among 30 cases with a low-numbered range. Work orders are assigned when repairs are completed, but the model is intended to flag new repairs. State the two rates and the appropriate treatment of the work-order number. (Answer: High range \(12/30=40\%\); low range \(3/30=10\%\). The number is assigned after completion, so retain it for record tracking but exclude it as a predictor for new repairs.)
- A hypothetical intake code is assigned before review. In the source population, code R has 6 positive outcomes and 24 negative outcomes; code S has 4 positive and 36 negative. Calculate both rates. (Answer: R is \(6/30=20\%\); S is \(4/40=10\%\). The rates show an association, but the code’s assignment process and stability still need checking.)
- In a hypothetical extract, code M has 8 positive and 12 negative outcomes, while code N has 10 positive and 40 negative outcomes. The extract contains only completed investigations. Calculate the rates and name one additional check needed before using the code. (Answer: M is \(8/20=40\%\); N is \(10/50=20\%\). Check whether investigation completion or inclusion differs by code and outcome before generalizing the rates.)
- A hypothetical service tier is fixed at enrollment and has 9 requests among 45 Standard members and 18 among 60 Enhanced members. Calculate the rates and state the code condition that must hold for it to be a defensible predictor. (Answer: Standard is \(9/45=20\%\); Enhanced is \(18/60=30\%\). The tier must be known by the prediction point and consistently defined for the records to which the model is applied.)
- A hypothetical administrative field uses codes 1 and 2. In the development data, 1 means Local and 2 means Regional; in a later extract, the mapping is reversed. State the necessary preparation before applying a model that used these categories. (Answer: Use a reliable, version-specific mapping to recover the semantic categories and encode them consistently; if the mapping cannot be verified, do not treat the raw values as stable predictors.)