Why Historical Decisions Matter
A training data set may appear to contain a simple outcome such as “fraud,” “accepted,” or “claim paid.” Yet a recorded outcome can reflect not only the condition an analyst wants to understand, but also the institution’s earlier rules for investigating, reviewing, approving, or recording cases. If the model treats that historical record as ground truth, it may learn the institution’s past decisions rather than the underlying condition.
This distinction matters whenever model predictions will inform decisions about people or claims. A model that accurately predicts past approvals may be useful if the objective is to reproduce those approvals. It is not automatically a good model of underlying risk, eligibility, or fraud. The target must be defined in terms of the question the analysis is meant to answer. This treatment is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
For example, “confirmed fraud” may mean that fraud was established after an investigation, not that every fraudulent case in the population was identified. Likewise, “accepted” records a past underwriting decision; it does not, by itself, define the applicant’s underlying risk. The distinction is not merely semantic. If the label-making process differs across populations or changes over time, the recorded training outcomes can differ even when the underlying condition does not.
How a Decision Process Shapes the Label Rate
Let \(C=1\) denote that a condition of interest is present, and let \(L=1\) denote a positive recorded label. Define the prevalence \(\pi=P(C=1)\), the positive-label rate among cases with the condition \(s=P(L=1\mid C=1)\), and the positive-label rate among cases without the condition \(f=P(L=1\mid C=0)\). These rates describe how the label-making process records the underlying condition.
This expression follows by splitting the population into the mutually exclusive cases \(C=1\) and \(C=0\). Every positive label must come from one of those two cases, so their contributions to the total positive-label rate add. The formula is useful only when the stated conditional rates describe the label process in the population being studied. It does not make a label historically neutral; it makes the effect of that process explicit.
If \(\pi\), \(s\), and \(f\) are known, the equation can be solved for the underlying prevalence:
The result is not a general-purpose correction. It depends on having defensible values for the label rates and on those values applying to the population at issue. If historical records do not reveal how decisions were made, the observed label rate alone generally cannot identify the condition’s prevalence.
Worked Example: Different Historical Classification Rates
A hypothetical organization reviews cases from Groups A and B. The underlying condition is present in 20% of each group. A positive historical label is recorded for 90% of Group A cases with the condition and 50% of Group B cases with it. For cases without the condition, the positive-label rate is 20% in both groups. What positive-label rates should be expected, and do they imply different underlying prevalence?
Method and assumptions. Let \(C\) indicate the condition and \(L\) indicate the recorded positive label. In each group, take \(C\) to have a Bernoulli distribution with parameter 0.20. Conditional on \(C\), use the stated Bernoulli label rates. The rates are treated as stable within each group. There is no interest-rate or mortality assumption in this classification problem.
Group A. The share with the condition is 0.20, and the share without it is \(1-0.20=0.80\). Therefore:
Group B. The same prevalence applies, but the positive-label rate among cases with the condition is 0.50:
The observed positive-label rates are 0.34 and 0.26, an 8-percentage-point difference, even though both groups have underlying prevalence 0.20. To check whether the assumed rates recover that prevalence, invert the formula. For Group A, \((0.34-0.20)/(0.90-0.20)=0.14/0.70=0.20\). For Group B, \((0.26-0.20)/(0.50-0.20)=0.06/0.30=0.20\). Both calculations return the stated prevalence.
Common mistake avoided. It is wrong to calculate \(0.80(0.20)\) as 0.04. The product is 0.16; 0.04 would be \(0.80(0.05)\), using a different rate. Substituting 0.04 would produce incorrect totals of 0.22 and 0.14. The non-condition contribution must use the specified 0.20 rate in both groups.
Historical Decisions as Model Targets
A label may directly represent an earlier institutional action rather than a finding about the underlying condition. Suppose a model is trained to predict whether an application was accepted. Even if acceptance was based partly on risk, the target is still the past acceptance decision. The model can learn patterns associated with the old rule, including differences in how that rule was applied. Calling its output a risk estimate would require separate evidence that acceptance is a valid measure of the risk of interest.
Worked Example: Predicting Historical Underwriting Acceptance
A hypothetical insurer reviews 1,000 applications in each of two groups. In each group, 40% meet a separately defined eligibility standard, so each group has 400 eligible and 600 ineligible applicants. Under an earlier process, Group A applicants meeting the standard were accepted 90% of the time, while Group B applicants meeting it were accepted 50% of the time. Ineligible applicants in either group were accepted 10% of the time. Calculate the expected acceptance rates and explain what an acceptance-label model targets.
Method and assumptions. Define \(E=1\) as meeting the specified eligibility standard and \(L=1\) as accepted under the historical process. Treat eligibility as Bernoulli with parameter 0.40 in each group, and apply the stated conditional acceptance probabilities. Counts below are expected counts from these rates. The example concerns decisions, so no interest-rate or mortality assumption applies.
For Group A, expected acceptances among eligible applicants are \(400(0.90)=360\). Among ineligible applicants, they are \(600(0.10)=60\). Total expected acceptances are \(360+60=420\), giving an acceptance rate of \(420/1{,}000=0.42\), or 42%.
For Group B, expected acceptances among eligible applicants are \(400(0.50)=200\). Among ineligible applicants, they are again \(600(0.10)=60\). Total expected acceptances are \(200+60=260\), giving an acceptance rate of \(260/1{,}000=0.26\), or 26%.
The eligibility rate is 40% in both groups, but the historical acceptance rates differ by 16 percentage points. A model trained on \(L\) is being trained to predict the old acceptance process. The calculation does not establish that the eligibility standard is the only relevant objective, nor does it establish what decision should replace the historical one. It does show why acceptance labels cannot automatically be interpreted as objective measurements of risk.
Common mistake avoided. It would be incorrect to describe the 42% and 26% acceptance rates as the groups’ eligibility rates. Eligibility is 40% for both; acceptance is a separate outcome generated by the earlier process.
When Labels Depend on Investigation
In claims or fraud settings, a positive label may require a sequence of institutional actions. A case may need to be selected for investigation and then confirmed before it receives a positive label. If cases not investigated are recorded as negative, the label combines the underlying condition with the investigation and confirmation rules. This is a specific form of historical decision dependence: the label is partly created by what the institution chose to do.
Worked Example: Changes in Fraud Investigation Practice
A hypothetical organization processes 1,000 files in each of two periods. The underlying fraud prevalence is 10% in both periods. A file is labeled positive only if it is investigated and fraud is confirmed. Among investigated files, the confirmation process records a positive label for 80% of fraudulent cases and 5% of nonfraudulent cases. In the earlier period, 50% of fraudulent files and 10% of nonfraudulent files are investigated. In the later period, those investigation rates rise to 90% and 30%, respectively. Assume uninvestigated files are stored as label zero. Calculate the expected positive-label rates.
Method and assumptions. Let \(C=1\) indicate actual fraud, \(I=1\) indicate investigation, and \(L=1\) indicate a positive confirmation label. Take fraud status as Bernoulli with parameter 0.10 in each period. Use the stated conditional probabilities for investigation and confirmation, and assume a positive label requires both investigation and confirmation. The results are expected rates under these assumptions, not observed counts. No interest-rate or mortality assumption applies.
In the earlier period, the positive-label rate among all files is the sum of two routes to a positive label: a fraudulent file is investigated and confirmed, or a nonfraudulent file is investigated and incorrectly confirmed.
Thus the expected positive-label rate is 4.45%, corresponding to \(1{,}000(0.0445)=44.5\) positive labels in expectation.
In the later period, the underlying prevalence and confirmation rates are unchanged, but the investigation rates differ:
The expected positive-label rate is now 8.55%, or \(1{,}000(0.0855)=85.5\) positive labels in expectation. The underlying fraud prevalence remains 10% in both periods. The increase in recorded positives follows from the changed investigation process under the assumptions stated.
Common mistake avoided. Treating every uninvestigated file stored as zero as a verified nonfraud case confuses “not confirmed” with “confirmed absent.” A model trained on these labels can learn the institution’s investigation and confirmation pathway as well as patterns associated with fraud. The calculations do not establish a corrected fraud rate unless the relevant process rates are known and applicable.
A Practical Audit of Historical Labels
Before fitting or interpreting a model, make the label-generation process explicit. The aim is not to assume that every historical decision was inappropriate. It is to determine what the recorded target means, whether it matches the analytical question, and whether past decision rules could explain important patterns in the data.
Write down the underlying condition of interest separately from the database field used as the target. For example, distinguish “fraud occurred” from “fraud was confirmed.”
Record which actions can create or prevent a positive label, such as referral, investigation, approval, denial, or confirmation. Identify how cases receiving no action are coded.
Check whether decision criteria or practices changed. A change in the label rate may reflect a change in the condition, the decision process, or both.
If the task is to predict historical decisions, say so. If the task concerns an underlying condition, establish why the available label is a defensible measure of that condition.
If using an adjustment, identify the required label rates and explain why they are applicable. If they cannot be justified, report that the underlying condition is not identified by the recorded labels alone.
A common analytical trap is to treat a large sample as a remedy for a poorly defined label. More data can reduce sampling variability in the observed label rate, but it does not remove the effect of a historical decision rule. A precisely estimated rate of past approvals is still a rate of past approvals. Likewise, a stable pattern of confirmed cases does not by itself show that the same share of all cases had the underlying condition.
Check Your Understanding
For each problem, distinguish the underlying condition from the recorded label and show the relevant calculation.
- A hypothetical review process has a condition prevalence of 30%, labels 80% of cases with the condition as positive, and labels 10% of cases without it as positive. What positive-label rate is expected? (Answer: 0.3100, or 31.00%)
- Two groups each have an underlying condition prevalence of 25% and a 10% positive-label rate among cases without the condition. The positive-label rates among cases with the condition are 80% for Group A and 40% for Group B. Find each observed positive-label rate and their difference. (Answer: Group A 0.2750; Group B 0.1750; difference 0.1000, or 10 percentage points)
- In two groups, 50% of applicants meet a stated eligibility standard. Historical acceptance among eligible applicants is 70% in Group A and 40% in Group B; acceptance among ineligible applicants is 5% in both groups. Calculate each group’s historical acceptance rate. (Answer: Group A 0.3750; Group B 0.2250)
- A group has a positive-label rate of 22%. Suppose the label process records 70% of cases with the condition as positive and 10% of cases without it as positive. Under these assumed rates, calculate the implied condition prevalence. (Answer: \((0.22-0.10)/(0.70-0.10)=0.2000\), or 20.00%)
- In a hypothetical period, 5% of files have an underlying condition. Of files with the condition, 80% are investigated; of files without it, 20% are investigated. Among investigated files, positive confirmation rates are 75% with the condition and 4% without it. If uninvestigated files are stored as negative, calculate the expected positive-label rate. (Answer: \(0.05(0.80)(0.75)+0.95(0.20)(0.04)=0.0376\), or 3.76%)