Why External Data Requires a Separate Fairness Review
A rating variable may originate outside an insurer’s policy and claims systems: a vendor might provide a consumer attribute, a score derived from public records, or a classification assembled from several sources. External origin does not make a variable inherently unfair, but it does create questions that cannot be answered by reviewing predictive performance alone. An insurer needs to understand what the attribute measures, how the data were assembled, whose records are represented, and how errors or proxies could affect consumers.
The review is distinct from, but connected to, the earlier material on causal pathways, proxy variables, and fairness diagnostics. A variable can be associated with expected loss and still raise concerns about data quality, indirect use of protected traits, or unequal effects. Conversely, a difference in group averages is a signal to investigate, not by itself proof that a factor is impermissible or unjustified. Applicable law and regulation vary by jurisdiction, line, and use; actuarial analysis does not replace legal review.
Trace the Data Before Evaluating the Score
A vendor label is not a full definition. A “stability index,” for example, could be based on payment history, address duration, account activity, or a combination of inputs. Each construction has different meanings and risks. The actuarial review should identify the original data sources, the time period represented, the population from which records were collected, the rules for matching records to insureds, the transformations and model versions applied, and the intended interpretation of the output.
The source-to-use path matters. A score calculated from multiple records may embed choices about which records are included, how missing information is handled, and how categories are assigned. Matching errors can attach one person’s attribute to another policyholder. A vendor’s assurance that an input is “consumer data” does not establish that it measures the proposed risk characteristic accurately or consistently.
- Ask what is measured. Distinguish a directly observed event from a vendor’s inference or composite score.
- Ask whose records are available. Check eligibility, coverage, missingness, and matching success for the intended book of business.
- Ask when and how values were created. Identify the observation window, update frequency, definitions, and transformations.
- Ask what use is permitted and explainable. Confirm contractual, privacy, regulatory, and operational requirements, including whether an affected consumer can receive a meaningful explanation where required.
Worked Example: Checking Vendor Coverage Before Comparing Outcomes
A hypothetical insurer evaluates an external record used to assign an applicant to a risk class. In a validation cohort of 1,000 policies, an audit category A contains 500 policies and category B contains 500. The vendor supplies a usable record for 450 policies in A and 350 in B. Among policies with records, 45 claims occur in A and 28 in B. Among policies without records, 8 claims occur in A and 15 in B. Calculate coverage and claim frequencies, then state what the comparison does and does not establish.
Method. Calculate record coverage using each audit category as the denominator, then compare claim frequency among records-present and records-absent policies. The audit categories are used here to check selection patterns; they are not proposed rating variables.
Coverage in A: \(450/500=0.90\), or 90%. Coverage in B is \(350/500=0.70\), or 70%. The 20 percentage-point difference means record availability is not uniform across the two categories.
Claim frequency among records-present policies: A has \(45/450=0.10\), or 10%; B has \(28/350=0.08\), or 8%. The combined records-present frequency is \((45+28)/(450+350)=73/800=0.09125\), or 9.125%.
Claim frequency among records-absent policies: A has \(8/(500-450)=8/50=0.16\), or 16%; B has \(15/(500-350)=15/150=0.10\), or 10%. The combined records-absent frequency is \((8+15)/(50+150)=23/200=0.115\), or 11.5%.
The records-present and records-absent groups differ in both coverage and observed claim frequency. This raises a selection concern: performance calculated only for policies with a vendor record may not describe the full intended population. These data do not show why claim frequencies differ, nor do they establish that record availability caused the difference. The appropriate conclusion is to investigate the missing-record mechanism and evaluate the proposed variable on a population and method that account for it.
Common mistake avoided: treating 73 claims among 800 linked policies as a validation result for all 1,000 policies. That calculation silently excludes 200 policies, whose observed frequency is 11.5%, not 9.125%.
Representativeness Is More Than a Large Sample
A large vendor file can still be unrepresentative. Record availability may vary by age, income, housing circumstances, language, product channel, or other characteristics; the relevant factors depend on the source. Even when an audit does not use protected traits as rating inputs, suitable and lawful analysis may examine whether coverage, error, or score distributions differ across groups. A sample can also be unrepresentative because the vendor collects data only for people who use a particular service or have a particular kind of account.
For each relevant group, compare the proportion with a usable record, the distribution of missing values, match rates, and the observed characteristics of included and excluded policies. Where outcomes are available, compare outcome rates carefully, accounting for exposure and relevant differences in risk. Do not infer that missing records indicate a low or high risk. If missingness is associated with expected loss, a model trained on available records can be miscalibrated when applied to the broader book.
The denominator is essential. A count of missing records alone gives no coverage rate; the rate requires the number of eligible policies in the group. Likewise, differences in claim counts are not interpretable without an appropriate exposure or policy denominator. These simple checks often reveal that an apparently technical score depends on who could be observed in the first place.
Measurement Error Can Change the Apparent Risk Difference
A third-party value may be inaccurate even when it is available for nearly every policy. The intended attribute could be measured at the wrong time, recorded in a coarse category, or inferred from incomplete records. Misclassification can weaken a real relationship, create a misleading one, or affect groups unequally. Its direction and magnitude depend on the error mechanism; it is not safe to assume that all errors merely make a rating factor more conservative.
To see why, let the true low- and high-risk categories each make up half the population, with outcome means \(\mu_L\) and \(\mu_H\). Among records labeled low, a fraction \(c\) are truly low and \(1-c\) are truly high. Their expected outcome is \(c\mu_L+(1-c)\mu_H\). Among records labeled high, the expected outcome is \((1-c)\mu_L+c\mu_H\). Subtracting the first mean from the second gives
Thus, when \(c=1\), the observed difference equals the true difference; when \(c=0.5\), the labels carry no information under these assumptions. This is a justification for examining validation accuracy, not a universal correction formula. In particular, one should not apply it mechanically if the vendor’s error rates vary across groups or are related to outcomes.
Worked Example: Quantifying Symmetric Category Error
A hypothetical insurer studies an external address-duration classification: “short” or “long.” Assume the true categories each occur in half the policies. The true claim frequencies are 4% for short duration and 10% for long duration. A validation study indicates that the recorded category is correct 80% of the time in each true category, with classification error independent of claims given the true category. Calculate the claim frequencies by recorded category and compare their difference with the true difference.
Method. Use the stated equal prevalence and symmetric accuracy to form the true-category mixture within each recorded category. This calculation is valid only under those assumptions.
For policies recorded as short, 80% are truly short and 20% truly long. The expected claim frequency is \(0.80(0.04)+0.20(0.10)=0.032+0.020=0.052\), or 5.2%.
For policies recorded as long, 20% are truly short and 80% truly long. The expected claim frequency is \(0.20(0.04)+0.80(0.10)=0.008+0.080=0.088\), or 8.8%.
The observed difference is \(0.088-0.052=0.036\), or 3.6 percentage points. The true difference is \(0.10-0.04=0.06\), or 6 percentage points. The formula gives \((2(0.80)-1)(0.06)=0.60(0.06)=0.036\), which checks the direct calculation.
The external classification still separates the observed averages, but the difference is smaller than the true-category difference under this particular error model. This does not establish that the factor is suitable: the accuracy may not be stable across groups, the assumed true prevalence may be wrong, and the classification’s source and meaning still need review.
Common mistake avoided: applying 80% accuracy as though the recorded “long” group has an 80% claim frequency. Accuracy describes classification correctness, not claim probability.
Assessing Potential Protected-Trait Proxies
An external attribute may correlate with a protected trait even when the trait is not supplied to the insurer. The relevant question is not only whether the vendor explicitly uses a protected trait. It is also whether inputs, coverage patterns, or constructed scores carry related information into the rate. Earlier material on proxy variables and fairness diagnostics provides tools for identifying associations; for external data, those tools must be paired with a clear account of the source and the specific rating use.
A proxy signal is a reason for a disciplined review, not a complete conclusion. Association alone does not establish why the association exists, whether the variable captures a legitimate risk characteristic, whether it is permitted for the use, or whether less problematic alternatives are available. Assess predictive value and limitations alongside group effects, the variable’s causal or operational rationale, and relevant regulatory constraints. Avoid treating a predictive score’s opacity as a reason to accept a vendor’s explanation without verification.
For a rate factor, calculate the model’s implied average impact across relevant audit groups, while keeping group membership out of the rating calculation unless its use is legally permitted. A difference can arise because groups have different score distributions, because the model assigns different relativities to score values, or both. That decomposition helps locate the concern; it does not alone determine whether the factor should be used.
Worked Example: Separating Score Distribution From Rate Impact
A hypothetical insurer considers a third-party consumer score with two categories. The proposed indicated relativity is 1.20 for the high-score category and 0.80 for the low-score category. In an audit, 70% of policies in group A and 30% in group B are assigned the high score. Assume the same base rate applies before this factor and that these percentages are based on the intended policy population. Calculate the average factor for each group and explain the fairness interpretation.
Method. Calculate a weighted average of the proposed relativities using each group’s score distribution. This measures the factor’s average multiplicative impact, not the groups’ total premium or expected losses from all rating variables.
For group A, the high-score share is 0.70 and the low-score share is \(1-0.70=0.30\). Its average factor is \(0.70(1.20)+0.30(0.80)=0.84+0.24=1.08\).
For group B, the high-score share is 0.30 and the low-score share is \(1-0.30=0.70\). Its average factor is \(0.30(1.20)+0.70(0.80)=0.36+0.56=0.92\).
The average factors differ by \(1.08-0.92=0.16\). Relative to group B’s factor, the difference is \(0.16/0.92=0.1739\), or approximately 17.39%. The disparity arises here from different score distributions because the proposed relativity is identical for a given score in both groups.
This result identifies a material group-level impact to investigate. It does not establish that the score is a protected-trait proxy, that the factor is unlawful, or that the score predicts expected loss. Those questions require evidence about the score’s construction, its relationship to outcomes, its measurement quality, and applicable constraints.
Common mistake avoided: reporting 1.20 as the average factor for group A because 1.20 is the high-score relativity. The low-score share must also be included in the weighted average.
Document a Defensible Use Decision
An external-data review should end with a decision and conditions, not merely a list of correlations. Depending on the findings, an insurer may use the variable with controls, seek additional validation, restrict the population or purpose, modify the construction, or decline to use it. Any decision should be consistent with applicable requirements and supported by evidence proportionate to the variable’s effect on rates.
Record the source, definitions, observation period, matching process, transformations, vendor version, and intended rating use.
Calculate record availability and match rates using clear denominators. Compare included and excluded records overall and across relevant audit groups.
Where possible, compare the external value with a suitable reference measure. Quantify error and check whether error rates vary across groups or relate to outcomes.
Evaluate predictive evidence and group impacts for the actual proposed use. Distinguish a screening signal from a conclusion about causation, fairness, or compliance.
Document limitations, permitted uses, monitoring, vendor-change review, and the conditions that would prompt revalidation or suspension.
This process matters because third-party data can change without a rating model changing. A vendor may revise a source, matching rule, or score definition, and the insurer may then be using a materially different attribute under the same label. Record lineage, version control, and revalidation are therefore part of fairness oversight as well as ordinary model governance.
Check Your Understanding
Work each problem using the stated assumptions. The audit groups below are diagnostic categories, not proposed rating variables.
- A vendor returns a usable record for 360 of 400 policies in group A and 240 of 400 policies in group B. Calculate coverage for each group, overall coverage, and the coverage gap in percentage points. (Answer: A 90%; B 60%; overall 600/800 = 75%; gap 30 percentage points.)
- True categories are equally prevalent, with claim frequencies 3% and 9%. A recorded category is correct 90% of the time in each category, and errors are independent of claims conditional on the true category. Calculate the observed claim frequencies for recorded low and recorded high, and their difference. (Answer: recorded low 3.6%; recorded high 8.4%; difference 4.8 percentage points.)
- A score has relativities 1.15 for high and 0.85 for low. The high-score share is 60% in group X and 30% in group Y. Calculate each group’s average factor and the absolute difference. (Answer: X \(0.60(1.15)+0.40(0.85)=1.03\); Y \(0.30(1.15)+0.70(0.85)=0.94\); difference 0.09.)
- In a cohort, 50 policies lack a vendor score and 10 have claims; 450 policies have a score and 36 have claims. Calculate claim frequencies for the missing-score and scored policies. Why is the scored-only rate not automatically valid for the whole cohort? (Answer: missing-score frequency \(10/50=20\%\); scored frequency \(36/450=8\%\). The groups differ substantially, so using only scored policies requires investigation of selection and missingness.)
- A vendor says its consumer score is “validated” but provides no source history, match-rate information, error rates, or description of transformations. State two distinct pieces of evidence needed before a fairness assessment can be complete. (Answer: examples include documented provenance and transformations; coverage and match rates for the intended population; validation against a suitable reference measure; or error and outcome comparisons across relevant groups.)