Why Provenance Matters
A model can use a field named claim amount, but that name alone does not establish what the field measures. It might be a payment, an incurred amount, a reserve, or a total aggregated across several records. Likewise, customer tenure might be calculated from a policy system, a customer relationship system, or a manually maintained file. These differences matter: the same label can refer to different quantities, while different labels can sometimes represent the same underlying concept.
Data provenance records the origin and construction of a variable. For each outcome and predictor, an analyst should be able to explain its business meaning, the source system or process that produced it, the relevant source field or event, and any transformations used to create the modeling value. This documentation helps reviewers assess whether a variable measures the intended concept and whether an analysis can be reproduced.
From Business Concept to Modeling Variable
Start with the quantity the analysis needs, not with whichever column is easiest to obtain. For example, a business question might concern the total cost of a claim event. A payment-system field called amount may record individual cash transactions, while the analysis needs an event-level total that also reflects a reserve at a specified valuation point. The source field and the modeling variable are related, but they are not interchangeable.
A useful way to express the distinction is to let \(Z\) represent the intended business quantity, \(R\) represent the source records, and \(g\) represent the documented processing rule. The value supplied to the analysis is then \(Y=g(R)\). The processing rule is part of the definition of \(Y\); it is not merely a technical detail. If another analyst uses a different source field or a different rule \(g\), that analyst may obtain a different variable even if both versions have the same column name.
For each variable, a compact provenance record should answer questions such as these:
- Meaning: What business concept is intended, and what does one value represent?
- Origin: Which system, process, form, or event first records the information?
- Source definition: What does the source field mean, including its unit and any stated conventions?
- Construction: Is the modeling value copied directly, selected from several fields, aggregated, converted, or otherwise derived?
- Accountability: Which business or data owner can clarify the definition and resolve questions about the source?
This is a documentation task, not a reason to assume that one system is always correct. A source may be authoritative for one purpose and unsuitable for another. The rule for selecting or reconciling sources should be explicit and connected to the intended variable.
A Short Justification: Why the Processing Rule Is Part of the Variable
Suppose a source record \(R\) can be transformed in two different ways, \(g_1\) and \(g_2\). The resulting modeling values are \(Y_1=g_1(R)\) and \(Y_2=g_2(R)\). Unless the rules produce the same value for every relevant record, the two variables are not equivalent.
That conclusion follows directly from applying each rule to the same source value: the output depends on both the input and the rule. Therefore, documenting only a column name or source system is insufficient when processing changes the value. To reproduce the variable, another analyst needs the source definition and the transformation rule.
Worked Examples
Worked Example: Defining a Claim-Cost Outcome
Hypothetical scenario. An insurer wants an outcome representing the total incurred cost for each property claim event at a stated valuation date. For one event, the claim accounting ledger shows two indemnity payments of $1,200 and $450, an outstanding case reserve of $350, and a recorded recovery of $100. The payment system also has a claim-header field showing $1,650. Define and calculate the intended outcome, and identify the provenance details that should be recorded.
Method. Begin with the business definition, then identify which source records contribute to it. For this example, define incurred cost at the valuation date as cumulative indemnity payments plus the outstanding case reserve less the recorded recovery. Assume all amounts are in U.S. dollars, the two payments belong to the same claim event, the reserve is measured at the stated valuation date, and the recovery is expressed as a positive amount to subtract.
Step 1: Total the payments. The payment records give \(1{,}200+450=1{,}650\) dollars. The claim-header field agrees with this payment total, but it does not include the reserve or recovery under the stated definitions.
Step 2: Add the reserve. The amount before recovery is \(1{,}650+350=2{,}000\) dollars.
Step 3: Subtract the recovery. The defined incurred-cost outcome is \(2{,}000-100=1{,}900\) dollars.
Provenance record. Document the outcome as event-level incurred cost in dollars at the specified valuation date; identify the accounting ledger as the source for payments, reserve, and recovery; state the source meanings of those fields; and record the calculation \(1{,}200+450+350-100\). Also note that the claim-header amount represents payments only in this scenario.
Check. The result includes the $350 reserve and excludes $100 of recovery, so it is $250 greater than the $1,650 payment total. The arithmetic gives \(1{,}900\), consistent with that comparison.
Common mistake avoided. Treating the header field as the outcome would give $1,650. That is a valid payment total under the stated source definition, but it is not the defined incurred-cost outcome.
Worked Example: Tracing a Derived Mileage Predictor
Hypothetical scenario. A pricing analysis needs an annualized vehicle-mileage predictor. A telematics process records cumulative odometer readings of 18,240 miles at the start of a 90-day measurement period and 23,040 miles at its end. Assume the readings refer to the same vehicle, the period contains exactly 90 days, and the intended derived value is the observed distance scaled to 365 days. Calculate the predictor and describe its provenance.
Method. Use the difference between the cumulative readings to obtain distance during the measurement period. Then apply the stated annualization rule. The rule is part of this example’s predictor definition; it should not be mistaken for a directly observed annual odometer reading.
Step 1: Calculate measured distance. The change in odometer reading is \(23{,}040-18{,}240=4{,}800\) miles over 90 days.
Step 2: Calculate the daily rate implied by the observation. The observed rate is \(4{,}800/90=53.3333\ldots\) miles per day.
Step 3: Scale to 365 days. The annualized value is \(4{,}800\times(365/90)=19{,}466.6666\ldots\) miles. Rounded to the nearest whole mile, the predictor is 19,467 miles per year.
Provenance record. State that the source is the telematics odometer-reading process, the source values are cumulative miles, and the derived variable is an annualized mileage estimate. Record the two readings, the 90-day period, the 365-day scaling convention, and the rounding rule.
Check. Multiplying the unrounded daily rate by 365 gives \(53.3333\ldots\times365=19{,}466.6666\ldots\), which agrees with the direct calculation.
Common mistake avoided. Dividing 4,800 by 365 would treat the 90-day distance as though it were a full year’s distance. The stated annualization instead scales the 90-day distance by \(365/90\).
Worked Example: Resolving Conflicting Source Definitions for Building Age
Hypothetical scenario. A property model uses building age in whole years as a predictor, measured at the start of 2025. The property record lists a construction year of 2010. A municipal assessor record lists 2008, and the insurer’s documented rule for this analysis designates the assessor’s construction year as the source for building age. Assume both sources refer to the same building and age is calculated as the index year minus construction year, with no partial-year adjustment. Calculate the predictor and state the provenance decision.
Method. Follow the stated source-authority rule rather than averaging conflicting values or choosing the value that seems more plausible. Then apply the defined age calculation.
Step 1: Select the source value. The rule designates the assessor record, so the construction year used is 2008. The insurer property-record value of 2010 is retained as a conflicting source value for documentation, not substituted into this calculation.
Step 2: Calculate age. The index year is 2025, so building age is \(2025-2008=17\) years.
Provenance record. Define the predictor as whole-year building age at the start of 2025; document the assessor record as the designated source for construction year; state the calculation rule; and record that the insurer property record gave 2010. Naming the difference makes the selection reviewable.
Check. Under the specified rule, the two year values differ by two years. The age using 2008 is 17; using 2010 would produce 15. The selected answer is therefore 17, not an average of 16.
Common mistake avoided. Averaging the two construction years would create a value of 2009 and an age of 16, even though the documented authority rule selects one source. Averaging is not a neutral resolution; it changes the variable’s definition.
Documenting Provenance in Practice
A provenance record can be maintained as a data dictionary or in a model-development specification. The appropriate format depends on the organization, but the information should be specific enough that a reviewer can distinguish a raw source field from a derived model variable. The following fields are often useful:
| Documentation item | Question it answers | Example of a precise entry |
|---|---|---|
| Model-variable name and meaning | What does the analysis use this value to represent? | Incurred property-claim cost per event at the stated valuation date |
| Source system and process | Where is the information first recorded or maintained? | Claim accounting ledger; payment, reserve, and recovery processes |
| Source field or event definition | What does each source value mean? | Payment amount is a cash payment; reserve is the outstanding case estimate |
| Unit and convention | How should values be interpreted? | U.S. dollars; recovery stored as a positive amount and subtracted |
| Transformation | How is the source converted to the model value? | Sum payments, add reserve, subtract recovery |
| Decision owner or authority rule | Who defines the process or resolves source conflicts? | Assessor construction year selected by the documented analysis rule |
For outcomes, be particularly explicit about which business event is represented and what the recorded value counts or measures. For predictors, describe the source concept and any conversion that gives it the meaning used by the model. Earlier material on outcome definitions and predictor availability addresses the analysis design questions of what is measured and when it may be used; provenance adds the complementary question of where the recorded value came from and how it was defined.
Provenance is also useful when source systems use familiar but ambiguous labels. A field called status might mean a workflow stage in one system and a coverage state in another. A field called date might refer to an event date, an entry date, or a processing date. The field name is not a definition. Record the meaning supplied by the source process and the rule used to turn it into the analysis variable.
Common Traps and Review Checks
- Assuming a column name is a complete definition. Names such as amount, count, and date need a business meaning, unit, and source convention.
- Confusing source values with derived values. If a predictor is calculated, document both the inputs and the calculation, including any scaling or rounding.
- Blending conflicting sources without a rule. Selecting, averaging, or prioritizing values affects what the variable means. State the rule and why it is appropriate for the intended use.
- Recording only the system name. A system can contain fields created by different processes. Identify the relevant field or event and its definition.
- Failing to check units and signs. A distance recorded in kilometers cannot be treated as miles without conversion, and a recovery may be stored with a sign convention that changes the calculation.
A practical review is to take one modeled value and trace it backward: identify the modeling definition, reproduce any transformation, locate the source fields or events, and explain what those source values mean. If that path cannot be described clearly, the variable’s provenance is incomplete.
Check Your Understanding
For each situation, identify the provenance issue or calculate the defined modeling value. Each scenario is hypothetical.
- A model outcome is defined as paid benefits plus an outstanding reserve. The source ledger records paid benefits of $2,400 and reserve of $600, while a summary field contains $2,400. What is the defined outcome, and why is the summary field insufficient? (Answer: $3,000; the summary field records paid benefits only under the stated definition.)
- A source process records 3,600 miles over 60 days. The documented predictor is distance annualized to 365 days. Calculate it to the nearest whole mile. (Answer: 21,900 miles per year.)
- Two systems report a customer’s start year as 2014 and 2016. The documented rule selects the membership system’s value, 2014, and tenure is measured in whole years at the start of 2025. What tenure is used, and what should happen to the other value in the provenance documentation? (Answer: 11 years; record 2016 as a conflicting source value rather than averaging or silently discarding it.)
- A predictor is described only as “monthly account balance.” Name three provenance details needed before another analyst can reliably interpret it. (Answer: Examples include the source system and field or process, whether the value is a month-end or other balance, and the currency or unit; the extraction or aggregation rule should also be documented if the model value is derived.)
- A recorded value is calculated as 2,750 plus 500 minus a recovery of 125. The source definition says the first amount is paid cost, the second is an outstanding reserve, and recovery is stored as a positive amount to subtract. What is the modeled value, and what transformation belongs in its provenance record? (Answer: $3,125; document paid cost plus reserve minus recovery, or \(2{,}750+500-125\).)