Why Data Provenance Matters
A prediction is only as interpretable as the data that produced it. A model may calculate a score correctly while the underlying records combine different definitions, omit part of the target population, or contain transformations whose effects are undocumented. In those situations, the numerical output can be precise yet fail to answer the operational question being asked.
Data provenance describes where data came from and how they were collected, extracted, linked, transformed, and supplied to a model. A source name alone is not enough. For example, “claims database” does not specify whether the records count paid claims or reported claims, which dates determine inclusion, or how exposure was measured. Those details determine what a model’s input actually represents.
Provenance and quality are related but distinct. Provenance answers “where did this value come from, and what happened to it?” Quality asks whether the resulting data are suitable for the intended use. A documented transformation can still be unsuitable; an undocumented one is difficult to assess. Useful checks include whether records are complete for the intended population, whether definitions are consistent across sources and time, whether units are valid, and whether the data are current enough for the decision.
For a predictive workflow, a practical lineage trace follows the data through five points: the original source, the extraction rule, any joins, each transformation, and the final model input. The trace should also identify the relevant dates and the responsible system or process. This helps distinguish a limitation in the source from one introduced during data preparation.
From Data Limitation to Decision Caveat
A useful caveat is specific and decision-relevant. “The data may be incomplete” leaves a decision-maker unable to judge the consequence. A stronger statement identifies which records may be absent, how they enter the analysis, what conclusion is uncertain, and what action is appropriate while the limitation remains.
Name the system, record type, time period, and unit of observation or exposure. Check whether the unit is the one needed for the proposed prediction.
Record filters, joins, deduplication, code mappings, and derived-variable definitions. Each rule can change which records or meanings reach the model.
Where possible, report counts, denominators, and a range of plausible operational consequences. Do not replace unknown values with assumed values without stating the assumption.
State which comparison or action is supported, which is not, and what reconciliation or documentation would reduce the uncertainty.
Reconciling Source Units Before Comparing Rates
A common provenance problem is that two systems store the same general quantity in different units. A rate can be combined only after its numerator and denominator have compatible definitions and units. If a count of events is divided by exposure, the resulting rate is events per unit of exposure. Pooling valid rates requires adding their event counts and their exposure amounts, not averaging the rates without regard to their denominators.
Worked Example: Combining Claim Rates With Different Exposure Units
A hypothetical insurer is assessing a portfolio prediction. An older extract contains 120 claims over 600 policy-years. A newer extract contains 90 claims over 4,800 policy-months. Assume the two extracts use the same claim definition, neither includes duplicate records, and each month represents one-twelfth of a policy-year. Calculate the pooled claim rate per policy-year. Name the method and a common mistake it avoids.
Method. Convert both exposure amounts to policy-years, then pool the counts and exposures. This avoids combining denominators expressed in different units.
Step 1: Convert the newer exposure. The 4,800 policy-months correspond to \(4{,}800/12=400\) policy-years.
Step 2: Check each source rate. The older rate is \(120/600=0.20\) claims per policy-year. The newer rate is \(90/400=0.225\) claims per policy-year.
Step 3: Pool compatible quantities. The combined claim count is \(120+90=210\). The combined exposure is \(600+400=1{,}000\) policy-years. Thus the pooled rate is \(210/1{,}000=0.21\) claims per policy-year.
Check. The pooled rate is between the two source rates, 0.20 and 0.225. The newer source contributes 40% of total exposure, so the pooled rate also equals \(0.60(0.20)+0.40(0.225)=0.21\).
Common mistake. Adding the raw denominators gives \(600+4{,}800=5{,}400\), which incorrectly treats policy-months as policy-years. That calculation would produce \(210/5{,}400=0.0389\) claims per “combined unit,” a unit that has no consistent interpretation. The converted-denominator calculation is the valid comparison under the stated assumptions.
Incomplete Records and What Can Be Concluded
A data extract may contain fewer records than the operational source from which it was created. The gap might arise from a filter, a failed join, a missing required field, or a system that records only certain cases. The count of records that reached the model should therefore be compared with a suitable source count, using the same period and unit. A difference is a reason to investigate; it does not by itself establish why records are absent.
When a model input is unavailable for some cases, the result calculated on available cases describes those cases. It does not automatically describe the full operational volume. One transparent diagnostic is to calculate a range under extreme assignments for cases whose status is unknown. Such a range describes uncertainty from the missing records alone; it is not a forecast or a claim about which extreme is likely.
Worked Example: Bounding Alert Volume When Scores Are Missing
A hypothetical service team receives 500 referrals in a week according to its intake log. The model-input extract contains scores for 440 referrals, of which 88 exceed the action threshold. The remaining 60 referrals have no score in the extract. Estimate the observed alert rate among scored referrals and the possible alert-count range for all 500 referrals. State what the range means.
Method. Calculate the rate using only the records with scores, then bound the total count by assigning none or all of the 60 unscored referrals to the alert group. This avoids silently treating an absent score as a negative alert.
Step 1: Calculate the observed rate. Among scored referrals, \(88/440=0.20\), or 20%.
Step 2: Find the lower count. If none of the 60 unscored referrals would exceed the threshold, the total is \(88+0=88\) alerts.
Step 3: Find the upper count. If all 60 unscored referrals would exceed the threshold, the total is \(88+60=148\) alerts.
Step 4: Express the range against all referrals. The lower rate is \(88/500=0.176\), or 17.6%. The upper rate is \(148/500=0.296\), or 29.6%. Thus the data support a range of 88 to 148 alerts, or 17.6% to 29.6% of referrals, under these extreme assignments.
Check. There are 500 referrals in total, and the 440 scored cases plus 60 unscored cases account for all 500. The range includes the 88 known alerts and adds between zero and 60 possible alerts.
Common mistake. Applying the observed 20% rate to all 500 referrals gives 100 alerts. That is a possible estimate only under an additional assumption that the scored referrals’ alert rate is applicable to the unscored referrals. The extract alone does not establish that assumption. The range makes the uncertainty visible; the intake-to-extract reconciliation is needed to determine why the 60 scores are absent.
Transformations Can Change Meaning
A transformation is not merely a technical detail. A code mapping can change the meaning of a feature, and a join can change which records contribute to a result. Two columns with the same displayed label are not necessarily comparable if their source definitions differ. Provenance documentation should preserve the source field, the mapping rule, and the meaning assigned to the value in each source and period.
Before interpreting a combined result, ask whether the values being combined represent the same concept. If the answer is uncertain, report the source-specific results separately or withhold the combined interpretation until the definitions are reconciled. A numerical difference between categories is not informative if the categories were constructed differently.
Worked Example: A Legacy Code With Two Meanings
A hypothetical operations team combines two systems to describe the share of cases marked “R” that receive a high model score. In the older system, “R” means a renewal; among 100 such cases, 40 receive a high score. In the newer system, “R” means a reinstatement; among 80 such cases, 48 receive a high score. Calculate the combined share, then assess whether it supports a single interpretation of “R.”
Method. First calculate the pooled numerical share, then inspect the provenance of the category before interpreting that share.
Step 1: Calculate the combined numerator and denominator. There are \(40+48=88\) high-score cases among \(100+80=180\) cases coded “R.”
Step 2: Calculate the combined share. The numerical result is \(88/180=0.4889\), or 48.9% when rounded to one decimal place.
Step 3: Compare the source-specific shares. In the older system, the share is \(40/100=40\%\). In the newer system, it is \(48/80=60\%\).
Interpretation. Although 48.9% is arithmetically correct for the combined rows, “R” does not identify one common case type: it means renewal in one source and reinstatement in the other. The pooled number therefore cannot be described as the high-score share for a single, consistently defined “R” group. The source-specific results should remain separate until a defensible common definition is established.
Check. The pooled share, 48.9%, lies between the two source-specific shares, 40% and 60%, as expected from pooling counts with positive denominators. That arithmetic check does not resolve the definition problem.
Common mistake. Reporting “48.9% of R cases receive a high score” treats the code as if it had one meaning. The correction is to state the two source definitions and avoid an unsupported combined interpretation.
Writing a Caveat That Helps the Decision
A provenance caveat should let a reader see the path from evidence to action. It is useful to state the limitation, its measured extent, the inference it affects, and the next step. Keep the wording proportional to the evidence: a record-count discrepancy identifies an unresolved gap, not its cause; a unit mismatch invalidates a pooled rate until units are reconciled; and a code-definition conflict prevents a common category interpretation even when the arithmetic is correct.
For example, a decision-ready statement might read: “The weekly intake log lists 500 referrals, but the model extract contains scores for 440. The observed 20% alert rate applies to scored referrals only; without resolving the 60-record gap, the total alert count could range from 88 to 148 under extreme assignments. Staffing plans based on a single projected count should therefore be treated as provisional while the extraction process is checked.” This wording distinguishes what is known from what remains uncertain and identifies why the limitation matters.
A strong report also avoids implying that documentation alone fixes a data-quality issue. A data dictionary, extraction log, or transformation record makes a process auditable, but the analyst must still assess whether the resulting records and definitions fit the proposed decision. Provenance supports that assessment; it does not substitute for it.
Check Your Understanding
For each problem, distinguish the calculation from the conclusion justified by the data.
- A hypothetical portfolio extract records 75 events over 300 policy-years in one system and 42 events over 2,400 policy-months in another. Assume event definitions match and one month is one-twelfth of a year. Find the pooled rate per policy-year. (Answer: \(117/(300+200)=0.234\) events per policy-year.)
- An intake log lists 240 cases. Scores are present for 210, and 63 of those exceed a threshold. If the remaining cases could all be above or all below the threshold, give the possible alert-count and alert-rate ranges for the 240 cases. (Answer: 63 to 93 alerts; 26.25% to 38.75%.)
- Two hypothetical systems use the same code “C.” System A means “current account” and has 30 high scores among 60 cases; System B means “closed account” and has 18 among 30. Calculate the pooled share and state whether it has a single category interpretation. (Answer: \(48/90=53.3\%\); no, the code has different meanings in the two sources.)
- A source file has 1,200 records, but a model extract contains 1,080. The extraction log documents that 50 records were excluded by a stated date filter. How many records remain unaccounted for, and what is the appropriate immediate conclusion? (Answer: \(1{,}200-1{,}080-50=70\) records remain unaccounted for; investigate the discrepancy before assuming the extract represents all eligible source records.)
- One system reports 24 events over 120 exposure units; another reports 36 events over 240 exposure units. The units match, and definitions are consistent. Calculate the pooled rate and the unweighted mean of the two rates; explain which rate represents combined exposure. (Answer: pooled \(60/360=0.1667\) events per unit; unweighted mean \((24/120+36/240)/2=0.175\); the pooled rate represents combined exposure.)