Recording Data Provenance

Why Data Provenance Matters

A dataset rarely appears at the start of an analysis in exactly the form in which it was first collected. It may have been copied from an archive, downloaded on a particular date, restricted to certain records, or summarized before analysis. If those steps are not recorded, a reader may be unable to tell what the numbers represent or how to reproduce the analysis dataset. Data provenance is the documented history of a dataset’s origin and changes.

Provenance is not a guarantee that data are accurate or representative. It lets an analyst and reader see where the data came from, what population they concern, when the relevant events or measurements occurred, and what happened to the data before analysis. This context is necessary for interpreting a result responsibly.

Definition: A provenance record is a concise account of a dataset’s source, collection context, population, access or acquisition, and transformations leading to the dataset used in an analysis. It should identify the relevant version of the data and make the path from source to analysis dataset understandable.

What a Provenance Record Should Capture

A useful record answers a small set of practical questions. It should be specific enough for someone else to identify the source and understand the dataset’s scope, but it need not repeat every detail stored elsewhere. For a small analysis, include:

Source: the organization, system, person, or publication that created or supplied the original data. If you received a copy from an intermediary, distinguish that intermediary from the original source when known.
Source identification: a title, file name, stable address, archive reference, or other locator, plus a version or release date when available.
Collection context: who collected or recorded the data, how, and during what period. If this information is unavailable, state that rather than guessing.
Population and coverage: the people, places, events, or units the data are intended to describe, and any known limits on which of them were observed.
Key dates: distinguish when the observations were made from when the source was published, when you obtained it, and the period represented by the analysis.
Transformations: list the operations that changed which records or values were included or how they were represented, in the order they occurred.
Analysis dataset: identify the resulting file or table and briefly state what it contains or is used to calculate.

These fields are connected but not interchangeable. For example, a file downloaded in October may contain records collected during the previous calendar year. Its download date describes your access to the file; it does not tell you when the recorded events occurred. Similarly, “all patients in the file” describes the observed records, not necessarily the population the study was meant to represent.

Worked Example: Recording a Public Data Source

Suppose a student uses a hypothetical municipal open-data file to calculate the mean monthly bicycle counts at five monitoring stations during April. The file is named BikeCounts_2025_v2.csv. The archive says counts were recorded daily from January through December 2025, that the stations are municipal counters, and that version 2 was released on February 10, 2026. The student downloads it on March 3, 2026, selects April records from the five named stations, and calculates one mean for each station.

A concise provenance record might read:

Source: Municipal Transportation Office, “Daily Bicycle Counter Totals,” open-data archive; file BikeCounts_2025_v2.csv, version 2, released February 10, 2026.
Collection context: automated daily totals from municipal bicycle counters; the source describes observations from January 1 through December 31, 2025.
Population and coverage: bicycle counts recorded by the municipality’s listed counters. The analysis covers the five named stations only, not every bicycle trip in the municipality.
Access date: downloaded March 3, 2026.
Transformations: selected records dated in April 2025 for the five specified stations; calculated the arithmetic mean of each station’s daily counts for that month.
Analysis dataset: a five-row summary table of station-level mean daily counts for April 2025, measured in bicycles per day.

The result is described as counts at five municipal stations, not as the average number of bicycle trips made by all residents. A counter measures traffic at its location; the provenance record makes the coverage boundary visible. The access date also cannot be mistaken for the date of the bicycle counts.

Trace Transformations in Order

A provenance record should let a reader follow the data from the identified source to the analysis dataset. A compact way to express that path is:

D_{\text{analysis}} = T_k\bigl(T_{k-1}(\cdots T_2(T_1(D_{\text{source}}))\cdots)\bigr)

Here \(D_{\text{source}}\) is the identified source dataset, each \(T_i\) is one documented transformation, and \(D_{\text{analysis}}\) is the data actually used. The notation is simply a reminder that operations happen in sequence: the output of one step becomes the input to the next. The record need not use this equation, but it should state the steps clearly enough that the sequence is not ambiguous.

This chain matters because a result depends on the data that remain after each operation, not just on the original source. For example, “restricted to the selected region, then retained records from the specified year, then calculated a group mean” says more than “cleaned the data.” The latter does not tell a reader what changed. A transformation description should name the action and the rule used, without trying to replace detailed technical documentation where that is needed.

Worked Example: Documenting a Survey Summary

A hypothetical campus survey was offered to all 240 students enrolled in an introductory course from September 8 through September 12, 2026. A total of 72 students submitted responses to the question, “How many hours did you study for this course last week?” An analyst uses the submitted responses to calculate the mean reported study time. The survey platform exports a file on September 13, and the analyst retains the 72 submitted responses for the calculation.

The provenance record should distinguish the intended group from the respondents:

Source: survey responses collected through the course’s survey platform; response export downloaded September 13, 2026.
Collection context: a survey open from September 8 through September 12, 2026, asking enrolled students to report study hours for the previous week.
Population and coverage: the intended group was the 240 students enrolled in the course. The analysis uses 72 submitted responses; these are respondents, not all enrolled students.
Transformations: retained submitted responses to the study-hours question and calculated their arithmetic mean. The result describes reported hours among those respondents.
Analysis dataset: the response values used for the mean, with units of reported hours per student for the previous week.

Suppose the 72 responses sum to 504 reported hours. The mean among respondents is

\bar{x} = \frac{504\ \text{hours}}{72\ \text{respondents}} = 7.00\ \text{hours per respondent}

The arithmetic supports a mean of 7.00 reported hours per respondent. It does not establish that all 240 enrolled students averaged 7.00 hours: the 168 students who did not submit a response are not represented in this calculation. The provenance record makes that distinction explicit, without claiming that the respondents represent the full enrolled group.

A Short Check on Provenance Completeness

A provenance record is useful when it permits a reader to answer three questions: Where did these data come from? What do they cover? What was done to produce the analysis dataset? This gives a practical completeness check. If any answer depends on an unstated assumption, the record is incomplete at that point.

The logic is direct. A result is calculated from an analysis dataset. To interpret that result, a reader needs to know which data entered the calculation and what the data represent. The transformation history links those analysis data to their source; the source and collection context establish their origin and scope. Thus, omitting any link can leave an important ambiguity. A source without a population description does not establish who or what the records represent. A population description without transformation details does not establish which part of it was analyzed.

This is not a proof that the data are suitable for every question. Rather, it is a justification for why source, scope, and transformation history belong together: each answers a different question needed to interpret the analysis.

Worked Example: From Individual Measurements to a Summary

A hypothetical community garden project records the weights, in kilograms, of tomatoes harvested from plots on August 20, 2026. The coordinator supplies a spreadsheet containing observations from 12 plots. An analyst selects the six plots in the north section and calculates their average recorded yield. The six yields are 8, 9, 10, 10, 11, and 12 kg.

A compact record could state:

Source: harvest spreadsheet supplied by the community garden coordinator; observations recorded August 20, 2026.
Population and coverage: the source contains recorded yields for 12 garden plots. The analysis covers the six plots designated as the north section; it does not summarize all plots in the garden.
Transformations: selected the six north-section plot records and calculated their arithmetic mean yield in kilograms.
Analysis dataset: the six selected plot yields and their mean for August 20, 2026.

The calculation is

\bar{x} = \frac{8 + 9 + 10 + 10 + 11 + 12}{6} = \frac{60}{6} = 10.0\ \text{kg per plot}

The mean of 10.0 kg per plot describes those six selected plots on the stated harvest date. The selection step is important provenance: reporting only that the source contained 12 plots would not explain why the analysis used six or what area those six represented.

Writing a Record That Is Concise and Useful

A provenance record is not a long narrative. Prefer short, concrete statements to broad descriptions. “Downloaded the April 2025 records for stations A through E from version 2 of the municipal archive on March 3, 2026” is more useful than “obtained relevant data online.” If the source does not report its collection method or population, note that the information was not provided. This preserves the difference between what is known and what is inferred.

The amount of detail can depend on the analysis. A brief class exercise may need only a few bullets. A consequential analysis may need a more complete source citation, version identification, access restrictions, and enough detail to locate the exact data. In either case, the record should avoid publishing private information or credentials. Describe the dataset and its approved location without exposing sensitive contents.

Keep provenance focused on origin and lineage. A data dictionary explains variables and their meanings; a provenance record explains where the data came from and how the analysis version was obtained. Detailed decisions about correcting values, handling missing observations, or other cleaning operations should be recorded as such rather than hidden behind a vague provenance phrase such as “prepared the file.”

Common Mistakes

Confusing dates. “Downloaded on March 3” does not mean the data were collected on March 3. Record access and collection dates separately.
Calling the observed records the population. A dataset may contain only respondents, selected locations, or records available to an agency. State the intended population and actual coverage distinctly when both are known.
Describing a transformation vaguely. “Filtered the data” is incomplete unless the record says what rule determined which data were kept.
Treating a source as proof of representativeness. A well-known source can still cover only a subset of the population relevant to a question. Provenance documents scope; it does not remove coverage limitations.
Filling gaps with guesses. If the collection method, population, or version is unclear, mark it as unknown or not reported. A cautious statement is more reliable than an invented detail.

Key takeaway: A useful provenance record connects an identifiable source to the data used in an analysis. State what the data cover, distinguish when they were collected from when they were accessed, and list the transformations that produced the analysis dataset.

Check Your Understanding

Use the provenance principles to answer these questions.

A file downloaded in May contains measurements recorded the previous November. Which date describes access, and which describes collection?
A survey was sent to 500 people and received 120 responses. What is the intended group, and what group is actually represented by the submitted responses?
Why is “the data were filtered” not a sufficient description of a transformation?
A dataset contains records from all clinics in one city, but the analysis uses only three clinics. What coverage limitation should the provenance record make clear?
Name three pieces of information that would let another analyst identify a source dataset and understand how an analysis version was derived from it.

What You'll Learn