Recording Data Provenance and Source Files

Why Data Provenance Matters

A regression result depends not only on the model and analysis, but also on the particular data used to fit it. A file called survey_final.csv does not say who supplied it, which release it belongs to, or whether it has been edited since receipt. If a result is questioned later, those missing details can make it difficult to determine which data supported the analysis.

Data provenance is the record of where data came from and how they moved from their source to the form used in an analysis. For regression work, this record helps answer practical questions: Was the file downloaded from the stated source? Which release was used? Did the data describe the intended time period? What changes were made before analysis? Provenance does not establish that the source data are accurate or that an analysis is valid. It makes the data’s history inspectable.

A data dictionary describes the meaning and format of variables; provenance instead describes the history and identity of data files. These records can refer to one another, but one does not replace the other. Similarly, recording transformations is not the same as writing a data-preparation script: the purpose here is to keep a clear account of what happened to the files and which outputs came from which inputs.

What to Record About a Source

A useful provenance record identifies a specific data release and the specific file received, rather than naming only a broad organization or website. Record enough detail that someone can distinguish this file from earlier or later versions. The relevant details depend on the source, but commonly include the items below.

• Origin: the organization, repository, system, or person that supplied the data, along with the dataset or collection name. Include a stable webpage, catalog entry, or contact route when available.
• Release or reference date: the release number, publication date, or other version identifier supplied by the source. Also note the time period the records represent when that is different from the release date.
• Receipt details: when the project obtained the file and how, such as a direct download, an emailed attachment, or an authorized export from an internal system. For a download, record the access date and the relevant download location.
• File identity: the exact received file name, file type, and, where useful, its size and a checksum. A checksum is a computed string used to check whether a file’s contents have changed.
• Conditions and context: any source-provided notes needed to understand the extract, such as its coverage, known limitations, or restrictions on access and use.

Dates need labels. The date a file was downloaded is not necessarily the date the data represent, the date they were collected, or the date the source published the release. If a source reports records for January through June but publishes an extract in August, write down both the coverage period and the publication or receipt date when known. If a date or version is unavailable, say that it is unavailable rather than guessing.

A record should also distinguish direct observations from descriptions supplied by a provider. For example, “release 3, dated 12 May” is appropriate if that identifier appears on the source. “Latest release” is less useful unless the record also states when and where it was checked. The goal is not to claim more certainty than the available documentation supports.

Keep an Unchanged Source Copy

Once a source file has been received, preserve an unchanged copy before opening, editing, converting, or otherwise preparing it for analysis. Store this copy in a clearly designated source-data location, separate from derived files. Avoid overwriting it when making corrections or creating a more convenient format. If the provider later issues a revision, retain that as a separate received version rather than replacing the earlier file without a record.

“Unchanged” refers to the bytes in the received file, not to a claim that the provider’s data are error-free. A checksum can help check whether the copy still matches the file that was first recorded. If the checksum calculated later matches the recorded checksum, that is evidence that the file contents are unchanged. A matching checksum does not prove who created the file, whether it came from the claimed provider, or whether its contents were true. Source details and secure handling are needed for those separate questions.

When data are sensitive or access is restricted, preservation must follow the applicable privacy and security requirements. An unchanged copy should not be kept in a broadly accessible folder merely for convenience. Record where the authorized copy is held and who may access it, without placing confidential data in a public provenance log.

Record the Path From Source to Analysis File

An analysis file is often not identical to any single source file. It may result from importing a spreadsheet, changing date formats, converting measurement units, joining two sources, or selecting particular records. Provenance should make those steps visible at a useful summary level: name the input files, describe the important operation, identify the resulting file, and note when or why the change was made if that context matters.

A compact way to think about the record is as a chain:

Each arrow should be understandable. If the record names only the final analysis file, the reader cannot identify the source release or determine how the data reached that state. If it names a source but omits an intermediate transformation, the connection between the source and the analyzed data is uncertain. This is why the record should identify both files and the steps linking them. Complete provenance is necessary for tracing the data, although it is not by itself sufficient to reproduce an entire analysis: other records may be needed for the model, software, and analysis decisions.

Describe transformations factually and specifically. “Cleaned the data” is too vague to tell a later reader what changed. A more informative note might say that dates were converted from day-month-year text to a standard date format, or that two named files were joined using a stated county identifier. The aim is to document the nature of the change and its inputs and output, not to provide a full variable-by-variable account or the preparation code.

A Practical Provenance Record

A small manifest, such as a table or a structured text record, can keep source details together. It should be stored with the project documentation and kept consistent with the actual files. One row per received source file is often a workable starting point; transformation notes can refer to the input and output filenames. Do not rely on memory, temporary download folders, or a filename such as final_final2.csv to explain the data history.

A manifest is only useful if it is accurate and understandable. Use stable, descriptive filenames; avoid ambiguous labels such as “new” or “latest” without a date or release identifier. If a file is replaced, document the replacement as a new received version and explain the relationship to the earlier one. Keep the provenance record itself free of unnecessary personal or confidential information.

Common Mistakes and Their Corrections

• Recording only the download date. That date says when the project obtained a file, not what time period the data represent or which release they came from. Record each date with its meaning.
• Overwriting the received file. Editing a source file destroys the reference copy and makes later comparisons difficult. Preserve it unchanged and save modifications as separate derived files.
• Assuming a filename proves identity. Two different files can share a filename, and a filename can be changed without changing the contents. Use source details and, when appropriate, a checksum to identify and check the file.
• Writing “cleaned” as the whole transformation record. That description does not say what input was changed or what the change involved. Name inputs and outputs and state the important operations in plain language.
• Treating a checksum as proof of authenticity or accuracy. A checksum can help detect changed contents relative to a recorded value; it does not verify the provider or the truth of the data.
• Keeping only the final analysis file. Without the received sources and the links between source and output, the data history cannot be checked reliably. Preserve authorized source copies and document the lineage.