Data Provenance, Revisions, and Comparability

Why Provenance and Comparability Matter

An observed value is not self-explanatory. A reported employment rate, for example, depends on which records were used, how employment was defined, when the records were collected, and how the rate was calculated. If any of those details change, a difference between two values may reflect a change in the economy, a change in measurement, or both.

Data provenance is the documented history of where data came from and how they were produced and changed before analysis. Comparability is the degree to which observations refer to the same defined quantity under sufficiently consistent collection and processing rules. Comparability is not guaranteed just because two columns have the same label or units.

For an empirical analysis, provenance should make it possible to answer practical questions: Which source file was used? Which release or download date does it represent? What does each variable mean? Were values converted, recoded, aggregated, or joined to other records? Were any observations changed by a later revision? These details matter even when the data come from a respected statistical agency: authoritative data can still be revised or defined differently across releases.

Tracing a Value From Source to Analysis

A useful provenance chain follows a value through each stage where its meaning or numerical value could change. The source might be a survey, administrative records, a published statistical table, or a database extract. Subsequent steps might include downloading a particular release, selecting a subset of records, changing units, combining categories, or calculating a rate. Record enough information to identify the original material and to reproduce each consequential transformation.

A compact record for a variable can include its name, definition, observational unit, population or coverage, reference period, unit of measurement, coding rules, source and release, and any processing applied in the analysis. For a dataset, also record its file or table identifier and the date it was obtained. If a source publishes revised vintages, identify the vintage actually used rather than relying on a general source name.

This record is not a substitute for careful data analysis. It is what allows an analyst or reviewer to determine what the analysis actually describes. A data file with a clear filename but no release, definition, or processing history may not be enough to reconstruct the meaning of its values.

Revisions: New Values Do Not Always Mean New Events

Published values may be revised when late reports arrive, records are corrected, seasonal adjustment is updated, or a method is changed. Some revisions update a small part of a series; others recalculate many past values. A revision is a change to the reported data value or its production, not necessarily a change to the underlying event that was measured.

For comparisons over time, use values from a consistent release when possible. If the research question concerns what decision-makers knew at a particular date, an earlier release may be the relevant version. If the question concerns the best current estimate of historical conditions, a later revised series may be more appropriate. State which question the chosen vintage addresses. Do not silently combine early values from one release with revised values from another.

A revision can also arise from a changed method rather than corrected records. In that case, differences between vintages may reflect both new information and a new measurement procedure. Documentation should be checked before treating a revised historical series as directly interchangeable with its earlier version.

Comparability Across Definitions, Places, and Time

A comparison needs more than a matching variable name. Check whether the observations refer to the same kind of unit and population, the same concept and coding rules, comparable reference periods, and compatible measurement units. For geographic comparisons, check whether boundaries and population coverage align. For time comparisons, check whether survey questions, administrative systems, or classification standards changed.

When a definition changes, the same word can refer to different quantities. When a unit changes, values may be convertible but should not be compared until converted. When a geographic boundary changes, values for the named place may cover different populations. These problems differ: some can be fixed with a documented conversion, while others require additional information or make a direct comparison indefensible.

A bridge is information that links measurements made under two procedures, such as observations collected under both definitions during an overlap period. A bridge can support a conversion or adjustment only to the extent that it is appropriate for the periods, populations, and quantities being compared. An overlap in one year does not automatically show that the relationship between definitions stayed constant in every year.

When combining rates, combine the numerators and denominators before calculating the overall rate. Taking a simple average of the two area rates would give each area equal weight regardless of labor-force size, and would not generally produce the combined district rate.

A Compact Check for Comparability

Before treating two observations as a meaningful comparison, ask what each one represents and whether the answer changes across observations. Record the result rather than relying on an impression that the data “look consistent.”

• Source and version: Are the source, release, and revision status known? Are the values drawn from consistent vintages?
• Unit and population: Do the observations describe the same type of unit and a comparable population or coverage area?
• Definition and coding: Do the variable definition, category codes, and inclusion rules match?
• Time and place: Are reference periods and geographic boundaries aligned with the question?
• Processing: Have units, denominators, filters, or aggregation rules changed? Can the effect be reproduced?
• Evidence for adjustment: If procedures differ, is there a documented conversion or overlap measurement that supports harmonization?

The result of this check need not be a simple yes or no. A comparison may be directly comparable, comparable after a transparent adjustment, or not adequately supported. State which case applies and why. If available records do not resolve a definition break, the limitation should remain visible rather than be hidden by a numerical adjustment.

Why a Small Documentation Difference Can Matter

A value can be numerically precise and still have an uncertain interpretation if its source, definition, or history is unclear. A difference between two rates might appear small, but it could be of the same size as a revision or a measurement break. Conversely, a conspicuous change may be real even when the measurement procedure also changed; the documentation helps distinguish what can be established from what remains uncertain.

A useful way to express the issue is to imagine, for illustration, that an observed value \(Y_t\) equals a target quantity \(\theta_t\) plus a measurement-regime component \(b_t\):

If this additive representation applies, subtracting the values in two periods gives

Thus an observed difference contains both a change in the target quantity and a change in the measurement regime. This is an illustration, not a universal measurement model: changes may not be additive or quantifiable from available records. Its practical lesson is that a comparison needs evidence that the measurement component is stable, or a defensible way to account for its change.

Common Mistake

Wrong reasoning: “The variable has the same name in both files, so its values are comparable. The later value is higher, therefore the underlying quantity increased by that amount.”

Correction: A shared label does not establish a shared definition, population, boundary, vintage, or calculation. Trace both values to their sources, identify changes in how they were made, and state whether the difference supports the intended comparison. If a change cannot be resolved with the available evidence, report the limitation rather than interpreting the raw difference as a change in the underlying quantity.