Why Data Lineage Matters
A population PK result is only reviewable if its data path can be followed. A reviewer needs to see not merely which dataset was used, but how source records became analysis records, how those records became model-ready input, and which output supports a reported result. If that path is incomplete, it may be impossible to determine whether a result came from the intended analysis population, whether a transformation altered a value, or whether an unexplained exclusion affected the analysis.
Data lineage is a structured account of those relationships. It supports review of the analysis without replacing the analysis plan, the record of amendments and deviations, or the underlying data and programs. The goal is to make the path verifiable: identify each important artifact, identify the transformation that links it to another artifact, and provide enough information to locate and check both.
What a Reviewable Lineage Records
A useful lineage account treats the workflow as connected steps rather than as one broad statement such as “data were cleaned and analyzed.” For each artifact, record a stable name or identifier, its purpose, its version or data-cut identity, its location, and the artifact from which it was derived. For each transformation, record what changed, why it changed, who was accountable, when it occurred, which program or controlled procedure implemented it, and what checks were performed.
The level of detail should allow a reviewer to answer practical questions. Which source extract supplied the concentration values? How were dose and observation records combined? Were times converted to a common reference? Which records were excluded, and for what reason? Which exact model-ready dataset was used for the run supporting a table in the report? Do not assume that a general description of the method answers these questions for a particular artifact.
- Identify inputs and outputs. Name each upstream and downstream artifact, including its relevant version or cut.
- Describe the transformation. State the rule applied, not just that records were “processed.” Include units, key variables, merge criteria, and exclusion rules when relevant.
- Assign accountability. Name a responsible role or person and the date of the transformation or review.
- Show checks and exceptions. Record reconciliations, validation checks, unresolved discrepancies, and their disposition.
- Connect results back to their inputs. Identify the model run and output supporting each reported result, then identify the dataset supplied to that run.
Version identification needs to distinguish the artifact actually used from a later file with a similar name. A data-cut date or version label may be useful; a content fingerprint can provide an additional check that two files are identical. Neither a label nor a fingerprint explains the scientific meaning of a transformation. The lineage record must also say what was done and why. Recording software versions and computational settings is a related but distinct documentation task.
Building the Chain and Checking Its Logic
Represent the workflow as a directed graph: artifacts are nodes, and transformations are the links between them. The direction of each link runs from input to output. A result can have several paths back to source artifacts—for example, concentrations, dosing, and subject characteristics may be supplied by different source extracts. A complete record identifies all relevant inputs rather than implying that one file supplied everything.
A short justification for this structure follows directly from the review question. Suppose an output \(O\) is produced from inputs \(I_1,\ldots,I_k\) by transformation \(T\). To verify \(O\), a reviewer must be able to identify the \(I_j\), understand the relevant parts of \(T\), and check that the stated output corresponds to the stated inputs. If any input or link is missing, the path is not fully specified. Applying this reasoning repeatedly from a reported result back to its sources gives a traceable chain. Counts, keys, and independent checks help test whether the documented links match the actual artifacts.
This approach depends on several practical conditions. Source extracts must be identified and retained in a controlled way; record identifiers must be stable enough to support reconciliation; transformation rules must be documented accurately; and exceptions must not be silently dropped. If a process is not reproducible from the recorded inputs and rules, state that limitation and document the available evidence. A lineage record is not proof that every source value is correct, but it makes the route and the checks open to review.
List the source extracts and downstream datasets, outputs, and reported results needed to answer the review question. Identify each by a specific version, cut, or other distinguishing information.
Link inputs to outputs and describe the rules, rationale, accountable person or role, date, and controlled program or procedure used.
Compare counts, keys, and selected values across stages. Explain every difference, including exclusions, duplicate handling, and corrections.
Connect each result of interest to its supporting output and model run, then follow the chain back to the exact model-ready dataset and its sources.
Worked Examples
Worked Example: Reconciling a Model-Ready Dataset
Consider a hypothetical study with source extracts containing 612 concentration records, 384 dose records, and demographic information for 48 participants. The documented merge creates an event-level dataset with one row per concentration or dose event. The model-ready dataset has 982 rows. Determine whether the row counts reconcile and what additional information should be recorded.
The expected number of rows before exclusions is:
The observed reduction is:
Suppose the transformation log identifies 6 concentration records with invalid collection times and 8 concentration records with missing concentration values, all excluded under documented rules. Then \(6+8=14\), which accounts for the difference. The model-ready dataset contains \(612-14=598\) concentration rows and 384 dose rows, giving \(598+384=982\). If all 48 participants remain represented, the subject-level reconciliation is consistent too; that fact should be checked from the records rather than inferred solely from the row totals.
The lineage record should identify the three source artifacts and their versions, the merge keys and rules, the exclusion criteria, the count before and after each operation, and the accountable reviewer. The arithmetic establishes that the stated counts balance; it does not alone establish that the exclusions were appropriate or that the correct records were removed. Those points require review of the record-level exception list and the governing analysis decisions.
Worked Example: Documenting a Time Transformation
In a hypothetical source file, the first dose is recorded at 08:00 and a concentration sample is recorded at 11:30 on the same day. The model-ready dataset uses elapsed time in hours from the first dose. Show the calculation and the lineage details needed to review it.
The elapsed time is calculated as:
The resulting model-ready time is 3.5 hours. The transformation record should identify the source date-time fields, the reference dose event, the time zone or convention applied, the calculation rule, and the output variable and unit. It should also identify how missing or ambiguous date-times and samples preceding the reference dose are handled. A check can recompute elapsed time for selected records and compare the result with the model-ready value. Without a stated time-zone and date-handling convention, the arithmetic shown for this same-day example would not be enough to review records that cross midnight or daylight-saving transitions.
Worked Example: Tracing a Reported Result Back to Its Data
A hypothetical report gives a typical clearance estimate of 4.20 L/h. The supporting output is identified as result table T-04 from model run R-07. R-07 used model-ready dataset M-03, with 982 rows and 598 concentration records. Show how to document a traceable path without treating those identifiers as sufficient evidence by themselves.
The lineage entry for T-04 should point to output T-04 and run R-07; the entry for R-07 should identify M-03 as its input. The entry for M-03 should identify the analysis dataset from which it was derived and the transformation that produced it. That transformation should point to the concentration, dosing, and demographic source extracts, each with its specific version or cut. For example, the data reconciliation should show \(598+384=982\) rows if M-03 contains 598 concentration rows and 384 dose rows, and it should account for the 14 exclusions described in the preceding example.
This chain lets a reviewer check that the report points to the stated output and that the output points to the stated run and dataset. The estimate’s value and unit should agree between the report and the cited output. The chain does not, by itself, establish that the model was scientifically appropriate or that 4.20 L/h is an accurate estimate. It establishes where the reported value came from and makes the relevant inputs and transformations inspectable.
Diagnostics, Comparisons, and Failure Modes
A lineage audit is going wrong when links cannot be followed or when simple cross-checks disagree. Warning signs include a dataset name without a version, an output that cannot be tied to a particular run, unexplained count changes, identifiers that do not match across stages, or a transformation described only as “cleaning.” Repeatedly overwriting files makes it difficult to establish which artifact was actually used. A count that balances is useful, but it can conceal a wrong merge or a compensating pair of errors; check keys and selected records as well.
| Approach | Strength | Limitation |
|---|---|---|
| Narrative workflow description | Provides context and explains rationale in readable form. | Can omit exact artifact versions, record-level changes, or links between specific inputs and outputs. |
| Structured lineage table or graph | Makes artifacts, transformations, and links easier to inspect and reconcile. | Can become a list of labels without meaningful rules, rationale, or checks. |
| File names and version labels alone | Quick to record and communicate. | May not distinguish changed contents or explain how an artifact was created. |
| Counts and content fingerprints | Help detect missing records or unexpected file changes. | Do not explain scientific decisions, transformation logic, or whether the chosen data are appropriate. |
The strongest documentation combines these approaches: a structured map for navigation, concise narrative for rationale, and quantitative and record-level checks for verification. Keep the scope proportionate. Lineage should identify the chain needed to review the analysis, not duplicate every operational detail already available in controlled source records. It also cannot fix inadequate source data or justify an undocumented analytical decision. Those limitations should be made visible rather than hidden by a polished diagram.
Common Mistake
Wrong reasoning: “The report names the final dataset, so the analysis is traceable.” A dataset name does not establish which version was used, how it was made, which records changed, or which reported output came from it.
Correction: Link the result to its supporting output and run, the run to its exact model-ready input, and each derived dataset to its inputs and documented transformations. Reconcile changes with counts and record-level evidence, and identify unresolved gaps explicitly.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A dataset changes from 240 source records to 233 analysis records. What must be documented before that difference is considered reconciled?
- Why are a file name and a content fingerprint not sufficient descriptions of a transformation?
- A result table points to a model run, but the run record does not identify its input dataset. What part of the lineage is missing?
- A concentration file has 100 records and a dose file has 60 records. The merged event-level dataset has 160 rows, and 5 concentration records are excluded under documented criteria. How many rows should the model-ready dataset contain if no other rows are added or removed?
Solutions. (1) Identify the 7-record difference by reason and, where possible, record identifiers; state the rules and accountable transformation, then verify the exclusions against the source and output. A net count alone does not show which records changed. (2) The file name and fingerprint help identify an artifact or detect a content change, but neither explains the rule, rationale, inputs, or accountability for creating the output. (3) The link from the run to the exact input artifact is missing, so a reviewer cannot verify which dataset supported the result. (4) The expected count is \(100+60-5=155\) rows. This assumes the 5 excluded records are among the concentration records and there are no other additions, exclusions, or duplicate-handling changes.