Input Data Lineage in QSP Workflows

Why Input Data Lineage Matters

A QSP result depends not only on model equations and software, but also on the data supplied to the workflow. A concentration measured in a clinical sample may pass through unit conversion, quality-control rules, aggregation, and a final selection step before it is used for calibration. If the analysis records only the final table, it may be impossible to tell which observations were included, how values were changed, or whether a later rerun used the same inputs.

Input data lineage makes those connections explicit. It is not the same as documenting model parameters, model code, or the computational environment. It focuses on the identity and history of the experimental or clinical data that enter model development and simulation. The earlier material on parameter provenance concerns the evidence supporting parameter values; data lineage concerns the observations and transformations that produced an analysis input.

Representing Sources, Transformations, and Outputs

A useful lineage record treats data and processing steps as different kinds of objects. Data objects include the source file, an extracted table, and a final analysis dataset. Processing activities include unit conversion, exclusion, aggregation, and joining tables. Each activity should identify its inputs and outputs. Together, these relationships form a directed graph: edges point from source data through processing activities toward derived data.

For a particular analysis input, a complete lineage has a traceable path back to each originating source. The record should also state what the processing did, which version of the processing specification was used, and which output it produced. A filename alone is not an adequate identity: files can be overwritten, copied, renamed, or stored in different locations while appearing to have the same name.

The record should preserve the distinction between original observations and their processed forms. A unit-converted value is not the original measurement, even if the numeric difference is only a scale factor. A row excluded by a quality rule should remain identifiable in the source data, with the exclusion decision documented in the derived-data history rather than silently deleting the row from the only retained copy. This separation supports reprocessing when a rule changes and makes it possible to audit how a result was constructed.

Selection, Cleaning, and Aggregation Are Data Changes

Transformations that affect which records appear in an analysis input need the same care as numeric transformations. Examples include excluding samples that fail a prespecified assay criterion, selecting a visit window, averaging replicate measurements, or joining biomarker values to subject characteristics. Each operation should be stated in terms that allow a reviewer to determine which records it applies to and why.

A lineage record should distinguish an observed value from a missing value, an imputed value, and a value calculated by aggregation. These may all appear as numbers in a final table, but they have different evidential meanings. If missing values are imputed, identify the affected records and method. If replicates are averaged, retain the replicate-level source data and record the aggregation rule and the number of observations contributing to each mean.

Versions and Reconstructability

A source release and a processed-data version answer different questions. The source release identifies what was obtained from the provider; the processed-data version identifies what a particular workflow produced from that source under stated rules. If a source is corrected or reissued, it should receive a new source version rather than silently replacing the earlier one. If the processing rule changes, the resulting derived dataset should also have a distinguishable version.

Checksums can help detect whether a file’s contents have changed: matching checksums provide evidence that the checked bytes are identical, while a changed checksum signals that the bytes differ. A checksum does not explain what changed, establish that the data are valid, or prove that a transformation was scientifically appropriate. It supports identity checking and should be used alongside descriptive version records and retained processing information.

The reason for linking each step can be stated simply. A derived input depends on both its parent data and the transformation applied to them. If either changes, the output may change. Thus, recording only the transformation code is insufficient if the source version is unknown, and recording only the source file is insufficient if the selection or conversion rule is unknown. A linked record covers both dependencies.

Here \(D_{\mathrm{in}}\) is the identified input dataset, \(T\) is the documented transformation, \(v\) is its version, and \(\theta\) denotes relevant settings or rules. Reproducing \(D_{\mathrm{out}}\) requires identifying all four, as well as any additional parent datasets used by \(T\). This is a practical dependency statement, not a claim that a correct record guarantees a scientifically correct result.

Diagnostics, Alternatives, and Limits

Before relying on a QSP analysis input, check whether its lineage is internally consistent. Confirm that every derived dataset has identifiable parents, every stated transformation has an output, and the analysis points to a specific data version. Check units before and after conversions, row counts before and after filters, and the number of contributing records for aggregations. Where joins are used, check whether records were duplicated or dropped and document the keys and expected matching behavior.

Warning signs include files overwritten in place, unexplained row-count differences, a final table without source identifiers, missing unit information, and an analysis record that points only to a folder or mutable filename. Another warning is a single column that mixes measured, imputed, and aggregated values without a marker or supporting record. These issues do not prove that the analysis is wrong, but they make its input history difficult to evaluate and may conceal consequential changes.

A spreadsheet-based log can be adequate for a small, stable workflow if identifiers are unique, records are maintained, and the links are complete. A database or automated provenance system can help when many datasets and transformations are involved. Both approaches serve the same purpose: preserving the connection between data objects, processing activities, and analyses. Automation can reduce manual omissions, but it cannot decide whether a filtering rule is scientifically justified or whether a dataset is fit for a particular use.

Lineage has limits. It does not establish that source measurements are accurate, that a quality-control decision is appropriate, or that a calibration strategy is sound. Nor does a traceable data input by itself establish model validity. It makes the data history inspectable so those questions can be addressed. Some source data may be restricted or unavailable for redistribution; in that case, record the access conditions and stable references while protecting confidential information, and retain permitted derived outputs and documentation.

Common Pitfalls

• Replacing the original file with a cleaned file. This destroys the distinction between observations as received and later processing. Preserve the original source version and create a separately identified derived dataset.
• Calling a transformed value an observation. A converted, averaged, or imputed value has a processing history. Label its derivation and do not imply that it was directly measured.
• Recording a filename but not a version. Names can remain constant while contents change. Record a specific release or identifier and use an integrity check where appropriate.
• Recording code without the data it used. A script cannot reconstruct an input if its source version, settings, or parent datasets are unknown.
• Treating complete lineage as proof of correctness. Lineage supports traceability and review; it does not validate measurements or justify scientific decisions.