Duplicate and Replicate Samples

Why Repeated Records Need Interpretation

A row that looks repeated is not necessarily redundant. Two identical rows may have been created by importing the same source record twice. But two measurements at the same collection time may instead represent distinct specimens, technical replicates, or separate records that happen to have the same measured value. Treating every repeated value or timestamp as a duplicate can erase valid information; keeping every repeated row without investigation can count one observation more than once.

The first task is therefore to determine what one row represents. In a pharmacometric data set, a row might represent a collection event, a specimen, an assay measurement, or a source-file record. The correct duplicate check depends on that intended unit, which is called the data grain. This tutorial focuses on distinguishing accidental duplication from genuine repeated samples, preserving the identifiers that make that distinction possible, and specifying checks that flag suspicious repetitions for review.

Define the Data Grain Before Defining Duplicates

A duplicate rule is meaningful only after the row’s intended meaning is specified. For example, if each row represents one assay measurement, two assay measurements from the same specimen may be legitimate rows. If each row represents one collection event, the same two measurements may belong in a separate measurement-level table rather than being treated as two collection events. The data grain should be stated in plain language, such as “one row per recorded sample measurement.”

A key is a field or combination of fields intended to identify one record at the stated grain. A source record identifier can help identify repeated imports. A sample or specimen identifier can distinguish collected material. A replicate identifier can distinguish repeated measurements of the same material. A collection-event identifier can distinguish separate collection events, including events that share a recorded time. These identifiers serve different purposes and should not be collapsed into one field.

For a measurement-level data set, one possible key might combine subject, specimen, and measurement-replicate identifiers. That choice is valid only if those fields are populated consistently and actually distinguish every intended measurement. A combination such as subject and time alone is often too coarse: it can label two legitimate same-time samples as if they were the same record. Conversely, a source-row identifier may distinguish imported records without establishing whether those records represent different biological samples.

What Repeated Values and Times Can Tell You

A repeated value is a useful signal, not a diagnosis. The same subject can have equal measurements at different times. Two separate samples collected at one time can also have equal measurements, particularly when the recorded values are rounded. And duplicate source rows can disagree if one copy was edited or transformed differently. Thus, checking only the measured value will miss some duplicates and incorrectly flag some legitimate records.

A repeated collection time is similarly ambiguous. It may occur because the data contain legitimate replicates, because separate samples were collected at the same recorded time, because time values were rounded or recorded at limited precision, or because a record was duplicated. Time should be checked in relation to subject and sample or event identifiers, not as an isolated column.

A practical check separates three questions:

• Are the source records repeated? Compare source record identifiers and the values carried by each record. An identical source identifier appearing more than once warrants review of the import or merge.
• Are the sample or measurement identifiers repeated? Check whether the proposed key is unique at the stated grain. Repeated keys may indicate duplicate rows, a missing replicate identifier, or an invalid key definition.
• Are values or times repeated within a subject? Count or list these cases and inspect sample, event, and replicate identifiers. Repetition is a review trigger, not proof of error.

These checks are complementary. A unique key does not guarantee that a record is scientifically correct, and repeated values do not establish that it is incorrect. In practice, checks should preserve enough fields to trace each flagged row to its source and should record the disposition of investigated cases.

Preserve Replicate Structure in the Data

Replicate identifiers prevent distinct observations from being mistaken for repeated copies. Their meaning should be explicit. A specimen identifier identifies collected material; a measurement-replicate identifier distinguishes repeated measurements of a specimen; and a collection-event identifier distinguishes collection events. These can be separate fields. If the source supplies only one identifier, document what it identifies rather than assuming it captures every kind of repetition.

Do not replace distinct replicate rows with a single average merely to make a key unique. Aggregation changes the data and can conceal variation or collection structure. If a later analysis requires one value per subject and time, the choice to aggregate, select, or otherwise represent replicates must be made explicitly, with its rationale and effects documented. A duplicate-cleaning step should not silently make that modeling decision.

Specify Checks That Flag Rather Than Erase

A reproducible quality check should report what was checked, at what grain, which records were flagged, and what action followed. The check should not silently delete rows. For example, a workflow can produce separate reports for repeated source identifiers, repeated proposed keys, repeated subject-time combinations, and repeated values within a subject. Keeping these reports separate helps identify whether a concern is an import problem, an identifier problem, or a potentially legitimate replicate pattern.

When checking times, use the recorded collection-time field at its available precision and inspect the identifiers that accompany it. Do not assume that two displayed times are identical to the same underlying instant if values have been rounded. Also do not infer the collection time from a different time field: the check should state precisely which recorded field it uses. Questions about planned versus actual time are distinct and are addressed when those fields are specified.

Deriving a Safe Duplicate Rule

The logic for a safe rule follows from the difference between a record’s observed attributes and its identity. Let \(R\) denote the set of source records and let \(g(r)\) be the displayed fields for record \(r\), such as subject, time, and value. If two records have the same displayed fields, then \(g(r_1)=g(r_2)\). That equality does not establish \(r_1=r_2\): distinct records can map to the same displayed values. For example, two samples can have the same subject, time, and rounded measurement.

A deletion rule based only on displayed-value equality is therefore unsafe: it treats a many-to-one representation as if it uniquely identified a record. A stable source identifier or a well-defined key at the intended grain supplies additional identity evidence. Even then, a repeated identifier signals a problem to investigate; it may reflect a repeated import, an incorrectly assigned identifier, or an incomplete key. Deletion is justified only after the records are confirmed to represent the same intended record.

This is why a robust workflow preserves both the original identifiers and the fields used in review. It allows a reviewer to distinguish “same values” from “same source record,” and to revise a decision if later evidence changes the interpretation.

Common Pitfalls, Alternatives, and Limits

• Deleting every repeated subject-time row. This assumes one sample per subject and time. It can erase distinct same-time collection events or replicates. Define the grain and inspect sample or event identifiers first.
• Deleting identical measurements. Equal values can arise in distinct samples, particularly when values are rounded. Compare record and sample identifiers instead of treating value equality as identity.
• Using a key that is too coarse. Subject and time alone may collapse distinct samples. Use fields that identify one record at the intended grain.
• Using a key that is too fine or unstable. Including an import-row number can make repeated imports appear unique. Retain provenance, but do not confuse a new processing-row number with a stable source-record identity.
• Aggregating replicates during cleaning. Averaging or selecting a replicate changes the observation structure. Keep this separate from deduplication and document any analysis-specific rule.

Automated deduplication is appropriate only for cases with a defensible identity rule and sufficient evidence, such as a confirmed repeated import of the same stable source record. Manual review is appropriate when identifiers conflict or the source is ambiguous. Retaining all rows with a review flag can be safer while a discrepancy is unresolved, but unresolved rows should not be treated as validated merely because they remain in the file.

No single check can establish that samples are scientifically valid. A unique key does not verify the correctness of the recorded time or value, and a duplicate report cannot determine whether a repeat was planned. Checks are limited to the fields and provenance available; their purpose is to make potential issues visible and traceable, not to substitute for source review.