Why Duplicate Records Matter in Population PK
Population pharmacokinetic data are assembled from records describing subjects, dose events, biological samples, and covariates. A repeated row can look harmless, especially if its values match another row exactly. But if it represents a second copy of the same sample, it may cause that concentration to be treated as an additional observation. A duplicated dose record can likewise misrepresent the dosing history. Conversely, two similar-looking rows may be legitimate separate samples, doses, or measurements. The central task is therefore not simply to find repeated values: it is to decide whether records refer to the same underlying event or measurement.
A reliable review separates candidate detection from resolution. A key or comparison rule can flag rows for review, but the flag alone does not establish that they are duplicates. Resolution depends on the meaning of the records, their source and available identifiers, and a rule that can be explained and reproduced.
The distinction matters in all three record families. Two rows for one subject and sample may be duplicate imports, but different assay replicates from one specimen are not automatically duplicates. Two dose rows may describe one administration entered twice, or two administrations close together. Two covariate rows may be repeated copies of one measurement, or separate measurements taken at different times. Interpret the record type before comparing values.
Keys and Comparisons
A key is a set of fields used to identify a record's intended entity. A useful key reflects the study's data structure and the definition of the record. For example, a sample key might use subject identifier, occasion, and sample identifier; a dose key might use subject identifier, occasion, and dose-event identifier; and a covariate key might use subject identifier, covariate name, and the measurement's effective date or visit identifier. The precise key depends on how those records were defined and linked.
A key is a screening device, not a guarantee of uniqueness or truth. A key that omits event identity can combine two legitimate records. One that includes a row number created during file import can conceal duplicate records, because the copies will have different row numbers. Values that are meaningful for comparing records but not for identifying them—such as dose amount or a recorded sample time—can be compared after generating candidates.
A useful comparison has two parts. First, identify records that share the chosen key or a plausible alternative key. Second, compare fields that should agree if the rows represent the same entity. Decide in advance which fields must match exactly and which may differ within a justified tolerance. Tolerances must be specific to the field and its recording process; they should not be chosen after seeing which records a reviewer wants to combine. If a difference could change the event's meaning, it is not a harmless near-match.
For example, two samples from one subject can have the same concentration and recorded time while being separate specimens. Conversely, two rows with slightly different recorded times can be copies of one sample if the source records establish that one time was transcribed differently. The decision rests on event identity and evidence, not on a single matching or differing field.
A Defensible Review and Resolution Process
State what one row represents and select fields that identify that entity. Keep record families separate: a sample key should not be treated as a dose key.
Find repeated keys, identical rows, and near matches using specified comparisons. Retain enough information to see which fields agree and which differ.
Review event identifiers, source records, record type, and whether repeated entries could be legitimate replicate or separate events. Do not infer identity solely from numerical similarity.
Retain distinct records, exclude a confirmed redundant copy, merge complementary fields when they describe one record, or leave a conflict unresolved pending review.
Keep the original records, identify which rows were linked or removed from the analysis dataset, state the rule and evidence, and record who or what applied the rule and when.
The logic behind this process is straightforward. A key narrows the search to records that might refer to the same entity. Field comparisons then test agreement with the expected record structure. If the key is incomplete, different events can collide; if a field used as a key is unstable or missing, copies can fail to match. Since neither comparison stage is infallible, a flagged pair needs a decision rule grounded in the data's meaning and provenance.
Worked Examples
Worked Example: Excluding an Exact Duplicate Sample Row
Suppose a hypothetical study's sample file contains these rows:
| Subject | Occasion | Sample ID | Recorded time (h) | Concentration (mg/L) |
|---|---|---|---|---|
| P014 | 1 | S081 | 4.00 | 2.36 |
| P014 | 1 | S081 | 4.00 | 2.36 |
The proposed sample key is subject, occasion, and sample ID. Both rows have the same key and identical values in the compared fields. The sample ID identifies one sample, and a source-file check confirms the second row was introduced by importing the same record twice. These facts support a confirmed duplicate, not merely a pair of similar observations.
The file has two rows but one distinct sample. After excluding one redundant copy from the analysis dataset, the sample contributes one observation: \(2.36\ \text{mg/L}\) at \(4.00\ \text{h}\), not two observations at those values. The resolution log should identify both source rows, specify which copy was retained, and state the import evidence and rule. If the rows instead had different sample IDs or were documented assay replicates, identical values alone would not justify exclusion.
Worked Example: Reviewing a Near-Duplicate Dose Event
In a hypothetical dose file, the proposed key is subject, occasion, and administration identifier. Two rows share subject P022, occasion 2, and administration identifier A17. One row gives a dose amount of \(500\ \text{mg}\) and time \(08{:}00\); the other gives \(500.0\ \text{mg}\) and time \(08{:}01\). The source administration record identifies one administration, and the second row is a transcription from a secondary file.
The amount values agree numerically: \(500 = 500.0\). The recorded times differ by one minute, so the rows are not exact duplicates. If the prespecified review rule treats differences of no more than two minutes as candidate matches, the time comparison is:
This calculation flags the pair under that rule; it does not establish that the times are interchangeable in every context. Here, the administration identifier and source record support a single event. The reviewer can retain one canonical dose record using the designated authoritative source for time, and link the secondary row to it in the resolution log. The time should not be averaged to \(08{:}00.5\): averaging would create a value that appears to be an observed administration time but is supported by neither source. If the identifiers or source evidence instead indicated two administrations, both rows would remain.
Worked Example: Merging Complementary Covariate Records
A hypothetical covariate file has two rows for subject P031, visit V3, and the same measurement form. The first contains height \(168\ \text{cm}\) and no weight value; the second contains height \(168\ \text{cm}\) and weight \(62.4\ \text{kg}\). The form identifier and audit trail establish that the rows are two exports of the same form, not repeated measurements.
The shared subject, visit, and form identifier make the rows candidates. Their height values agree, and the second row supplies a weight value absent from the first. Given the evidence that both rows represent the same form, the two can be merged into one covariate record with height \(168\ \text{cm}\) and weight \(62.4\ \text{kg}\). The merge should retain links to both source rows and state which fields came from each.
If the two height values had instead been \(168\ \text{cm}\) and \(171\ \text{cm}\), the records would contain a conflict. The reviewer should not silently choose one, average them, or label the difference a duplicate without checking the source and measurement context. Until the conflict is resolved by evidence and a stated rule, preserve the distinct source values and flag the pair for review.
Diagnostics, Alternatives, and Limitations
Useful diagnostics include counts of repeated keys, the number of exact row matches, the number and type of near-match disagreements, and examples reviewed from each candidate category. Summaries should be produced separately for dose, sample, and covariate records. After resolution, check that the resulting dataset has the intended record counts and that retained rows still have valid links to their source identifiers. A sudden large increase in excluded rows, or many conflicts in one source, may indicate a faulty key, a changed file format, or an overly broad comparison rule rather than widespread true duplication.
Three common approaches have different strengths and risks:
| Approach | Useful for | Main risk |
|---|---|---|
| Exact full-row comparison | Finding repeated copies with identical compared values | Misses copies with formatting or transcription differences; can flag legitimate identical events |
| Key-based comparison | Finding repeated entities even when some fields differ | A weak or incorrect key can combine distinct records |
| Near-match screening with review | Finding likely copies with small, specified differences | Thresholds can create false matches; evidence is still needed for resolution |
Automatic exclusion is appropriate only when the rule reliably identifies redundant copies in the relevant data source and the action remains auditable. Manual review is more appropriate for conflicting values, uncertain event identity, or records that may be replicates. A hybrid approach often works well: use reproducible code to generate and classify candidate pairs, then review ambiguous or consequential cases.
Duplicate handling has limits. A comparison cannot recover an event identifier that was never recorded, establish which conflicting source is correct without supporting evidence, or guarantee that all duplicates were found. Identical values do not prove identical events, and a one-field difference does not prove distinct events. Record-level deduplication also does not replace checks of whether subjects, events, and samples were linked correctly.
Common Mistakes
- Deleting every repeated value. Two distinct samples or dose events can have identical values. Confirm shared event identity before excluding a row.
- Using a key that omits event identity. Subject and occasion alone may identify many samples or dose events, not one record. Add fields that distinguish the intended entity.
- Treating a tolerance as proof. A one-minute difference may trigger a candidate rule, but it does not prove that two dose records represent one event.
- Resolving conflicts by averaging or choosing the latest row. These operations can invent a value or hide uncertainty. Use source evidence and a prespecified hierarchy, or leave the case unresolved.
- Removing records without an audit trail. Keep the source records and document the rule, decision, and links to affected rows so another analyst can reproduce the result.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two sample rows have the same subject, time, and concentration but different sample identifiers. Should one automatically be excluded? Explain.
- A dose key matches across two rows, but the dose amounts conflict. What should happen before either row is removed or the values are combined?
- Two covariate rows share a key and contain complementary fields. What evidence is needed before merging them?
- A near-match rule flags two dose times that differ by one minute, within a prespecified two-minute threshold. What does the flag establish, and what does it not establish?
Solutions. (1) No. Different sample identifiers may represent distinct samples, and matching time and concentration do not establish that the rows describe the same specimen. Check the sample identifiers and source records. (2) Treat the rows as a conflicting candidate pair. Review event identity and provenance, then apply a documented authoritative-source rule if evidence supports one; otherwise retain the source records for adjudication rather than silently choosing or averaging values. (3) Evidence should establish that both rows describe the same covariate measurement or source form, such as a shared form identifier and supporting audit trail. Without that evidence, the fields may come from separate measurement occasions. (4) The flag establishes that the pair meets the specified screening criterion. It does not prove that the rows represent one administration; event identifiers and source evidence must support that decision.