Build Automated Dataset Quality-Control Reports

Why Automate Dataset Quality-Control Reports?

A quality-control report should make it possible to answer, consistently and quickly: what was checked, how many records were eligible, what failed, which thresholds were crossed, and what remains unresolved? A list of error messages alone cannot answer these questions. Nor does a clean-looking summary establish quality if the checks, denominators, or input files are unclear.

In population pharmacokinetics, the checks span fields that jointly define an analysis record: dose, time, concentration, censoring information, and covariates. Earlier material on reconciling source files with analysis records helps establish which records are being checked. An automated report then turns specified checks into a repeatable account of the resulting dataset. It supports review; it does not decide by itself whether an unusual value is an error or authorize a correction.

Definition: An automated dataset quality-control report is a reproducible summary generated by applying versioned, explicitly defined checks to a specified dataset. For each check, it reports the checked record unit, eligibility rule, denominator, number and identity of failures, failure rate when defined, warning rule and trigger status, and resolution status.

Define the Check Before Counting

A check is interpretable only when its unit and denominator are explicit. One check might assess one dose record per row; another might assess one concentration observation. A subject-level covariate check might count subjects, not observations. If those units are mixed, the resulting rates do not have a clear meaning. State the unit in the report and keep the count consistent with it.

For a given check, let \(n\) be the number of eligible records and \(k\) the number of eligible records that fail that check. The failure rate is \(k/n\) when \(n>0\). Eligibility must be rule-specific. For example, a rule about concentration censoring may apply only to concentration records, while a rule about dose amount applies only to dose records. A record excluded from a check is not a pass: report it as ineligible or not evaluated, with the reason where useful.

For a required-field check, a missing value is usually a failure, so the denominator includes every record expected to have that field. For a conditional check, the denominator includes only records meeting the condition. For instance, a rule requiring a censoring indicator to agree with a censoring category applies to records for which that information is expected. Write this distinction into the rule rather than allowing software defaults to determine it silently.

Key reporting rule: Report both the number of failing records and the number checked. If \(n=0\), the failure rate is undefined, not zero; report “not evaluated” or “no eligible records” and explain why.

A report should also distinguish failures per rule from unique records failing at least one rule. One record can fail a dose check and a time check, so adding the two rule counts can count that record twice. If an overall count is included, calculate it as the number of distinct records in the union of the failure sets, and label it accordingly.

Warning Thresholds and Their Boundaries

A failure and a warning are not interchangeable. A failed record is one that violates a defined rule. A warning is a report-level signal that a chosen count or rate has crossed a stated threshold. A threshold can prompt review even when every recorded failure is already understood; conversely, a small failure rate does not make an individual failure acceptable.

Suppose a warning is triggered only when a failure rate is greater than a proportion threshold \(T\). For \(n>0\) and \(0\leq T<1\), the smallest integer failure count that triggers the warning is:

k_{\min}=\lfloor nT\rfloor+1

To see why, a warning requires \(k/n>T\), or \(k>nT\). The smallest integer strictly greater than \(nT\) is \(\lfloor nT\rfloor+1\). The condition \(T<1\) matters: when \(T\geq1\), no feasible failure count can trigger a strictly-greater-than warning, because \(k\leq n\) and therefore \(k/n\leq1\). At \(T=1\), the rate can equal 1 but cannot exceed it. If the rule instead says “at least \(T\),” its boundary is different and must be implemented as specified; do not substitute one comparison for the other.

Worked Example: Checking a Dose-Record Warning Threshold

In a hypothetical dataset, a rule checks whether each of 120 eligible dose records has a positive dose amount. Three records fail. The report warns if the failure rate is greater than 2%.

The denominator is 120 eligible dose records, and the observed rate is \(3/120=0.025\), or 2.50%. For \(n=120\) and \(T=0.02\), the smallest triggering count is:

k_{\min}=\lfloor120(0.02)\rfloor+1=\lfloor2.4\rfloor+1=3

Since \(k=3\), the warning triggers: 2.50% is greater than 2%. The report should show the three failing record identifiers and their resolution status, not merely state “warning.” If there had been two failures, the rate would have been \(2/120=1.67\%\), below the threshold, and no warning would trigger.

Specify Checks for PK Fields

Automated checks are most useful when each rule can be read as a precise statement about a field and a defined record set. The following examples illustrate the kinds of checks that can be reported; the actual permitted values and conditions must come from the study’s data definitions and analysis conventions.

Field group	Possible report check	Details the report should preserve
Dose	Required dose amount is present and satisfies the specified range or coding rule.	Which dose records are eligible; units; failed record identifiers.
Time	Required event or sample time is present and follows the defined time convention.	Time origin and units; which event types are checked; rule-specific exclusions.
Concentration	Required result and its analysis status follow the specified data rules.	Observation identifiers; whether records are quantified, missing, or otherwise classified.
Censoring	Censoring status and associated fields are mutually consistent under the study’s coding rules.	Applicable records; expected combinations; failures separated from inapplicable records.
Covariates	Required covariate values, units, categories, or imputation flags meet their definitions.	Subject or observation as the unit; source or imputation status where relevant.

These checks should not impose a universal numerical rule where the study has not defined one. For example, a concentration that looks unusually high is not automatically a data error. A quality-control report can identify a range-screen flag and provide the record for review; it should preserve the distinction between a flag and a confirmed correction.

For censored concentrations, the report should apply the study’s stated representation consistently. It should check the intended relationship among the result, censoring indicator, and any bound or category required by that representation. It should not silently convert or replace values as part of reporting. For covariates, checks can report missingness, unexpected units or categories, and whether imputed values are identified as required. They should respect the distinction between observed and imputed values.

Worked Example: Separating Censoring Failures From Ineligible Records

Consider a hypothetical set of 80 concentration records. The study’s specified coding rule requires a censoring indicator and a compatible censoring category for each record. All 80 records are eligible. Two records have combinations that violate the rule; the other 78 are consistent. The warning threshold is strictly greater than 2.5%.

Here \(n=80\), \(k=2\), and the failure rate is \(2/80=0.025\), or 2.50%. The integer trigger count is:

k_{\min}=\lfloor80(0.025)\rfloor+1=\lfloor2\rfloor+1=3

The warning does not trigger: two failures give a rate equal to the threshold, not greater than it. Both records still appear as failures and require a recorded disposition. If, instead, a rule applied only to records carrying a censoring indicator, its denominator would be the number meeting that condition; the report must not use 80 unless all 80 are eligible for that particular check.

Resolution Status and Reproducibility

A reliable report keeps three questions separate: did a record fail a rule, did the aggregate count trigger a warning, and what happened to each failure? Resolution status might identify a failure as open, corrected, or reviewed with a documented reason for retaining the value. The permitted status labels should be defined for the project. A documented exception is not the same as a pass, and a correction should remain traceable to its source and rationale.

The report should retain enough information to reproduce its results: the input dataset identity or version, rule-set version, run date, and relevant configuration such as threshold values and whether comparison is strict or inclusive. It should show each rule’s denominator, failures, rate, threshold, trigger result, and status counts. Where a check is not run, the report should state that rather than imply it passed.

A concise run summary can show total records processed, records eligible for each rule, unique records with one or more failures, and unresolved items. However, these summaries must not replace rule-level results. When a rule has no eligible records, display its denominator as zero and its rate as undefined. When a record fails multiple rules, preserve the separate rule-level findings while avoiding double counting in any distinct-record total.

Worked Example: Reporting Covariate Failures and Their Status

A hypothetical report checks the units of a required baseline covariate for 50 eligible subject records. The allowed unit is specified as kilograms. One record has an unrecognized unit and one has no unit recorded; the other 48 pass. A warning is defined as a failure rate greater than 2%.

The failure count is \(k=2\), the denominator is \(n=50\), and the rate is \(2/50=0.04\), or 4.00%. The threshold count is:

k_{\min}=\lfloor50(0.02)\rfloor+1=\lfloor1\rfloor+1=2

The warning triggers because two failures exceed the 2% rate threshold. Suppose review establishes that the first record’s unit was verified from source and corrected with an audit trail, while the second remains unresolved pending source review. The report therefore shows one corrected and one open failure. The warning remains triggered for this run because its result describes the observed check; recording a resolution does not retroactively change the count that triggered it.

Diagnostics, Alternatives, and Limitations

Several symptoms indicate that an automated report may be misleading or malfunctioning:

An unexpected denominator: eligibility conditions may be wrong, or records may have been dropped before the check.
A rate with no denominator: the report may be dividing by zero or treating “not evaluated” as zero failures.
Counts that change after an unrelated edit: the check may depend on an undocumented field, input order, or mutable source.
Overall failures greater than the number of eligible records: rule-level counts may have been added despite overlapping records.
Warnings at the wrong boundary: the implemented comparison may use “at least” where the rule specifies “greater than,” or vice versa.
Failures disappearing after correction: the report may have overwritten history instead of preserving the original finding and its disposition.

Manual review is valuable for unusual records and contextual judgment, but it is difficult to repeat consistently across many records. A static one-time summary is easy to read but can become stale when data or rules change. Automated, versioned checks make repeat runs comparable and make routine violations easier to locate; they do not replace source review, clinical interpretation, or decisions about whether an exception is justified. Thresholds are operational triggers, not proof that data below the threshold are error-free or that every flagged record is wrong.

The report is only as sound as its definitions. A rule can consistently produce the wrong answer if its denominator, permitted values, or time convention is incorrect. Changes to source data, transformations, or rule definitions should therefore be traceable. Reviewers should be able to identify the records behind each count and verify that the displayed result follows the stated rule.

Key takeaway: A reproducible quality-control report makes checks auditable: define the unit and eligible denominator, report failures and warning logic separately, and preserve a traceable resolution status for each finding.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A rule checks 24 eligible records and warns when the failure rate is greater than 100%. Can any feasible count trigger it? Explain.
A rule has 30 eligible records, 3 failures, and a warning threshold of 10% with a strictly greater-than comparison. Does the warning trigger?
One record fails both a dose check and a time check. How should rule-level counts and a unique-record total represent it?
A check has no eligible records. What denominator and rate should the report show?

Solutions. (1) No. A failure count cannot exceed 24, so the largest possible rate is \(24/24=1\), or 100%, which is not greater than 100%. (2) No. The observed rate is \(3/30=0.10\), exactly 10%, so it does not exceed the threshold. (3) Count the record once in each of the two rule-level failure counts, and once in the distinct-record total for records failing at least one rule. Do not obtain the unique total by simply adding rule counts. (4) Show \(n=0\) and a rate of undefined or “not evaluated,” not zero; state why no records were eligible.