Why Spot-Check Dose and Observation Records?
A dataset can be reproducible in the narrow sense that the same code produces the same output, yet still contain records that do not agree with their sources. A dose entered at the wrong time, a recorded route that differs from the source, or an observation assigned the wrong collection time can affect how a model input represents the clinical record. Spot-checking is a practical independent review: select records, compare specified fields with source records, and document what the comparison establishes.
The earlier material on tracing analysis records back to source data addresses how to locate the supporting source. Here the emphasis is on checking selected dose and observation fields accurately and on interpreting the scope of the resulting evidence. A spot-check is not a substitute for reviewing the dataset’s construction or for resolving every ambiguity in the source.
Set the Review Rules Before Comparing Records
A sound check starts by defining its scope. Specify the dataset version, the record types and fields to review, the source records to use, and how the sample will be selected. Also state how to treat source precision and unresolved conflicts. For example, a source may record a dose time only to the nearest minute. A difference of seconds in a system-generated timestamp should not automatically be called an error if the source does not support that precision. Conversely, a difference of several minutes should not be dismissed as harmless unless a documented rule permits it.
Use a stable way to identify the analysis record and its corresponding source entry. Compare like with like: a dose record with evidence about that dose, and an observation record with evidence about that observation. Record the exact values as displayed in both places, including date and time where applicable. If the source is ambiguous, conflicting, or incomplete, label the result as requiring clarification rather than deciding which value must be correct.
For each comparison, the reviewer should record enough information for another person to repeat it: the dataset version, record identifier, source location or reference, compared fields, reviewer, review date, result, and any follow-up. Do not silently edit the dataset during the check. An identified discrepancy should enter the applicable correction and review process so that the change, its basis, and its effect can be tracked.
List the eligible dose and observation records, the fields to check, and the source records that support the comparison.
Use a recorded random selection, a stated risk-based rule, or a documented combination. Preserve the selection method and the selected identifiers.
Transcribe the source and analysis values, apply only documented precision or data-handling rules, and classify each item as matching, discrepant, or unresolved.
Describe the evidence, route confirmed discrepancies for resolution, and state clearly what the checked sample does and does not establish.
Worked Comparisons
Worked Example: Comparing Dose and Observation Fields
Suppose a reviewer selects two dose records and one observation record from a hypothetical analysis dataset. The relevant source entries show: dose A was administered at 08:00, amount 100 mg, oral route; dose B was administered at 20:00, amount 100 mg, oral route; and an observation was collected at 10:15. The dataset contains the following corresponding values: dose A at 08:00, 100 mg, oral; dose B at 20:30, 100 mg, oral; and the observation at 10:51.
Dose A. The time, amount, and route match the source. Record this as a match for the fields checked. This conclusion is limited to those fields and this record; it does not establish that other fields or records are correct.
Dose B. The amount and route match, but the times differ. The time difference is \(20{:}30-20{:}00=30\) minutes. If the source records the time to the minute and no documented rule explains this difference, classify the time as a discrepancy and preserve both values. Do not change the dataset value during the comparison.
Observation. The time difference is \(10{:}51-10{:}15=36\) minutes. If the source supports minute-level precision and there is no applicable documented transformation, classify the observation time as discrepant. If the source time is ambiguous or a documented convention could explain the difference, classify it as unresolved until that evidence is checked.
The check does not decide which value should be used merely because one looks more plausible. It establishes where the recorded values agree or differ and provides a traceable basis for resolution.
What a Random Sample Can Detect
A random sample can provide a transparent chance-of-detection calculation when the review frame and the number of erroneous records are treated as fixed. Let \(N\) be the number of records in the frame, \(m\) the number of records with an error, and \(n\) the number selected by simple random sampling without replacement. Suppose each selected record can be classified correctly and every error in the frame would be detected if its record were reviewed. These assumptions describe the calculation, not a guarantee that an actual audit will meet them.
There are \(\binom{N}{n}\) possible samples of size \(n\), all equally likely under simple random sampling without replacement. A sample contains no erroneous record only if all \(n\) records come from the \(N-m\) error-free records. There are \(\binom{N-m}{n}\) such samples, provided \(n\leq N-m\). Thus the probability of finding at least one error is one minus the probability of selecting only error-free records.
This is a probability conditional on the stated frame, sample design, and fixed number of errors. It is not the probability that the dataset contains an error, nor does it tell us how many errors are present when the true \(m\) is unknown.
Worked Example: Detection Probability in a Random Sample
Consider a hypothetical frame of \(N=100\) dose records, of which \(m=5\) are erroneous. A reviewer selects \(n=10\) records by simple random sampling without replacement. Assume that reviewing any erroneous record would identify it.
The probability of detecting no error is the probability that all ten records come from the 95 error-free records:
Therefore,
Under these assumptions, a random check of 10 records has about a 41.62% chance of finding at least one error when exactly 5 of the 100 records are erroneous. It still has about a 58.38% chance of finding none. A clean sample therefore does not prove that the other records are error-free.
Sampling Choices and Their Limits
Simple random sampling gives each record in the defined frame an equal chance of selection. It is useful when the review objective is broad coverage and a reproducible sample. It can, however, miss rare errors, especially when the sample is small. A risk-based sample deliberately gives more attention to records with reasons for concern, such as records from a known problematic transfer or a period with a high rate of manual entry. That can make the check more useful for finding plausible problems, but its results do not have the same interpretation as an equal-probability random sample.
Stratified sampling divides the frame into meaningful groups and selects records within each group. For example, dose records and observation records can be sampled separately so both types receive review. The allocation should be stated in advance, and the results should be reported by stratum. A convenience sample—such as checking only the first records in a file—may be quick, but it can systematically omit other parts of the frame and gives no defensible probability-of-detection calculation unless its selection design supports one.
A complete review checks every record in the defined frame. It removes sampling uncertainty about whether an unreviewed record in that frame contains an error, but it still depends on accurate record matching, correct interpretation of the source, and a reliable review process. When the consequences of an error are high, the frame is small, or initial findings suggest a pattern, expanding the sample or reviewing the full frame may be warranted.
Worked Example: Reporting a Stratified, Risk-Based Check
A hypothetical reviewer defines a frame of 40 records: 20 dose records flagged by a prior process for extra attention and 20 routine dose records. The reviewer randomly selects 5 records from each group. One discrepancy is found among the 5 flagged records; none is found among the 5 routine records.
The checked fraction in each group is \(5/20=0.25\), or 25%. Across the two groups, 10 of 40 records were checked, so the overall checked fraction is \(10/40=0.25\), also 25%. The direct finding is that one of the five checked flagged records had a discrepancy and none of the five checked routine records did.
It would be incorrect to report that exactly four of the 20 flagged records are discrepant by multiplying the observed \(1/5\) by 20. That calculation treats a small sample proportion as if it were the known population proportion. The sample supports documenting the one observed discrepancy and considering follow-up; it does not establish the total number or location of discrepancies in either group. If estimating overall rates is an objective, the sampling design and appropriate uncertainty calculations must be specified for that purpose.
Diagnostics, Common Mistakes, and Follow-Up
Review quality can fail even when a sample was selected correctly. Watch for a mismatch between the record identifier and the source entry, a source with insufficient time precision, conflicting source documents, or a comparison that applies an undocumented tolerance. Also check that selected records have not been quietly replaced because they were inconvenient to locate. These issues should be logged as limitations or unresolved cases, not converted into matches.
- Calling a clean sample proof of a clean dataset. A sample can miss errors. State the sample design and its scope; do not claim unreviewed records were verified.
- Using a targeted sample as if it were random. Risk-based selection may improve the chance of finding particular problems, but it does not support the same population-wide probability statements as simple random sampling.
- Resolving conflicts by intuition. A value that seems clinically plausible is not necessarily the source-supported value. Preserve the conflict and request documented resolution.
- Combining unlike fields or record types without reporting them separately. A total count of checked records can conceal that few observation times—or few dose routes—were actually reviewed. Report coverage by the field or record group relevant to the review objective.
- Making silent corrections. A spot-check is a review, not an undocumented editing step. Keep the comparison evidence and route proposed corrections through controlled resolution.
If a confirmed mismatch appears, determine whether it is isolated, repeated, or associated with a common source or processing step. Expand the review when the finding suggests a broader issue, and document why the additional records were selected. The aim is not to make a small sample look conclusive; it is to make the evidence useful, reproducible, and appropriately limited.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A random sample of 8 records is selected from a frame of 50. There are exactly 3 erroneous records in the frame. Write the probability that the sample detects at least one error.
- A reviewer deliberately selects records from a time period with known data-entry concerns. Can the reviewer use the simple-random-sample detection formula to describe the chance of finding an error? Explain.
- The source supports an observation time only to the nearest 15 minutes, while the analysis record differs by 4 minutes. What should the reviewer conclude?
- A check of 12 records finds no discrepancy. State one valid conclusion and one conclusion that would be too strong.
Solutions. (1) With \(N=50\), \(m=3\), and \(n=8\), the probability is \(1-\binom{47}{8}/\binom{50}{8}\). The numerator counts samples with no erroneous records; subtracting that probability from 1 gives the chance of detecting at least one. (2) Not by default. The formula assumes equal-probability selection without replacement from the stated frame; deliberate risk-based selection does not meet that design unless its actual selection probabilities are accounted for. (3) The 4-minute difference is finer than the source’s stated 15-minute precision, so the reviewer should not call it a confirmed discrepancy on that basis alone. Record the values and apply the review’s documented precision rule or seek clarification. (4) A valid conclusion is that no discrepancy was found among the 12 checked records for the fields reviewed. It would be too strong to conclude that the entire dataset contains no discrepancies.