Verifying Analysis Population and Exclusion Counts

Why Population Counts Need an Audit

A model can run successfully on the wrong set of subjects. A report may state that 112 participants were eligible, while the input file contains 110 unique identifiers; alternatively, a table may count 112 subjects even though the analysis file contains many more rows. Neither mismatch is explained by a successful run. An independent review therefore needs to reconstruct how the declared population becomes the population actually represented in the model input.

This audit is about who is included, who is excluded, and whether the counts reconcile. It does not establish that the eligibility criteria are scientifically appropriate. It checks whether the stated rules were applied to identifiable subjects and whether the resulting subject and record counts agree with the analysis artifacts. The earlier material on tracing analysis records back to source data and auditing units and derived dataset variables provides useful context: here, the focus is the population flow and its accounting.

Define the Counting Unit Before Counting

A count is meaningful only after its unit and scope are stated. A subject count counts distinct subject identifiers. A record count counts rows at a specified level in a specified dataset. One subject may contribute multiple records, so a row count is not a substitute for a unique-subject count. The audit should report these quantities separately and name the file or dataset version from which each was obtained.

The candidate set \(C\) must also be explicit. It might be all subjects on a screening roster, all subjects satisfying a protocol-level criterion, or another population declared by the analysis specification. Those are not interchangeable choices. Identify the roster, the date or version, the identifier field, and any rule that determines whether a subject belongs in the candidate set. If the scope is ambiguous, the reviewer cannot know whether an apparent discrepancy is a counting error or a difference in population definition.

For a staged flow, define the starting set at each stage. For example, a protocol-screened set may be reduced by one set of exclusions to form an eligible set, and a later analysis-specific rule may reduce that set again. Each stage needs its own count and rule. Do not add exclusion counts from different stages to a starting total unless the categories are defined over the same starting set and do not overlap.

In practice, the final equality can be checked in both directions: identify subjects expected in the model input but absent from it, and identify subjects present in the model input but not in the declared population. Equal totals alone are insufficient: two sets can have the same size and different members. Compare the identifiers, not only the summary numbers.

Exclusions Must Have a Traceable Basis

For each excluded subject, the review should be able to connect the identifier to a documented exclusion rule and evidence that supports applying that rule. A usable audit trail identifies the criterion, the source field or record used to evaluate it, the rule or programming version, and the resulting subject-level decision. The evidence should be specific enough for another reviewer to reproduce the classification from the referenced material.

A label such as “not evaluable” is not, by itself, a traceable basis. The reviewer needs to know what condition “not evaluable” represents, how it was determined, and whether that condition was part of the declared population definition. A derived flag can be a useful implementation, but it should be possible to trace that flag to its source and rule. The audit is not complete if a count can be reproduced only by trusting an unexplained flag.

Exclusion reasons may overlap. A subject can meet more than one exclusion condition, but must be counted only once in the overall number of excluded subjects. There are two sound ways to report this: show the overlapping reason flags and separately report the number of unique excluded subjects, or assign each excluded subject one primary reason using a documented priority rule. Do not sum overlapping flag totals and present the result as a unique-subject total.

Justifying the Count Reconciliation

The arithmetic behind a non-overlapping exclusion table is a partition argument. For each candidate subject \(i\), define an indicator \(I_A(i)\) that equals 1 if the subject is in the analysis population and 0 otherwise. Define \(I_{E_j}(i)\) similarly for exclusion category \(j\). If every candidate is assigned to exactly one of the analysis population or the mutually exclusive exclusion categories, then for each candidate:

Summing over all candidates gives the count identity. The sum of an indicator over a set is the number of members for which it equals 1:

This identity depends on the partition assumption. If categories overlap, a subject contributes more than once to the sum; if a candidate is not classified anywhere, the sum is less than one for that subject. Thus a failed equation is a signal to investigate the classification, not a reason to adjust a count until the totals appear to agree.

Reconcile Records Separately From Subjects

A complete population check also states the record counts, but it does not treat them as subject counts. If a subject-level exclusion removes a subject from the analysis population, that subject’s records may also be absent from the model input. Separately, a record-level rule may remove particular rows while leaving the subject in the analysis population. Each operation needs a stated unit, a defined rule, and its own accounting.

The arithmetic for records depends on the actual data flow. A row count can decrease by the number of rows removed at a particular step, but the decrease cannot be inferred from the number of subjects excluded: subjects may contribute different numbers of rows. Record counts should therefore be obtained from the named datasets or a reproducible count, and the unique-subject count should be calculated independently from identifiers in the resulting model input.

A Reproducible Review Workflow

A reliable review combines a population-flow table, identifier-level comparisons, and evidence for each exclusion. The following sequence makes the logic visible and helps isolate the source of a discrepancy.

Diagnostics, Alternatives, and Limits

A mismatch can arise from a missing or duplicate identifier, a stale roster, a changed eligibility rule, an unrecorded exclusion, a filter applied at the wrong unit, or a difference between the declared population and the actual model input. Check these possibilities before concluding that the model input is incorrect. Duplicate rows can inflate a row count without changing the number of subjects; inconsistent identifier formatting can make one subject appear to be two; and an unclassified candidate can make an otherwise plausible flow fail to balance.

There are several useful ways to perform the count check. A manual flow table is easy to inspect and communicate, but can hide overlapping categories or transcription errors. A scripted count is repeatable and can compare identifier sets, but is only as trustworthy as its input files, code, and stated counting unit. An independent recomputation from the roster and exclusion evidence provides a stronger cross-check than merely re-running the submitted count code. These approaches complement one another: a table communicates the result, while reproducible set comparisons test membership.

A count match is necessary evidence of consistency, not proof of a valid analysis population. It cannot show that the criteria were justified, that source information was correct, or that an unobserved exclusion was appropriate. Similarly, if the specification does not define the candidate universe or how overlapping reasons are counted, there may be no unique count to verify. Record the ambiguity and seek clarification rather than silently choosing a definition.