Why Cohort Construction Needs to Be Documented
An observational analysis depends on which people enter the cohort. Readers need to know not only the final number analyzed, but also how records moved from the available data to the group meeting the study’s eligibility rules. A count without its construction history is difficult to reproduce and may conceal important selection decisions.
Eligibility documentation should let another analyst answer practical questions: What was the starting population? What did each criterion mean in the data? At what time was it evaluated? How many records remained after each step, and why were others removed? These questions concern the cohort’s construction, not the later definition of treatment strategies, outcomes, or analytical methods.
The unit being counted must remain clear throughout. In a person-level study, a data source may initially contain records rather than unique people. State when records are deduplicated or otherwise linked into person-level units, and report counts in a way that does not confuse records with people. The earlier material on record linkage is relevant to identifying units across data sources; here the emphasis is on showing how the resulting units satisfy or fail eligibility rules.
Make Eligibility Rules Operational
A criterion is useful for reproducibility only if it can be applied consistently to the data. “Adults with adequate prior history,” for example, leaves unanswered what age threshold defines an adult, what data indicate that a person has a prior history, and how much history is required. Replace broad descriptions with operational definitions, including the data fields, codes, dates, or other evidence used to implement each rule.
For each inclusion and exclusion criterion, document at least its substantive meaning, operational definition, time of assessment, and order in the cohort algorithm. Where relevant, specify the code set or version, observation window, source table, and rule for resolving conflicting records. A criterion based on age, for instance, should state the age threshold and the date at which age is calculated. A criterion requiring prior data availability should state the required duration and the dates over which availability is assessed.
Timing matters because a person can satisfy a rule at one point and fail it at another. Identify the study’s relevant index date as established by its protocol, then say whether each criterion is assessed before, on, or after that date. This tutorial does not define the treatment strategies or comparator; those must be specified separately. The aim here is to make the timing of cohort eligibility unambiguous.
Some people fail more than one criterion. A report should say whether exclusion categories are mutually exclusive or overlapping. If the algorithm assigns one reason per person, give the priority order used to assign that reason. If it reports overlapping counts, label them as such and do not add them as though they were separate people. Without this detail, readers cannot tell whether the displayed totals are supposed to reconcile.
Worked Example: Turning Broad Eligibility Language Into Rules
Suppose a hypothetical study plans to identify adults with a specified diagnosis in a health-system database. The initial protocol description says “adults with sufficient prior enrollment and complete source records.” That sentence is not yet reproducible.
A more operational description might specify that a person must be at least 18 years old on the protocol-defined index date; have at least 365 days of observable enrollment before that date; and have a qualifying diagnosis recorded in the specified clinical data during the study’s stated calendar period. It should identify the data field or code set used for the diagnosis and specify how duplicate person records are resolved.
The criteria also need a stated assessment time. Age can be calculated on the index date, while the enrollment requirement is assessed over the 365 days before it. The diagnosis criterion can be assessed over the specified calendar period. These rules describe eligibility without implying that the database perfectly measures age, enrollment, or diagnosis in every case.
Finally, the report should provide the order in which failures are counted. If a person is under 18 and also lacks 365 days of enrollment, a mutually exclusive flow diagram needs a rule for which failure is assigned first. The person still fails eligibility either way, but the reported reason counts depend on that priority rule.
Reconcile Counts From One Step to the Next
A participant-flow diagram is a compact account of cohort assembly. Each displayed count should have a defined unit and denominator. For a sequential algorithm in which each removed unit is assigned to exactly one step, let \(N_{j-1}\) be the number entering step \(j\) and \(r_j\) the number removed at that step. The number remaining is \(N_j=N_{j-1}-r_j\). Repeating this accounting yields the final eligible cohort.
This identity follows by substituting each step into the next: \(N_1=N_0-r_1\), \(N_2=N_1-r_2=N_0-r_1-r_2\), and, continuing through step \(K\), \(N_K=N_0-\sum_{j=1}^{K}r_j\). It applies when the removals \(r_j\) are mutually exclusive, sequential counts. If categories overlap, their counts cannot simply be summed in this equation; instead, report the overlap structure or make the sequential assignment rule explicit.
A useful flow diagram names each step and gives the number entering, the number removed, the reason, and the number remaining. Add the source period or relevant population definition where needed. If the final eligible cohort is later reduced for analysis-specific reasons, do not silently fold those reductions into eligibility exclusions: distinguish cohort construction from subsequent analytic-sample selection.
Worked Example: Checking a Sequential Cohort Flow
In a hypothetical database extraction, 12,480 records are identified. Deduplication produces 11,960 unique people. The eligibility algorithm then removes, in order, 640 people below the age threshold, 410 without the required prior observation period, 275 outside the calendar period, and 835 who fail the specified clinical criterion. Assume every person is assigned to the first applicable exclusion step.
After deduplication, the cohort count is 11,960 unique people. Applying the first criterion gives \(11{,}960-640=11{,}320\). The next step leaves \(11{,}320-410=10{,}910\), and the calendar-period step leaves \(10{,}910-275=10{,}635\).
The final step gives \(10{,}635-835=9{,}800\) eligible people. As a check, the total removed by the four mutually exclusive eligibility steps is \(640+410+275+835=2{,}160\), and \(11{,}960-2{,}160=9{,}800\), matching the sequential calculation.
The 520 difference between the 12,480 extracted records and 11,960 unique people is a record-processing count, not an eligibility exclusion, unless the study explicitly defines it otherwise. The report should describe deduplication and keep the record count separate from the person-level eligibility flow.
12,480 extracted records from the specified hypothetical data period.
520 repeated records removed through the stated deduplication rule; 11,960 unique people remain.
Remove 640 below the age threshold, then 410 without the required prior observation period, then 275 outside the calendar period, then 835 failing the specified clinical criterion.
9,800 unique people meet all stated eligibility criteria.
Choose a Clear Reporting Format
The participant-flow diagram is strongest when it shows the sequence and makes the count arithmetic visible. A short accompanying narrative should describe rules that do not fit neatly into boxes, such as the deduplication procedure, the operational definition of a criterion, or the priority rule for overlapping failures. A table can be useful when many criteria need precise definitions or when a diagram would become difficult to read.
| Format | Best use | Common weakness |
|---|---|---|
| Participant-flow diagram | Showing the sequence of counts from records or candidates to the eligible cohort | May omit the detailed operational definition of a criterion |
| Eligibility table | Listing each rule, its data-based definition, and assessment time | May not make the total count progression easy to follow |
| Narrative description | Explaining unusual decisions, ordering, or data-processing rules | Counts and exclusion steps can be hard to audit in prose alone |
These formats complement rather than replace one another. A diagram can show what happened to the counts; a table or narrative can explain how each decision was implemented. Use consistent terminology and the same unit of analysis across all of them.
Worked Example: Handling Overlapping Exclusions
A hypothetical study has 1,000 candidate people. A preliminary review finds that 120 lack the required prior observation period, 90 fail a clinical eligibility rule, and 30 fail both. These are overlapping counts, so adding \(120+90\) would count the 30 people who fail both rules twice.
The number who fail at least one of these two criteria is \(120+90-30=180\), using the inclusion-exclusion calculation. Thus, if these are the only two criteria, \(1{,}000-180=820\) people pass both. The 820 is the final count for these criteria, not \(1{,}000-120-90=790\).
A sequential flow could instead assign people who fail the observation-period criterion first: remove 120, of whom 30 also fail the clinical rule. Among the remaining 880, remove the 60 who fail only the clinical rule. The count remaining is \(1{,}000-120-60=820\). The diagram must say that the second removal is among people remaining after the first step. Alternatively, the report may show overlapping criterion counts and state that those counts are not additive.
Diagnostics, Failure Modes, and Limitations
Before finalizing a flow diagram, check the arithmetic and the meaning of each count. Every transition should reconcile with the prior count, and the final number should agree with the stated eligibility rules and the cohort used in the next stage of reporting. Confirm that people, records, and linked units have not been mixed, and that repeated records or failed linkages are described at the stage where they occur.
A discrepancy does not necessarily mean the underlying analysis is invalid, but it does mean the cohort construction is not yet auditable. Common causes include counting overlapping exclusion categories as if they were unique, omitting a processing step, changing the denominator between boxes, or reporting a count from a later analysis as though it were the eligible cohort. If a criterion’s denominator is not the full set entering the step, label the denominator explicitly.
- Vague rules: “Incomplete records” is not reproducible unless the report states which fields or records are required and how completeness is assessed.
- Unclear timing: A rule without a reference date or window may be applied differently by different analysts.
- Hidden precedence: If failure categories overlap but the diagram implies they are mutually exclusive, the displayed reasons are misleading.
- Changing units: A record count cannot be subtracted from a unique-person count without explaining the transition.
- Unexplained final-count differences: If the analyzed cohort is smaller than the eligible cohort, report the further step and its reason rather than leaving the difference unexplained.
Documentation improves transparency and reproducibility; it does not, by itself, eliminate bias or establish that the included people represent a broader target population. Eligibility criteria can affect who is included, and the flow diagram shows the resulting counts, not the causal consequences of those decisions. Interpretation still depends on the design, data quality, and analysis.
A common mistake is to assume that exclusion counts can always be added: “The study removed 120 people for one reason and 90 for another, so 210 people were excluded.” That conclusion is correct only if the categories are mutually exclusive. When 30 people are in both groups, the number excluded is \(120+90-30=180\). State whether categories overlap, or use a sequential priority rule that assigns each person to one reason.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A report says that 4,000 records yielded 3,600 people, then lists 100, 150, and 50 exclusions, leaving 3,300 eligible people. What should be checked?
- Two exclusion criteria identify 80 and 60 people, with 20 people failing both. How many fail at least one criterion?
- Why should a report state the date on which an age criterion is assessed?
- What is the difference between the eligible cohort and the analytic sample?
- Give two ways to report overlapping exclusion categories without double-counting.
Solutions. (1) The arithmetic for the eligibility steps is internally consistent because \(3{,}600-100-150-50=3{,}300\). The report should also explain the 400-record to 3,600-person transition, confirm that the counts are unique people assigned to mutually exclusive steps, and verify that the final count agrees with the stated cohort. (2) By inclusion-exclusion, \(80+60-20=120\) people fail at least one criterion. (3) Age can change over time; stating the assessment date makes the rule reproducible and clarifies who meets the threshold. (4) The eligible cohort meets the documented eligibility rules; the analytic sample is the set used in a particular analysis and may be smaller after additional analysis-specific steps. (5) Report overlapping counts and identify the overlap so they are not treated as additive, or apply a stated sequential priority rule and report mutually exclusive removals at each step.