Accounting for Exclusions, Attrition, and Analysis Samples

Why the Analysis Sample Needs Its Own Accounting

A comparative observational study does not move directly from a database query to an effect estimate. Records may be removed during eligibility assessment, treatment and comparator groups may be restricted by the analytic design, matching may leave records unused, and outcome data may be unavailable for some otherwise eligible people. A reader needs to know how many people remain at each stage, and why the counts change.

A clear accounting trail is more than a presentation aid. It lets readers check whether the reported analysis follows the stated design, see how much data contribute to each outcome, and assess whether follow-up differs between groups. Counts alone do not establish that an analysis is unbiased, but unexplained changes in counts make its evidence difficult to evaluate.

Make Every Change in the Flow Checkable

A useful flow starts with the records identified and proceeds through eligibility, treatment and comparator classification, any design-specific selection, and outcome availability. At each step, state the count entering the step, the count excluded or retained, and the reason. If counts are reported separately by treatment group, readers can see where group sizes diverge.

For a simple sequential flow, the accounting identity is:

The identity applies only when each person is counted once at that step. If a report gives several reasons for exclusion and one person has more than one reason, those reason-specific counts overlap and must not be added as if they were distinct people. Instead, say that reasons are nonexclusive, or use a stated rule to assign one primary reason per person. State whether exclusions were applied sequentially, since the order can affect the count attributed to each reason.

For each outcome analysis and group \(g\), a basic outcome-availability count is:

If the analysis additionally requires other variables, such as complete covariates, say so and show those losses separately. A reader should not have to infer whether the denominator is the eligible cohort, the matched cohort, or only those with an observed outcome.

Report Matching and Weighting Without Hiding the Counts

After matching, distinguish the people eligible for matching from those actually matched and those who remain unused. In a one-to-one match without replacement, for example, 2,100 matched treated people paired with 2,100 matched comparators form 2,100 pairs and involve 4,200 records. State the matching ratio and whether matching used replacement: with replacement, the same comparator can appear in more than one matched set, so the number of matched sets and the number of unique comparator people are not necessarily equal.

Also distinguish people who were not selected as matches from people who were ineligible for the analysis. “Not matched” describes the design’s selection; it does not by itself mean a person lacked an outcome or was lost to follow-up. If matching used a caliper or other restriction, give the number not matched and the relevant reason rather than presenting only the final matched count.

Weighting usually retains a record-level analytic set but gives records different contributions. Therefore a weighted analysis should report the number of people with usable outcomes in each group, clarify how weights were handled when outcomes were unavailable, and provide useful summaries of the weights or weighted totals. A weighted total is not a headcount. One diagnostic summary of weight concentration is the effective sample size:

where \(w_i\) is the analysis weight for record \(i\). This quantity can be smaller than the number of records when weights are unequal; it is a summary of weight concentration, not a literal count of people. State how weights were normalized and whether the summary is for the full weighted cohort or the outcome-specific analysis sample. Do not substitute \(n_{\text{eff}}\) for the actual record count.

Attrition and Follow-Up Are Group-Specific

When an outcome is intended to be assessed over a follow-up window, report the number with outcome information and the number without it separately for treated and comparator groups. For a clearly defined starting group, the observed-outcome proportion and attrition proportion are:

The starting denominator must be named: it might be all eligible people, the matched group, or another explicitly defined analysis cohort. The two proportions sum to one only when each person in that starting group is classified as either having an observed outcome or having an unavailable outcome for that window. Explain other categories, such as an outcome that is not yet due to be measured.

“Attrition” should not collapse distinct processes into one vague label. Administrative end of data, loss of contact, withdrawal, death, and missing outcome measurement can have different meanings. Report the follow-up window and how outcome status was determined; where relevant, summarize actual follow-up by group using an appropriate statistic and state its units. A person whose outcome is unavailable is not automatically known to have experienced no outcome.

A group difference in attrition is a reason for closer scrutiny, not proof by itself that an estimate is biased. The concern depends on why outcomes are unavailable and how that relates to treatment, prognosis, and the outcome. Conversely, similar attrition percentages do not prove that missingness is innocuous. The report should describe the pattern and the handling of unavailable outcomes rather than treating a percentage comparison as a correction.

A Practical Reporting Sequence

Use a consistent sequence, with counts attached to each reason and group where possible. This helps a reader reproduce the path from records identified to the analysis sample.

Diagnostics, Alternatives, and Common Pitfalls

Before finalizing a flow, check that each sequential step reconciles arithmetically and that the totals agree with the analysis data. Check that group-specific counts sum to reported totals, that the outcome-analysis denominators match the stated cohort, and that matched-set counts are not confused with unique people. For weighted analyses, verify that the reported weight summaries refer to the same records and weight version used for the stated analysis.

A flow diagram can make a long sequence of exclusions easier to follow, while a table can make group-specific outcome availability and attrition easier to compare. These are presentation alternatives, not substitutes for an accounting rule: both should display the same denominators, explain overlapping reasons, and identify the analysis sample. A brief text summary can orient readers, but text alone is often difficult to audit when many stages or outcomes are involved.

• Pitfall: treating all exclusion reasons as separate people. Adding nonexclusive reason counts can overstate the number excluded. State that reasons overlap, or report mutually exclusive primary reasons.
• Pitfall: reporting only the final matched sample. The reader cannot tell how many eligible people were unused. Show the pre-match and post-match group counts and explain unmatched records.
• Pitfall: calling a weighted total a sample size. Weights change contributions, not the number of people observed. Report record counts and weight summaries separately.
• Pitfall: using one overall attrition percentage. It can hide group differences. Give group-specific numerators, denominators, and percentages for the stated follow-up window.
• Pitfall: assuming the analysis sample is the same for every outcome. Outcome availability can differ. Reconcile counts for each outcome or clearly state when outcomes share the same analysis sample.

A transparent accounting trail does not settle questions about confounding, missing-data assumptions, or the population represented by a design. It does allow readers to see which observations contributed and where information was lost. Keep the account descriptive: report the flow and follow-up accurately, without treating a count difference as a causal finding.