Exclusion Rules And Their Consequences

Why Exclusion Rules Matter

After defining who may enter a study, an analyst may apply additional rules that remove some otherwise eligible records from the analysis. Those rules can be operationally necessary, such as removing duplicate rows, or they can make the analysis easier, such as retaining only records with every desired field present. In either case, an exclusion changes which observations contribute to the results.

The central question is not simply whether a rule reduces the data set. It is which records it removes, and what population the remaining records describe. A model built after excluding records is directly informed by the retained records, not automatically by every entity that met the original cohort definition.

Eligibility and exclusion are related but distinct. Eligibility rules define entry into the cohort; exclusion rules then identify which eligible records will not be used for a specified analysis. For example, a study might define eligible policy-periods as policies active on a particular date, then exclude a subset because a predictor field is unavailable. The retained data can therefore describe a narrower group than the eligible cohort.

The intended prediction population should guide these choices. If the intended use includes every eligible policy-period, excluding records with a predictor that will often be unavailable in practice may make the analysis less relevant to that use. Conversely, if the model is explicitly intended only for a well-defined subset, a narrower analysis can be appropriate. The important requirement is to state the scope rather than letting exclusions define it silently.

Counting What the Rules Remove

When several rules apply, a record that fails two rules is still one excluded record. Adding the separate rule counts without checking overlap can therefore overstate the total number removed. The calculation follows the same inclusion-exclusion principle used to count overlapping sets.

Here, \(A\) and \(B\) are the sets of records failing two different rules. The intersection is subtracted because those records appear in both separate counts. For more than two rules, a table that records combinations of rule failures is often the clearest audit trail.

How Exclusions Change a Summary

Removing records can change the composition of the analyzed group, even when no retained value has been altered. A useful way to see this is to split the original records into the retained and excluded groups. For an unweighted mean, the original mean is a size-weighted average of the means in those two groups.

Here, \(n_R\) and \(n_E\) are the numbers retained and excluded, \(n=n_R+n_E\), and \(\bar{y}_R\) and \(\bar{y}_E\) are their respective outcome means. This identity follows by partitioning the total outcome sum: \(\sum_{\text{all}} y_i=\sum_R y_i+\sum_E y_i=n_R\bar{y}_R+n_E\bar{y}_E\). Dividing by \(n\) gives the displayed formula. It applies to an ordinary unweighted mean of the same outcome measured on all records; it does not by itself describe weighted estimates or outcomes unavailable for some records.

If the excluded group has a different outcome average or a different mix of characteristics, the retained-sample average need not equal the full-cohort average. This difference is not automatically an error: the retained group may be the intended target. But it is important to distinguish a result for the retained group from a claim about the broader eligible cohort.

Common Exclusion Rules and Their Interpretation

A rule should be assessed in context: what records does it remove, why are those records being removed, and is the resulting scope consistent with the intended prediction use? Common examples include:

• Unavailable predictor: records lacking a field needed by the chosen analysis are omitted. The remaining sample is limited to records for which that field is available.
• Invalid or inconsistent value: a record may be removed when a documented data-quality check establishes that a value cannot be used as recorded. If a value can be corrected reliably, correction may be preferable to dropping the whole record.
• Duplicate record: repeated rows may be removed when they are confirmed to represent the same intended observation. The analyst must first check that the rows are not valid repeated observations with distinct meanings.
• Extreme value: a threshold may remove unusually large or small values. An extreme observation is not necessarily erroneous; the rule can remove genuine cases that matter to the prediction task.
• Restricted operational scope: records outside a stated use case may be excluded, such as a class of products the model is not intended to score. This can be appropriate when the intended scope is clearly specified.

The rule should be described in terms of records, not just a technical action. “Removed rows with missing values” is incomplete if the analyst has not said which fields were checked, how many rows were affected, whether rules overlapped, and what kinds of records remained.

A Defensible Exclusion Process

A disciplined process makes exclusions auditable and helps prevent convenience from silently changing the analysis population. It also separates the reason for a rule from its numerical consequences.

A sensitivity comparison does not make an unjustified rule acceptable. It shows whether an analytical conclusion depends on that choice. The final report should identify the rule used and state the population to which the resulting analysis applies. That documentation is part of the data design, not merely a technical appendix.

Common Mistakes

• Adding rule counts without subtracting overlap. A record failing two rules is one record removed, not two.
• Treating exclusions as neutral. A smaller sample can have a different mix of records and a different outcome summary.
• Equating extreme with erroneous. A high value may be a valid observation. Use evidence and the intended outcome definition to decide how it should be handled.
• Reporting only the final sample size. Without counts by rule and overlap, readers cannot tell how the retained set was formed.
• Generalizing beyond the retained scope. State whether a result describes all eligible records or only those satisfying the analysis rules.