Comparing Included and Excluded Records

Why Compare Included and Excluded Records?

A complete-case regression uses records with observed values for every variable required by the model. The resulting analysis may therefore describe a different subset of the original records. Comparing included and excluded records can reveal whether the observed composition changes in ways that matter for interpreting the analysis. For example, included records might have lower average ages, or excluded records might cover a wider range of a predictor.

These comparisons are diagnostics. A difference may prompt questions about how records became incomplete, but it does not by itself prove that the regression estimate is biased. Nor does finding similar summaries prove that the estimate is unbiased. The comparison describes observed data; it cannot reveal values that were not recorded or settle whether the conditions for valid complete-case regression hold.

Definition: For a specified regression model, a record is included if it has observed values for all variables required by that model, and excluded otherwise. A comparison of included and excluded records summarizes observed values in the two groups to describe possible changes in sample composition.

Make the Comparison Interpretable

Start by defining the groups from the model's complete-case rule, then summarize the outcome and each predictor in the same units and coding used for the model. For each variable, report the number of observed values in each group. An excluded record may still have an observed value for some variables, while lacking another value that led to its exclusion. Do not treat a missing value as zero or silently count it as an observed value.

For a quantitative variable, useful summaries include the observed sample size, mean, standard deviation, and minimum and maximum. For a binary variable, report the observed number in each category and the proportion in a clearly stated category. For a categorical predictor with several levels, show the observed count or proportion at each level. These summaries complement one another: means describe location, standard deviations describe spread, and ranges show the observed extremes.

Always state which group is subtracted from which when reporting differences. Here, differences and standardized differences are defined as included minus excluded. A positive value therefore means the included group has the larger value or proportion; a negative value means it has the smaller one.

Standardized Differences

A raw difference in means has the variable's original units, which makes it concrete but can make comparisons across variables difficult. A standardized mean difference expresses the difference relative to a typical amount of variation in the two groups. For a quantitative variable, define it as follows, using the observed values in each group:

\text{SMD} = \frac{\bar{x}_{\text{included}}-\bar{x}_{\text{excluded}}} {\sqrt{(s_{\text{included}}^2+s_{\text{excluded}}^2)/2}}

The denominator is the square root of the average of the two sample variances. This gives the two groups equal weight in defining the scale, even if their observed sample sizes differ. The SMD has no units. Its sign preserves the direction of the raw difference, while its magnitude describes the difference relative to within-group variation. It is a descriptive measure, not a test statistic, and it has no universal cutoff that determines whether composition has changed in a consequential way.

For a binary variable, let \(p_{\text{included}}\) and \(p_{\text{excluded}}\) be the observed proportions in the category of interest. A corresponding standardized difference is:

\text{SMD} = \frac{p_{\text{included}}-p_{\text{excluded}}} {\sqrt{\{p_{\text{included}}(1-p_{\text{included}}) +p_{\text{excluded}}(1-p_{\text{excluded}})\}/2}}

State the category whose proportions are compared. If the denominator is zero, the standardized difference is undefined; report the proportions and counts directly instead. For a categorical variable with more than two levels, level-by-level proportions can be compared, with the category being contrasted stated each time.

Worked Example: Comparing an Outcome and a Predictor

Suppose a hypothetical cohort contains 100 records, of which 80 meet the complete-case rule for a regression of a follow-up score on age. The remaining 20 are excluded. Among excluded records, the score is observed for 15 and age for 18. The table gives hypothetical summaries calculated from observed values only.

Variable	Group	Observed n	Mean	SD	Range
Follow-up score (points)	Included	80	72	12	43–96
Follow-up score (points)	Excluded	15	65	15	38–91
Age (years)	Included	80	42	10	21–67
Age (years)	Excluded	18	48	12	29–74

For the score, the raw mean difference is \(72-65=7\) points. The standardized difference is:

\text{SMD}_{\text{score}} = \frac{72-65}{\sqrt{(12^2+15^2)/2}} = \frac{7}{\sqrt{184.5}} = \frac{7}{13.58} \approx 0.515

The included records have a mean score 7 points higher, or about 0.515 of the equal-weight within-group standard-deviation scale. For age, the raw difference is \(42-48=-6\) years:

\text{SMD}_{\text{age}} = \frac{42-48}{\sqrt{(10^2+12^2)/2}} = \frac{-6}{\sqrt{122}} \approx -0.543

Thus, the included records are younger on average by 6 years, about 0.543 of the corresponding standard-deviation scale. The sample sizes differ across groups for the score comparison because only 15 excluded records have an observed score, and only 18 have an observed age. These calculations compare observed values; they do not supply the missing scores or ages. The differences describe composition but do not establish whether the fitted regression is biased.

Range Plots Show Observed Coverage

A simple range plot places the included and excluded groups on separate rows and draws a horizontal segment from each group's observed minimum to its maximum. The scale and units should be shared across the two rows. It is helpful to label each segment with its observed sample size, and it can be useful to mark the group means as separate points. This makes it easier to see whether observed values occupy similar parts of the variable's scale or whether one group extends farther into a tail.

A range plot is deliberately limited: it shows only the endpoints, not how values are distributed between them. A single unusual observation can greatly extend a range, and overlapping ranges do not imply that the groups have similar means or distributions. Read the plot alongside the means, standard deviations, proportions, and observed sample sizes rather than treating it as a complete description.

Worked Example: A Small Mean Difference but a Wider Range

In another hypothetical comparison, a baseline biomarker is observed for 60 included records and 12 excluded records. Its values are measured in the same units in both groups. The included records have mean 5.2, standard deviation 0.8, and range 3.1 to 7.4. The excluded records have mean 5.3, standard deviation 1.6, and range 2.0 to 9.0.

The raw mean difference is \(5.2-5.3=-0.1\) units. The standardized difference is:

\text{SMD} = \frac{5.2-5.3}{\sqrt{(0.8^2+1.6^2)/2}} = \frac{-0.1}{\sqrt{1.6}} \approx -0.079

The means are close relative to the average within-group variation, but the observed range among excluded records is much wider: 2.0 to 9.0, compared with 3.1 to 7.4 among included records. A range plot would show the excluded group's segment extending farther at both ends. The plot and standard deviations therefore add information not captured by the small difference in means. They do not show why those records were excluded, and the smaller excluded group means its observed extremes should be interpreted with its sample size in mind.

A Short Justification: Why Standardize?

Standardizing makes the mean difference independent of the measurement unit. Suppose a quantitative variable is converted from \(x\) to \(x'=a+bx\), where \(b\) is positive. Each group mean becomes \(a+b\bar{x}\), so the difference between means becomes \(b\) times the original difference. Each group standard deviation becomes \(b\) times its original standard deviation. Substituting into the SMD gives:

\frac{(a+b\bar{x}_{\text{included}}) -(a+b\bar{x}_{\text{excluded}})} {\sqrt{\{b^2s_{\text{included}}^2+b^2s_{\text{excluded}}^2\}/2}} = \frac{b(\bar{x}_{\text{included}}-\bar{x}_{\text{excluded}})} {b\sqrt{(s_{\text{included}}^2+s_{\text{excluded}}^2)/2}} = \text{SMD}

The shift \(a\) cancels in the numerator, and the positive scale factor \(b\) cancels between numerator and denominator. Thus, changing a measurement from years to months changes the raw mean difference and standard deviations but not the SMD. This is useful for comparing differences across variables, but it does not make the variables substantively interchangeable or the SMD a measure of bias.

Worked Example: Comparing a Binary Predictor

Suppose smoking status is observed for 80 included records and 18 excluded records. In this hypothetical example, 20 included records and 9 excluded records are smokers. The proportions of smokers are \(20/80=0.25\) and \(9/18=0.50\), respectively. The raw difference, included minus excluded, is \(0.25-0.50=-0.25\), or 25 percentage points.

Using the binary-variable formula:

\text{SMD} = \frac{0.25-0.50} {\sqrt{\{0.25(1-0.25)+0.50(1-0.50)\}/2}} = \frac{-0.25}{\sqrt{(0.1875+0.25)/2}} = \frac{-0.25}{\sqrt{0.21875}} \approx -0.535

The negative sign indicates a smaller observed proportion of smokers among included records; the observed proportion is 25 percentage points lower. This is a descriptive comparison based on the stated observed counts. It does not establish that smoking caused records to be excluded or that the regression estimate is biased.

What These Comparisons Can—and Cannot—Tell You

Differences in observed outcomes or predictors can suggest that the complete-case sample has a different composition from the records excluded from the model. The direction and size of those differences may help identify variables worth examining more closely. For example, a large difference in an outcome's observed mean can be relevant to questions about the analysis sample, while a difference in predictor ranges can indicate that parts of the observed predictor scale are less represented among included records.

But these are marginal summaries: each variable is examined separately. They do not show whether the relationship between outcome and predictors differs between groups, or what values the missing outcomes and predictors would have taken. Consequently, even a set of small standardized differences cannot prove that complete-case regression is unbiased. Conversely, a noticeable difference does not by itself prove bias; the implications depend on the target of the analysis and the missing-data process. Conditions for valid complete-case regression must be considered separately.

No distributional assumption, such as normality, is needed to calculate these descriptive summaries or plot observed ranges. The key practical requirements are to define inclusion consistently with the regression being considered, use the same variable definitions and units in both groups, and make clear which observed records contribute to each summary.

Common Mistakes

Comparing every excluded record as though every variable were observed. An excluded record may have an observed outcome but a missing predictor. Report the observed sample size for each variable and group.
Using a difference without naming its direction. State whether the calculation is included minus excluded or the reverse, and retain that convention in the interpretation.
Treating a standardized difference as a significance test. It is a descriptive scale-free comparison, not a p-value and not proof of an important difference. Interpret its direction and magnitude in context.
Reading overlapping ranges as evidence of identical groups. A range plot shows endpoints only. Check means, standard deviations, proportions, and sample sizes as well.
Calling a difference proof of bias—or similarity proof of no bias. Observed summaries diagnose sample composition; they cannot establish the missing values or the validity of the regression estimate.

Key takeaway: Compare included and excluded records using observed outcome and predictor summaries, standardized differences, and range plots. Report the available sample sizes and units, and treat every observed difference as a diagnostic clue—not a verdict about bias.

Check Your Understanding

Answer each question using the definitions and conventions in this tutorial.

A model uses an outcome and three predictors. State the rule for deciding whether a record is included in a complete-case analysis.
Why might the observed sample size for a variable among excluded records be smaller than the total number of excluded records?
An included-minus-excluded standardized difference is negative. What does its sign tell you, and what does it not tell you?
What additional information can standard deviations and ranges provide when the two groups have similar means?
Why can neither large nor small observed differences, by themselves, establish whether a complete-case regression estimate is biased?