Why Missingness and the Analysis Sample Belong in a Report
A regression result is based on the records that enter the fitted model, not necessarily on every person or unit originally eligible for the analysis. If some values are absent, the rule for handling them can change the analysis sample. Readers therefore need to know both how much information was missing and which records were analyzed.
These details matter for reproducibility: another analyst should be able to apply the stated rule and determine which records belong in the analysis. They also matter for interpretation. If the analyzed records differ from the broader eligible group, the results directly describe the analyzed group; applying them to a wider population requires appropriate justification.
The eligible group and analysis sample should be defined for the particular analysis being reported. Different models may require different variables, so they may have different analysis samples. A record missing a value for a variable that is not required by a given model need not be excluded from that model.
Report Counts With Their Denominators
A useful missingness summary names the variables that matter to the model, gives the number of missing values for each, and states the denominator used for the percentages. The denominator is often the eligible group for that analysis, but the report should say so explicitly. Counts without denominators can be difficult to interpret: 12 missing values mean different things in groups of 60 and 6,000.
Variable-specific missing counts are not generally additive. The same record may be missing values on more than one variable, so summing the counts can count that record more than once. If the report gives a total number excluded, it should be the number of distinct records excluded under the stated rule, not simply the sum of the variable-specific missing counts.
Worked Example: Summarizing Missing Values by Variable
Suppose a hypothetical analysis begins with 240 eligible records. The model requires an outcome, an exposure, and age. An audit finds 18 missing outcome values, 9 missing exposure values, and 6 missing age values. The pairwise overlaps are 3 records missing both outcome and exposure, 2 missing both outcome and age, and 1 missing both exposure and age; 1 record is missing all three.
For each variable, divide its missing count by the 240 eligible records. The outcome has \(18/240=0.075\), or 7.5%, missing; the exposure has \(9/240=0.0375\), or 3.75%, missing; and age has \(6/240=0.025\), or 2.5%, missing. These percentages all use the same explicit denominator.
To find the number missing at least one required value, count the union of the three missing-value groups. The inclusion-exclusion calculation is:
Thus 28 distinct records are incomplete on at least one required variable. Under the stated complete-case rule, the analysis sample contains \(240-28=212\) records. The individual missing counts sum to 33, but reporting 33 as the number excluded would be wrong because records missing more than one value would be counted repeatedly.
State the Rule That Forms the Analysis Sample
A clear report gives the inclusion rule in words precise enough to reproduce. For a complete-case analysis, identify the variables that must be observed and state that a record is included only when it has observed values for all of them. Then give the number of eligible records, the number excluded by the rule, and the final analysis-sample size. If the rule includes any additional restrictions, such as a specified range of dates or removal of duplicate records, describe those too.
Do not rely on a phrase such as “after cleaning” to explain sample formation. It does not tell the reader which records were removed or why. Nor should a report leave it unclear whether missingness was assessed for all variables in the data set or only those required for the particular model.
Worked Example: Calculating the Complete-Case Sample
Consider a hypothetical set of 200 eligible records. A particular model requires observed values for the outcome, treatment group, age, and clinic. Among these records, outcome is missing for 12, treatment group for 8, age for 15, and clinic for 0. The audit also shows that the pairwise intersections of the first three missing-value groups contain 2 records missing both outcome and treatment group, 4 missing both outcome and age, and 3 missing both treatment group and age. One record is missing all three values.
Because the model requires all four variables, a record is excluded if it is missing any one of them. The clinic variable has no missing values, so it does not add to the excluded set. The number of distinct records missing at least one required value is:
The complete-case sample therefore has \(200-27=173\) records. A reproducible report could state: “Of 200 eligible records, 27 had a missing value for the outcome, treatment group, or age and were excluded; the complete-case model used 173 records. Clinic was observed for all eligible records.”
The report should not imply that 12, 8, and 15 exclusions occurred in separate groups. The overlap information shows why the total excluded is 27 rather than \(12+8+15=35\). It should also not treat the 27 as the number excluded from every possible model: a model requiring a different set of variables could produce a different sample.
Consider Which Population the Results Describe
The analysis sample is the group whose observed records contributed to the fitted model. The eligible group is often the population the analysis was intended to address, but a complete-case rule can make the two groups differ. A report should describe that difference when important, rather than assuming that exclusions leave the sample unchanged.
One practical check is to compare the eligible group and analysis sample on important characteristics that are observed for both groups. This can reveal changes in composition, such as a subgroup forming a larger or smaller share of the analyzed records. Such a comparison does not, by itself, prove that a regression estimate is biased or establish the direction or size of any difference in the results. It does help readers judge how directly the analyzed records reflect the eligible group.
Worked Example: Comparing an Eligible Group With the Analysis Sample
Suppose 200 records are eligible for a hypothetical analysis. A baseline characteristic divides them into a higher-score group of 50 and a lower-score group of 150. Under the model’s stated complete-case rule, 20 records from each group are excluded. The analyzed sample therefore contains 30 higher-score records and 130 lower-score records, for a total of 160.
In the eligible group, the higher-score group makes up \(50/200=0.25\), or 25.0%. In the analysis sample, it makes up \(30/160=0.1875\), or 18.75%. The lower-score group’s share changes from \(150/200=75.0\%\) to \(130/160=81.25\%\).
The exclusion rates also differ: \(20/50=40.0\%\) of the higher-score group was excluded, compared with \(20/150=13.33\%\) of the lower-score group. Thus, although the same number of records was excluded from each group, the exclusions were not proportionate to group size. A concise report could state that the analysis sample included 160 of 200 eligible records and that the higher-score group made up 18.75% of the analysis sample, compared with 25.0% of the eligible group.
This comparison establishes a change in sample composition; it does not establish that the regression results are biased by a particular amount. It does mean that a claim about the broader eligible group needs to be considered in light of who was excluded and the purpose of the analysis.
A Short Justification for Comparing Group Shares
The reason group composition is informative can be shown with a simple calculation. Suppose an eligible group contains \(N_H\) records in a subgroup of interest and \(N\) records overall. If \(E_H\) subgroup records and \(E\) records overall are excluded, the subgroup’s share changes from \(N_H/N\) to \((N_H-E_H)/(N-E)\). These two shares are equal only when exclusions preserve the original composition.
For example, if every subgroup has the same exclusion rate \(r\), then the subgroup retains \((1-r)N_H\) records and the full group retains \((1-r)N\) records. The subgroup share after exclusion is:
The common retention factor cancels, so the composition is unchanged. If subgroup exclusion rates differ, that cancellation does not generally occur, and the analysis sample can have a different composition. This argument explains why comparing inclusion rates or subgroup shares is useful; it does not prove that composition is the only feature relevant to generalizing a regression result.
What a Complete Report Should Make Clear
A reader should be able to answer three questions from the report: what was missing, how the analysis sample was selected, and what group the resulting model directly describes. A compact table can help, provided its denominators and sample rule are stated.
| Report item | What to state |
|---|---|
| Eligible group | Number of records before the model-specific missing-value rule |
| Missingness | Missing count for each important variable, with the denominator for each percentage |
| Sample rule | Variables required for inclusion and any other exclusion conditions |
| Analysis sample | Number of distinct records excluded and number used in the model |
| Composition | Important observed differences between eligible records and analyzed records |
| Interpretive scope | The group directly represented by the analyzed records and any caution about broader claims |
If reasons for missing values are known, report them accurately and distinguish known reasons from speculation. If the cause is unknown, say so rather than assigning a cause. Similarly, do not describe a complete-case sample as representative merely because it is large: sample size alone does not show that excluded and included records are alike.
Common Mistakes
- Adding missing counts as though they were separate records. The counts may overlap. Use the distinct number excluded under the rule, accounting for overlap where needed.
- Giving percentages without a denominator. State whether each percentage uses the eligible group, the analysis sample, or another clearly defined group.
- Leaving the inclusion rule vague. “Complete cases” is not enough unless the report identifies which variables had to be observed.
- Using one sample count for every model. Different models may require different variables and therefore have different analysis samples.
- Claiming that analyzed records automatically represent everyone eligible. Describe the analyzed group and relevant differences; do not infer representativeness from the sample count alone.
- Claiming that observed differences prove a particular bias. A composition difference is important context, but it does not by itself quantify how estimates would change.
Check Your Understanding
Answer each question using the definitions and reasoning in this tutorial.
- A model requires an outcome and two predictors. Why might the sum of the three variable-specific missing counts differ from the number of records excluded?
- In a group of 300 eligible records, 24 are missing an outcome value. What denominator would you use for the missing-outcome percentage if reporting it among all eligible records, and what is the percentage?
- What information must accompany the phrase “complete-case analysis” for another analyst to reproduce the sample rule?
- A subgroup accounts for 40% of the eligible group but 25% of the analysis sample. What does this comparison establish, and what does it not establish by itself?
- Why might two regression models fitted to the same eligible group use different analysis samples?