Why Diagnostic Counts Need Their Own Table
A diagnostic plot can look complete while representing only a fraction of the records or subjects in an analysis. Conversely, a plot with fewer points may simply be using a diagnostic quantity that is available for fewer records. Counts make that difference visible: they tell readers how much data was eligible, how much was usable for a particular diagnostic, and how many subjects contributed.
The key is to define the denominator before reporting a percentage. A count such as “12 usable observations” is hard to interpret without knowing whether 12 out of 13 or 12 out of 120 records could have appeared. Likewise, a subject count and a record count answer different questions: one subject may contribute many records.
These definitions depend on a reproducible scope. For example, the analysis may already have excluded records outside a planned population or a specified study period. Those decisions should be applied consistently before counting \(N_0\) and \(M_0\). Each record should have a stable identifier, and the subject identifier should be available so that distinct subjects can be counted. If duplicated rows are counted as separate records by mistake, both the numerator and denominator can be distorted.
Usability is diagnostic-specific. A residual-versus-time plot needs both a residual and a time value; a residual-versus-prediction plot needs a residual and a prediction. A record may therefore be usable for one plot but not another. A numerical value that is extreme is not, for that reason alone, missing or unusable. Counts should reflect stated eligibility and availability rules, not silently remove inconvenient diagnostic values.
Build the Table From a Defined Scope
A useful diagnostic-count table states its unit of counting and denominator. For each diagnostic or summary, report the in-scope record count, the usable record count, the number of distinct contributing subjects, and the corresponding coverage percentages where appropriate. A compact table can also show counts of missing required quantities. If several quantities are missing on the same record, those missingness counts overlap and must not be added as though they represented distinct records.
For a record-level plot, the primary percentage is usually record coverage. For a subject-level summary, record coverage alone does not show how many subjects have enough information to receive a summary. If a summary requires a minimum number of usable records per subject, report subject-summary coverage using that eligibility rule explicitly.
Worked Example: Counts for Several Diagnostic Plots
Consider a hypothetical data set with 12 in-scope records from four subjects, each contributing three records. Residuals are missing on two records, time is missing on one, and predictions are missing on two. Suppose the two residual-missing records are A1 and B2; the time-missing record is C3; and the prediction-missing records are A2 and D1. A plot requires all quantities named in its columns.
For a residual-only diagnostic, \(12-2=10\) records have residuals. Record coverage is \(100 \times 10/12=83.3\%\). For residual versus time, the missing residual and time records are A1, B2, and C3, so \(12-3=9\) paired records are usable, giving \(100 \times 9/12=75.0\%\). For residual versus prediction, the missing records across the required pair are A1, B2, A2, and D1. There are four distinct records in that union, so \(12-4=8\) pairs are usable, giving \(100 \times 8/12=66.7\%\).
The two missing-residual records and two missing-prediction records cannot simply be added to get the number missing for the paired plot: A1 and B2 are already missing residuals, while A2 and D1 are additional records. The distinct-record union is what determines usable pairs. In this example, all four subjects have at least one usable record for each of the three diagnostics, so subject coverage is \(4/4=100\%\) for each. That does not make record coverage 100%; the two measures have different units and denominators.
| Diagnostic | In-scope records | Usable records | Record coverage | Contributing subjects | Subject coverage |
|---|---|---|---|---|---|
| Residual only | 12 | 10 | 83.3% | 4 of 4 | 100% |
| Residual versus time | 12 | 9 | 75.0% | 4 of 4 | 100% |
| Residual versus prediction | 12 | 8 | 66.7% | 4 of 4 | 100% |
Record Coverage and Subject-Summary Coverage
A record-level coverage percentage describes the fraction of eligible records that contribute to a diagnostic. It does not reveal whether usable data are spread across subjects or concentrated in a few. Subject coverage addresses that second question, but its rule must match the output. For a plot that can show a single usable point from a subject, “at least one usable record” may be a suitable contributing-subject definition. For a subject-level summary that requires at least \(k\) usable records, a subject contributes only if that threshold is met.
Worked Example: Reporting Record and Subject Coverage
In a hypothetical analysis, five subjects each have three in-scope records, for \(N_0=5 \times 3=15\) records and \(M_0=5\) subjects. The subject-level prediction-error summary is produced only for a subject with at least two usable records. The usable-record counts for subjects A through E are 3, 3, 0, 2, and 1.
The total usable-record count is \(U_d=3+3+0+2+1=9\). Therefore record coverage is \(100 \times 9/15=60.0\%\), and \(15-9=6\) records are not usable for this summary. Subjects A, B, and D meet the two-record threshold, so \(S_d=3\) of \(M_0=5\) subjects receive a summary. Subject-summary coverage is \(100 \times 3/5=60.0\%\). Subjects C and E do not meet the threshold. The fact that both percentages happen to be 60.0% is specific to these counts; the percentages generally need not agree.
The table should not describe all five subjects as having a summary merely because all five have records in scope. Nor should it use the number of usable records as the denominator for subject coverage. Each reported percentage answers a different question and needs its own denominator.
Summarizing Counts Across Subgroups
When counts are reported by subgroup, retain the subgroup denominators and calculate the overall coverage from the pooled counts. A pooled percentage is a denominator-weighted result, not generally the arithmetic mean of subgroup percentages. This matters when subgroup sizes differ.
Worked Example: Pooling Subgroup Coverage
Suppose a hypothetical diagnostic table has two mutually exclusive subject groups. Group A has eight in-scope records, of which seven are usable; group B has 12 in-scope records, of which six are usable. Group A has four in-scope subjects, all four contributing at least one usable record; group B has six in-scope subjects, five contributing.
Group A record coverage is \(100 \times 7/8=87.5\%\), and group B record coverage is \(100 \times 6/12=50.0\%\). The pooled record coverage is calculated from totals: \(100 \times (7+6)/(8+12)=100 \times 13/20=65.0\%\). Simply averaging 87.5% and 50.0% gives 68.75%, which is not the pooled coverage because the groups have different record denominators.
Subject coverage is also calculated from subject counts: \(100 \times (4+5)/(4+6)=90.0\%\). The two types of coverage should remain separately labelled; nine contributing subjects out of ten does not mean that 90% of records are usable.
| Group | In-scope records | Usable records | Record coverage | In-scope subjects | Contributing subjects | Subject coverage |
|---|---|---|---|---|---|---|
| A | 8 | 7 | 87.5% | 4 | 4 | 100% |
| B | 12 | 6 | 50.0% | 6 | 5 | 83.3% |
| Overall | 20 | 13 | 65.0% | 10 | 9 | 90.0% |
Why the Pooled Percentage Is Weighted
The pooled record-coverage formula follows directly from adding usable and in-scope counts across mutually exclusive subgroups. For subgroup \(g\), let \(N_g\) be its in-scope record count and \(U_g\) its usable count. Its coverage as a proportion is \(U_g/N_g\). The pooled coverage is the total usable records divided by total in-scope records:
Thus each subgroup proportion is weighted by its in-scope record count. An unweighted average gives each subgroup equal influence regardless of size and does not, in general, answer the pooled-record question. The same justification applies to subject coverage when subgroup subject sets are mutually exclusive and their subject counts are the denominators.
Apply the declared analysis and reporting filters, then count in-scope records and distinct subjects.
List the quantities required for each plot or summary and the rules for a record to contribute.
Count usable records, distinct contributing subjects, and any useful missing-quantity counts without treating overlapping missingness as disjoint.
Divide usable records by in-scope records, or contributing subjects by in-scope subjects, and verify each numerator does not exceed its denominator.
Show the counts and denominators beside percentages, and state whether the percentage is record or subject coverage.
Diagnostics, Alternatives, and Limits
Count tables are a check on automated diagnostics as well as a reporting aid. For each diagnostic, verify that the usable count is between zero and the in-scope count, that the distinct-subject count is no greater than the in-scope subject count, and that a reported percentage agrees with its counts after rounding. Investigate unexpected changes in denominators across runs. A jump can signal a changed data scope, a join that duplicated or dropped records, a mismatched identifier, or an unintended change in a diagnostic’s required quantities.
A table of percentages alone is less informative than counts with denominators. A table of raw missing counts alone can also mislead when diagnostic panels have different eligible totals. A single “complete-case” count across every diagnostic is another possible summary, but it may discard records usable for some diagnostics and conceal the diagnostic-specific coverage. Reporting separate counts for each plot or summary makes those differences explicit. The appropriate presentation depends on the question, but the counting rules and scope should remain reproducible.
Coverage is not a measure of diagnostic quality. High coverage does not establish that a residual pattern is acceptable, that missing quantities are unrelated to the data, or that the model is adequate. Low coverage does not by itself identify why quantities are unavailable. Likewise, a total missing count does not describe where missingness occurs. Interpret the table alongside the diagnostics and the stated data-processing rules; do not infer a cause from a count alone.
Common Mistakes
- Using different denominators without saying so. A percentage based on records in a plot is not interchangeable with one based on subjects. Label both the unit and denominator.
- Adding overlapping missing counts. If one row lacks both a residual and a prediction, it appears in both field-specific missing counts but is only one unusable paired record. Count distinct rows that fail the joint requirement.
- Comparing percentages without their counts. The same percentage can represent very different amounts of information. Report the numerator and denominator, such as 9 of 15 records.
- Averaging subgroup percentages to get pooled coverage. Pool the numerators and denominators first, unless the intended estimand is explicitly an equally weighted average of subgroup rates.
- Equating usability with typicality. Do not exclude an extreme diagnostic value merely because it looks unusual. Apply declared eligibility rules consistently and retain the count trail.
Check Your Understanding
Use the stated counts and rules to answer each question. Solutions follow the questions.
- A diagnostic has 24 in-scope records and 18 usable records. What is record coverage, and how many records are not usable?
- There are 10 in-scope subjects. A subject summary requires at least three usable records; seven subjects meet that threshold. What is subject-summary coverage?
- Two required quantities are each missing on three records. Is the number of records unusable for their paired plot necessarily six? Explain.
- Group A has 4 usable records out of 5; group B has 6 usable records out of 15. Calculate pooled record coverage and explain why the unweighted average of subgroup percentages differs.
Solutions. (1) Record coverage is \(100 \times 18/24=75.0\%\); \(24-18=6\) records are not usable. (2) Subject-summary coverage is \(100 \times 7/10=70.0\%\). The three-record threshold determines which subjects receive summaries; it does not change the denominator of ten in-scope subjects. (3) Not necessarily. Some records may be missing both quantities, so count the distinct records missing at least one required quantity. The union could be as small as three if the same three records are missing both, or as large as six if the sets do not overlap. (4) Pooled coverage is \(100 \times (4+6)/(5+15)=50.0\%\). The subgroup rates are 80.0% and 40.0%, whose unweighted average is 60.0%; that gives the two groups equal weight even though their denominators are five and 15 records. The pooled record percentage weights each group by its record count.