Why Summary Output Can Reveal Data Problems
A summary is more than a compact description of a data set. It is also a first-pass check on whether values appear to have been recorded and stored as intended. A negative value in a field expected to contain completed policy durations, a category that differs by one letter from the others, or a monetary field stored as character text can each be a reason to investigate.
The key distinction is between a warning sign and a confirmed error. Summary output can show that a value or pattern deserves attention; it usually cannot establish what the correct value should have been. The appropriate next steps are to check the field definition, the data-entry rules, and the source record, while retaining the original value until the issue is resolved. This kind of disciplined data review is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Earlier material on summaries and category frequencies provides the tools for this review. Here the focus is how to interpret those outputs as diagnostic evidence, alongside the variable’s R class. An output line is useful only in context: a numerical extreme may be a real observation, while a small difference in category spelling may represent either a typo or a genuinely separate category.
Read the Summary Together With the Class
For a numeric variable, R’s summary() output commonly reports the minimum, first quartile, median, mean, third quartile, and maximum. It may also report the number of missing values. These statistics can expose patterns worth checking: a surprising minimum or maximum, a large gap between the mean and median, or a high missing-value count. None of these alone proves that a record is wrong.
The class helps establish what kind of object R is summarizing. For example, a field intended to represent a measured amount but stored as character text may not receive the numeric summary a user expects. Conversely, a factor may contain numbers as labels, but those labels are categories unless the field definition says otherwise. Check both the class and the observed values; do not infer meaning from the printed numbers alone.
Missing-value counts also need context. If a data set has \(n\) rows and \(n_{\text{obs}}\) non-missing values for a field, then its missing count is \(n_{\text{miss}}=n-n_{\text{obs}}\). This follows because every row has either a recorded value or a missing value, not both. The check is useful for detecting unexpected loss of information, but it does not explain why values are missing.
Worked Example: Investigating a Negative Recorded Age
A hypothetical portfolio data set has 120 rows. The field age_at_application is documented as age in completed years. R reports that the field is numeric, with this summary: Min. = -1, 1st Qu. = 29, Median = 41, Mean = 40.8, 3rd Qu. = 53, Max. = 82, NA’s = 4. What should an analyst conclude and check?
Step 1: Check completeness. The data set has \(n=120\) rows and 4 missing values, so the number of recorded ages is \(120-4=116\). This calculation checks the count reported in the summary against the stated row total.
Step 2: Compare the value with the field definition. The minimum is \(-1\). A negative number is inconsistent with the documented meaning “age in completed years,” so that value is a clear reason to investigate. The numeric class confirms that R is summarizing a numeric field, but it does not tell us what \(-1\) means.
Step 3: Form a testable hypothesis rather than changing the data. The value could be a special code for “unknown,” an entry error, or a valid value under a different field definition. Check the data dictionary, any documented missing-value conventions, and the source record for the row containing \(-1\). If the codebook confirms that \(-1\) means unknown, recode it as missing according to the documented rule. If the source record confirms another age, correct it with an audit trail.
Step 4: Check the conclusion. The summary establishes that at least one recorded value conflicts with the stated definition and that 4 values are missing. It does not establish the correct replacement for \(-1\). Leaving the value unchanged until verified avoids converting a diagnostic flag into an unsupported correction. The common mistake here is to replace every negative value automatically with missing; that is appropriate only if the coding rule or source evidence supports it.
Category Counts and Factor Levels
For a categorical field, summary output and frequency tables can reveal unexpected labels, rare categories, and categories that exist as factor levels but have no observed records. For example, “Active” and “Actve” might indicate an inconsistent spelling. Yet those values should not be merged solely because they look similar: the data dictionary or source records must confirm whether they represent the same category.
When categories are mutually exclusive and every observed record belongs to exactly one of them, their counts provide a useful internal check. If category \(j\) occurs \(n_j\) times among \(n_{\text{obs}}\) non-missing observations, its observed proportion is \(p_j=n_j/n_{\text{obs}}\). Because each observed record contributes to exactly one category, adding all the category counts counts each observed record once; therefore, \(\sum_j n_j=n_{\text{obs}}\), and \(\sum_j p_j=1\). This check applies to non-missing observations; missing values are not another substantive category unless the analysis explicitly treats them as one.
An unused factor level is not necessarily an error. It may remain after filtering data, or it may be included to preserve a common set of levels across data sets. A level with no observations is different from a rare observed level, and neither should be removed or combined without checking its purpose.
Worked Example: Checking an Unexpected Policy Status
A hypothetical extract has 248 rows. The field status is a factor. A frequency check reports 151 Active, 72 Lapsed, 11 Pending, 4 Actve, and 2 Unknown values; 8 values are missing. The factor also has an unused level, Withdrawn. Determine what the output does and does not establish.
Step 1: Reconcile the counts. The observed category counts sum to \(151+72+11+4+2=240\). Adding 8 missing values gives \(240+8=248\), which agrees with the row total. The observed proportions use 240 as their denominator, not 248.
Step 2: Calculate a category proportion as a check. The observed proportion labelled Actve is \(4/240=0.0167\), rounded to four decimal places. The corresponding Active proportion is \(151/240=0.6292\). These calculations describe the recorded labels; they do not show whether the four Actve records should be counted as Active.
Step 3: Interpret the spelling and the unused level. Actve is a plausible spelling inconsistency and should be checked against the allowed-value list and source records. Withdrawn having zero observations only shows that the level is present but unused in this extract; it does not imply a bad record. The two Unknown values are observed category values, whereas the 8 missing values have no recorded category. Do not treat them as interchangeable without a defined rule.
Step 4: Check the conclusion. The counts reconcile with all 248 rows. The output flags Actve for investigation and identifies Withdrawn as unused, but it does not prove that the spelling is an error or that the unused level should be deleted. Automatically merging Actve into Active is the common mistake; first establish the intended coding.
When the Storage Class Is the Warning Sign
Sometimes the main issue is not an unusual numeric summary, but that R cannot summarize the field in its intended form. A column meant to contain amounts may be stored as character because some entries contain commas, currency symbols, or text such as “NA”. In that case, a character summary may report only the length, class, and mode rather than the numeric distribution. The displayed string “NA” is not automatically the same thing as an actual missing value in R.
Worked Example: A Monetary Field Stored as Text
A hypothetical table has eight records in a field called paid_amount, documented as dollars. The class is character, and its values are “85.00”, “120.00”, “95.00”, “1,250.00”, “NA”, “0.00”, “75.00”, and “140.00”. A character summary reports Length 8, Class character, Mode character. Diagnose the issue without assuming which amount is wrong.
Step 1: Identify the mismatch. The documented variable is a dollar amount, but the class is character. The summary therefore gives no numeric minimum, median, or mean. This is evidence that the field needs review before numeric analysis.
Step 2: Inspect the values and count them. There are eight recorded strings. One includes a thousands separator, and one is the text “NA”. The string “NA” is a recorded value unless it has been converted to R’s missing-value representation. The amount “1,250.00” may be a legitimate value or a data-entry issue; the class output cannot decide.
Step 3: Show why blind conversion can mislead. Directly applying numeric conversion to the unmodified strings will fail to convert “1,250.00” because of the comma, and will also fail on “NA”. If those failures become missing values, six amounts remain: 85, 120, 95, 0, 75, and 140. Their sum is \(85+120+95+0+75+140=515\), and their mean is \(515/6=85.8333\) dollars. This mean excludes the potentially valid amount of 1,250 dollars.
Step 4: Compare with a documented interpretation. Suppose the source specification confirms that commas are thousands separators and that the text “NA” means missing. Removing the separator and converting the seven actual amounts gives a sum of \(85+120+95+1250+0+75+140=1765\) dollars. The mean of the seven observed amounts is \(1765/7=252.1429\) dollars, rounded to four decimal places. This is a conditional calculation based on the stated interpretation, not evidence that the source specification is correct.
Step 5: Check the conclusion. Class information and the strings reveal a format mismatch and a special text value that need documented handling. They do not show whether the 1,250-dollar record is correct. Blind conversion is the common mistake: it can silently exclude a valid value and change the summary substantially. Confirm the format and missing-value convention before using a converted field.
A Practical Investigation Sequence
A reliable review proceeds from the evidence in the output to a question that can be checked against documentation or source data. It does not jump directly from an unusual summary to a correction.
Read the variable definition, units, and allowed coding rules. A value can only be judged against an intended meaning that is known.
Determine whether the object is numeric, character, or a factor, and reconcile observed and missing counts with the number of rows.
Record the exact summary statistic, category, spelling, or class mismatch that prompted concern. Distinguish an observed value from a missing value and an unused level.
Check source records and documented rules. Preserve the original data and record any confirmed correction or recoding decision.
The sequence makes the output actionable while keeping the conclusion appropriately limited. A high or low summary value, a rare category, or a type mismatch can each justify a targeted check. Whether the value is wrong depends on evidence beyond the summary itself.
Common Mistakes
- Treating a flag as proof. A value that conflicts with a field description deserves review; it is not automatically safe to delete, cap, or recode.
- Ignoring the class. A character column of numeric-looking strings may not be summarized or analyzed numerically as intended.
- Confusing a missing value with a text code. The string “NA” and an actual R missing value are different until a documented conversion makes them equivalent.
- Merging similar labels automatically. Similar spellings may be inconsistent entry, but the allowed-value definition should determine whether they represent the same category.
- Removing unused factor levels without checking. An unused level may be intentional or may result from filtering; its presence alone does not identify a data error.
Check Your Understanding
For each problem, decide what the output establishes and what should be checked next.
- A numeric field has 75 rows and summary output reports 9 missing values. How many non-missing values are there? (Answer: 66 non-missing values.)
- A factor has observed counts of 87 for Renewal, 12 for New, and 1 for Renwal, with 4 missing values in a 104-row table. What is the observed proportion labelled Renwal, to four decimal places, and what does it warrant? (Answer: \(1/100=0.0100\); investigate the label against the allowed values or source record, but do not merge it without confirmation.)
- A field documented as a numeric code has class factor and observed labels 1, 2, and 9. What can be concluded about label 9 from this information alone? (Answer: It is an observed factor level, but whether 9 is valid or a special code requires the coding definition.)
- A categorical field has counts 45, 30, and 15 across its three observed categories, plus 10 missing values in 100 rows. What are the proportions among non-missing records, rounded to four decimal places? (Answer: 0.5000, 0.3333, and 0.1667; they sum to 1.0000.)
- A character field intended to hold dollar amounts contains “200”, “1,100”, and “NA”. Why is a numeric summary not yet justified, and what should be verified first? (Answer: The field is character, the comma may need documented separator handling, and “NA” may be text rather than a true missing value; verify the format and missing-value convention before conversion.)