Why Distinct Values Need Careful Counting
A quick count of distinct values can reveal whether a field behaves like a small set of categories or contains many different values. In predictive analytics, that first check can help identify data-entry inconsistencies, unexpected labels, or a field that may function more like an identifier than a useful grouping variable. But the count depends on what is being counted: values that actually appear in the data, or categories declared in an R factor.
R provides three simple tools for this inspection. The function unique() returns distinct values, length() counts how many elements are in the returned vector, and nlevels() counts the declared levels of a factor. Their answers need not be the same. In particular, an unused factor level is declared but not observed, while a missing value can be observed as missing without being a factor level.
What the Three Functions Return
For a vector x, unique(x) returns one occurrence of each distinct value, in the order in which each value first appears. Applying length() to that result gives the number of distinct values. For example, if a vector contains the character values "A", "B", "A", then unique() returns "A", "B", and length() of that result is 2.
A factor has both observed values and a set of declared levels. A factor can retain a level even when no current observation uses it. Therefore, length(unique(x)) and nlevels(x) answer different questions for a factor. A missing value, NA, is not a declared factor level; if present, it is counted once by length(unique(x)). When the goal is to count distinct nonmissing values, remove missing observations before applying unique().
In R, the corresponding expression is length(unique(x[!is.na(x)])). The expression is valid for a factor as well as for a character or numeric vector. For a factor, removing missing observations does not automatically remove unused declared levels; that is why this expression counts observed values, whereas nlevels() still reports the declared level count.
Worked Examples
Worked Example: A Factor with Unused Levels and a Missing Value
A hypothetical portfolio extract records a policy’s channel as a factor. Its observed values, in order, are "Agent", "Direct", "Agent", NA, and "Broker". The factor declares the levels "Agent", "Broker", "Direct", and "Partner". Find (a) the distinct observed-value count including missingness, (b) the distinct nonmissing category count, and (c) the declared-level count.
Method. Use length(unique(x)) for the count including any observed NA, filter with !is.na() for the nonmissing count, and use nlevels(x) for all declared factor levels.
(a) Including missingness. The distinct observed values are "Agent", "Direct", NA, and "Broker". There are four, so length(unique(x)) is 4. Repeated "Agent" contributes only once.
(b) Excluding missingness. Removing NA leaves "Agent", "Direct", "Agent", and "Broker". The distinct nonmissing values are "Agent", "Direct", and "Broker", so length(unique(x[!is.na(x)])) is 3.
(c) Declared levels. The declared levels are "Agent", "Broker", "Direct", and "Partner". Thus nlevels(x) is 4. "Partner" contributes to this count even though it is not observed.
Check. The counts are 4 including NA, 3 nonmissing observed categories, and 4 declared levels. The difference between the nonmissing observed count and declared count is the one unused level, "Partner". The common mistake avoided here is treating nlevels() as a count of categories that actually appear.
Worked Example: A Character Field Is Not a Factor
A hypothetical claims review records the character vector c("Glass", "Water", "Glass", "Wind", "Water", "Earth"). Determine the distinct observed values and explain what nlevels() reports for this vector.
Method. Apply unique() to list distinct values and length() to count them. Use nlevels() only to ask for the declared levels of a factor.
The values in order of first appearance are "Glass", "Water", "Wind", and "Earth". The first value, "Glass", appears again in position three, and "Water" appears again in position five; those repetitions do not add distinct values. Therefore, unique(x) contains four elements, and length(unique(x)) is 4.
The vector is character, not factor. It has no declared factor levels, so nlevels(x) returns 0. This does not mean there are zero observed categories: the observed distinct-value count is 4. If a factor representation is needed for a specific analysis, it should be created deliberately, with its levels checked; converting to a factor changes the type and introduces a declared-level set.
Check. The result is four observed distinct values and zero declared factor levels. The common mistake avoided is using nlevels() on a character vector as though it counted the distinct strings.
Worked Example: A Subset Retains Unused Factor Levels
A hypothetical service file has a factor for resolution type with observations c("Repair", "Refund", "Repair", NA, "Replace") and declared levels c("Repair", "Refund", "Replace", "Credit", "Escalate"). An analyst inspects only rows 2, 4, and 5. Find the distinct values in that subset including NA, its nonmissing observed-category count, and the declared-level count before and after droplevels().
Method. First subset the factor, then count its observed values with unique(). Use nlevels() to inspect declared levels. Apply droplevels() only when the intention is to remove unused levels from the subset.
The selected observations are "Refund", NA, and "Replace". These three values are all distinct, so length(unique(x[c(2, 4, 5)])) is 3. After excluding NA, the observed values are "Refund" and "Replace", giving length(unique(x[c(2, 4, 5)][!is.na(x[c(2, 4, 5)])])) equal to 2.
The subset still declares the original five levels, so nlevels() of the subset is 5. Only "Refund" and "Replace" are observed among nonmissing values; "Repair", "Credit", and "Escalate" are unused in this subset. Applying droplevels() removes those unused levels. The resulting factor has two declared levels, so nlevels(droplevels(x[c(2, 4, 5)])) is 2.
Check. The subset has three distinct values including NA, two distinct nonmissing observed categories, five declared levels before dropping unused levels, and two afterward. The common mistake avoided is assuming that selecting rows automatically deletes factor levels.
Why the Counts Work
The counting rule follows directly from what unique() does. Suppose the observed entries of a vector are considered one at a time. The first time a value appears, it contributes one element to unique(x). Every later occurrence of that same value contributes none. Consequently, each different observed value contributes exactly one element to the result, and no value contributes more than one. Counting the elements of unique(x) therefore counts the distinct observed values.
The same reasoning applies to NA: if no missing value occurs, none is returned; if one or many missing values occur, the distinct result includes one NA. To count distinct nonmissing values, filtering first ensures that NA is not among the values passed to unique(). For a factor, nlevels() uses the factor’s declared level set instead of inspecting which values occur. An unused level is therefore included by nlevels() but cannot appear as an observed value in unique(x).
Decide whether the question asks for distinct values including NA, nonmissing observed categories, or all declared factor levels.
Check whether the column is character or factor. nlevels() answers a declared-level question for factors, not a distinct-string question for character vectors.
Use unique() and length() for observed distinct values, filter missing values if required, and use nlevels() for the factor’s declared categories.
Interpreting Cardinality in Data Exploration
A categorical field with only a few observed values is often easy to inspect as a set of groups. A field with many distinct values deserves closer attention. It might be a genuine fine-grained category, a free-text field with inconsistent spelling, or an identifier such as a claim number. A large observed count is a clue to investigate, not proof that a field is an identifier or that it should be excluded from modeling.
Always interpret the count in context. For instance, "North", "north", and "NORTH" are different character values to R because their spellings differ. Likewise, trailing spaces can produce apparently different strings. This count reports exact distinct stored values; it does not decide whether two strings should be treated as equivalent. Any standardization should follow a documented data-cleaning rule rather than an assumption made from the count alone.
For numeric values, exact distinctness also has a practical limitation: two values that differ by a small amount are distinct, even if they represent nearly the same measurement. A numeric column with many distinct values is not automatically categorical. Consider the variable’s meaning and measurement scale before labeling it. These checks are exploratory descriptions, not evidence that a variable predicts an outcome or that its categories have a causal interpretation.
Common Exam Trap
Suppose a factor has three observed nonmissing categories, one missing observation, and six declared levels. The calculation nlevels(x) = 6 is correct only for the declared-level count. It is wrong to report six as the number of categories observed in the records. Likewise, length(unique(x)) includes the one distinct NA, so it would return 4, not 3. For the nonmissing observed count, the correction is length(unique(x[!is.na(x)])) = 3.
A second trap is to treat a character vector’s nlevels() result of 0 as evidence that it has no distinct values. First check the vector’s class; then use length(unique(x)) for the observed distinct count. State the target precisely: for example, "three observed nonmissing categories" rather than simply "three levels" when the variable is not a factor.
Check Your Understanding
For each problem, distinguish observed values from declared factor levels when necessary.
- A character vector is c("Plan A", "Plan B", "Plan A", "Plan C", "Plan B"). What does length(unique(x)) return, and what does nlevels(x) return? (Answer: 3 distinct observed values; nlevels(x) is 0.)
- A factor has observations c("Low", "High", "Low", NA) and declared levels c("Low", "Medium", "High"). What are length(unique(x)), length(unique(x[!is.na(x)])), and nlevels(x)? (Answer: 3 including NA, 2 nonmissing observed categories, and 3 declared levels.)
- A factor with declared levels c("Red", "Blue", "Green", "Gold") is subset to observations c("Blue", "Blue"). What are the distinct observed count and nlevels() of the subset before droplevels()? (Answer: 1 distinct observed value and 4 declared levels.)
- A character vector contains c("East", "east", "East ", NA). How many distinct values does length(unique(x)) count, and how many nonmissing distinct values are there? (Answer: 4 including NA and 3 nonmissing values; capitalization and the trailing space make the strings distinct.)
- A factor has 5 distinct observed nonmissing categories, 2 missing observations, and 8 declared levels. What do length(unique(x)), length(unique(x[!is.na(x)])), and nlevels(x) return? (Answer: 6 including the single distinct NA, 5 nonmissing observed categories, and 8 declared levels.)