Near-Zero Variance Predictor Screening

Why Screen for Near-Zero Variance?

A predictor that takes almost the same value for every observation often contributes little to a fitted model. A predictor that is exactly constant cannot distinguish observations at all. Identifying such variables can simplify data preparation and prevent avoidable numerical problems.

But rarity is not the same as irrelevance. A flag that appears in only a handful of records may identify a subgroup with distinctive operational, financial, or safety implications. A useful screen therefore has two parts: calculate how little a predictor varies, then consider what its rare values represent. These are diagnostic steps, not a decision to discard every flagged variable.

Definition: A zero-variance predictor has the same observed value for every record in the data being screened. A near-zero-variance predictor varies, but its observed distribution is so concentrated that it may provide little information for general modeling. The term is a practical screening label, not a universal statistical cutoff.

Numeric Predictors: Variance and Scale

For numeric observations \(x_1,\ldots,x_n\), the sample mean is \(\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i\). The sample variance and sample standard deviation are

s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2, \qquad s=\sqrt{s^2}.

In R, var(x) returns the sample variance, using the \(n-1\) denominator, and sd(x) returns the sample standard deviation. For a constant vector, both are zero. Variance is measured in squared units, so its numerical size depends on the units used: changing dollars to cents multiplies variance by 10,000. A small variance, by itself, is not evidence that a predictor is unimportant.

For a numeric predictor, also inspect its range, distinct-value count, and scale. If a variable is a count or a coded flag, frequency summaries can reveal concentration that a variance alone does not describe. For a continuous variable with many distinct values, a frequency ratio may be uninformative even when its spread is small relative to a meaningful scale.

Worked Example: A Numeric Predictor With a Small Spread

A hypothetical system records a service score for ten cases: 100 for nine cases and 101 for one case. Calculate the sample variance, standard deviation, percent unique, and frequency ratio. Explain what the diagnostics do and do not establish.

Method. Calculate the numeric summaries and the frequency summaries separately. The values are \(100,100,100,100,100,100,100,100,100,101\), so \(n=10\), with two distinct values.

Mean. The sum is \(9(100)+1(101)=1001\), giving \(\bar{x}=1001/10=100.1\).

Sample variance. Nine deviations are \(-0.1\), and one is \(0.9\). The sum of squared deviations is \(9(0.1^2)+0.9^2=0.09+0.81=0.90\). Thus \(s^2=0.90/(10-1)=0.10\), and \(s=\sqrt{0.10}\approx0.3162\).

Frequency diagnostics. The percent unique is \(100(2/10)=20\%\). The most common value occurs nine times and the second most common once, so the frequency ratio is \(9/1=9\).

In R, x <- c(rep(100, 9), 101), followed by var(x), sd(x), length(unique(x))/length(x)*100, and sort(table(x), decreasing=TRUE), gives the same quantities. The variance describes the observed spread in score units squared; the frequency ratio describes the imbalance between the two observed values. Neither summary tells us, without the modeling objective and context, whether the score is useful.

Check and common mistake. The squared deviations sum to \(0.90\), and division by nine gives \(0.10\). Dividing by ten instead would calculate the population variance of these ten recorded values, not the sample variance returned by R's var().

Categorical Predictors: Frequency and Uniqueness

Variance is not the usual first diagnostic for a categorical predictor. Instead, tabulate the levels and compare their frequencies. Two useful summaries are the frequency ratio and percent unique:

\text{frequency ratio} =\frac{\text{count of the most frequent level}} {\text{count of the second most frequent level}}, \qquad \text{percent unique} =100\frac{\text{number of distinct observed values}}{n}.

A large frequency ratio indicates that the most common value substantially outnumbers the next most common value. A small percent unique indicates that few distinct values occur relative to the number of records. These measures answer different questions: a variable can have several distinct levels but still be dominated by one level, or have many levels without one dominating.

If there is only one observed level, there is no second positive count, so the frequency ratio is not finite. Treat this as a zero-variance case, not as an ordinary near-zero-variance calculation. In R, table(x) displays observed counts; table(x, useNA="ifany") can also show missing values. Decide how missing values are handled before comparing predictors, because summaries based on different subsets of records are not directly comparable.

Practical screening rule: One commonly used convention flags a nonconstant predictor when its frequency ratio exceeds 19 and its percent unique is below 10%. These are configurable screening thresholds, not a theorem or a universal definition. A package may use different defaults, and results should be interpreted in the context of the data and the intended use.

The R package caret provides nearZeroVar() for this type of screen. Its commonly used defaults correspond to a frequency cutoff of \(95/5=19\) and a uniqueness cutoff of 10%. The function is a convenient way to identify candidates for review; it does not decide that they should be removed. Always distinguish a constant predictor from a nonconstant predictor flagged by a near-zero-variance rule, and check the package's settings when reproducibility matters.

Worked Example: A Rare Operational Flag

A hypothetical claims workflow has 1,000 records. An indicator is “routine” for 995 records and “manual escalation” for five. Calculate the frequency ratio and percent unique. If the indicator is coded 1 for manual escalation and 0 otherwise, calculate its mean and sample variance.

Frequency diagnostics. The largest count is 995 and the second largest is 5, so the frequency ratio is \(995/5=199\). There are two distinct observed values, hence percent unique is \(100(2/1000)=0.2\%\). Under the stated example convention, both thresholds are crossed: 199 exceeds 19, and 0.2% is below 10%.

Indicator moments. Let \(Z=1\) for manual escalation and \(Z=0\) otherwise. Its mean is \(\bar{Z}=5/1000=0.005\). The sum of squared deviations is \(5(1-0.005)^2+995(0-0.005)^2\). The first term is \(5(0.990025)=4.950125\), and the second is \(995(0.000025)=0.024875\), so the sum is \(4.975\). Therefore, \(s_Z^2=4.975/999\approx0.004980\).

In R, a factor can be summarized with table(flag); for a 0/1 numeric version, mean(flag) gives 0.005 and var(flag) gives approximately 0.004980. The frequency results show that the escalation value is rare. They do not show that it is unimportant: if those five records correspond to a consequential workflow or review requirement, dropping the flag could erase precisely the distinction that matters.

Check and common mistake. The second-highest count is five, not the number of distinct levels, two. Also, the small variance is consistent with a rare binary value; it does not establish that the value has no practical significance.

Why a Rare Indicator Has Low Variance

The relationship between rarity and variance can be derived directly. Suppose \(Z\) is a binary indicator, with \(Z=1\) with probability \(p\) and \(Z=0\) with probability \(1-p\). Since \(Z^2=Z\), its expected value is \(E[Z]=p\), and \(E[Z^2]=p\). Using \(\operatorname{Var}(Z)=E[Z^2]-(E[Z])^2\) gives

\operatorname{Var}(Z)=p-p^2=p(1-p).

When \(p\) is close to zero, \(p(1-p)\) is close to zero. This explains why a rare indicator has little overall variance even though it may identify a small group of special interest. In a sample with \(k\) ones among \(n\) records, the sample variance is \(\frac{n}{n-1}\hat{p}(1-\hat{p})\), where \(\hat{p}=k/n\). The factor \(n/(n-1)\) appears because sample variance uses the \(n-1\) denominator.

This derivation explains a property of the diagnostic; it does not establish a relationship with an outcome. Whether a predictor is associated with a response is a separate question. At this screening stage, the appropriate conclusion is that the feature is rare and deserves context-sensitive review—not that it predicts an outcome.

Worked Example: Applying a Frequency Rule in R

A hypothetical equipment file has 50 records: 48 have status “standard” and two have status “special inspection.” Determine whether the status would be flagged by the example convention, and show the corresponding R calculations.

Method. Use the observed level counts to calculate the frequency ratio and percent unique, then compare each with its stated threshold.

Frequency ratio. The largest count is 48 and the second is 2, so the ratio is \(48/2=24\). Since \(24>19\), the frequency condition is met.

Percent unique. There are two observed levels among 50 records, so percent unique is \(100(2/50)=4\%\). Since \(4\%<10\%\), the uniqueness condition is also met. The predictor is therefore a candidate for a near-zero-variance flag under this convention.

In R, let status <- factor(c(rep("standard", 48), rep("special inspection", 2))). Then table(status) returns counts 48 and 2. The calculation max(table(status))/sort(table(status), decreasing=TRUE)[2] returns 24, while length(unique(status))/length(status)*100 returns 4. A package function such as caret::nearZeroVar(status) can automate candidate identification, but the manually calculated diagnostics make the basis of the flag transparent.

Check and common mistake. Both conditions are met, so this is a candidate under the stated rule. Do not discard “special inspection” solely because it occurs twice: its operational meaning and the consequences of losing it need review.

Interpret Flags Before Acting

A screening flag is a prompt to inspect a variable, not an automatic deletion instruction. Review the following before making a modeling decision:

Meaning and consequences. Determine what a rare level represents. A low-frequency category may indicate an unusual process, a regulatory condition, or a potentially important subgroup.
Data quality. Check whether the rare value is a valid level, a coding error, or an artifact of inconsistent labels. Different spellings of the same category can inflate the number of levels and distort counts.
Observed sample. A level absent from the current records cannot be assessed by its frequency there. Also check whether factor levels were declared in advance but do not occur in the observed data; unused levels are not observed variation.
Numeric scale. A variance can look small or large merely because of units or a large baseline. Use scale-aware summaries only when their assumptions make sense, and do not substitute an arbitrary threshold for subject-matter interpretation.
Analysis purpose. A feature that is unhelpful for one modeling objective may be important for monitoring, subgroup review, or operational decisions. Screening criteria should match the intended use.

For a quick R review, use var() and sd() for numeric predictors, and table(), frequency ratios, and percent unique for discrete or categorical predictors. Check missingness and factor levels explicitly. Keep a record of thresholds and decisions so that screening can be repeated consistently. The central judgment is simple: low variation describes the observed distribution; it does not, by itself, determine the value of the predictor.

Key takeaway: Calculate variance for numeric predictors and frequency-based diagnostics for sparse predictors. Treat any near-zero-variance flag as a reason to investigate, especially when rare values may identify a consequential subgroup.

Check Your Understanding

For each problem, calculate the requested diagnostic and interpret the result using the stated convention where applicable.

A numeric predictor has values 7, 7, 7, 7, and 8. Calculate its sample mean, sample variance, and percent unique. (Answer: mean \(=7.2\), sample variance \(=0.2\), percent unique \(=40\%\).)
A categorical variable has counts 72, 8, and 0 across its declared levels, with 80 records observed. Calculate its observed frequency ratio and percent unique. (Answer: frequency ratio \(=72/8=9\), percent unique \(=100(2/80)=2.5\%\); the unused declared level is not an observed value.)
A binary predictor equals 1 for 4 of 200 records and 0 otherwise. Calculate its sample mean, sample variance, frequency ratio, and percent unique. (Answer: mean \(=0.02\), sample variance \(=\frac{200}{199}(0.02)(0.98)\approx0.0197\), frequency ratio \(=196/4=49\), percent unique \(=1\%\).)
A factor has 90 records in its most common level and 10 in its second most common level. Under the convention of frequency ratio greater than 19 and percent unique below 10%, should it be flagged? (Answer: no; its frequency ratio is \(9\), so the frequency condition is not met, even though percent unique is \(2\%\).)
A system has 990 routine cases and 10 cases marked “critical review.” Its frequency ratio is 99 and its percent unique is 0.2%. State one conclusion supported by these diagnostics and one conclusion they do not support. (Answer: the category is highly imbalanced and meets both example screening thresholds; the diagnostics alone do not show that “critical review” is irrelevant or that it predicts an outcome.)

What You'll Learn