Distance Choices for Categorical Predictors

Why Categorical Distance Choices Matter

Nearest-neighbor methods compare observations using distances between their predictor values. For numeric predictors, a distance can use differences in measured amounts. For a nominal category such as region or policy type, however, subtracting category labels has no meaningful interpretation. A distance method must specify what counts as a mismatch and how that mismatch contributes relative to other predictors.

This choice is not merely a matter of notation. Two encodings of the same categorical information can give different distances and therefore different nearest observations. The calculations below focus on distance definitions and neighborhood membership, not on how a nearest-neighbor prediction is formed.

Definition: For two observations \(x\) and \(z\), a categorical mismatch on predictor \(j\) occurs when their category values differ. A categorical distance summarizes these mismatches across predictors. A nominal predictor has categories with no natural ordering, such as region or payment method; assigning numbers to its labels does not make those numbers quantities.

Simple Matching for Categorical Predictors

Suppose there are \(p\) categorical predictors and no missing values. Simple matching counts the number of predictors on which two observations differ. Its unnormalized form is a count; its normalized form is the proportion of predictors that differ. Normalization puts the result on a zero-to-one scale but does not change the ordering of pairs when all pairs use the same \(p\).

d_{\mathrm{SM}}(x,z)=\frac{1}{p}\sum_{j=1}^{p} I(x_j\ne z_j)

Here \(I(\cdot)\) equals 1 when the condition is true and 0 otherwise. Thus, identical category profiles have distance 0, and profiles differing on every predictor have distance 1. Simple matching treats every predictor mismatch equally. That is an assumption of the chosen measure, not a claim that every mismatch is equally important in the application.

Worked Example: Counting Categorical Mismatches

A hypothetical insurer compares a quote profile with five categorical predictors: region, payment frequency, coverage tier, renewal status, and distribution channel. The quote profile is (North, monthly, standard, new, agent). Candidate A is (North, annual, standard, renewing, agent). Candidate B is (South, annual, premium, new, online). Calculate the simple matching distances and identify which candidate is closer under this measure.

Method. Compare the five fields one at a time, count mismatches, and divide by five. No ordering or numerical meaning is assigned to the labels.

For Candidate A, the matches are region, coverage tier, and distribution channel. Payment frequency and renewal status differ, so there are \(2\) mismatches:

d_{\mathrm{SM}}(x,A)=\frac{2}{5}=0.4

For Candidate B, region, payment frequency, coverage tier, and distribution channel differ. Renewal status matches. There are \(4\) mismatches:

d_{\mathrm{SM}}(x,B)=\frac{4}{5}=0.8

Candidate A is closer by simple matching, since \(0.4<0.8\). As a check, the distances are proportions of five fields and both lie between 0 and 1.

Indicator Encoding and Euclidean Distance

A common way to represent a nominal predictor with \(q\) categories is full indicator encoding: create \(q\) columns, one for each category, and mark the observed category with 1 while setting the other columns to 0. For example, the category “North” might be represented as \((1,0,0)\), “South” as \((0,1,0)\), and “West” as \((0,0,1)\). These vectors encode labels, not numeric levels.

Under ordinary Euclidean distance, equal categories contribute zero squared distance. Different categories contribute two: one indicator changes from 1 to 0 and another from 0 to 1. This gives a precise connection between full indicator encoding and simple matching.

Result: With \(p\) nominal predictors, full indicator encoding for each predictor, and ordinary Euclidean distance across all resulting columns, the squared distance is twice the number of categorical mismatches. Therefore the Euclidean distance is \(\sqrt{2m}\), where \(m\) is the mismatch count. It ranks pairs in the same order as simple matching when all categorical predictors are treated equally and there are no other predictors in the distance.

To see why, consider one predictor. If both observations have the same category, their indicator vectors are identical, so their squared Euclidean difference is 0. If their categories differ, the two vectors each have one 1 in a different position. The squared differences at those two positions are \(1^2+(-1)^2=2\); all other positions contribute 0. Summing this contribution over the \(m\) mismatching predictors gives squared distance \(2m\), and taking the square root gives \(\sqrt{2m}\). Since both this expression and simple matching increase with \(m\), they produce the same ordering in this all-categorical, equally treated setting.

This equivalence concerns rankings, not identical distance values. Simple matching is \(m/p\), whereas full-indicator Euclidean distance is \(\sqrt{2m}\). When numeric predictors are added, or when category blocks are rescaled or weighted, the two approaches need not produce the same neighborhoods.

Worked Example: How Integer Codes Can Change the Nearest Candidate

A hypothetical service team compares a new account with two existing accounts using a product category and a numeric risk score that has already been scaled to the interval from 0 to 1. The query is category A with score 0.0. Candidate P has category B and score 0.8; Candidate Q has category C and score 0.1. Compare full indicator encoding with two possible integer encodings of the nominal category. Which candidate is nearer under each distance?

Method. Calculate Euclidean distance from the query in each representation. For full indicators, use three columns for A, B, and C; the query is \((1,0,0)\), P is \((0,1,0)\), and Q is \((0,0,1)\).

For P, the categorical squared difference is \(1^2+(-1)^2+0^2=2\), and the numeric squared difference is \((0.8-0.0)^2=0.64\). Thus:

d(x,P)=\sqrt{2+0.64}=\sqrt{2.64}\approx 1.6248

For Q, the categorical squared difference is also 2, and the numeric squared difference is \((0.1-0.0)^2=0.01\). Thus:

d(x,Q)=\sqrt{2+0.01}=\sqrt{2.01}\approx 1.4177

With full indicators, Q is nearer. Now encode A, B, and C as \(0,1,2\). P’s categorical difference is 1, while Q’s is 2. The distances become:

d(x,P)=\sqrt{1^2+0.8^2}=\sqrt{1.64}\approx 1.2806 $$ $$ d(x,Q)=\sqrt{2^2+0.1^2}=\sqrt{4.01}\approx 2.0025

Under this integer encoding, P is nearer. If the same categories are instead coded as A = 0, B = 100, and C = 1, then the distances are \(\sqrt{10000.64}\approx 100.0032\) for P and \(\sqrt{1.01}\approx 1.0050\) for Q, so Q is nearer. The ranking has changed solely because arbitrary labels were changed. The calculation confirms the central warning: integer subtraction imposes artificial spacing and ordering on nominal categories.

Common mistake avoided: Treating the codes 0, 1, and 2 as if a two-unit difference represented twice the categorical difference of a one-unit difference. That interpretation is not justified for nominal labels.

Combining Numeric and Categorical Predictors

A mixed-data distance must put numeric and categorical differences on a comparable scale. One standard option is Gower’s distance. For complete data, define a component distance for each predictor: for a numeric predictor, use the absolute difference divided by that predictor’s range; for a categorical predictor, use 0 for a match and 1 for a mismatch. The overall distance is the average of the component distances.

d_{\mathrm{G}}(x,z)=\frac{1}{p}\sum_{j=1}^{p} d_j(x,z), \qquad d_j(x,z)= \begin{cases} \dfrac{|x_j-z_j|}{R_j}, & \text{numeric predictor }j,\\[6pt] I(x_j\ne z_j), & \text{categorical predictor }j, \end{cases}

Here \(R_j\) is the range used to scale numeric predictor \(j\), typically its maximum minus minimum in the training data. This construction makes each component lie between 0 and 1 when the observed values lie within the range used. Each predictor, rather than each encoded indicator column, contributes one component to the average. A numeric predictor with zero range provides no observed variation and cannot be divided by its range; it should be removed from this calculation or handled by an explicitly stated alternative. If values are missing, the denominator and included components must be defined consistently; the complete-data formula above assumes no missing values.

Worked Example: A Gower Distance for a Mixed Profile

A hypothetical analyst compares a query with two service locations using age and two nominal predictors, region and service plan. The numeric age range in the training data is 20 to 60 years, so \(R=40\). The query is age 40, North region, standard plan. Candidate A is age 50, North, premium; Candidate B is age 44, South, standard. Calculate both Gower distances and determine which profile is closer.

Method. Calculate the scaled absolute age difference and the 0-or-1 mismatch for each categorical predictor, then average the three predictor-level components.

For Candidate A, the age component is \(|40-50|/40=10/40=0.25\). Region matches, giving 0. The plan differs, giving 1. Therefore:

d_{\mathrm{G}}(x,A)=\frac{0.25+0+1}{3} =\frac{1.25}{3} \approx 0.4167

For Candidate B, the age component is \(|40-44|/40=4/40=0.10\). Region differs, giving 1. The plan matches, giving 0. Therefore:

d_{\mathrm{G}}(x,B)=\frac{0.10+1+0}{3} =\frac{1.10}{3} \approx 0.3667

Candidate B is closer under this Gower distance because \(0.3667<0.4167\). As a check, each component is between 0 and 1 and the average is also between 0 and 1. The result reflects the stated equal contribution of the three predictors; it does not imply that the predictors have equal practical importance in every application.

Choosing and Interpreting a Distance

A distance rule should match the meaning of the predictors and the intended comparison. Simple matching is transparent for a set of equally treated categorical predictors. Full indicator encoding works with Euclidean distance while preserving equal mismatch contributions for each categorical predictor. Gower’s distance provides a direct way to average scaled numeric differences and categorical mismatches at the predictor level.

The encoding and scaling decisions should be made as part of model development, using information available in the training data. In particular, a range used for numeric scaling should not be calculated from held-out observations when those observations are meant to evaluate generalization. This is consistent with leakage-safe preprocessing and with the earlier material on feature scaling.

Use simple matching when the predictors are categorical and the intended rule is one equal penalty per mismatch.
Use full indicator encoding when a Euclidean calculation is desired for nominal categories; remember that each mismatching category contributes squared distance 2 before any rescaling.
Use a mixed-data measure when numeric and categorical predictors both contribute, and state how numeric differences are scaled and how predictor contributions are combined.

A frequent exam trap is to compare raw integer codes for nominal categories, or to count indicator columns as if they were separate predictors. The former invents distances between labels; the latter can give a categorical predictor with many categories a different total influence simply because it requires more indicator columns. When the goal is one equal contribution per predictor, calculate or interpret contributions at the predictor-block level.

Key takeaway: Distances depend on representation. For nominal categories, use a mismatch-based rule or a justified indicator representation rather than arbitrary integer labels. For mixed data, specify the scaling and aggregation rule before comparing neighborhoods.

Check Your Understanding

For each problem, state the distance rule being used and check that your result follows from the stated encoding or scaling.

Two profiles have four categorical predictors and differ on exactly one. What is their normalized simple matching distance? (Answer: \(1/4=0.25\).)
Two observations differ on three of five categorical predictors. Under full indicator encoding and ordinary Euclidean distance, what is their distance? (Answer: \(\sqrt{6}\approx 2.4495\).)
A query has category A and scaled numeric value 0.2. Candidate R has category B and value 0.5; Candidate S has category C and value 0.1. With full indicators for A, B, and C plus the numeric predictor, which is nearer, and what are the distances to four decimals? (Answer: S is nearer; \(d_R=\sqrt{2+0.09}\approx1.4457\), \(d_S=\sqrt{2+0.01}\approx1.4177\).)
A mixed-data comparison uses one numeric variable with training range 10 to 50 and two categorical variables. The query is (age 20, East, basic); candidate T is (age 30, East, plus). Using Gower’s distance, what is the distance? (Answer: \((10/40+0+1)/3=0.4167\).)
Two nominal categories are coded 1 and 4. Is their categorical difference necessarily three times the difference between categories coded 1 and 2? State the appropriate simple-matching contribution for a mismatch. (Answer: No; the codes do not define spacing, and a mismatch contributes 1 under simple matching.)

What You'll Learn