Feature Weighting And Its Effect On Cluster Structure

Why Feature Weights Change Cluster Structure

Standardizing features puts them on comparable scales, but it does not guarantee that every feature should have equal influence on a clustering. In some applications, a feature may be less relevant to the grouping objective than another; in others, a particular feature may deserve greater emphasis. Feature weighting makes this choice explicit. Since k-means assigns observations by distance, changing the weights can change which centroid is nearest and therefore can change the cluster structure.

The key arithmetic point is that a feature is first multiplied by its weight, and then the transformed coordinates are squared when Euclidean distance is calculated. Thus a multiplier of $w_j$ produces a squared-distance coefficient of $w_j^2$, not $w_j$. We will treat the standardized feature values as given; the earlier material on standardization explains how they are obtained.

Definition: Suppose observation $i$ has standardized features $z_{i1},\ldots,z_{ip}$, and a centroid has coordinates $c_1,\ldots,c_p$ on those same features. Assign a nonnegative weight $w_j$ to feature $j$, and transform that coordinate to $w_jz_{ij}$. The resulting weighted squared Euclidean distance is $$ d_w^2(i,c)=\sum_{j=1}^{p}w_j^2(z_{ij}-c_j)^2. $$ A zero weight removes that feature from the distance calculation. A positive weight increases or decreases its contribution according to its square.

This definition assumes that the same feature weights are applied to observations and centroids, and that the distance used for assignment is squared Euclidean distance in the transformed coordinates. In k-means, an observation is assigned to a cluster whose centroid minimizes this distance. For a fixed assignment, the centroid in each original standardized coordinate is the mean of the assigned observations; multiplying every value of that coordinate by the same weight also multiplies its mean by that weight. Consequently, the usual mean update remains consistent with the transformed-coordinate calculation.

Deriving the Weighted Distance

The formula follows directly from ordinary Euclidean distance. Transform each coordinate by $z'_{ij}=w_jz_{ij}$, and transform the centroid coordinate by $c'_j=w_jc_j$. The squared distance between the transformed observation and transformed centroid is

\sum_{j=1}^{p}(z'_{ij}-c'_j)^2 =\sum_{j=1}^{p}(w_jz_{ij}-w_jc_j)^2 =\sum_{j=1}^{p}w_j^2(z_{ij}-c_j)^2.

The last equality holds because $w_j$ is a common multiplier for the observation and centroid coordinate, so it factors out of the difference and is then squared. This is why a multiplier of $0.5$ gives a coefficient of $0.25$ on the squared difference, while a multiplier of $2$ gives a coefficient of $4$. If the desired coefficient on a squared difference is $\lambda_j$, the corresponding feature multiplier is $\sqrt{\lambda_j}$, not $\lambda_j$.

Weights are modeling choices, not automatically estimated measures of importance. Standardization and weighting answer different questions: standardization changes units and scale, while weighting deliberately changes the relative contribution of features after that scaling. Multiplying every feature weight by the same positive constant multiplies all squared distances by the same constant, so it does not change nearest-centroid assignments. Relative weights are what matter for those assignments.

Worked Example: A Weight Reverses the Nearest-Centroid Assignment

A hypothetical risk-profile analysis uses two already-standardized features: claim frequency $z_1$ and claim severity $z_2$. A profile has coordinates $(1.5,0.7)$. Two candidate centroids are $A=(0,0)$ and $B=(2,2)$. Initially, each feature has multiplier $1$. Then the analyst reduces the multiplier on frequency to $0.5$ and increases the multiplier on severity to $2$. Find the nearest centroid under each scheme.

Method. Calculate the squared distances to both centroids under each set of weights. Comparing squared distances is sufficient because square root is increasing and therefore preserves which distance is smaller.

Unweighted comparison. With both weights equal to $1$, the squared distance to $A$ is

$$ d^2((1.5,0.7),A)=(1.5-0)^2+(0.7-0)^2=2.25+0.49=2.74. $$

The squared distance to $B$ is

$$ d^2((1.5,0.7),B)=(1.5-2)^2+(0.7-2)^2=0.25+1.69=1.94. $$

Since $1.94<2.74$, the unweighted assignment is to $B$.

Weighted comparison. The multipliers are $w_1=0.5$ and $w_2=2$, so the squared-distance coefficients are $0.25$ and $4$. For $A$,

$$ d_w^2((1.5,0.7),A)=0.25(1.5-0)^2+4(0.7-0)^2 =0.25(2.25)+4(0.49)=0.5625+1.96=2.5225. $$

For $B$,

$$ d_w^2((1.5,0.7),B)=0.25(1.5-2)^2+4(0.7-2)^2 =0.25(0.25)+4(1.69)=0.0625+6.76=6.8225. $$

Now $2.5225<6.8225$, so the weighted assignment is to $A$. The changed assignment is not an arithmetic anomaly: the weighting deemphasizes the frequency difference and emphasizes the severity difference. The observation is much closer to $A$ on severity than to $B$, and severity now contributes more to the distance.

Check. The weighted distances are positive, and the feature contributions add to the displayed totals: $0.5625+1.96=2.5225$ and $0.0625+6.76=6.8225$. The common mistake to avoid is using $0.5$ and $2$ as the squared-distance coefficients; the correct coefficients are their squares, $0.25$ and $4$.

Assignments and Centroid Updates

For fixed centroids, feature weights affect each observation’s assignment through the weighted squared distance. After assigning observations, k-means updates each centroid to the coordinate-wise mean of its assigned observations. It then compares assignments again using the updated centroids. Weighting does not change the fact that centroids are means; it changes the geometry used to assign observations and the objective being minimized.

Worked Example: Assigning Observations and Updating the Means

Four hypothetical standardized profiles have coordinates $P_1=(0,0)$, $P_2=(0,2)$, $P_3=(3,1)$, and $P_4=(4,3)$. Start with centroids $C_1=(0,1)$ and $C_2=(4,2)$. Use feature multipliers $w_1=1$ and $w_2=0.5$. Assign each profile, update the centroids, and check whether the assignments are still nearest-centroid assignments after the update.

Method. The distance formula is $d_w^2=(z_1-c_1)^2+0.25(z_2-c_2)^2$. Calculate both distances for each observation, assign it to the smaller one, and then average the coordinates within each resulting cluster.

For $P_1=(0,0)$, the distances to $C_1$ and $C_2$ are

d_w^2(P_1,C_1)=0^2+0.25(-1)^2=0.25,\qquad d_w^2(P_1,C_2)=(-4)^2+0.25(-2)^2=16+1=17.

Thus $P_1$ is assigned to cluster 1. For $P_2=(0,2)$,

d_w^2(P_2,C_1)=0^2+0.25(1)^2=0.25,\qquad d_w^2(P_2,C_2)=(-4)^2+0.25(0)^2=16.

Thus $P_2$ is also assigned to cluster 1. For $P_3=(3,1)$,

d_w^2(P_3,C_1)=3^2+0.25(0)^2=9,\qquad d_w^2(P_3,C_2)=(-1)^2+0.25(-1)^2=1+0.25=1.25.

Thus $P_3$ is assigned to cluster 2. For $P_4=(4,3)$,

d_w^2(P_4,C_1)=4^2+0.25(2)^2=16+1=17,\qquad d_w^2(P_4,C_2)=0^2+0.25(1)^2=0.25.

Thus $P_4$ is assigned to cluster 2. The assignments are $\{P_1,P_2\}$ and $\{P_3,P_4\}$.

Update. The first cluster mean is \[ \left(\frac{0+0}{2},\frac{0+2}{2}\right)=(0,1), \] so its centroid remains $C_1=(0,1)$. The second cluster mean is \[ \left(\frac{3+4}{2},\frac{1+3}{2}\right)=(3.5,2), \] so its updated centroid is $C_2'=(3.5,2)$.

Check the assignments. For $P_1$, the distance to $C_1$ is $0.25$; to $C_2'$ it is $3.5^2+0.25(2^2)=12.25+1=13.25$. For $P_2$, the distances are $0.25$ and $3.5^2+0.25(0^2)=12.25$. For $P_3$, the distances are $9$ and $0.5^2+0.25(1^2)=0.25+0.25=0.5$. For $P_4$, the distances are $17$ and $0.5^2+0.25(1^2)=0.5$. Every observation remains assigned to its current cluster.

The weighted within-cluster sum of squared distances at the updated centroids is also directly checkable. Cluster 1 contributes $0.25+0.25=0.5$. Cluster 2 contributes $0.5+0.5=1$. The total is $1.5$. The common mistake to avoid is updating a centroid using a weighted median or weighting observations by feature weights; the k-means update here is the ordinary coordinate-wise mean of the assigned observations.

What Weights Mean—and What They Do Not Mean

A larger multiplier makes differences on that feature count more heavily in the distance, but the resulting assignments still depend on all weighted feature differences and the locations of the centroids. A feature with a large weight does not automatically determine every assignment. For a particular pair of centroids, its contribution may be small if the observation and the centroids are close on that coordinate.

Weights also do not have to sum to one. Multiplying all weights by the same positive constant scales every squared distance by the same constant squared and leaves the nearest centroid unchanged. A zero weight is a substantive choice to exclude a feature from assignment distances. A negative multiplier would produce the same squared distances as its absolute value, so weights are conventionally taken as nonnegative; a negative sign does not encode a reversed preference in Euclidean distance.

The weighted objective is the sum, over observations, of their squared weighted distances to their assigned centroids. Reducing one feature’s weight reduces its influence on this objective, while increasing another’s raises its relative influence. This can be useful when the analyst has a defensible reason to emphasize some standardized features. It can also produce clusters that reflect that choice rather than an objective, universal definition of similarity. The weights should therefore be justified and reported alongside the resulting interpretation.

Common Exam Traps

Using the multiplier instead of its square. If the feature is multiplied by $0.5$, its contribution is $0.25(z-c)^2$, not $0.5(z-c)^2$.
Weighting the observation but not the centroid. The transformed distance compares $w_jz_{ij}$ with $w_jc_j$. Equivalently, factor the same weight out of their difference.
Comparing distances computed under different weights. To assess a weighted assignment, calculate every candidate centroid distance under that same weighting scheme.
Assuming a weight alone determines membership. Compare the total weighted squared distances to all candidate centroids; do not assign based only on the feature with the largest weight.
Changing the centroid update rule. Given a fixed assignment, update each feature coordinate with the mean of the observations assigned to that cluster. The weight affects the distance geometry, not the definition of that mean.

Key takeaway: After multiplying standardized feature $j$ by $w_j$, its contribution to squared Euclidean distance is $w_j^2(z_{ij}-c_j)^2$. Recalculate all candidate distances consistently before assigning observations, then update centroids by coordinate-wise means.

Check Your Understanding

For each problem, apply the stated weights to the standardized coordinates and compare squared distances. Treat each observation and centroid as given unless the question asks for a mean update.

An observation is $(1,2)$, with candidate centroids $A=(0,0)$ and $B=(0,1)$. The feature multipliers are $2$ and $0.5$. Calculate both weighted squared distances and give the assignment. (Answer: $d_w^2(A)=4+1=5$, $d_w^2(B)=4+0.25=4.25$; assign to $B$.)
An observation is $(1.5,0.7)$, and centroids are $A=(0,0)$ and $B=(2,2)$. Use multipliers $0.5$ and $2$. Which centroid is nearer? (Answer: $d_w^2(A)=2.5225$, $d_w^2(B)=6.8225$; assign to $A$.)
An observation is $(2,1)$, with centroids $A=(0,1)$ and $B=(3,0)$. The multipliers are $1$ and $2$. Calculate the weighted squared distances and assign the observation. (Answer: $d_w^2(A)=4$, $d_w^2(B)=1+4=5$; assign to $A$.)
Two observations assigned to one cluster are $(1,0)$ and $(3,4)$. Find the updated centroid, then calculate the weighted squared distance from $(1,0)$ to it using multipliers $1$ and $0.5$. (Answer: the centroid is $(2,2)$, and the distance is $1+0.25(4)=2$.)
An observation is $(1,1)$, with centroids $A=(0,0)$ and $B=(2,0)$. Compare assignments with multipliers $(1,1)$ and $(0.5,2)$. (Answer: unweighted distances are $2$ to $A$ and $2$ to $B$, a tie; weighted distances are $4.25$ to $A$ and $4.25$ to $B$, also a tie.)

What You'll Learn