A Hierarchical Clustering Workflow In R

From Prepared Data to Interpretable Groups

Hierarchical clustering turns a set of pairwise dissimilarities into a nested sequence of merges. In R, a useful workflow is more than a call to a clustering function: the analyst must select and prepare the features, compute a suitable distance object, fit one or more linkage methods, inspect the dendrogram, choose a cut, and describe the resulting groups in the original feature context.

This tutorial focuses on that workflow. It assumes the observations and variables have already been selected for the analysis. As in the earlier material on clustering, the distance and linkage determine what “close” means; the dendrogram does not discover a uniquely correct grouping on its own. The methods shown here are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A hierarchical clustering is a nested sequence of groups formed by merging observations or existing groups. An agglomerative procedure starts with each observation in its own group and repeatedly merges the two groups judged closest under a specified dissimilarity and linkage rule. A distance object stores pairwise dissimilarities, and a dendrogram displays the merge sequence and merge heights.

Prepare the Matrix and Compute Distances

In the standard numeric workflow, the input to dist() is a numeric matrix or data frame with observations in rows and features in columns. Row names identify observations in printed output; they do not enter the distance calculation. Columns should contain only the features intended to define similarity. Exclude identifiers and outcome variables that do not belong in the clustering question.

Euclidean distance between two feature vectors $x_i$ and $x_j$, each with $p$ numeric features, is

d(i,j)=\sqrt{\sum_{r=1}^{p}(x_{ir}-x_{jr})^2}.

A feature measured on a much larger numerical scale can dominate this distance. Standardizing columns is often appropriate when features should have comparable influence, but it is a modeling choice: it changes the meaning of distance. A constant column cannot be standardized, and missing values require deliberate treatment before relying on a complete numeric matrix. For mixed variables, do not treat category codes as ordinary numeric measurements; the earlier material on Gower distance covers an alternative for mixed data.

Worked Example: Standardize Features and Inspect the Distance Object

A hypothetical analyst compares three service locations using customer age and the number of follow-up visits. The data are A: (20, 0), B: (30, 1), and C: (50, 1). The analyst decides that the two features should have equal influence and uses R's sample-standard-deviation scaling.

Method. Put locations in rows, apply column-wise centering and scaling, and calculate Euclidean distances on the scaled values. In R, the commands are: X <- data.frame(age=c(20,30,50), visits=c(0,1,1), row.names=c("A","B","C")); Z <- scale(X); D <- dist(Z); as.matrix(D).

For age, the mean is $(20+30+50)/3=33.3333$. The sample standard deviation is $\sqrt{(177.7778+11.1111+277.7778)/2}=\sqrt{233.3333}=15.2753$. The scaled age values are approximately $-0.8729,-0.2182,1.0911$. For visits, the mean is $2/3$, the sample standard deviation is $\sqrt{(0.4444+0.1111+0.1111)/2}=0.5774$, and the scaled values are approximately $-1.1547,0.5774,0.5774$.

The pairwise distances are then calculated from differences in both scaled features:

d(A,B)=\sqrt{(-0.6547)^2+(-1.7321)^2}\approx1.8516, \qquad d(B,C)=\sqrt{(-1.3093)^2+0^2}\approx1.3093,

d(A,C)=\sqrt{(-1.9640)^2+(-1.7321)^2}\approx2.6186.

Thus the printed distance matrix has zero diagonal and symmetric off-diagonal entries $d(A,B)\approx1.8516$, $d(A,C)\approx2.6186$, and $d(B,C)\approx1.3093$. B and C are closest under the chosen standardized Euclidean distance. The check is that all three distances are nonnegative and the largest is A-to-C, which differs on both features.

Common mistake avoided. Calculating directly from the raw columns would let the age differences, measured in tens, overwhelm the visit differences, measured in units. Standardization is not automatically right in every application; this example explicitly chooses it because equal feature influence is intended.

Fit and Compare Linkage Methods

A distance object alone does not specify how to measure the distance between two groups. For groups $G$ and $H$, single linkage uses the smallest cross-group distance, complete linkage the largest, and average linkage the mean of all cross-group distances. Ward linkage instead merges groups to minimize the increase in within-group sum of squares, under its Euclidean-distance interpretation.

In R, hclust(D, method="single"), method="complete", and method="average" fit the corresponding methods. For Ward clustering with Euclidean distances, use method="ward.D2". The returned object stores the merge sequence, merge heights, labels, and method. Using the same distance object for a comparison helps isolate how the linkage choice changes the hierarchy.

Justification: Average linkage is the mean of every cross-group pairwise distance. If $G$ contains $n_G$ observations and $H$ contains $n_H$, there are $n_Gn_H$ such pairs. Summing their distances and dividing by that count gives

d_{\mathrm{average}}(G,H)= \frac{1}{n_Gn_H}\sum_{i\in G}\sum_{j\in H}d(i,j).

Each cross-group pair appears once in the double sum, so the expression gives every pair equal weight. This also explains why average linkage is not generally the average of two previous cluster-level distances unless those distances represent the same number of observation pairs.

Worked Example: Compare Linkage Heights for Four Locations

Four hypothetical locations have one numeric feature, recorded as A: 0, B: 1, C: 4, and D: 5. Using Euclidean distance, the pairwise distances are $d(A,B)=1$, $d(A,C)=4$, $d(A,D)=5$, $d(B,C)=3$, $d(B,D)=4$, and $d(C,D)=1$. The R workflow is X <- matrix(c(0,1,4,5), ncol=1, dimnames=list(c("A","B","C","D"), "score")); D <- dist(X); hc.single <- hclust(D, "single"); hc.complete <- hclust(D, "complete"); hc.average <- hclust(D, "average"); hc.ward <- hclust(D, "ward.D2").

First merges. A-B and C-D are both at distance 1. The tie means either pair may appear first in the displayed merge order, but both must merge before the two pairs join. Their first merge heights are 1 for all four methods.

Single linkage. After the two pairs have formed, the smallest distance between $\{A,B\}$ and $\{C,D\}$ is $\min(4,5,3,4)=3$. The final merge height is 3.

Complete linkage. The largest cross-group distance is $\max(4,5,3,4)=5$. The final merge height is 5.

Average linkage. The mean cross-group distance is $(4+5+3+4)/4=4$. The final merge height is 4.

Ward linkage. The two pair means are 0.5 and 4.5. Merging these size-2 groups increases the within-group sum of squares by $(2\cdot2/4)(4.5-0.5)^2=16$. With ward.D2, the corresponding displayed height is $\sqrt{2\cdot16}=\sqrt{32}\approx5.6569$. The initial pair merges have height 1. These values illustrate why Ward heights are not simply the maximum or mean cross-group distance.

The check is that all methods first form the two nearest pairs, while the final heights differ because their group-to-group criteria differ. In R, inspect hc.single$height, hc.complete$height, hc.average$height, and hc.ward$height to verify the merge heights. The tied first merges may be ordered differently without changing the resulting two pairs.

Common mistake avoided. Do not describe the complete-linkage final height as the average of cross-group distances: that would give 4, the average-linkage height. Complete linkage uses the largest cross-group distance, which is 5 here.

Inspect the Dendrogram and Choose a Cut

Plotting a fitted object makes its merge sequence visible: for example, plot(hc.complete, main="Complete-linkage clustering", xlab="", sub=""). The vertical position of a merge is its height under the selected distance and linkage. A large vertical gap between successive merges can be a useful feature to examine, but it is not by itself proof that a particular number of groups is correct. Check the observation labels, the intended use, and the resulting group profiles as well.

A tree can be cut by a chosen number of groups, $k$, or by a height, $h$. In R, cutree(hc, k=2) returns a group assignment for every input row; cutree(hc, h=3.5) cuts at height 3.5. The choice should be stated and justified in the analysis rather than hidden in code. Group numbers are labels, not ranks, and can change if the tree or row order changes.

Fit the chosen hierarchy.
Pass the prepared distance object and an explicit linkage method to hclust().

Inspect the tree.
Plot the dendrogram and check the merge sequence and heights against the selected linkage.

Apply a stated cut.
Use cutree() with a specified group count or height and retain the observation labels.

Summarize the groups.
Join assignments back to the original features and report summaries that make the groups interpretable.

Worked Example: Cut a Tree and Summarize the Groups

Use the four-location data from the preceding example, fit complete linkage, and select two groups. The commands are hc <- hclust(dist(X), method="complete"); grp <- cutree(hc, k=2); split(rownames(X), grp); aggregate(as.data.frame(X), list(cluster=grp), mean).

The two pairwise merges occur at height 1, and the final merge occurs at height 5. A cut at $k=2$, or equivalently at a height such as 3.5, therefore gives the groups $\{A,B\}$ and $\{C,D\}$. Which group is printed as 1 or 2 is arbitrary.

For $\{A,B\}$, the mean score is $(0+1)/2=0.5$. For $\{C,D\}$, it is $(4+5)/2=4.5$. The group sizes are 2 and 2. The summaries confirm that the cut separates the lower scores from the higher scores in this small example. They do not establish that the same grouping would be useful for a larger population or a different set of features.

Check. There are four assignments in total, each observation appears once, and the two group means average to $(0.5+4.5)/2=2.5$, the mean of all four scores. This agrees with the original data.

Common mistake avoided. Do not interpret cluster label 1 as “more” or “better” than label 2. The numerical labels merely identify groups; their feature summaries provide the interpretation.

Make the Workflow Reproducible

A reproducible analysis should keep the feature preparation, distance calculation, linkage, cut, and summaries connected. In particular, retain the row names and the exact selected columns so that assignments can be matched back to the right observations. When comparing linkages, use the same prepared matrix and distance definition unless the purpose is explicitly to compare different distance choices.

Before interpreting a result, verify that the input is numeric and that each row represents one observation. Check for missing values and constant columns, and confirm that any scaling decision is intentional. A simple, transparent R sequence is Z <- scale(X); D <- dist(Z); hc <- hclust(D, method="average"); plot(hc); grp <- cutree(hc, k=3), followed by summaries of the original, unscaled features. The example uses average linkage and three groups only as a sequence of commands; those choices are not defaults for every analysis.

The central interpretation is conditional: the resulting groups depend on the observations, features, scaling, distance, linkage, and cut. Reporting those choices alongside a concise group profile makes the analysis understandable and repeatable. This workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

Work through each problem by identifying the distance or linkage rule before calculating.

Three observations have one-dimensional values 2, 5, and 9. Under complete linkage, which pair merges first and what is the final merge height? (Answer: The pair 2 and 5 merges first at height 3; the final merge height is 7.)
Four observations have one-dimensional values 0, 2, 7, and 11. Under single linkage, cut the hierarchy into two groups. Which observations belong together? (Answer: The groups are $\{0,2\}$ and $\{7,11\}$.)
Two observations have two-feature vectors (1, 2) and (4, 6). Find their Euclidean distance. (Answer: $\sqrt{3^2+4^2}=5$.)
Ward linkage is considering a merge of groups of sizes 3 and 2, with respective one-dimensional means 10 and 16. Find the increase in within-group sum of squares and the corresponding Ward.D2 height. (Answer: The increase is $(3\cdot2/5)(16-10)^2=43.2$, and the Ward.D2 height is $\sqrt{2\cdot43.2}=\sqrt{86.4}\approx9.2952$.)
A complete-linkage tree has merge heights 1, 1, and 5. What groups result from a cut at height 3.5, and what should be remembered about their printed numeric labels? (Answer: The cut gives the two groups formed by the first two merges; their numeric labels are arbitrary identifiers, not ranks.)

What You'll Learn