Cophenetic Distances and Dendrogram Fidelity

Why Compare a Dendrogram with the Original Dissimilarities?

A hierarchical clustering produces a nested sequence of merges, which can be displayed as a dendrogram. The dendrogram summarizes the original pairwise dissimilarities, but it cannot generally preserve every one of them exactly. A useful diagnostic is to compare each original dissimilarity with the height at which that pair of observations first belongs to the same cluster. These merge heights are the cophenetic distances.

The cophenetic correlation summarizes the linear association between the original dissimilarities and the cophenetic distances across all unordered pairs. It helps assess whether the dendrogram’s merge structure reflects the relative sizes of the input dissimilarities. It does not establish that the clustering is substantively correct or that the chosen dissimilarity measure is appropriate.

Definition: For two distinct observations \(i\) and \(j\), their cophenetic distance, \(c_{ij}\), is the dendrogram height at which they first belong to the same cluster. Equivalently, it is the merge height of the first cluster containing both observations. For \(n\) observations, there are \(m=n(n-1)/2\) unordered pairs. The cophenetic correlation is the Pearson correlation between the \(m\) original pairwise dissimilarities \(d_{ij}\) and their corresponding cophenetic distances \(c_{ij}\).

The diagonal values \(c_{ii}\) are conventionally zero, but they are not included in the correlation: each observation is paired with the other distinct observations only. In a dendrogram whose merge heights are nondecreasing, cophenetic distances have the ultrametric property: for any three observations, the two largest of their three cophenetic distances are equal. This property describes the tree representation; it does not mean that the original dissimilarities must be ultrametric.

Reading Cophenetic Distances from a Dendrogram

To find \(c_{ij}\), trace the two observations upward from their leaves until they first meet at a common branch. Read the height of that merge. The pair’s original dissimilarity is not used to read the cophenetic distance; it is a separate value that will be compared with the merge height.

Worked Example: Constructing the Cophenetic-Distance Vector

A hypothetical clustering of four service units, A, B, C, and D, first merges A with B at height 2. It merges C with D at height 3. The two resulting clusters merge at height 8. List the six cophenetic distances in the order AB, AC, AD, BC, BD, CD.

Method. For each pair, find the first node in the dendrogram that contains both members.

For A and B, the first shared cluster is formed at height 2, so \(c_{AB}=2\). For C and D, the first shared cluster is formed at height 3, so \(c_{CD}=3\).

Each cross-cluster pair, AC, AD, BC, and BD, remains in separate clusters until the two larger clusters merge at height 8. Therefore, \(c_{AC}=c_{AD}=c_{BC}=c_{BD}=8\).

In the requested order, the cophenetic-distance vector is \((2,8,8,8,8,3)\). As a check, the two pairs merged within the initial clusters have smaller cophenetic distances than the four pairs that only meet at the final merge.

Common mistake avoided: The cophenetic distance for AC is 8, not the height of either earlier merge. A pair’s value comes from the first merge that joins the two observations, not from the first merge involving either one separately.

Calculating the Cophenetic Correlation

Let \(d_1,\ldots,d_m\) be the original dissimilarities for the unordered pairs, listed in the same order as the corresponding cophenetic distances \(c_1,\ldots,c_m\). Their means are \(\bar d\) and \(\bar c\). The cophenetic correlation is the ordinary Pearson correlation:

r_{\mathrm{coph}}= \frac{\sum_{k=1}^{m}(d_k-\bar d)(c_k-\bar c)} {\sqrt{\left[\sum_{k=1}^{m}(d_k-\bar d)^2\right] \left[\sum_{k=1}^{m}(c_k-\bar c)^2\right]}}

The coefficient is defined only if both vectors vary, so that both sums of squared deviations in the denominator are positive. A positive value indicates that pairs with larger original dissimilarities tend, in a linear sense, to have larger cophenetic distances. A value near 1 indicates a strong positive linear association; a value near 0 indicates little linear association. A negative value indicates an inverse linear association.

Interpretation: The cophenetic correlation measures linear association across unordered pairs, not exact preservation of all pairwise dissimilarities. It is a descriptive summary of the particular dissimilarity matrix and dendrogram being compared.

One way to justify the formula is to center both vectors. The numerator is their centered dot product, while each sum of squares is the squared length of one centered vector. Dividing the dot product by the product of the lengths gives the cosine of the angle between the centered vectors. Thus the coefficient is large and positive when the centered original dissimilarities and centered cophenetic distances point in similar directions. This also explains why a constant vector makes the coefficient undefined: after centering, its length is zero.

Worked Example: Computing Cophenetic Correlation for Four Observations

For four hypothetical components, a dendrogram merges A and B at height 2, C and D at height 3, and the two clusters at height 8. The original dissimilarities in the order AB, AC, AD, BC, BD, CD are \((2,5,6,6,7,3)\). Calculate the cophenetic correlation.

Method. Pair each original dissimilarity with the merge height for that pair. From the dendrogram, the cophenetic vector in the same order is \((2,8,8,8,8,3)\).

The original-dissimilarity mean is \(\bar d=(2+5+6+6+7+3)/6=29/6\). The cophenetic-distance mean is \(\bar c=(2+8+8+8+8+3)/6=37/6\).

Calculate the centered sum of squares for the original dissimilarities: \(\sum d_k^2=4+25+36+36+49+9=159\), so \(S_{dd}=159-29^2/6=159-841/6=113/6\). For the cophenetic distances, \(\sum c_k^2=4+64+64+64+64+9=269\), so \(S_{cc}=269-37^2/6=269-1369/6=245/6\).

The cross-product sum is \(\sum d_kc_k=4+40+48+48+56+9=205\). Therefore, \(S_{dc}=205-(29)(37)/6=205-1073/6=157/6\).

Substituting these centered sums gives \(r_{\mathrm{coph}}=(157/6)/\sqrt{(113/6)(245/6)} =157/\sqrt{27685}\approx0.9436\). The coefficient is close to 1, indicating a strong positive linear association for these six pairs. The calculation is consistent with the data: the within-cluster pairs have the smaller cophenetic distances, and the pairs whose clusters meet only at height 8 have larger original dissimilarities overall.

Common mistake avoided: The pairs must be matched in the same order in both vectors. Pairing an original dissimilarity with another pair’s merge height changes the cross-product sum and gives the wrong correlation.

What a High Correlation Does—and Does Not—Say

A high cophenetic correlation says that the tree’s merge heights tend to increase as the original dissimilarities increase. It is a compact summary, not a guarantee that every pair is represented closely. A small number of pairs with large discrepancies can be obscured by an overall association, and the coefficient does not identify which pairs are poorly represented. Inspecting the pairwise differences \(d_{ij}-c_{ij}\), or a plot of original dissimilarities against cophenetic distances, can reveal patterns that a single coefficient hides.

The coefficient also depends on the numerical scale of the merge heights. Multiplying all cophenetic distances by a positive constant and adding a constant to them leaves the correlation unchanged, because centering removes the shift and standardization removes the positive scale factor. A nonlinear transformation of heights need not preserve the correlation. In particular, a high correlation is not a statement that the dendrogram reproduces the original values exactly.

The cophenetic correlation is calculated over \(n(n-1)/2\) pairs, not over \(n\) observations. Those pairwise values share observations and are therefore not independent. The coefficient is useful as a descriptive measure, but the usual interpretation should not treat all pairs as independent observations for an ordinary correlation test.

Worked Example: A Three-Observation Correlation and Its Limits

A hypothetical dendrogram merges P and Q at height 2, then adds R at height 6. The original dissimilarities are \(d_{PQ}=1\), \(d_{PR}=4\), and \(d_{QR}=5\). Calculate the cophenetic correlation and state what it supports.

Method. The cophenetic distances in the order PQ, PR, QR are \((2,6,6)\). The original vector in that order is \((1,4,5)\). Their means are \(\bar d=10/3\) and \(\bar c=14/3\).

For the original vector, \(\sum d_k^2=1+16+25=42\), giving \(S_{dd}=42-10^2/3=26/3\). For the cophenetic vector, \(\sum c_k^2=4+36+36=76\), giving \(S_{cc}=76-14^2/3=32/3\). The cross-product sum is \(\sum d_kc_k=2+24+30=56\), so \(S_{dc}=56-(10)(14)/3=28/3\).

Thus \(r_{\mathrm{coph}}=(28/3)/\sqrt{(26/3)(32/3)} =28/\sqrt{832}=7/\sqrt{52}\approx0.9707\). This is a strong positive linear association among the three pairs. It does not prove that the tree is the only reasonable clustering, nor does it establish that the original dissimilarities capture the right notion of similarity for the application.

Common mistake avoided: There are three pairs here, not three observations plus their diagonal entries. Including diagonal values would change the calculation and would not be the cophenetic correlation defined above.

Common Calculation Traps

Using the wrong merge. Read the height where the two leaves first share a cluster. Do not use a later merge that contains their already-joined cluster.
Misaligning pair order. Make a single pair list, such as AB, AC, AD, BC, BD, CD, and use it for both vectors.
Counting ordered pairs or diagonal entries. Use each distinct unordered pair once, for \(n(n-1)/2\) values.
Confusing association with equality. A high correlation does not mean \(d_{ij}=c_{ij}\), nor does it say that all individual discrepancies are small.
Rounding too early. Keep exact centered sums or unrounded means until the final coefficient. For example, \((76/3)/\sqrt{(70/3)(106/3)}=76/\sqrt{7420}\approx0.8823\), not 0.8824.
Reporting a coefficient when it is undefined. If all original dissimilarities or all cophenetic distances are equal, the corresponding centered sum of squares is zero and the correlation is undefined.

Key takeaway: Build the cophenetic vector by reading one first-common-merge height for every unordered pair, then calculate its Pearson correlation with the original dissimilarity vector in exactly the same pair order. Interpret the result as a descriptive summary of linear association, not as proof of a uniquely correct clustering.

Check Your Understanding

For each problem, keep the pair order explicit and interpret the result in context.

A dendrogram first merges P and Q at height 1, merges R and S at height 2, and joins those clusters at height 7. Give the cophenetic distances in the order PQ, PR, PS, QR, QS, RS. (Answer: \((1,7,7,7,7,2)\))
For three observations, original dissimilarities in the order AB, AC, BC are \((2,4,5)\). The dendrogram merges A and B at height 1 and then adds C at height 5. Calculate the cophenetic correlation to four decimal places. (Answer: \(5/\sqrt{28}\approx0.9449\))
A comparison has centered cross-product sum \(S_{dc}=12\), original centered sum of squares \(S_{dd}=18\), and cophenetic centered sum of squares \(S_{cc}=32\). Calculate the correlation and state its direction. (Answer: \(12/\sqrt{18\cdot32}=0.5000\), positive)
Two dendrograms are compared with the same original dissimilarities. One has a cophenetic correlation of 0.91 and the other 0.78. What limited conclusion is supported, and what conclusion is not justified by these values alone? (Answer: The first has stronger positive linear association with the original dissimilarities; this alone does not establish that its clusters are substantively correct or that every pair is better represented.)
A dendrogram merges A and B at height 5, C and D at height 3, and the two clusters at height 8. In the order AB, AC, AD, BC, BD, CD, give the cophenetic vector and identify the issue with the merge heights. (Answer: \((5,8,8,8,8,3)\); the merge heights are not nondecreasing because the second merge is below the first.)

What You'll Learn