Why Represent Pairwise Comparisons as a Matrix?
Hierarchical clustering begins with comparisons between observations. A pairwise dissimilarity matrix collects those comparisons in one organized object: each row and column refers to an observation, and each entry reports how dissimilar one observation is from another. The matrix makes it possible to check the input before a clustering procedure uses it.
This representation is useful when the observations are actuarial risk profiles, customer records, or other objects whose pairwise differences have already been defined. The entries need not be calculated from raw features in this tutorial; the central task is to represent and validate the resulting comparisons. The specific formulas for common distance measures are a separate topic.
The index order matters: rows and columns must refer to the same observations in the same order. The entry in row \(i\), column \(j\) compares \(x_i\) with \(x_j\). Thus, \(d_{ij}\) and \(d_{ji}\) describe the same unordered pair when dissimilarity is symmetric.
Reading the Diagonal and Off-Diagonal Entries
A diagonal entry compares an observation with itself. Under the usual convention, that comparison has dissimilarity zero, so every diagonal entry is \(d_{ii}=0\). An off-diagonal entry compares two different indexed observations. Those entries contain the pairwise information used to start agglomerative clustering.
When the matrix is symmetric, the two positions \(d_{ij}\) and \(d_{ji}\) repeat the same comparison. Consequently, an \(n \times n\) matrix displays each distinct pair twice, even though there are only \(n(n-1)/2\) distinct pairs.
To see why, there are \(n(n-1)\) ordered choices of two different indices: \(n\) choices for the first and \(n-1\) for the second. Each unordered pair appears in two orders, \((i,j)\) and \((j,i)\), so dividing by two gives \(n(n-1)/2\). The \(n\) diagonal positions are not included because they are self-comparisons, not pairs of distinct observations.
Worked Example: Constructing a Matrix from Pairwise Scores
A hypothetical analyst has four risk profiles, A, B, C, and D. A review process has already assigned these pairwise dissimilarities: A–B is 2, A–C is 7, A–D is 5, B–C is 6, B–D is 4, and C–D is 3. Put the scores into a matrix in the order A, B, C, D, and check the basic properties.
Method. Put each supplied pairwise score in both corresponding positions, since the pair is unordered. Put zero on the diagonal, since each profile is compared with itself.
For instance, the A–C score of 7 gives \(d_{13}=7\) and \(d_{31}=7\). The B–D score of 4 gives \(d_{24}=4\) and \(d_{42}=4\). The four diagonal entries are all zero. Each listed score is nonnegative, and each off-diagonal entry matches its reflection across the diagonal; therefore the matrix is symmetric and nonnegative.
There are \(4(4-1)/2=4\cdot3/2=6\) distinct pairs, exactly the six scores supplied. The matrix contains 12 off-diagonal positions because each of the six scores appears twice. Common mistake avoided: counting all 12 off-diagonal cells as 12 independent comparisons.
Check. The matrix is \(4 \times 4\), has four zeros on its diagonal, and represents all six unique comparisons exactly once in the upper triangle.
Which Properties Are Required?
For a standard pairwise dissimilarity input to agglomerative clustering, the essential checks are symmetry, a zero diagonal, and nonnegative entries. These checks ensure that a pair has one consistent comparison value, self-comparisons are not treated as separation, and larger entries do not represent negative dissimilarity.
Some definitions of a metric also require that \(d_{ij}=0\) only when \(x_i=x_j\). A dissimilarity used in clustering may assign zero to two distinct records—for example, two records with identical measured profiles. That is not a contradiction if the records are distinct but indistinguishable under the chosen comparison rule.
These are general input checks, not a guarantee that every clustering method is suitable for every matrix. A particular linkage rule may require further structure or assumptions. For example, whether a particular method has a geometric interpretation is a separate question from whether the matrix is symmetric and nonnegative.
Worked Example: Checking a Matrix Without Requiring the Triangle Inequality
A hypothetical three-profile comparison produces the matrix below, with profiles P, Q, and R in that order. Determine whether it meets the basic requirements for a general dissimilarity matrix, and check whether it is a metric by testing the triangle inequality.
Step 1: Check the diagonal. The diagonal entries are \(d_{PP}=0\), \(d_{QQ}=0\), and \(d_{RR}=0\).
Step 2: Check symmetry and nonnegativity. The three unique off-diagonal values are \(d_{PQ}=d_{QP}=4\), \(d_{PR}=d_{RP}=11\), and \(d_{QR}=d_{RQ}=6\). All are nonnegative, and each pair of reflected entries agrees. The matrix therefore meets the basic requirements for a general dissimilarity matrix.
Step 3: Test the triangle inequality. For the P–R pair, the direct dissimilarity is \(d_{PR}=11\), while the route through Q gives \(d_{PQ}+d_{QR}=4+6=10\). Since \(11\) is greater than \(10\), the triangle inequality fails for this triple.
The result is still a valid general dissimilarity matrix under the stated basic checks, but it is not a metric. Common mistake avoided: rejecting every dissimilarity matrix that fails the triangle inequality. That condition is needed for a metric, not for every general dissimilarity input.
Check. The matrix passes the symmetry, zero-diagonal, and nonnegativity checks; it fails the metric test because \(11\) is greater than \(4+6\).
Using a Similarity Matrix
Sometimes the available table records similarity rather than dissimilarity: larger values mean that two observations are more alike. Such a table cannot be treated as a dissimilarity matrix without checking its meaning and scale. If a similarity \(s_{ij}\) is symmetric, has \(s_{ii}=1\), and lies between 0 and 1, one possible conversion is \(d_{ij}=1-s_{ij}\). This conversion reverses the direction: the greatest similarity, 1, becomes dissimilarity 0.
The conversion is appropriate only when the similarity values have the stated interpretation and scale. It is not a universal rule for arbitrary scores. After converting, check the resulting matrix directly; in particular, do not assume that a transformed similarity necessarily satisfies every property of a metric.
Worked Example: Converting a Bounded Similarity Matrix
A hypothetical comparison system reports similarities between four policy profiles, with larger values indicating greater similarity. The profiles are ordered A, B, C, D. The similarity matrix is symmetric, has diagonal entries 1, and all values are between 0 and 1. Convert it to a dissimilarity matrix using \(d_{ij}=1-s_{ij}\), then verify the basic checks.
Step 1: Convert each distinct off-diagonal value. The values are \(1-0.8=0.2\), \(1-0.3=0.7\), \(1-0.5=0.5\), \(1-0.4=0.6\), \(1-0.7=0.3\), and \(1-0.2=0.8\). Each converted value is placed in both reflected positions.
Step 2: Convert the diagonal. For each profile, \(1-s_{ii}=1-1=0\).
Step 3: Verify. The matrix is symmetric because the similarity matrix was symmetric and the same transformation was applied to both positions of every pair. Its diagonal is zero because every original diagonal value was 1. All converted values are nonnegative because each original similarity was at most 1. Thus the converted matrix meets the basic checks for a general dissimilarity matrix.
Check. There are six distinct pairs, and the six converted pair values are \(0.2, 0.7, 0.5, 0.6, 0.3,\) and \(0.8\), each appearing twice off the diagonal. Common mistake avoided: using the similarity values themselves as dissimilarities, which would reverse the intended interpretation.
Validation Before Clustering
A matrix that looks approximately symmetric may still contain a data-entry error, an inconsistent row order, or a score recorded on the wrong scale. Verify the source and construction before changing entries. Averaging mismatched values or replacing a negative entry with zero may produce a matrix that passes superficial checks but no longer represents the comparisons that were intended.
Check that row \(i\) and column \(i\) both refer to the same observation \(x_i\), and that the order is consistent throughout the matrix.
For \(n\) observations, the matrix should have \(n\) rows and \(n\) columns, with \(d_{ii}=0\) for all \(i\).
Compare \(d_{ij}\) with \(d_{ji}\) for each pair and inspect every entry for negative values. Investigate discrepancies against their source rather than silently editing them.
Ensure that larger values really indicate greater dissimilarity. If the table instead records similarity, use a conversion only when its assumptions and scale are justified.
Test the triangle inequality only when a metric is required. Do not impose it as a universal condition on general dissimilarity matrices.
The central exam-style task is not merely to recognize a square table. It is to interpret what every entry compares, distinguish the repeated symmetric entries from the unique pairs, and identify exactly which conditions are required. This matrix representation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, check the matrix properties and interpret the number or type of pairwise comparisons.
- A matrix describes five observations. How many diagonal entries and how many distinct unordered pairs should it contain? (Answer: 5 diagonal entries and \(5(4)/2=10\) distinct pairs.)
- For three observations U, V, and W, a matrix has \(d_{UV}=3\), \(d_{VU}=3\), \(d_{UW}=8\), \(d_{WU}=8\), \(d_{VW}=2\), and \(d_{WV}=2\); all diagonal entries are zero. Does it meet the basic requirements for a general dissimilarity matrix? (Answer: Yes; it is symmetric, has a zero diagonal, and all entries are nonnegative.)
- A symmetric, zero-diagonal matrix has nonnegative entries \(d_{AB}=2\), \(d_{BC}=3\), and \(d_{AC}=7\). Does it meet the basic requirements for a general dissimilarity matrix, and is it a metric? (Answer: It meets the basic requirements, but it is not a metric because \(7\) is greater than \(2+3=5\).)
- A symmetric similarity matrix has diagonal entries 1 and off-diagonal entries 0.25 and 0.9 for two different pairs. Under \(d_{ij}=1-s_{ij}\), what are the dissimilarities for those pairs? (Answer: \(0.75\) and \(0.10\), respectively.)
- A proposed dissimilarity matrix has \(d_{12}=4\) and \(d_{21}=5\), with all diagonal entries zero and all other entries nonnegative. What basic check fails, and what should be done before changing either value? (Answer: Symmetry fails; check the data source and observation ordering before making any correction.)