Distance Matrices And Their R Representations

What a Distance Matrix Represents

After choosing variables and preparing their scales, a distance-based analysis compares observations in pairs. A distance matrix records those pairwise distances in a square table: observation \(i\) labels a row, observation \(j\) labels a column, and the entry in row \(i\), column \(j\) is the distance between them. It is a useful representation for inspection, but it can be large even when the original data set is modest.

For \(n\) observations with numeric measurements \(x_{i1},\ldots,x_{ip}\), the Euclidean distance between observations \(i\) and \(j\) is the square root of the sum of squared differences across the \(p\) variables. The observations must have measurements on the same variables for this calculation. The examples below use complete numeric data and focus on how the resulting distances are represented, not on how to prepare missing data.

The diagonal is zero because an observation has no difference from itself. Symmetry means that the distance from observation \(i\) to observation \(j\) does not depend on which one is named first. These properties are useful checks on a computed matrix. A nonzero diagonal or unequal values on opposite sides of it would be a reason to check the data and calculation.

Why the Matrix Is Symmetric

For Euclidean distance, write the measurements for observations \(i\) and \(j\) as vectors \(\boldsymbol{x}_i\) and \(\boldsymbol{x}_j\). Their distance is the length of the difference vector. Reversing the order changes the sign of each coordinate difference, but squaring removes that sign.

Swapping \(i\) and \(j\) gives \((x_{jk}-x_{ik})^2=(x_{ik}-x_{jk})^2\) for every variable \(k\). Therefore \(d(j,i)=d(i,j)\). Setting \(i=j\) makes every difference zero, so \(d(i,i)=0\). This justifies symmetry and the zero diagonal for Euclidean distance. The same checks apply to other standard symmetric distance measures, but not every quantity called a dissimilarity is necessarily a metric with all the same properties; check the definition being used.

Only one value is needed for each distinct pair. There are \(n\) choices for a first observation and \(n-1\) choices for a different second observation. This counts every pair twice, once in each order, so the number of unique distances is

A full \(n\)-by-\(n\) matrix contains \(n^2\) entries, including the diagonal and both copies of every off-diagonal distance. The formula above explains why R can store the distinct distances in a compact form rather than storing every matrix entry.

Computing and Inspecting Distances in R

In base R, `dist()` computes pairwise distances between rows of a numeric data object. With `method = "euclidean"`, it uses Euclidean distance. The returned object has class `dist`; it is a compact representation of the unique pairwise distances, not a full square matrix. Use `as.matrix()` when a square matrix is needed for display or for an operation that expects one.

The following code reproduces the first example. Row names provide readable observation labels. They do not change the distance calculation.

A complete R version is:

To attach the labels and compute the matrix, use:

The matrix `D_mat` contains the entries shown in the worked example. In R, `D_mat["A", "B"]` retrieves the distance between A and B, which is 5. The compact object also retains information such as its size and labels; `attr(D_dist, "Size")` gives the number of observations, and `labels(D_dist)` returns the labels when supplied. A common practical check is to compare selected matrix entries with direct calculations for the corresponding rows.

Memory and Computation Grow Quadratically

A compact `dist` object reduces storage compared with a full square matrix, but it still contains a number of values proportional to \(n^2\). If the number of observations doubles, the number of unique pairs becomes approximately four times as large for large \(n\). In R, a double-precision number typically occupies 8 bytes, so a first-order estimate for the compact numeric values is \(8\binom{n}{2}\) bytes. A full double matrix requires \(8n^2\) bytes for its numeric entries.

These are estimates for the numeric values, not exact total memory use. R objects also need metadata, and labels or copies of the original data take additional space. Converting a compact object to a full matrix can require the full matrix to exist in memory alongside the compact object and the input data. The maximum usable allocation also depends on the computer and R environment.

Practical Checks and Common Traps

Before using a computed distance object, verify that the number of observations is the number of rows intended, that the feature columns are the intended numeric inputs, and that labels remain in the expected row order. When converting to a matrix, check that its dimensions are \(n\) by \(n\), its diagonal is zero, and entries are symmetric. For a small data set, recomputing one or two distances by hand is a direct validation of the chosen method and inputs.

A particularly tempting but incorrect storage calculation is to multiply the number of observations by itself and assume that this is the number of values in a compact `dist` object. That is the count for a full square matrix, not the compact object. Conversely, using only \(\binom{n}{2}\) to estimate a full matrix omits the mirrored entries. State which representation is being estimated before calculating memory.

These distinctions matter in practice because a conversion made only for convenient printing can substantially increase memory use. At larger sample sizes, estimate the pair count and storage before calling `as.matrix()`. If the estimate is close to the available memory, a full matrix may not be practical. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.