Why the Distance Choice Matters
An embedding can only arrange cells according to the relationships it is given. For t-SNE and UMAP, those relationships are commonly built from distances between expression vectors. Changing the distance definition can change which cells count as neighbors, and therefore change the input structure the embedding attempts to represent. Two cells with similar relative expression patterns but very different overall expression levels, for example, may be close under one distance and far apart under another.
The word “metric” is often used broadly in software settings. Strictly, a mathematical metric must satisfy properties including the triangle inequality. Euclidean distance does; \(1-\)cosine similarity and \(1-\)correlation are commonly used dissimilarities, but need not satisfy the triangle inequality. Here, “distance choice” includes these standard dissimilarities. For neighborhood construction, their rankings and the software’s handling of them are especially important.
Euclidean distance responds to both profile direction and overall magnitude. Cosine dissimilarity compares direction after normalizing each vector to length one, so positive rescaling of either vector does not change it. Correlation dissimilarity compares centered direction: adding the same constant to every gene value in a vector does not change its correlation with another vector, and positive rescaling does not change that correlation either. These invariances are properties of the formulas, not claims that magnitude or baseline expression is biologically irrelevant.
Cosine dissimilarity lies between 0 and 2 for nonzero vectors; correlation dissimilarity also lies between 0 and 2 when correlation is defined. A value of 0 indicates identical direction for cosine, or perfect positive linear association across genes for correlation. Neither conclusion requires the expression vectors to have equal values gene by gene.
A Worked Comparison of Three Distances
Worked Example: One Cell Pair, Three Geometries
Suppose three hypothetical cells have expression vectors \(x=(1,2,3)\), \(y=(2,4,6)\), and \(z=(2,3,4)\), in arbitrary transformed expression units. Treat \(x\) as the query cell and compare the other two cells with it.
For Euclidean distance, the differences from \(x\) are \(y-x=(1,2,3)\) and \(z-x=(1,1,1)\). Thus
Euclidean distance ranks \(z\) closer to \(x\). For cosine dissimilarity, \(x^\mathsf{T}y=28\), \(\|x\|=\sqrt{14}\), and \(\|y\|=\sqrt{56}=2\sqrt{14}\). The vectors \(x\) and \(y\) point in exactly the same direction, so their cosine similarity is 1 and their dissimilarity is 0. For \(z\), \(x^\mathsf{T}z=20\), \(\|z\|=\sqrt{29}\), giving
Cosine dissimilarity ranks \(y\) closer: the large difference in their expression magnitudes does not matter when their directions match. For correlation, \(y=2x\), and \(z=x+(1,1,1)\). Both are positive linear transformations of \(x\), so each has correlation 1 with \(x\). Therefore \(d_R(x,y)=0\) and \(d_R(x,z)=0\): correlation ties these cells, despite their different Euclidean distances.
This example shows that the three choices encode different notions of similarity. If a neighbor list is cut off after one cell, Euclidean selects \(z\), cosine selects \(y\), and correlation cannot distinguish \(y\) from \(z\). In a larger data set, such changes at the neighbor boundary can alter the graph supplied to t-SNE or UMAP.
What the Distances Imply for PCA, t-SNE, and UMAP
Ordinary PCA is not a method that accepts a user-selected pairwise distance metric. It centers the feature columns and finds directions of high variance, with gene scaling affecting the covariance structure as covered in the earlier material on gene scaling and PCA. Its geometry is Euclidean geometry in the chosen, centered feature space. Replacing Euclidean distance with cosine or correlation dissimilarity is not, by itself, a change to ordinary PCA.
PCA can still be part of a distance-based workflow. For example, cells may first be represented by scores on selected principal components, and Euclidean distances may then be calculated between those score vectors for a later neighborhood method. If all components are retained, the orthogonal change of coordinates preserves Euclidean distances. If components are discarded, distances can shrink because differences along omitted directions no longer contribute. Using cosine or correlation on scores is another possible downstream choice, but that is a distance calculation on PCA scores, not a different ordinary PCA objective.
With only the first \(K\) components, the matrix of retained basis vectors \(V_K\) instead gives \(\|V_K^\mathsf{T}(x-y)\|^2\leq\|x-y\|^2\). The omitted component of the difference accounts for any reduction. This explains why Euclidean neighborhoods among truncated PC scores need not match neighborhoods in the full feature space.
In t-SNE, the chosen input dissimilarities contribute to the pairwise similarities that define the target neighborhood relationships. In UMAP, the input distances help determine nearest neighbors before those relationships are encoded in its graph. Thus, for both methods, changing the distance can change which cells enter local neighborhoods even when the embedding settings are held fixed. The embedding algorithm does not make different input distances equivalent.
Worked Example: Why a Magnitude Change Can Alter a Neighbor
Consider hypothetical query-cell vector \(q=(1,0,0)\) and two candidate cells \(a=(100,0.1,0)\) and \(b=(1,1,0)\). These values are illustrative transformed expression values. The vectors are nonzero, so cosine dissimilarity is defined.
Euclidean distances are
Euclidean distance therefore selects \(b\) as the closer cell. For \(a\), the cosine similarity is \(100/\sqrt{10000.01}\), so \(d_C(q,a)\approx 0.00000050\). For \(b\), the cosine similarity is \(1/\sqrt{2}\), so \(d_C(q,b)=1-1/\sqrt{2}\approx0.2929\). Cosine dissimilarity selects \(a\), whose direction is nearly identical to \(q\), despite its much larger magnitude.
If \(a\) and \(b\) are the only candidates and a one-neighbor graph is requested, the two choices produce different edges from \(q\). A t-SNE or UMAP neighborhood construction based on these distances consequently receives different local relationships. This does not determine the final plotted coordinates on its own; it identifies a change in the input neighborhood structure.
Worked Example: A Truncated PCA Projection Can Change Euclidean Neighbors
Suppose a hypothetical PCA representation retains only the first coordinate. In that retained space, compare query \(u=(0,0)\), candidate \(v=(3,4)\), and candidate \(w=(3,0)\). In the full two-dimensional space,
Thus \(w\) is closer before projection. After retaining only the first coordinate, the projected vectors are \(u'=(0)\), \(v'=(3)\), and \(w'=(3)\). Their distances from the projected query are both 3, so the distinction disappears. The missing second-coordinate difference contributed \(4^2=16\) to the squared distance for \(v\). A subsequent cosine or correlation calculation on retained scores would also use only the retained representation; it cannot recover that omitted difference.
This arithmetic illustrates a general point, not a claim that an arbitrary first coordinate is necessarily a PCA component. PCA chooses its axes from the data. Once components are retained or discarded, however, any later distance is conditional on that representation.
Useful Relationships, Diagnostics, and Limitations
Cosine and correlation have useful connections to Euclidean distance after normalization. For nonzero vectors, set \(u=x/\|x\|\) and \(v=y/\|y\|\). Then
Therefore, Euclidean distance between unit-normalized vectors and cosine dissimilarity give the same neighbor ranking: the squared Euclidean distance is twice the cosine dissimilarity. Similarly, if each vector is first centered across genes and then normalized to length one, the squared Euclidean distance between the resulting vectors is \(2\{1-r(x,y)\}\). These relationships explain the geometry but do not make the preprocessing decisions interchangeable: normalization changes which aspects of expression contribute.
Before choosing a distance, inspect the representation used to calculate it. The genes should be aligned in the same order, and the transformed values should have a meaningful scale for the comparison. Library-size normalization, log transformation, gene selection, and gene scaling can all change the vectors and hence their neighborhoods. Euclidean distance can be dominated by high-magnitude cells or genes. Cosine can discard a potentially meaningful difference in overall expression level. Correlation deliberately ignores a gene-wise constant offset and scale, which may remove biologically relevant information as well as nuisance variation.
Check for undefined or degenerate cases. Cosine dissimilarity is undefined if either vector has zero norm. Pearson correlation is undefined if either vector is constant across genes, because its centered norm is zero. Near-zero norms or near-constant profiles can also make the calculation numerically unstable. In addition, \(1-r\) and \(1-\cos\) are dissimilarities that may not obey the triangle inequality; software implementations can differ in which forms they accept and how they use them. Confirm the documented definition and preprocessing in the selected implementation rather than assuming that every option labeled “correlation” is identical.
State which genes and transformed values define each cell vector, including whether distances are computed on genes or on retained PCA scores.
Choose Euclidean when absolute differences in the chosen feature space matter; choose cosine when direction should matter more than overall vector length; choose correlation when centered pattern across genes is the target.
For representative cells, compare neighbor identities or distances across plausible choices. Look for zero or constant vectors, ties, and changes concentrated among cells with unusual magnitude or baseline.
Ordinary PCA uses a variance-based Euclidean geometry; t-SNE and UMAP use distance-dependent neighborhood construction. Record the distance and representation used for each embedding.
There is no universally best choice. Euclidean distance is direct and natural when the chosen expression scale makes absolute differences meaningful. Cosine and correlation can reduce sensitivity to cell-wide scale, but their invariances may erase information of interest. Comparisons should therefore be framed as sensitivity checks on neighborhood relationships, not as a search for a metric that guarantees a biologically correct embedding.
Common Mistake
A common mistake is to say, “Cosine distance fixes library-size effects, so it is always preferable for RNA-seq.” Cosine is invariant to multiplying a whole vector by a positive constant, but that does not establish that all relevant technical effects are removed, nor that total expression is irrelevant. Conversely, correlation also ignores a constant offset across genes, but that does not mean it is simply a more complete cosine distance. The correction is to state which differences should count as biologically meaningful, calculate the resulting neighborhood changes on the chosen representation, and check that the software handles the selected dissimilarity as intended.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For nonzero \(y=3x\), what are the cosine and correlation dissimilarities between \(x\) and \(y\)? Must their Euclidean distance be zero?
- Why can ordinary PCA not be described as “PCA using correlation distance” merely by changing a distance option in a later t-SNE step?
- For \(x=(1,2)\) and \(y=(2,1)\), calculate the Euclidean and cosine dissimilarities, and state which is smaller in each comparison if a candidate \(z=(1,1)\) is also available as the query’s reference vector \(x\)?
- Name one condition that makes cosine dissimilarity undefined and one condition that makes correlation dissimilarity undefined.
Solutions. (1) Since \(y\) is a positive scalar multiple of \(x\), cosine similarity and correlation are both 1, so both dissimilarities are 0. Euclidean distance is \(\|x-3x\|=2\|x\|\), which is positive for nonzero \(x\). (2) Ordinary PCA finds high-variance directions of the centered feature matrix; it does not accept a configurable pairwise distance. A later t-SNE calculation can use distances between a chosen representation, but that does not change how the PCA axes were obtained. (3) For the pair \(x=(1,2)\), \(y=(2,1)\), \(d_E=\sqrt{(1-2)^2+(2-1)^2}=\sqrt{2}\approx1.414\). Their cosine similarity is \(4/5=0.8\), so \(d_C=0.2\). If \(x=(1,2)\) is compared with \(z=(1,1)\), then \(d_E=1\) and \(d_C=1-3/\sqrt{10}\approx0.05132\). Thus Euclidean ranks \(z\) closer than \(y\), and cosine also ranks \(z\) closer than \(y\); the two distances differ in values, not this particular ranking. (4) Cosine is undefined for a zero vector. Correlation is undefined for a vector constant across genes, because its centered vector has zero norm.