Correlation-Based Similarity For Profile Clustering

Comparing Profile Shape Rather Than Magnitude

Euclidean distance compares the coordinate values of two observations directly. That is useful when differences in level and scale are meaningful. In some clustering problems, however, the question is whether two observations have a similar pattern across variables, even if one observation has consistently larger values. Correlation-based similarity is designed for that profile-comparison goal.

For example, a hypothetical analyst might compare the monthly activity patterns of two service regions. If both regions have busy and quiet months in the same relative order, their profiles may be considered similar even when one region has a much higher overall activity level. Whether that is sensible depends on the purpose of the clustering: correlation deliberately discounts overall level and positive scale.

Definition: Let two observations have numeric profiles \(x=(x_1,\ldots,x_p)\) and \(y=(y_1,\ldots,y_p)\), recorded on the same \(p\) variables. Their Pearson profile correlation is the ordinary Pearson correlation calculated across those \(p\) paired coordinates. If both profiles have nonzero variation, their correlation distance is \(d_{\mathrm{corr}}(x,y)=1-r_{xy}\), where \(r_{xy}\) is their profile correlation.

The coordinates must be aligned: \(x_j\) and \(y_j\) must refer to the same variable for every \(j\). Correlation is undefined if either profile is constant, because its centered sum of squares is zero. With only one coordinate, there is no variation across coordinates from which to calculate a correlation. These are not cases where the correlation distance is zero; the correlation is simply undefined.

Calculating Profile Correlation and Distance

For complete profiles with \(p\) shared numeric coordinates and nonzero variation in both profiles, define their coordinate means as \(\bar{x}\) and \(\bar{y}\). The sample Pearson correlation formula is:

r_{xy}= \frac{\sum_{j=1}^{p}(x_j-\bar{x})(y_j-\bar{y})} {\sqrt{\left[\sum_{j=1}^{p}(x_j-\bar{x})^2\right] \left[\sum_{j=1}^{p}(y_j-\bar{y})^2\right]}}

The denominator rescales the centered cross-product so the correlation is between \(-1\) and \(1\). A correlation of \(1\) means the centered profiles point in the same direction; a correlation of \(-1\) means they point in opposite directions. The corresponding correlation distance \(1-r_{xy}\) ranges from \(0\) to \(2\): smaller values indicate more similar profile shapes, and a value of \(2\) represents perfectly reversed profiles.

Worked Example: Profiles That Differ in Level and Scale

A hypothetical analyst records two regions' values on four aligned monthly indicators. Region A has profile \(x=(2,4,6,8)\), and Region B has profile \(y=(11,14,17,20)\). Calculate their correlation and correlation distance. The method is to center both profiles, calculate the correlation, and then use \(1-r_{xy}\). This avoids mistaking different levels or units of magnitude for different profile shapes.

Step 1: Calculate the means. The means are \(\bar{x}=(2+4+6+8)/4=5\) and \(\bar{y}=(11+14+17+20)/4=15.5\).

Step 2: Center the profiles. The centered values are \(x-\bar{x}=(-3,-1,1,3)\) and \(y-\bar{y}=(-4.5,-1.5,1.5,4.5)\).

Step 3: Calculate the numerator and sums of squares. The cross-product sum is \((-3)(-4.5)+(-1)(-1.5)+(1)(1.5)+(3)(4.5)=13.5+1.5+1.5+13.5=30\). The centered sums of squares are \(9+1+1+9=20\) for \(x\), and \(20.25+2.25+2.25+20.25=45\) for \(y\).

Step 4: Calculate and check. The denominator is \(\sqrt{20\cdot45}=\sqrt{900}=30\), so \(r_{xy}=30/30=1\). Therefore \(d_{\mathrm{corr}}=1-1=0\). This is consistent with the profiles having exactly the same centered shape: \(y_j=1.5x_j+8\) for every coordinate. Their levels and scales differ, but their correlation distance is zero.

What Correlation Distance Ignores

Correlation is unchanged when a constant is added to every coordinate of one profile. It is also unchanged when every coordinate is multiplied by the same positive constant. Consequently, two profiles can have correlation distance zero even though their raw values are far apart. Euclidean distance, by contrast, uses the raw coordinate differences and is sensitive to those changes.

Worked Example: A Large Shift With No Change in Profile Shape

A hypothetical analyst compares a four-month profile \(x=(1,2,3,4)\) with \(y=(2,4,3,5)\), then considers a shifted profile \(z=(102,104,103,105)\). Calculate the correlation distance from \(x\) to \(y\), and compare the Euclidean distances from \(x\) to \(y\) and from \(y\) to \(z\). The method is to calculate correlation across the four aligned coordinates and separately calculate the raw Euclidean distances. This makes explicit which differences each measure retains.

Step 1: Calculate the correlation of \(x\) and \(y\). Their means are \(\bar{x}=2.5\) and \(\bar{y}=3.5\). The centered profiles are \((-1.5,-0.5,0.5,1.5)\) and \((-1.5,0.5,-0.5,1.5)\). The cross-product sum is \(2.25-0.25-0.25+2.25=4\). Each centered sum of squares is \(5\), so \(r_{xy}=4/\sqrt{5\cdot5}=4/5=0.8\). Thus \(d_{\mathrm{corr}}(x,y)=1-0.8=0.2\).

Step 2: Calculate the Euclidean distance from \(x\) to \(y\). The coordinate differences are \((-1,-2,0,-1)\). Therefore the distance is \(\sqrt{(-1)^2+(-2)^2+0^2+(-1)^2}=\sqrt{6}\), approximately \(2.449\).

Step 3: Compare \(y\) and \(z\). Since \(z=y+100\) in every coordinate, their centered profiles are identical. Thus \(r_{yz}=1\) and \(d_{\mathrm{corr}}(y,z)=0\). Their Euclidean distance is \(\sqrt{100^2+100^2+100^2+100^2}=\sqrt{40000}=200\).

The result checks the distinction: the shift leaves correlation distance at zero but creates a Euclidean distance of \(200\). Use correlation-based similarity only if a uniform level shift is not important to the intended grouping.

Why the Formula Relates to Euclidean Geometry

There is a useful geometric justification for correlation distance. It explains why correlation compares profile direction after removing the profile mean and normalizing its size. Assume both profiles have nonzero variation and the same \(p\) coordinates. Center each profile, then divide it by its Euclidean length to form unit vectors:

u=\frac{x-\bar{x}\mathbf{1}}{\sqrt{\sum_{j=1}^{p}(x_j-\bar{x})^2}}, \qquad w=\frac{y-\bar{y}\mathbf{1}}{\sqrt{\sum_{j=1}^{p}(y_j-\bar{y})^2}}

The dot product \(u\cdot w\) is exactly the Pearson correlation, because its numerator is the centered cross-product sum and each vector's denominator is its centered length. Expanding the squared Euclidean distance between these normalized centered vectors gives:

\|u-w\|^2=(u-w)\cdot(u-w) =u\cdot u+w\cdot w-2u\cdot w =1+1-2r_{xy} =2(1-r_{xy})

Therefore the Euclidean distance between these centered, normalized profiles is \(\sqrt{2(1-r_{xy})}\). This is sometimes called the chord distance between the normalized profiles. It has the same ordering of pairs as correlation distance, since the square root is increasing, but it is not numerically equal to \(1-r_{xy}\).

This distinction matters: \(1-r_{xy}\) is commonly used as a dissimilarity, but it should not automatically be treated as an ordinary Euclidean distance or assumed to satisfy every metric property. The chord distance above is an actual Euclidean distance between the transformed vectors. If an analysis requires Euclidean geometry, that distinction should be respected.

Worked Example: Interpreting a Negative Correlation

Two hypothetical four-coordinate profiles are \(a=(1,2,3,4)\) and \(b=(8,6,4,2)\). Calculate their correlation distance and chord distance, and explain what each says. The method is to identify the exact linear relationship, then apply the two distance formulas. This avoids interpreting a large correlation distance as a calculation error when the profiles are oppositely ordered.

Step 1: Identify the profile relationship. For every coordinate, \(b_j=10-2a_j\). This is a negative linear relationship, so the centered profiles are exact negatives up to a positive scale factor. Hence \(r_{ab}=-1\).

Step 2: Calculate correlation distance. \(d_{\mathrm{corr}}(a,b)=1-(-1)=2\), the maximum possible correlation distance.

Step 3: Calculate chord distance and check. The chord distance is \(\sqrt{2(1-r_{ab})}=\sqrt{2(1-(-1))}=\sqrt{4}=2\). Both results indicate perfectly reversed patterns. Whether such profiles should be considered dissimilar is a decision about the application: correlation distance does not treat an increasing pattern and its decreasing reverse as similar.

When Profile Similarity Is Appropriate

Correlation-based similarity is appropriate when the variables form a meaningful, aligned profile and the analysis is meant to group observations by their relative pattern rather than their absolute level or overall scale. Examples could include comparing the shape of monthly demand across regions or the relative response pattern of hypothetical devices across a fixed set of test conditions. In each case, the coordinates must represent corresponding conditions for every observation.

It is less appropriate when a high overall level is substantively different from a low one, when scale differences are important, or when the coordinates do not form comparable profiles. It can also be unstable or uninformative when profiles have very few coordinates. Correlation uses only the pattern across coordinates, so its meaning should be justified by the analysis objective rather than selected merely because the resulting values look convenient.

Key takeaway: Correlation distance \(1-r\) compares centered profile shape and ignores uniform shifts and positive rescaling. Use it when those ignored differences are unimportant; use a magnitude-sensitive measure when absolute levels and coordinate differences matter. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

A Common Exam Trap

A frequent error is to calculate \(1-r\) and then describe it as the Euclidean distance between the original observations. Those are different calculations. For the profiles in the preceding example with \(x=(1,2,3,4)\) and \(y=(2,4,3,5)\), the correlation distance is \(0.2\), while the Euclidean distance between the original profiles is \(\sqrt{6}\), approximately \(2.449\). The Euclidean chord distance between centered, normalized profiles is instead \(\sqrt{2(0.2)}=\sqrt{0.4}\), approximately \(0.632\).

The correction is to name the measure and its transformation: \(1-r\) is correlation distance; \(\sqrt{2(1-r)}\) is Euclidean distance after centering and normalizing each profile; the ordinary Euclidean distance uses the original coordinate differences. Do not interchange them.

Check Your Understanding

For each problem, assume the listed coordinates are aligned and complete unless the question says otherwise.

Profiles \(a=(2,4,6)\) and \(b=(5,7,9)\) are compared by Pearson profile correlation. Find \(r_{ab}\) and \(d_{\mathrm{corr}}\). (Answer: \(r_{ab}=1\); \(d_{\mathrm{corr}}=0\).)
Profiles \(x=(1,2,3)\) and \(y=(3,2,1)\) are exact reversals. Find their correlation distance and chord distance. (Answer: \(r_{xy}=-1\); correlation distance \(=2\); chord distance \(=2\).)
For \(a=(1,2,3,4)\) and \(b=(2,3,5,6)\), calculate the correlation distance to four decimal places. (Answer: \(r_{ab}=7/\sqrt{50}\approx0.9899\); \(d_{\mathrm{corr}}\approx0.0101\).)
A profile has four coordinates, all equal to 6, while a second profile varies across its four coordinates. Can their Pearson profile correlation and correlation distance be calculated? (Answer: No. The constant profile has zero centered sum of squares, so the correlation and correlation distance are undefined.)
Two nonconstant profiles have correlation \(r=0.75\) across \(p=4\) coordinates. Find their correlation distance and the squared Euclidean distance between their centered, unit-length profiles. (Answer: Correlation distance \(=0.25\); squared chord distance \(=2(1-0.75)=0.5\).)

What You'll Learn