Why Standardization Changes the PCA Question
Principal component analysis (PCA) uses variation among numeric variables to create new features. The covariance matrix records both how variables vary and their units of measurement. That can be appropriate when the original scales are meaningful and comparable. But if one variable is annual benefit amount in dollars and another is claim duration in days, their numerical scales may differ greatly. A covariance-based analysis can then give more influence to the variable with the larger numerical variance, even if that difference is mainly due to measurement units.
Correlation-based PCA addresses this scale issue by centering each variable and dividing by its standard deviation. Each transformed variable then has sample variance one, so the PCA is based on the correlation matrix. This is a choice about how to represent variation, not a universal improvement: standardization removes original scale information, which may itself be important.
Here \(i\) indexes observations and \(j\) indexes variables. The formulas use the sample convention with denominator \(n-1\). The same convention must be used consistently when calculating the sample covariance and correlation matrix. Standardization as defined here requires numeric observations and a nonzero sample standard deviation for every variable being scaled. If a variable is constant, \(s_j=0\), so the standardized values and correlations involving that variable are undefined.
From Standardized Data to the Correlation Matrix
Let \(X\) denote the \(n\)-by-\(p\) data matrix, with observations in rows and numeric variables in columns. Centering subtracts each column mean. Covariance-based PCA uses the sample covariance matrix of those centered variables. Correlation-based PCA instead standardizes each column and calculates the sample covariance matrix of the resulting standardized variables.
The link follows directly from the definitions. The sample covariance between variables \(j\) and \(k\) is \(s_{jk}=\frac{1}{n-1}\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)(x_{ik}-\bar{x}_k)\). Substituting the standardized values into their sample covariance gives:
The last expression is the sample correlation between variables \(j\) and \(k\). For \(j=k\), the standardized variable has sample variance one, so the diagonal entries are one. Therefore, the covariance matrix of the standardized data is the sample correlation matrix of the original variables. This is why PCA performed on standardized columns is called correlation-based PCA.
This relationship assumes the same observations are used for each covariance and standard deviation, as in a complete numeric data matrix. If different pairs of variables use different sets of observations because of missing values, pairwise correlations need not form the covariance matrix of one consistently standardized data matrix. A preprocessing decision about missing values is then needed before this identity can be used as stated.
Worked Example: Standardizing Features in Different Units
A hypothetical insurer records four policies. For each policy, the features are annual premium in dollars, number of claims, and claim-processing days. Standardize each feature using the sample mean and sample standard deviation. Report standardized values to four decimal places.
| Policy | Premium ($) | Claims | Processing days |
|---|---|---|---|
| A | 1,000 | 0 | 2 |
| B | 2,000 | 1 | 1 |
| C | 3,000 | 1 | 3 |
| D | 4,000 | 2 | 2 |
Method. Calculate each column’s mean and sample standard deviation, then subtract the mean and divide by the standard deviation. The calculation is performed separately for each feature, so the original units do not carry into the standardized values.
Premium. The mean is \((1{,}000+2{,}000+3{,}000+4{,}000)/4=2{,}500\). The squared deviations sum to \(2{,}250{,}000+250{,}000+250{,}000+2{,}250{,}000=5{,}000{,}000\). Thus \(s=\sqrt{5{,}000{,}000/3}=1{,}000\sqrt{5/3}\approx1{,}290.9944\) dollars. The standardized premiums are approximately \((-1.1619,-0.3873,0.3873,1.1619)\).
Claims. The mean is \((0+1+1+2)/4=1\). The squared deviations sum to \(1+0+0+1=2\), so \(s=\sqrt{2/3}\approx0.8165\) claims. The standardized values are \((-1.2247,0,0,1.2247)\).
Processing days. The mean is \((2+1+3+2)/4=2\). The squared deviations sum to \(0+1+1+0=2\), giving the same sample standard deviation, \(\sqrt{2/3}\approx0.8165\) days. The standardized values are \((0,-1.2247,1.2247,0)\).
Check. Each standardized column has mean zero. Its squared standardized values sum to three, so its sample variance is \(3/(4-1)=1\). The premiums’ dollar units, the claims’ count units, and the processing-time units have all been removed from the standardized values. The common mistake avoided here is dividing by the variance rather than the standard deviation.
Covariance-Based and Correlation-Based PCA
Covariance-based PCA retains each variable’s original scale. Correlation-based PCA first rescales the variables to unit variance. The covariance matrix is sensitive to measurement units: multiplying a variable by a constant changes its variance and its covariances with other variables. In contrast, correlation is unchanged by a positive change of units. Thus converting dollars to thousands of dollars changes a covariance-based analysis but not a correlation-based analysis.
Neither approach is automatically correct in every application. Covariance-based PCA is reasonable when the variables use comparable units or when larger absolute variation is intended to carry greater influence. Correlation-based PCA is often useful when variables have different units or when the analysis should treat a one-standard-deviation change in each feature as comparable. Standardizing does not make the features equally important in a substantive sense; it makes their marginal sample variances equal before PCA.
| Choice | Matrix used | Effect of changing a variable’s units | Typical consideration |
|---|---|---|---|
| Covariance-based PCA | Sample covariance matrix of centered variables | Can change the matrix and resulting PCA | Original scale and absolute variation are meaningful |
| Correlation-based PCA | Sample correlation matrix, equivalently covariance of standardized variables | Unaffected by positive unit conversions | Variables have unlike units or should have equal initial variance |
Worked Example: Comparing Covariance and Correlation
A hypothetical portfolio study records annual account balance in thousands of dollars and review time in days for four accounts. The observations are \((1,1),(2,2),(3,2),(4,3)\). Calculate the sample covariance matrix and correlation matrix, then explain what changes if balance is recorded in dollars instead. Round correlations to four decimal places.
Method. Use the sample covariance formula with denominator \(n-1=3\), then divide the covariance by the product of the two sample standard deviations to obtain the correlation.
The balance mean is \((1+2+3+4)/4=2.5\) thousand dollars, with deviations \((-1.5,-0.5,0.5,1.5)\). Their squared deviations sum to \(5\), so the sample variance is \(5/3\) thousand-dollars squared. The review-time mean is \((1+2+2+3)/4=2\) days, with deviations \((-1,0,0,1)\). Their squared deviations sum to \(2\), so the sample variance is \(2/3\) days squared.
The cross-products of deviations are \(1.5,0,0,1.5\), summing to \(3\). The sample covariance is \(3/3=1\) thousand-dollar-days. Therefore the covariance matrix is:
The sample standard deviations are \(\sqrt{5/3}\) thousand dollars and \(\sqrt{2/3}\) days. The correlation is \(1/(\sqrt{5/3}\sqrt{2/3})=3/\sqrt{10}\approx0.9487\). The correlation matrix is therefore:
Unit conversion. Recording balance in dollars multiplies that column by \(1{,}000\). Its variance becomes \(1{,}000{,}000(5/3)\), and its covariance with review time becomes \(1{,}000(1)=1{,}000\). The correlation remains \(0.9487\), because both the covariance and balance standard deviation are multiplied by \(1{,}000\), and those factors cancel in the correlation formula.
Check. Both diagonal entries of the correlation matrix are one, and the off-diagonal value is between \(-1\) and \(1\), as required for a correlation. The common mistake avoided is comparing the numerical size of a covariance entry with a correlation: they have different scales and units.
Standardizing New Values Consistently
When a new observation must be expressed on the same standardized scale as a reference data set, use the reference means and standard deviations. Do not recalculate them using the new observation. Re-estimating the scaling constants would change the coordinate system and make standardized values from different records incomparable.
Worked Example: Applying Training-Set Scaling
A hypothetical analyst has a reference set of three policy records. Their annual benefit amounts, in thousands of dollars, are \(40,60,80\), and their claim durations, in months, are \(8,12,16\). A new policy has benefit \(90\) thousand dollars and duration \(8\) months. Calculate its standardized values using the reference set’s statistics, and state what each value means. Round to four decimal places.
Method. Find the reference-set mean and sample standard deviation for each feature, then apply \(z=(x-\bar{x})/s\) to the new record.
For benefit amount, the mean is \((40+60+80)/3=60\) thousand dollars. The squared deviations are \(400,0,400\), so the sample variance is \(800/(3-1)=400\) and the sample standard deviation is \(20\) thousand dollars. The new policy’s standardized benefit is \((90-60)/20=1.5000\).
For claim duration, the mean is \((8+12+16)/3=12\) months. The squared deviations are \(16,0,16\), so the sample variance is \(32/2=16\) months squared and the sample standard deviation is \(4\) months. The new policy’s standardized duration is \((8-12)/4=-1.0000\).
Interpretation and check. The benefit is \(1.5\) reference-set standard deviations above its mean; the duration is one standard deviation below its mean. The new record is not included when computing either reference mean or standard deviation. This preserves the same scaling rule for the reference observations and the new policy.
Common Mistakes and Practical Checks
Before choosing correlation-based PCA, check what the scaling decision means in context. A variable with larger variance may have a larger variance because it is measured in larger units, or because it genuinely varies more. Standardization removes both effects; it cannot distinguish them.
- Dividing by variance. Standardization divides by \(s_j\), not \(s_j^2\). If a value is \(30\), its mean is \(20\), and its standard deviation is \(5\), the correct standardized value is \((30-20)/5=2\). Dividing by the variance \(25\) gives \(0.4\), which is not the standardized value.
- Assuming standardization changes only units. It also changes the relative influence of variables in the PCA input. A low-variance feature is scaled up relative to a high-variance feature.
- Scaling a constant feature. A constant variable has \(s_j=0\); its standardized value and correlation are undefined. It must be handled before correlation-based PCA.
- Mixing scaling conventions. If a correlation is calculated using one set of rows but a standard deviation uses another, the covariance-of-standardized-data identity may fail.
Check Your Understanding
Use the sample standard deviation convention with denominator \(n-1\). Give numerical answers to four decimal places where needed.
- A feature has observed values \(2,4,6\). Find its mean, sample standard deviation, and standardized value for the observation \(6\). (Answer: mean \(=4\), sample standard deviation \(=2\), standardized value \(=1.0000\))
- Two variables have sample standard deviations \(3\) and \(4\), and sample covariance \(6\). Calculate their sample correlation and state whether it is positive or negative. (Answer: \(6/(3\cdot4)=0.5000\), positive)
- A cost feature is recorded first in thousands of dollars and then in dollars, a positive factor-of-\(1{,}000\) conversion. State the effect on its variance and on its correlation with another variable. (Answer: its variance is multiplied by \(1{,}000{,}000\); its correlation is unchanged)
- A numeric feature has sample mean \(10\) and sample variance \(9\). For an observation with value \(16\), calculate its standardized value. (Answer: \(s=3\), so \(z=(16-10)/3=2.0000\))
- A data column contains the values \(5,5,5,5\). Can it be standardized using the sample formula, and what is the reason? (Answer: no; its sample standard deviation is \(0\), making division by the standard deviation undefined)