Why scaling can change the answer
A dimension-reduction procedure summarizes predictors, but its result can depend on how those predictors are measured. A distance calculation involving dollars and counts may be driven almost entirely by the dollar values. A variance-based procedure may likewise give more influence to a predictor whose numerical values happen to be large. Changing dollars to thousands of dollars can then change the result, even though the underlying information has not changed.
The remedy is not always to scale every predictor. The right choice depends on the procedure and the meaning of the variables. Centering and scaling are distinct operations: centering shifts a predictor’s origin, while scaling changes its unit and relative numerical spread. Both must be considered using the training data, with the resulting transformation then applied consistently to other data.
Distance calculations and measurement units
The Euclidean distance between two observations with \(p\) numeric predictors is the square root of the sum of their squared predictor differences. It is sensitive to units: expressing a predictor in dollars rather than thousands of dollars multiplies its differences by 1,000 and its squared contribution by 1,000,000. Consequently, distance-based comparisons can be dominated by the measurement scale rather than by a deliberate judgment about which predictors matter.
Subtracting the same training mean from every observation does not, by itself, change pairwise Euclidean distances. For two observations \(a\) and \(b\), the centered difference is \((x_{aj}-\bar{x}_j)-(x_{bj}-\bar{x}_j)=x_{aj}-x_{bj}\). Dividing by the standard deviation does change the distance: it expresses differences in units of training-set standard deviations. This is useful when predictors are measured in incomparable units, but it is a modeling choice, not a universal requirement.
Worked Example: Comparing Distances Across Different Units
A hypothetical pricing analysis uses two numeric predictors: number of prior claims and annual repair cost in dollars. In the training data, the claims predictor has mean 2 and sample standard deviation 1; repair cost has mean $1,000 and sample standard deviation $1,000. Compare observations A = (1 claim, $1,000) and B = (3 claims, $2,000), first using raw Euclidean distance and then using standardized predictors.
Method. Calculate each predictor’s difference and combine the squared differences. For the standardized distance, divide each difference by that predictor’s training standard deviation.
Raw distance. The claims difference is \(3-1=2\), and the repair-cost difference is \(2000-1000=1000\) dollars. Thus, $$ d_{\mathrm{raw}}(A,B)=\sqrt{2^2+1000^2} =\sqrt{4+1{,}000{,}000} =\sqrt{1{,}000{,}004} \approx 1000.002. $$ The displayed distance is rounded to three decimal places.
Standardized distance. A has standardized coordinates \(((1-2)/1,(1000-1000)/1000)=(-1,0)\). B has coordinates \(((3-2)/1,(2000-1000)/1000)=(1,1)\). Therefore, $$ d_{\mathrm{standardized}}(A,B) =\sqrt{(1-(-1))^2+(1-0)^2} =\sqrt{4+1} =\sqrt{5} \approx 2.236. $$ This distance is in standard-deviation units rather than a mixture of claims and dollars.
Check and trap. The raw calculation is dominated by the $1,000 cost difference, while the standardized calculation gives the differences a scale-relative interpretation. A common mistake is to report the raw distance as evidence that cost is intrinsically more important. The raw result reflects the chosen units; standardizing avoids that unit effect, but does not establish that both predictors should have equal substantive importance.
Variance-driven dimension reduction
Principal component analysis (PCA) is a variance-based dimension-reduction method. In its usual centered form, it finds linear combinations of predictors that capture variation in the predictor data. When PCA is based on the covariance matrix, a predictor with much larger numerical variance can dominate the first component. If predictors have different units, this dominance may be an artifact of measurement.
For centered training data with \(n\) observations and \(p\) predictors, the sample covariance matrix is \(S=X_c^\mathsf{T}X_c/(n-1)\), where \(X_c\) is the centered data matrix. The first principal component direction is a unit vector \(v\) that maximizes \(v^\mathsf{T}Sv\). Standardizing each predictor before forming the covariance matrix instead gives the correlation matrix, whose diagonal entries are one when all predictors have positive sample variance. This prevents a predictor from dominating solely because its units create a large standard deviation. It does not guarantee that the resulting components are better for a particular prediction objective.
Worked Example: How a Large Variance Can Control PCA
A hypothetical screening data set has four training observations and two already centered predictors: \(X_1=(-10,-10,10,10)\) and \(X_2=(-1,1,-1,1)\). Find the sample covariance matrix and the proportion of total variance captured by the first component using covariance-based PCA. Then state what happens after standardizing both predictors.
Method. Compute the sample variances and covariance using denominator \(n-1=3\). Because the predictors are centered, their sums and cross-products give the required quantities directly.
For \(X_1\), the sum of squared values is \(100+100+100+100=400\), so its sample variance is \(400/3\). For \(X_2\), the sum is \(1+1+1+1=4\), so its sample variance is \(4/3\). The cross-products are \(10,-10,-10,10\), which sum to zero; hence the sample covariance is \(0/3=0\). Therefore, $$ S= \begin{pmatrix} 400/3 & 0\\ 0 & 4/3 \end{pmatrix}. $$ The eigenvalues are \(400/3\) and \(4/3\). The first component is along \(X_1\), and the total variance is \(400/3+4/3=404/3\). Its proportion is $$ \frac{400/3}{404/3}=\frac{400}{404}=\frac{100}{101}\approx 0.9901. $$ Thus the first component captures about 99.01% of the total variance, rounded to four decimal places as a proportion.
After standardizing, each predictor has sample variance 1. Their sample covariance remains zero because their cross-products sum to zero. The standardized covariance matrix is therefore the identity matrix, with eigenvalues 1 and 1. Each direction captures half the total variance, and no unique first direction is selected by variance alone.
Check and trap. The calculations show that covariance-based PCA heavily favors \(X_1\), while standardized PCA treats the two predictors equally in marginal variance. A common mistake is to assume that a large percentage of variance captured necessarily means the original-unit result is appropriate. If \(X_1\)'s spread is meaningful and should receive greater influence, covariance-based PCA may be defensible; if its numerical scale is arbitrary, standardization is often more appropriate.
Centering is not the same as scaling
Centering is generally part of ordinary PCA because the goal is to describe variation around predictor means. Without centering, a large nonzero mean can influence a second-moment calculation and be confused with variation. Standardizing includes centering and divides by a scale estimate; centering alone leaves each predictor’s units and variance unchanged.
For Euclidean distances, centering all observations with the same training means leaves every pairwise difference unchanged. Scaling, in contrast, changes the relative contribution of each predictor. For PCA, centering is important to the usual covariance interpretation; scaling determines whether the covariance matrix or a standardized correlation matrix drives the result. These choices should be stated explicitly rather than treated as interchangeable preprocessing steps.
A predictor with zero training variance cannot be standardized by the formula above because its standard deviation is zero. It contributes no variation in that training set, so it should not be divided by zero. A near-zero-variance predictor can also be problematic: dividing by a very small standard deviation may magnify minor fluctuations or measurement noise. The scaling decision should therefore be made alongside the earlier material on near-zero variance screening and subject-matter understanding.
Fit the transformation on training data
Preprocessing parameters are part of the fitted analysis. Calculate the means and standard deviations from the training portion only, then use those same values for validation, test, or future observations. Recalculating them separately in each partition changes the coordinate system: an observation’s transformed value would then depend on which other observations happened to be placed beside it. Using assessment-set values to determine preprocessing parameters also lets information from that data influence model development.
The same principle applies if the eventual procedure is unsupervised. A transformation can use predictor values without using an outcome, yet fitting it on held-out observations still allows information from those observations to affect the analysis. In implementation, retain the training means and standard deviations with the fitted transformation, apply them unchanged to every later data set, and verify that no predictor has a zero or unsuitable scale.
Worked Example: Applying Training Scaling to a New Observation
A hypothetical model uses a predictor recording an account’s annual number of service requests. The training values are 8, 10, and 12. A later assessment observation has value 100. Calculate its standardized value using the training data, then compare it with the incorrect result from recomputing the mean and standard deviation after pooling the assessment observation with the training values.
Method. Use the training mean and sample standard deviation for the valid transformation. For the comparison, calculate the pooled sample statistics, using denominator \(n-1\).
The training mean is \((8+10+12)/3=10\). The deviations are \(-2,0,2\), so the training sample variance is \((4+0+4)/(3-1)=4\) and its standard deviation is 2. The valid standardized assessment value is $$ z_{\mathrm{valid}}=\frac{100-10}{2}=45. $$ It is far from the training mean, which is important information rather than a reason to change the scale.
If the assessment value is incorrectly pooled in, the four-value mean is \((8+10+12+100)/4=32.5\). The deviations from 32.5 are \(-24.5,-22.5,-20.5,67.5\); their squared values sum to \(600.25+506.25+420.25+4556.25=6083\). The pooled sample variance is \(6083/3=2027.6667\), and the sample standard deviation is approximately 45.0296. The assessment value would then be standardized as $$ z_{\mathrm{pooled}}=\frac{100-32.5}{45.0296}\approx 1.4990. $$ The results are rounded to four decimal places where shown.
Check and trap. The assessment value is 45 training standard deviations above the training mean, but about 1.4990 standard deviations above the pooled mean. The latter calculation uses the assessment value to alter its own scale, so it is not the fitted training transformation. Do not recompute scaling parameters on validation or test data just because a new value appears extreme.
Choosing whether to scale
Scaling is most compelling when predictors are measured in different units or when a method’s result depends on distances or marginal variances. It is not automatically beneficial in every analysis. Equal standard deviations do not mean equal predictive value, reliability, or practical importance. Standardization can give a low-variance predictor the same unit variance as a stable predictor, amplifying noise or small measurement errors. Conversely, leaving predictors unscaled may allow arbitrary units to decide the result.
Before reducing dimensions, ask what the procedure measures and what equal influence should mean in context. For a distance-based calculation, determine whether a one-unit difference should mean the same thing across predictors; if not, consider scaling by a defensible training-set scale. For variance-based reduction, decide whether absolute variation in original units is meaningful or whether relative variation is the intended basis. Record the choice, compute parameters only on training data, and use the same transformation throughout assessment.
Check Your Understanding
Work each problem using the stated training statistics and distinguish numerical scale from substantive importance.
- Two observations differ by 4 units on predictor A and 30 units on predictor B. Their training standard deviations are 2 and 10, respectively. Find their raw Euclidean distance and standardized Euclidean distance. (Answer: Raw distance \(=\sqrt{916}\approx 30.265\); standardized distance \(=\sqrt{13}\approx 3.606\).)
- A predictor has training mean 50 and training sample standard deviation 5. What is the standardized value of a new observation equal to 65? (Answer: \(3\).)
- A centered two-predictor data set has covariance matrix with diagonal entries 25 and 1, and off-diagonal entries 0. What proportion of total variance does the first covariance-based component capture? (Answer: \(25/26\approx 0.9615\), or 96.15%.)
- Two observations have predictor values (4, 20) and (9, 50). If the same constants are subtracted from both observations on each predictor, does their Euclidean distance change? Explain briefly. (Answer: No; the differences remain 5 and 30, so the distance remains \(\sqrt{925}\approx 30.414\).)
- A predictor’s training values are 3, 3, and 3. Can its standardized values be calculated with the usual training sample standard deviation? What should be done before using it in a scaling-based reduction? (Answer: No; its standard deviation is zero, so division is undefined. Exclude or otherwise handle the constant predictor before standardizing.)