Centering the Data Matrix

Why Center a Data Matrix?

A data matrix records several measured variables for a collection of observations. In principal components analysis (PCA), the location of the observations relative to their average matters: centering expresses each observation as a displacement from that average. It changes the reference point, or origin, used to describe the data, but it does not change the separation between any two observations.

The convention is important: observations are rows, variables are columns, and each column is centered using that column’s own sample mean. Subtracting a row mean or a single overall mean does something different. The earlier material on what principal components optimize motivates examining variation around the data’s center; here the task is the precise calculation and its geometric effect.

Definition: Let \(\mathbf{X}\) be an \(n \times p\) data matrix, with \(n\) observations in rows and \(p\) variables in columns. For variable \(j\), its sample mean is \(\bar{x}_j=\frac{1}{n}\sum_{i=1}^{n}x_{ij}\). The centered entry is \(x_{ij}^{(c)}=x_{ij}-\bar{x}_j\). Thus, centering subtracts each column’s sample mean from every entry in that column.

Write the vector of column means as \(\bar{\mathbf{x}}=(\bar{x}_1,\ldots,\bar{x}_p)^T\), and let \(\mathbf{1}_n\) be the \(n\)-element column vector of ones. The centered matrix can then be written compactly as

\mathbf{X}_c=\mathbf{X}-\mathbf{1}_n\bar{\mathbf{x}}^T.

The matrix \(\mathbf{1}_n\bar{\mathbf{x}}^T\) repeats the vector of column means in every row. Consequently, the subtraction is column-specific even though the matrix expression applies all the subtractions at once. A centered entry may be positive, negative, or zero: it records how far that observation lies above or below its variable’s sample mean.

A Reliable Centering Calculation

For a small matrix, work one column at a time. First calculate each column’s mean using all \(n\) observations. Then subtract that mean from each entry in the same column, retaining the original row and column positions. Finally, check that the entries in every centered column sum to zero. A zero column sum is a direct check that the correct column mean was subtracted.

Find the column means.
For each variable, add its \(n\) observed values and divide by \(n\).

Subtract within columns.
Subtract a variable’s mean from every observation of that variable.

Check the result.
Add the centered entries in each column; each sum should be zero, apart from possible rounding in decimal calculations.

Worked Example: Centering Two Measurements

A hypothetical analyst records two measurements for three facilities. The first variable is a facility score, and the second is a response time in minutes. Center the data matrix, verify the column sums, and compare the first and third facilities before and after centering.

Method. Treat facilities as rows and measurements as columns, then subtract each column’s sample mean. This avoids the common mistake of subtracting a facility’s row mean from both of its measurements.

\mathbf{X}= \begin{pmatrix} 2&10\\ 4&14\\ 6&12 \end{pmatrix}

For the first variable, the mean is \((2+4+6)/3=12/3=4\). For the second variable, the mean is \((10+14+12)/3=36/3=12\). Thus \(\bar{\mathbf{x}}=(4,12)^T\).

Subtract 4 from each entry in the first column and 12 from each entry in the second:

\mathbf{X}_c= \begin{pmatrix} 2-4&10-12\\ 4-4&14-12\\ 6-4&12-12 \end{pmatrix} = \begin{pmatrix} -2&-2\\ 0&2\\ 2&0 \end{pmatrix}.

The first centered column sums to \(-2+0+2=0\); the second sums to \(-2+2+0=0\). The check passes. From the first facility to the third, the original difference is \((6-2,12-10)=(4,2)\). The centered difference is \((2-(-2),0-(-2))=(4,2)\), exactly the same.

Check. Both centered column means are zero, and the first-to-third difference is unchanged. The centering operation has shifted the coordinate origin, not altered the facilities’ relative positions.

Why Centering Preserves Pairwise Differences

The key property follows directly from the definition. For two observations \(i\) and \(k\) and any variable \(j\), subtract their centered entries:

x_{ij}^{(c)}-x_{kj}^{(c)} =(x_{ij}-\bar{x}_j)-(x_{kj}-\bar{x}_j) =x_{ij}-x_{kj}.

The same column mean appears in both terms and cancels. This holds separately for every variable, so the vector difference between two centered observations is also their original vector difference. In particular, their ordinary Euclidean distance is unchanged, because every coordinate difference is unchanged.

Centering also guarantees that each centered column has mean zero. For any variable \(j\), summing the centered entries gives

\sum_{i=1}^{n}x_{ij}^{(c)} =\sum_{i=1}^{n}(x_{ij}-\bar{x}_j) =\sum_{i=1}^{n}x_{ij}-n\bar{x}_j =\sum_{i=1}^{n}x_{ij}-n\left(\frac{1}{n}\sum_{i=1}^{n}x_{ij}\right) =0.

Dividing that sum by \(n\) shows that the sample mean of the centered column is zero. These two results explain the role of centering: it moves the data so that the column averages are at the origin, while retaining the differences between observations. They do not imply that the entries, row means, or all other summaries of the matrix are zero.

Worked Example: Centering Three Facility Indicators

In a hypothetical study, four facilities are described by a count of incidents per reporting period and a response measure in minutes. The recorded matrix is

\mathbf{X}= \begin{pmatrix} 2&8\\ 4&10\\ 6&14\\ 8&16 \end{pmatrix}.

Center both variables and verify the centered matrix. Then compare the difference between the fourth and first facilities before and after centering.

Method. Compute one mean for each column and subtract it down that column. The trap is to use the grand mean of all eight entries: that would apply the same shift to both variables even though their column means differ.

The first column mean is \((2+4+6+8)/4=20/4=5\). The second column mean is \((8+10+14+16)/4=48/4=12\). Therefore, \(\bar{\mathbf{x}}=(5,12)^T\), and

\mathbf{X}_c= \begin{pmatrix} 2-5&8-12\\ 4-5&10-12\\ 6-5&14-12\\ 8-5&16-12 \end{pmatrix} = \begin{pmatrix} -3&-4\\ -1&-2\\ 1&2\\ 3&4 \end{pmatrix}.

The first centered column sums to \(-3-1+1+3=0\). The second sums to \(-4-2+2+4=0\). The original fourth-minus-first difference is \((8-2,16-8)=(6,8)\). The centered difference is \((3-(-3),4-(-4))=(6,8)\), confirming that it is preserved.

Check. The column sums are zero and the pairwise difference agrees before and after centering. A single grand-mean subtraction would fail the zero-column-mean check here because the two column means are 5 and 12, not the same.

Centering New Observations

When observations are divided into training and later data, calculate the column means from the training data and retain them. A later observation is centered by subtracting those stored training means, variable by variable. Recomputing means from the later observation or from a combined data set changes the reference point and no longer applies the same centering transformation.

For example, if the training mean vector is \((3,26,4)^T\), a new observation \((4,28,5)\) is represented relative to that training center as \((4-3,28-26,5-4)=(1,2,1)\). This is not a new calculation of sample means; it is the new observation’s displacement from the already established training center.

Worked Example: Centering a Later Observation

A hypothetical training set records three measurements for three service units:

\mathbf{X}_{\mathrm{train}}= \begin{pmatrix} 1&22&2\\ 3&26&4\\ 5&30&6 \end{pmatrix}.

Calculate the training means and centered training matrix. Then center a later unit with measurements \((4,28,5)\) using the training means. State why recalculating a mean using the later unit would be a mistake.

Method. Find each mean using only the three training rows. The common trap is to let the later observation alter the training reference point.

The training means are \((1+3+5)/3=3\), \((22+26+30)/3=78/3=26\), and \((2+4+6)/3=4\). Thus \(\bar{\mathbf{x}}_{\mathrm{train}}=(3,26,4)^T\), and

\mathbf{X}_{\mathrm{train},c}= \begin{pmatrix} 1-3&22-26&2-4\\ 3-3&26-26&4-4\\ 5-3&30-26&6-4 \end{pmatrix} = \begin{pmatrix} -2&-4&-2\\ 0&0&0\\ 2&4&2 \end{pmatrix}.

The later unit’s centered values are \((4-3,28-26,5-4)=(1,2,1)\). The training means remain the reference because they were calculated from the training set. If the later unit were included when calculating new means, all training rows would receive a different transformation, and the result would no longer use the stated training center.

Check. Each training column sums to zero: \(-2+0+2=0\), \(-4+0+4=0\), and \(-2+0+2=0\). The later unit is correctly centered relative to those means, not treated as an additional training observation.

Common Mistakes and Their Corrections

Centering is a simple operation, but a small indexing error can change its meaning. Keep the matrix orientation explicit: with observations in rows and variables in columns, compute and subtract means down columns.

Subtracting row means. Row-centering makes each observation’s row sum zero; it does not generally make each variable’s column mean zero. Use column means for the stated data layout.
Subtracting one grand mean. The grand mean of every matrix entry is generally not the mean of each individual variable. Calculate each column mean separately.
Changing the number of observations in the denominator. A sample column mean uses all \(n\) rows, so divide the column total by \(n\), not by the number of variables \(p\).
Assuming centering changes no values. It changes the entries and the origin. What it preserves is the difference between any two observations in each variable, not each observation’s coordinates.
Re-centering later data independently. For a later observation being compared with a training set, use the stored training means rather than means recalculated from the later data.

Worked Example: Spotting an Incorrect Centering

A two-observation, two-variable data matrix is

\mathbf{X}= \begin{pmatrix} 2&10\\ 6&14 \end{pmatrix}.

An analyst subtracts the first row mean, 6, from both entries in the first row, and the second row mean, 10, from both entries in the second row. The resulting matrix is \(\begin{pmatrix}-4&4\\-4&4\end{pmatrix}\). Explain why this is not column-centering and find the correctly centered matrix.

Method. Compare the proposed operation with the definition: each column must have its own mean subtracted from all rows. The analyst instead subtracts a separate mean within each row.

The first column mean is \((2+6)/2=4\); the second is \((10+14)/2=12\). Correct column-centering gives

\mathbf{X}_c= \begin{pmatrix} 2-4&10-12\\ 6-4&14-12 \end{pmatrix} = \begin{pmatrix} -2&-2\\ 2&2 \end{pmatrix}.

Each correct centered column sums to zero. In the analyst’s matrix, the first column sums to \(-8\), and the second sums to \(8\), so its column means are not zero. The row-centering has also changed the difference between the two observations from \((6-2,14-10)=(4,4)\) to \((0,0)\). Correct column-centering preserves \((4,4)\), as \((2-(-2),2-(-2))=(4,4)\).

Check. The correct result has zero column sums and retains the original pairwise difference. This diagnoses both errors in the proposed calculation: it used row means and did not preserve the separation between observations.

Key takeaway: Centering an \(n \times p\) data matrix means subtracting each variable’s sample mean from every observation in that column. The centered columns have mean zero, while pairwise differences between observations remain unchanged.

Check Your Understanding

For each problem, treat observations as rows and variables as columns unless stated otherwise.

A one-variable data set has observations \(3,7,8\). Find the sample mean and the centered values. (Answer: mean \(6\); centered values \(-3,1,2\).)
Center the two-variable matrix with rows \((10,2)\), \((14,6)\), and \((18,4)\). (Answer: column means \((14,4)\); centered rows \((-4,-2)\), \((0,2)\), and \((4,0)\).)
A single-variable data set has values \(4,8,10,14\). After centering, find the difference between the fourth and first observations. (Answer: mean \(9\); centered values \(-5,-1,1,5\); difference \(10\), unchanged from the original data.)
Center the matrix with rows \((2,5,9)\), \((4,7,11)\), and \((6,12,13)\). Verify the column sums. (Answer: means \((4,8,11)\); centered rows \((-2,-3,-2)\), \((0,-1,0)\), and \((2,4,2)\); all three column sums are zero.)
A training set has one-variable values \(2,6,10\), and a later observation has value \(8\). Find the training mean, the centered training values, and the later observation’s centered value using that training mean. What value would result if the mean were incorrectly recalculated using all four observations? (Answer: training mean \(6\); centered training values \(-4,0,4\); later centered value \(2\); mean using all four values \(6.5\), which would incorrectly give the later observation centered value \(1.5\).)

What You'll Learn