Decomposing Between-Group and Within-Group Scatter

Why Decompose Multivariate Scatter?

In a one-way multivariate design, each observational unit has several measured responses and belongs to one of several groups. The total variation in the response vectors has two sources: observations vary around their own group means, and the group means vary around the overall mean. Separating these sources gives a matrix-valued accounting identity, preserving variation in every response and the cross-products between responses.

This decomposition is descriptive and algebraic: it does not by itself test whether group means differ, nor does it require a distributional model. Its interpretation depends on the design, however. The groups must partition the observations, and the overall mean must weight each observation equally.

Definitions for a One-Way Multivariate Design

Suppose there are $k$ nonempty groups and $p$ responses. Group $g$ contains $n_g$ observed response vectors $x_{gi}\in\mathbb{R}^p$, for $i=1,\ldots,n_g$. Write $N=\sum_{g=1}^k n_g$, and treat each response vector as a column vector. The sample mean in group $g$ and the overall sample mean are

\bar{x}_g=\frac{1}{n_g}\sum_{i=1}^{n_g}x_{gi}, \qquad \bar{x}=\frac{1}{N}\sum_{g=1}^k\sum_{i=1}^{n_g}x_{gi} =\frac{1}{N}\sum_{g=1}^k n_g\bar{x}_g.

For a vector $u$, the outer product $uu'$ is a $p\times p$ matrix. It records squared deviations on the diagonal and cross-products off the diagonal.

Definition: The total scatter matrix, within-group scatter matrix, and between-group scatter matrix are, respectively, $$ T=\sum_{g=1}^k\sum_{i=1}^{n_g}(x_{gi}-\bar{x})(x_{gi}-\bar{x})', \quad W=\sum_{g=1}^k\sum_{i=1}^{n_g}(x_{gi}-\bar{x}_g)(x_{gi}-\bar{x}_g)', \quad B=\sum_{g=1}^k n_g(\bar{x}_g-\bar{x})(\bar{x}_g-\bar{x})'. $$

The factor $n_g$ in $B$ matters: group $g$'s mean represents $n_g$ observations. Thus a group with more observations contributes more to the between-group scatter when its mean is a given distance from the grand mean.

The Total Scatter Decomposition

Theorem: For any partition of $N$ observed vectors into $k$ nonempty groups, the scatter matrices above satisfy $$ T=W+B. $$ No distributional assumption is needed for this identity.

Proof. For each observation, add and subtract its group mean:

x_{gi}-\bar{x}=(x_{gi}-\bar{x}_g)+(\bar{x}_g-\bar{x}).

Set $a_{gi}=x_{gi}-\bar{x}_g$ and $d_g=\bar{x}_g-\bar{x}$. Expanding the outer product gives

(x_{gi}-\bar{x})(x_{gi}-\bar{x})' =a_{gi}a_{gi}'+a_{gi}d_g'+d_ga_{gi}'+d_gd_g'.

Sum over observations and groups. The first terms sum to $W$. For the cross-products, the within-group deviations sum to zero in every group:

\sum_{i=1}^{n_g}a_{gi} =\sum_{i=1}^{n_g}(x_{gi}-\bar{x}_g) =\sum_{i=1}^{n_g}x_{gi}-n_g\bar{x}_g =0.

Consequently, $\sum_g\sum_i a_{gi}d_g'=0$, and its transpose $\sum_g\sum_i d_ga_{gi}'$ is also zero. The remaining terms sum to

\sum_{g=1}^k\sum_{i=1}^{n_g}d_gd_g' =\sum_{g=1}^k n_g(\bar{x}_g-\bar{x})(\bar{x}_g-\bar{x})' =B.

Therefore $T=W+B$. $\square$

What the Matrices Measure

Each diagonal element of $T$ is the total sum of squared deviations for one response; the corresponding diagonal elements of $W$ and $B$ split that total into within-group and between-group sums of squares. For two distinct responses $r$ and $s$, the $(r,s)$ entry is a sum of cross-products of their deviations. It can be positive, negative, or zero. The same decomposition holds for these cross-products, not just for the diagonal sums of squares.

All three matrices are positive semidefinite: for any vector $c\in\mathbb{R}^p$, for example,

c'Tc=\sum_{g=1}^k\sum_{i=1}^{n_g}\{c'(x_{gi}-\bar{x})\}^2\geq 0.

The same argument applies to $W$ and $B$. This also shows that each matrix describes sums of squared deviations after projecting the response vectors onto any chosen direction $c$. A negative off-diagonal entry does not contradict positive semidefiniteness; that condition applies to the whole quadratic form, not to every individual entry.

Worked Example: Two Groups With Two Responses

Group 1 contains $(1,2)'$ and $(3,2)'$; group 2 contains $(4,4)'$ and $(4,6)'$. Calculate all three scatter matrices.

The group means are $\bar{x}_1=(2,2)'$ and $\bar{x}_2=(4,5)'$, so the grand mean is $\bar{x}=(3,3.5)'$. In group 1, the deviations from its mean are $(-1,0)'$ and $(1,0)'$. Their outer products sum to

W_1= \begin{pmatrix}1&0\\0&0\end{pmatrix} + \begin{pmatrix}1&0\\0&0\end{pmatrix} = \begin{pmatrix}2&0\\0&0\end{pmatrix}.

In group 2, the deviations are $(0,-1)'$ and $(0,1)'$, giving $W_2=\begin{pmatrix}0&0\\0&2\end{pmatrix}$. Hence $W=W_1+W_2=\begin{pmatrix}2&0\\0&2\end{pmatrix}$.

The group-mean deviations from the grand mean are $(-1,-1.5)'$ and $(1,1.5)'$. Each group has size two, so

B=2 \begin{pmatrix}1&1.5\\1.5&2.25\end{pmatrix} +2 \begin{pmatrix}1&1.5\\1.5&2.25\end{pmatrix} = \begin{pmatrix}4&6\\6&9\end{pmatrix}.

Thus $T=W+B=\begin{pmatrix}6&6\\6&11\end{pmatrix}$. The first response's total sum of squares is $6=2+4$; the second's is $11=2+9$. The total cross-product is $6$, entirely due here to the separation between group means.

Unequal Group Sizes and a Pairwise Expression

The grand mean is the observation-weighted mean of the group means. An unweighted average of group means is generally wrong when group sizes differ. The following equivalent expression for $B$ makes the weighting and the role of group separation explicit.

Result: The between-group scatter can also be written as $$ B=\frac{1}{N}\sum_{g<h}n_gn_h(\bar{x}_g-\bar{x}_h)(\bar{x}_g-\bar{x}_h)'. $$

Derivation. Using $\sum_g n_g\bar{x}_g=N\bar{x}$, expand the definition of $B$:

B=\sum_g n_g\bar{x}_g\bar{x}_g'-N\bar{x}\bar{x}' =\sum_g n_g\bar{x}_g\bar{x}_g' -\frac{1}{N}\sum_g\sum_h n_gn_h\bar{x}_g\bar{x}_h'.

Now expand the numerator of the proposed pairwise expression. For each unordered pair $g<h$, its outer product is

(\bar{x}_g-\bar{x}_h)(\bar{x}_g-\bar{x}_h)' =\bar{x}_g\bar{x}_g'+\bar{x}_h\bar{x}_h' -\bar{x}_g\bar{x}_h'-\bar{x}_h\bar{x}_g'.

In the sum over all pairs, the coefficient of $\bar{x}_g\bar{x}_g'$ is $n_g\sum_{h\ne g}n_h=n_g(N-n_g)$, while the cross terms include each ordered pair $g\ne h$ once. Equivalently, the entire pairwise sum is

\sum_{g<h}n_gn_h(\bar{x}_g-\bar{x}_h)(\bar{x}_g-\bar{x}_h)' =N\sum_g n_g\bar{x}_g\bar{x}_g' -\sum_g\sum_h n_gn_h\bar{x}_g\bar{x}_h'.

Dividing by $N$ gives the expanded expression for $B$, proving the identity. $\square$

Worked Example: Unequal Group Sizes

Group 1 has observations $(0,0)'$ and $(2,0)'$. Group 2 has $(3,1)'$, $(3,3)'$, and $(3,5)'$. The group sizes are two and three, and their means are $\bar{x}_1=(1,0)'$ and $\bar{x}_2=(3,3)'$. Therefore the grand mean is

\bar{x}=\frac{2(1,0)'+3(3,3)'}{5}=(2.2,1.8)'.

Within group 1, the deviations $(-1,0)'$ and $(1,0)'$ give $W_1=\begin{pmatrix}2&0\\0&0\end{pmatrix}$. Within group 2, the deviations $(0,-2)'$, $(0,0)'$, and $(0,2)'$ give $W_2=\begin{pmatrix}0&0\\0&8\end{pmatrix}$. Thus $W=\begin{pmatrix}2&0\\0&8\end{pmatrix}$.

The mean deviations from the grand mean are $(-1.2,-1.8)'$ and $(0.8,1.2)'$. Weighting their outer products by the group sizes yields

\begin{aligned} B &=2\begin{pmatrix}1.44&2.16\\2.16&3.24\end{pmatrix} +3\begin{pmatrix}0.64&0.96\\0.96&1.44\end{pmatrix}\\ &=\begin{pmatrix}4.8&7.2\\7.2&10.8\end{pmatrix}. \end{aligned}

Consequently $T=\begin{pmatrix}6.8&7.2\\7.2&18.8\end{pmatrix}$. The trace of $W$ is $10$, the trace of $B$ is $15.6$, and the trace of $T$ is $25.6=10+15.6$. As a check using the pairwise expression, there is just one pair of groups: $\bar{x}_1-\bar{x}_2=(-2,-3)'$, so

B=\frac{2\cdot3}{5} \begin{pmatrix}4&6\\6&9\end{pmatrix} =\begin{pmatrix}4.8&7.2\\7.2&10.8\end{pmatrix}.

Scatter Is Not Covariance

A scatter matrix is an unnormalized sum of outer products. It is not itself a covariance matrix. For example, the overall sample covariance matrix is $T/(N-1)$. Within each group, the usual sample covariance is $W_g/(n_g-1)$ when $n_g>1$, where $W_g$ is that group's contribution to $W$. Dividing by a denominator changes the scale and the target; the identity $T=W+B$ is an identity between scatter matrices, not a claim that covariance matrices add with the same denominators.

Worked Example: Within-Group Cross-Products and Between-Group Separation

Group 1 contains $(0,0)'$ and $(2,2)'$, while group 2 contains $(4,0)'$ and $(6,2)'$. The group means are $(1,1)'$ and $(5,1)'$, and the grand mean is $(3,1)'$.

In each group, the two deviations from its mean are $(-1,-1)'$ and $(1,1)'$. Their outer products sum to $\begin{pmatrix}2&2\\2&2\end{pmatrix}$, so the two groups together give $W=\begin{pmatrix}4&4\\4&4\end{pmatrix}$.

The group-mean deviations from the grand mean are $(-2,0)'$ and $(2,0)'$. With two observations per group,

B=2\begin{pmatrix}4&0\\0&0\end{pmatrix} +2\begin{pmatrix}4&0\\0&0\end{pmatrix} =\begin{pmatrix}16&0\\0&0\end{pmatrix}.

Therefore $T=\begin{pmatrix}20&4\\4&4\end{pmatrix}$. The positive off-diagonal entries of $W$ and $T$ arise from the within-group tendency for the two responses to deviate in the same direction; the between-group separation here is only in the first response. The trace check is $24=8+16$.

Common Mistakes

Giving every group mean equal weight. The unweighted average of group means equals the grand mean only when the group sizes are equal. Use $\bar{x}=\sum_g n_g\bar{x}_g/N$.
Leaving out the group-size factor in $B$. Each group's mean represents $n_g$ observations, so its outer product receives weight $n_g$.
Assuming the cross-products must be positive. Off-diagonal entries are signed sums; their signs describe the direction of paired deviations. Positive semidefiniteness does not require every entry to be nonnegative.
Calling $T$, $W$, or $B$ a covariance matrix. These are sums, not averages. State the divisor when a covariance estimate is intended.
Treating the identity as a statistical test. The decomposition holds for the observed vectors regardless of whether the group means are equal. Inference requires additional modeling and a separate inferential procedure.

Key takeaway: In a one-way multivariate design, each observation's deviation from the grand mean splits into a deviation from its group mean and a deviation of that group mean from the grand mean. Summing the resulting outer products gives $T=W+B$, with group-size weighting built into the between-group component.

Check Your Understanding

Use the definitions and decomposition to answer the following.

In the one-way multivariate design, why does the cross-product term vanish when the total scatter is expanded?
What does the $(r,s)$ entry of a within-group scatter matrix measure when $r\ne s$?
When group sizes differ, why is the grand mean not generally the unweighted average of the group means?
For the unequal-size example, verify that the pairwise expression gives the same between-group scatter matrix.
Explain why a negative off-diagonal entry in a scatter matrix does not imply that the matrix is not positive semidefinite.