Independence of Group Means and Within-Group Residuals

Why Separate Group Means From Residuals?

In a one-way layout, each observation contributes to a group average and to a deviation from that average. These are two different aspects of the data: the average describes the group-level location, while the deviations describe variation within the group. For exact normal-theory arguments, it is useful to know more than the fact that their corresponding vectors are orthogonal: under suitable assumptions, the group means and all within-group residuals are independent random objects.

The key conditions are independent Gaussian observations with a common variance. The group means need not be equal. The result follows because averaging and residual formation are orthogonal projections, and linear transformations of a Gaussian vector are jointly Gaussian. Orthogonality then gives zero cross-covariance, which in the jointly Gaussian setting implies independence.

Projection Setup and Definitions

Suppose there are \(k\) nonempty groups, with \(n_i\) observations in group \(i\), and let \(N=\sum_{i=1}^k n_i\). Write the observation in group \(i\) as \(Y_{ij}\), where \(j=1,\ldots,n_i\). Assume

The mean \(\mu_i\) may vary across groups, but every observation has the same variance. Stack the observations in a vector \(\mathbf Y\in\mathbb R^N\), in group order. Let \(Z\) be the \(N\times k\) group-indicator matrix: the entry in row \(r\), column \(i\) is 1 when observation \(r\) belongs to group \(i\), and 0 otherwise. Since each group is nonempty, \(Z'Z=\operatorname{diag}(n_1,\ldots,n_k)\) is invertible.

The matrix \(P\) is symmetric and idempotent, so it is an orthogonal projection. Its complement \(Q=I-P\) is also symmetric and idempotent, and \(PQ=0\). Thus \(P\mathbf Y\) and \(Q\mathbf Y\) lie in orthogonal subspaces for every realized data vector. This geometric statement alone is not yet a statement of statistical independence.

The Independence Result

To prove the theorem, let \(\boldsymbol\mu\in\mathbb R^N\) be the vector whose entries in group \(i\) all equal \(\mu_i\). Independence and the common variance give \(\mathbf Y\sim N_N(\boldsymbol\mu,\sigma^2 I)\). Since \(P\) and \(Q\) are linear maps, the pair \((P\mathbf Y,Q\mathbf Y)\) is jointly Gaussian, possibly with a singular covariance matrix. Its cross-covariance matrix is

Here \(Q'=Q\), and \(PQ=P(I-P)=P-P^2=0\). For completeness, zero cross-covariance gives independence here because the pair is jointly Gaussian. Its joint characteristic function has exponent determined by its mean and covariance matrix. With zero cross-covariance, that covariance matrix is block diagonal, so the joint characteristic function factors into the product of the characteristic functions of \(P\mathbf Y\) and \(Q\mathbf Y\). Factorization of the joint characteristic function implies independence. This argument permits singular covariance matrices, as can occur for the residual vector. Therefore \(P\mathbf Y\) and \(Q\mathbf Y\) are independent. \(\square\)

The same result can be seen at the level of each group. The mean \(\bar Y_i\) is a linear function of the observations in group \(i\), and each residual \(R_{ij}\) is also a linear function of those observations. Their covariance is

For \(l\ne j\) in the same group, the residual covariance is \(\operatorname{Cov}(R_{ij},R_{il})=-\sigma^2/n_i\). Thus residuals within a group are generally dependent on one another, even though they are independent of the group mean. Across different groups, the observations are independent by assumption.

Distributions and What the Theorem Does Not Say

Averaging independent normal observations gives \(\bar Y_i\sim N(\mu_i,\sigma^2/n_i)\). Different group means are independent because they are functions of disjoint sets of independent observations. Within group \(i\), the residual vector has mean zero and covariance matrix \(\sigma^2(I_{n_i}-J_{n_i}/n_i)\), where \(J_{n_i}\) is the \(n_i\times n_i\) matrix of ones. The zero row sums of this matrix reflect the constraint \(\sum_j R_{ij}=0\).

The theorem concerns independence between the group-level fitted values and the within-group residuals. It does not claim that the residual coordinates within a group are mutually independent. For instance, when a group has two observations, its residuals are negatives of each other, so they cannot be independent unless they are degenerate.

Unequal Group Sizes and a Group-Mean Contrast

Equal group sizes are not required. For example, let group 1 have size 2 and distribution \(N(3,4)\), and group 2 have size 4 and distribution \(N(1,4)\), with all six observations independent. Then \(\bar Y_1\sim N(3,2)\) and \(\bar Y_2\sim N(1,1)\). The contrast \(D=\bar Y_1-\bar Y_2\) has mean \(3-1=2\) and variance \(2+1=3\), so \(D\sim N(2,3)\).

Since \(D\) is a function of the fitted-value vector, it is independent of the full within-group residual vector. The different group sizes change the variances of the means, but they do not affect the projection argument: group averaging and within-group centering remain orthogonal in the observation space.

Common Mistake and Practical Significance

A common mistake is to argue, “The fitted values and residuals are orthogonal, so they are independent.” Orthogonality is a geometric property and, in general, only yields zero covariance for suitable random vectors. Zero covariance does not by itself imply independence. The correction is to check the distributional condition: here both components are linear transformations of a jointly Gaussian vector, and jointly Gaussian components with zero cross-covariance are independent.

The assumptions matter. If observations have unequal variances or are correlated, their covariance matrix is not \(\sigma^2 I\), and the calculation \(P(\sigma^2 I)Q=0\) need not hold. Without Gaussianity, even zero cross-covariance does not generally establish independence. Thus the conclusion is exact for the stated normal model, not a consequence of the algebra of group averages alone.

This separation matters because it allows probability statements about group-level estimates to be considered independently of within-group noise under the model. It also clarifies which part of an argument is geometric and which part is probabilistic: the projections determine orthogonality, while Gaussianity turns that orthogonality into independence.