Common Errors in Two-Sample and Multiple-Mean Inference

Why Small Inferential Errors Matter

In two-sample and multiple-mean analyses, a calculation can look familiar and still answer the wrong question. Pooling covariances when groups differ, using the wrong residual degrees of freedom, or treating a data-selected comparison as if it had been planned can all make uncertainty look smaller than it is. The resulting test statistic may be computed correctly from the wrong ingredients.

Use the following notation. Group $i$ has $n_i$ independent $p$-dimensional observations, sample mean vector $\bar X_i$, and unbiased sample covariance matrix $S_i$. Write $N=\sum_{i=1}^g n_i$. A planned contrast has fixed scalar weights $c_1,\ldots,c_g$ and estimates the vector $\sum_{i=1}^g c_i\bar X_i$. For a contrast among group means, the weights sum to zero.

Definition: Under independent multivariate normal sampling, group $i$ has mean vector $\mu_i$ and covariance matrix $\Sigma_i$. The common-covariance model assumes $\Sigma_i=\Sigma$ for every group. The heterogeneous-covariance model allows the $\Sigma_i$ to differ. Covariance pooling is justified by the former assumption, not simply by having several groups.

Error 1: Pooling Covariances Without a Common-Covariance Assumption

When all groups share a covariance matrix, the pooled within-group covariance estimate is

S_p=\frac{\sum_{i=1}^g(n_i-1)S_i}{N-g}.

The weights $n_i-1$ reflect the degrees of freedom contributed by each group. If the covariances differ, this is still a computable weighted average, but it is not a common covariance estimate in the model-based sense. More importantly, it generally does not give the correct covariance for a mean contrast. With independent groups and possibly unequal covariances, that covariance is estimated by $\sum_i c_i^2 S_i/n_i$. A James-type method or another procedure designed for unequal covariances is needed for inference; the pooled-covariance exact test is not made valid just by substituting the pooled matrix.

Theorem: If the groups are independent, each group has finite covariance $\Sigma_i$, and $S_i$ is the unbiased sample covariance for group $i$, then $$ \operatorname{Cov}\!\left(\sum_{i=1}^g c_i\bar X_i\right) =\sum_{i=1}^g\frac{c_i^2}{n_i}\Sigma_i. $$ Thus an unbiased estimate of this covariance is $\sum_i c_i^2 S_i/n_i$.

Proof. The covariance of the sample mean in group $i$ is $\Sigma_i/n_i$. Independence across groups makes every cross-covariance $\operatorname{Cov}(\bar X_i,\bar X_j)$, $i\ne j$, equal to zero. Expanding the covariance of the weighted sum therefore gives

\operatorname{Cov}\!\left(\sum_i c_i\bar X_i\right) =\sum_i c_i^2\operatorname{Cov}(\bar X_i) +\sum_{i\ne j}c_ic_j\operatorname{Cov}(\bar X_i,\bar X_j) =\sum_i\frac{c_i^2}{n_i}\Sigma_i.

Since $E[S_i]=\Sigma_i$, taking expectations of $\sum_i c_i^2S_i/n_i$ gives the same matrix. $\square$

Worked Example: A Pooled Estimate Can Misstate Contrast Uncertainty

Suppose two independent groups have $n_1=10$, $n_2=20$, and observed covariance matrices

S_1=\begin{pmatrix}4&1\\1&9\end{pmatrix}, \qquad S_2=\begin{pmatrix}16&4\\4&25\end{pmatrix}.

For the difference $\bar X_1-\bar X_2$, the unequal-covariance estimate is

\frac{S_1}{10}+\frac{S_2}{20} = \begin{pmatrix}0.4&0.1\\0.1&0.9\end{pmatrix} + \begin{pmatrix}0.8&0.2\\0.2&1.25\end{pmatrix} = \begin{pmatrix}1.2&0.3\\0.3&2.15\end{pmatrix}.

If one instead imposes a common covariance, the pooled estimate is

S_p=\frac{9S_1+19S_2}{28} =\begin{pmatrix}85/7&85/28\\85/28&139/7\end{pmatrix}.

Under that common-covariance model, the estimated covariance of the difference is $(1/10+1/20)S_p=0.15S_p$, or approximately

\begin{pmatrix}1.821&0.455\\0.455&2.979\end{pmatrix}.

This differs materially from the unequal-covariance estimate: its first diagonal entry is larger, while its second is smaller. There is no universal direction in which inappropriate pooling changes uncertainty. The problem is the unsupported assumption, not a guaranteed increase or decrease.

Error 2: Losing or Borrowing Degrees of Freedom

Under a common covariance matrix, group $i$'s centered observations contribute $n_i-1$ within-group degrees of freedom. Adding these contributions gives

Theorem: If the groups are independent multivariate normal samples with the same covariance $\Sigma$, then $$ W=\sum_{i=1}^g(n_i-1)S_i $$ has a Wishart distribution with scale $\Sigma$ and degrees of freedom $\nu=N-g$. Consequently $S_p=W/\nu$. For a fixed contrast with $q=\sum_i c_i^2/n_i$, the exact one-sample Hotelling transformation is $$ \frac{\nu-p+1}{\nu p}\, \frac{\left(\sum_i c_i\bar X_i\right)^\mathsf{T} (qS_p)^{-1} \left(\sum_i c_i\bar X_i\right)}{1} \sim F_{p,\nu-p+1} $$ under the null contrast, provided $\nu\ge p$.

Proof of the degrees-of-freedom statement. Within group $i$, normal sampling gives $(n_i-1)S_i$ a Wishart distribution with scale $\Sigma$ and $n_i-1$ degrees of freedom. The groups are independent, so these Wishart matrices are independent. Independent Wishart matrices with the same scale add to a Wishart matrix whose degrees of freedom are the sum of the component degrees of freedom. Hence $W$ has $ \sum_i(n_i-1)=N-g$ degrees of freedom, and division by $N-g$ gives the pooled covariance estimate. $\square$

The stated $F$ transformation additionally requires the common-covariance model, a fixed contrast, and the null $\sum_i c_i\mu_i=0$. It does not license using $N-g$ degrees of freedom with heterogeneous covariance estimates. For two groups, $N-g=N-2$; for a one-way analysis with $g$ groups, it is $N-g$, not $N-1$. Nor should the number of coordinates $p$ be mistaken for the number of groups $g$: they enter different parts of the reference distribution.

Worked Example: Counting Degrees of Freedom for a Planned Contrast

Consider $g=3$ independent groups with sizes $8,10,12$, and a two-dimensional response ($p=2$). Then $N=30$, so the pooled within-group degrees of freedom are

\nu=(8-1)+(10-1)+(12-1)=7+9+11=27,

not $N-1=29$. For the planned weights $c=(1,-1/2,-1/2)$, the contrast covariance multiplier under a common covariance is

q=\frac{1^2}{8}+\frac{(-1/2)^2}{10}+\frac{(-1/2)^2}{12} =\frac18+\frac1{40}+\frac1{48} =\frac{41}{240}.

If the observed squared Hotelling distance for the contrast is $T^2=6$, its exact $F$ statistic is

\frac{27-2+1}{27\cdot2}(6) =\frac{26}{54}(6) =\frac{26}{9} \approx 2.889,

with $2$ and $26$ degrees of freedom. Using $29$ in place of $27$ would change both the scale factor and the denominator degrees of freedom, so it would not be the stated exact test.

Error 3: Calling the Wrong Weights a Contrast

A contrast of group means is a weighted sum whose coefficients satisfy $\sum_i c_i=0$. This condition makes the estimate insensitive to adding the same vector to every group mean. If the weights do not sum to zero, the weighted sum is not a comparison that vanishes when all group means are equal.

Multiplying all contrast weights by the same nonzero constant changes the contrast's units, but not its Hotelling statistic when its covariance is scaled consistently. The numerator vector is multiplied by that constant and its covariance by the square of the constant, so the quadratic form is unchanged. A common error is to rescale the contrast estimate but retain the old covariance multiplier.

Worked Example: Checking and Rescaling Contrast Weights

Suppose three groups have sample means $\bar X_1=(4,2)^\mathsf{T}$, $\bar X_2=(6,3)^\mathsf{T}$, and $\bar X_3=(5,1)^\mathsf{T}$. The weights $c=(1/2,1/2,-1)$ sum to zero and compare the average of the first two group means with the third:

\frac12\bar X_1+\frac12\bar X_2-\bar X_3 =\begin{pmatrix}2\\1\end{pmatrix} +\begin{pmatrix}3\\1.5\end{pmatrix} -\begin{pmatrix}5\\1\end{pmatrix} =\begin{pmatrix}0\\1.5\end{pmatrix}.

With sample sizes $8,12,10$, the common-covariance multiplier is

q=\frac{(1/2)^2}{8}+\frac{(1/2)^2}{12}+\frac{(-1)^2}{10} =\frac1{32}+\frac1{48}+\frac1{10} =\frac{73}{480}.

If the weights are doubled to $(1,1,-2)$, the estimate becomes $(0,3)^\mathsf{T}$, and the multiplier becomes $4q=73/120$. Both the estimate and its covariance have been rescaled consistently, so the Hotelling quadratic form is unchanged. By contrast, weights $(1,1,1)$ sum to $3$, not zero; their weighted sum is not a contrast and does not test equality of the three means.

Error 4: Treating Many Tests as One Test

A significance level applies to a test or to a specified family of tests, depending on the procedure. Testing $m$ pairwise comparisons each at level $\alpha$ does not in general control the probability of at least one false rejection at $\alpha$. Under independence, that probability when all null hypotheses are true is $1-(1-\alpha)^m$. Pairwise multivariate tests are often dependent, so this expression is not generally their exact familywise error rate; the key point is that separate level-$\alpha$ tests do not guarantee familywise control.

For example, Bonferroni testing uses level $\alpha/m$ for each of $m$ comparisons. By the union bound, the probability of one or more false rejections is then at most $m(\alpha/m)=\alpha$, without requiring independence. Bonferroni regions, Dunnett-type procedures for comparisons with a common control, and Scheffé inference address different comparison families; choose the procedure to match the claims being made. An omnibus test of equality of means does not, by itself, provide simultaneous error control for every follow-up comparison.

Worked Example: Six Pairwise Tests

Suppose four groups are compared pairwise, giving $m=4(3)/2=6$ tests. If each is tested at $\alpha=0.05$, the independence benchmark for at least one false rejection when all six nulls are true is

1-(1-0.05)^6=1-0.95^6\approx 0.265.

This is not an exact calculation for dependent Hotelling tests, but it shows why six separate 5% tests should not be described as a 5% family. A Bonferroni allocation instead tests each comparison at $0.05/6\approx0.00833$. The union bound guarantees familywise error no greater than $6(0.05/6)=0.05$, regardless of dependence.

Error 5: Selecting a “Planned” Contrast After Seeing the Data

The exact calibration of a planned contrast assumes its weights were fixed independently of the data used for testing. If the analyst examines sample means, chooses the largest-looking difference, and then reports the ordinary planned-contrast $p$-value, the selection step has been ignored. The reported test is no longer the single fixed test whose null distribution was used to calculate that $p$-value. The same problem arises when only favorable comparisons are reported from a larger set.

The correction is to define the comparison family and selection rule in advance, or to use an inference procedure that accounts for the full family of comparisons. Scheffé inference is designed for inference over all linear contrasts under its assumptions; Bonferroni or a suitable simultaneous procedure can address a defined finite family. A genuinely independent validation sample can also be used for a comparison chosen using separate data, provided the validation analysis is specified appropriately.

A Practical Diagnostic Sequence

Check the covariance model.
Ask whether a common covariance is defensible. If not, do not treat pooled covariance formulas or their exact reference distributions as automatically valid.

Check the contrast.
Verify that the weights sum to zero for a mean comparison, and calculate its covariance using the same weights and sample sizes.

Check degrees of freedom.
For pooled within-group covariance, count $\sum_i(n_i-1)=N-g$. Use the degrees of freedom for the specified exact or approximate procedure.

Check the testing family and timing.
State which comparisons are included and whether they were chosen before seeing the data. Match error control to that family and selection process.

Key takeaway: Valid multivariate mean inference depends on matching the covariance estimator, contrast, degrees of freedom, and multiplicity adjustment to the model and the comparisons actually made. A familiar formula cannot repair a mismatch among those choices.

Check Your Understanding

For each question, identify the assumption or inferential step that needs checking.

Three groups have unequal sample covariance matrices. What assumption is required to justify using a pooled within-group covariance as a common covariance estimate?
For $g$ groups with total sample size $N$, why does the pooled within-group covariance use $N-g$, rather than $N-1$, degrees of freedom?
Do weights $(2,-1,-1)$ define a contrast? What quantity do weights $(1,1,1)$ define instead?
Six pairwise tests are each performed at level $0.05$. What does Bonferroni change, and what guarantee does the union bound provide?
An analyst chooses the contrast with the largest observed difference and reports its ordinary planned-test $p$-value. What step has that calculation failed to account for?