Simultaneous Pairwise Coordinate Comparisons

Why Compare Every Pair?

A mean vector can describe several outcomes, treatments, or groups on a common scale. Often the scientific question is not whether the vector differs from a particular value, but which coordinates differ from one another. For example, researchers may want to know which of several treatment means are distinguishable, while controlling the chance of making at least one false claim across the whole collection of comparisons.

The relevant parameters are pairwise differences. Because many comparisons are considered together, ordinary separate confidence intervals do not automatically provide simultaneous protection. We will construct intervals that cover all pairwise differences with probability at least \(1-\alpha\), under the stated sampling model.

Definition: Let \(\mu=(\mu_1,\ldots,\mu_p)'\) be a mean vector. For each pair \(i<j\), define the pairwise coordinate contrast \(\delta_{ij}=\mu_i-\mu_j\), with contrast vector \(c_{ij}=e_i-e_j\), where \(e_i\) is the \(i\)th coordinate vector. Thus \(\delta_{ij}=c_{ij}'\mu\). The complete family contains \(m=\binom{p}{2}=p(p-1)/2\) contrasts.

The orientation \(i<j\) fixes the sign: the estimated difference is the mean of coordinate \(i\) minus the mean of coordinate \(j\). Reversing the order changes the sign of the estimate and both interval endpoints, but not whether the interval contains zero.

Sampling Model and Simultaneous Intervals

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed \(p\)-dimensional observations from \(N_p(\mu,\Sigma)\), where \(\Sigma\) is positive definite and unknown. Write \(\bar X=n^{-1}\sum_{k=1}^n X_k\) for the sample mean vector and \(S=(n-1)^{-1}\sum_{k=1}^n(X_k-\bar X)(X_k-\bar X)'\) for the sample covariance matrix. For a contrast \(c_{ij}\), the estimate is \(c_{ij}'\bar X=\bar X_i-\bar X_j\), and its estimated standard error is \(\sqrt{c_{ij}'Sc_{ij}/n}\).

Let \(t_{\nu,q}\) denote the \(q\) quantile of the Student \(t\) distribution with \(\nu\) degrees of freedom. For familywise error level \(\alpha\), the two-sided Bonferroni interval for every pair is

(\bar X_i-\bar X_j)\ \pm\ t_{n-1,\,1-\alpha/(2m)} \sqrt{\frac{c_{ij}'Sc_{ij}}{n}},\qquad 1\leq i<j\leq p.

The factor \(2m\) appears because there are \(m\) intervals and each interval has two tails. These intervals are generally dependent; Bonferroni control does not require the comparisons to be independent.

Theorem: Under the independent multivariate normal sampling model above, the Bonferroni intervals have simultaneous coverage at least \(1-\alpha\): \(P\{\delta_{ij}\text{ lies in its interval for every }i<j\}\geq 1-\alpha\). This is a finite-sample guarantee for \(n>1\).

Here are the two steps behind the result. First, each individual contrast has an exact Student \(t\) pivot. To see this, fix any nonzero contrast \(c\) and set \(Y_k=c'X_k\). Then \(Y_1,\ldots,Y_n\) are independent \(N(c'\mu,c'\Sigma c)\) variables. An orthogonal transformation of the \(n\) observations, with first row proportional to \((1,\ldots,1)\), separates \(\sqrt n\,\bar Y\) from \(n-1\) residual coordinates. Normality makes those transformed coordinates independent. The standardized first coordinate is \(N(0,1)\), and the sum of squares of the other \(n-1\) standardized coordinates is \(\chi^2_{n-1}\), independent of the first. Since \(c'Sc\) is the residual sum of squares divided by \(n-1\), it follows that

\frac{c'\bar X-c'\mu}{\sqrt{c'Sc/n}}\sim t_{n-1}.

Consequently, the probability that a particular interval fails to cover its contrast is \(\alpha/m\): its two tails each have probability \(\alpha/(2m)\). Define \(A_{ij}\) as the event that interval \(ij\) misses \(\delta_{ij}\). The probability that at least one interval misses is bounded by the sum of the individual failure probabilities:

P\!\left(\bigcup_{i<j} A_{ij}\right) \leq \sum_{i<j}P(A_{ij}) =m\frac{\alpha}{m} =\alpha.

Taking the complement proves the claimed simultaneous coverage. No independence assumption between the \(A_{ij}\) events is used. \(\square\)

Calculating the Pairwise Standard Errors

The contrast form keeps the calculation systematic. If \(S_{ii}\) and \(S_{jj}\) are the sample variances and \(S_{ij}\) is the sample covariance of coordinates \(i\) and \(j\), then

c_{ij}'Sc_{ij}=S_{ii}+S_{jj}-2S_{ij}.

Thus the precision of a difference depends on the covariance as well as the two marginal variances. Positive covariance can reduce the variance of a difference; negative covariance can increase it. The number of intervals, and hence the Bonferroni critical value, depends on the number of coordinates \(p\), not on how many comparisons happen to look promising in the observed data.

Worked Example: Three Coordinates and Three Simultaneous Intervals

Suppose \(n=12\), \(\bar X=(10,14,17)'\), and \(S=\begin{pmatrix}4&1&0\\1&9&2\\0&2&16\end{pmatrix}\). Construct 95% Bonferroni intervals for all pairwise differences. There are \(m=3(2)/2=3\) comparisons, so the critical value is \(t_{11,\,1-0.05/6}=t_{11,\,0.991667}\approx2.820\).

For pair \(1,2\), the estimate is \(10-14=-4\). Its estimated contrast variance is \(4+9-2(1)=11\), so the standard error is \(\sqrt{11/12}\approx0.9574\). The margin is \(2.820(0.9574)\approx2.700\), giving the interval \((-6.700,-1.300)\).

For pair \(1,3\), the estimate is \(10-17=-7\). The contrast variance is \(4+16-2(0)=20\), and the standard error is \(\sqrt{20/12}\approx1.2910\). The margin is \(2.820(1.2910)\approx3.641\), giving \((-10.641,-3.359)\).

For pair \(2,3\), the estimate is \(14-17=-3\). The contrast variance is \(9+16-2(2)=21\), and the standard error is \(\sqrt{21/12}\approx1.3229\). The margin is \(2.820(1.3229)\approx3.731\), giving \((-6.731,0.731)\).

The first two intervals exclude zero, so those pairwise differences are distinguishable at the stated familywise level. The third interval includes zero, so this procedure does not establish a difference between coordinates 2 and 3. The conclusion is about the evidence under simultaneous control, not proof that the two population means are equal.

Reading the Intervals as a Family

An interval that lies entirely below zero supports \(\mu_i<\mu_j\); one entirely above zero supports \(\mu_i>\mu_j\). An interval containing zero does not declare the pair equal. It says that zero remains compatible with the data under this simultaneous procedure. The familywise guarantee concerns all \(m\) intervals at once: with probability at least \(1-\alpha\), every population difference is covered. It is not a guarantee that each individual interval has exactly \(1-\alpha\) coverage.

A useful consequence is that the conclusions should be read together. If several intervals exclude zero, they provide a set of pairwise findings whose overall false-exclusion probability is controlled. But the interval widths also matter. A statistically distinguishable pair may differ by an amount too small to matter scientifically, while an inconclusive interval may still rule out effects that would be practically important. Practical importance requires a context-specific threshold, not merely checking whether zero is included.

Worked Example: Six Comparisons Among Four Coordinates

Take \(n=20\), \(\bar X=(5,7,10,6)'\), and \(S=I_4\), the \(4\) by \(4\) identity matrix. For all six comparisons, \(c_{ij}'Sc_{ij}=2\), so every standard error is \(\sqrt{2/20}=\sqrt{0.1}\approx0.3162\). Here \(m=4(3)/2=6\), and the 95% Bonferroni critical value is \(t_{19,\,1-0.05/12}=t_{19,\,0.995833}\approx2.944\). Each margin is approximately \(2.944(0.3162)=0.931\).

The six oriented estimates and intervals are:

\(1-2:\ 5-7=-2\), giving \((-2.931,-1.069)\).
\(1-3:\ 5-10=-5\), giving \((-5.931,-4.069)\).
\(1-4:\ 5-6=-1\), giving \((-1.931,-0.069)\).
\(2-3:\ 7-10=-3\), giving \((-3.931,-2.069)\).
\(2-4:\ 7-6=1\), giving \((0.069,1.931)\).
\(3-4:\ 10-6=4\), giving \((3.069,4.931)\).

All six intervals exclude zero, so every pair is distinguishable by this familywise 95% procedure. Notice that the estimate of \(\mu_1-\mu_4\) is only \(-1\), but its interval still excludes zero: the conclusion follows from its estimate and uncertainty together, not from ranking the sample means alone.

Worked Example: Covariance Can Change a Pairwise Conclusion

Suppose \(n=16\), \(\bar X=(2,2.5,4)'\), and \(S=\begin{pmatrix}1&0.5&0.5\\0.5&1&0.5\\0.5&0.5&1\end{pmatrix}\). This covariance matrix is positive definite: its eigenvalues are \(2,0.5,0.5\). For each pair, the contrast variance is \(1+1-2(0.5)=1\), so the standard error is \(\sqrt{1/16}=0.25\). There are three intervals; the critical value \(t_{15,\,0.991667}\) is approximately \(2.694\), giving margin \(2.694(0.25)=0.674\).

For pair \(1,2\), the estimate is \(2-2.5=-0.5\), so the interval is approximately \((-1.174,0.174)\), which includes zero. For pair \(1,3\), the estimate is \(2-4=-2\), giving \((-2.674,-1.326)\). For pair \(2,3\), the estimate is \(2.5-4=-1.5\), giving \((-2.174,-0.826)\). The latter two intervals exclude zero, while the first does not.

If the covariance were incorrectly ignored, one might use a contrast variance of \(1+1=2\), inflating the standard error to \(\sqrt{2/16}\). The correct variance subtracts twice the covariance. This example shows why the full covariance information is needed even when the target is a simple difference of two coordinates.

Common Mistakes and Practical Interpretation

Using an unadjusted critical value for every pair. Separate 95% intervals do not in general give 95% coverage for the entire family. Use the tail probability \(\alpha/(2m)\) for each interval when applying this Bonferroni construction.
Counting ordered pairs twice. There are \(p(p-1)/2\) distinct unordered pairs, not \(p(p-1)\). The two orientations contain the same comparison with opposite signs.
Dropping the covariance term. The variance of a difference is \(S_{ii}+S_{jj}-2S_{ij}\), not simply \(S_{ii}+S_{jj}\).
Treating an interval containing zero as evidence of equality. It means the procedure has not distinguished the pair at this familywise level; it does not establish that the population difference is zero.
Interpreting a zero-excluding interval as a measure of importance. Statistical distinguishability and scientific importance are different questions.

Bonferroni intervals are straightforward and remain valid despite dependence among pairwise contrasts, but their critical value can be conservative, particularly when many comparisons are strongly related. Exact max-\(t\) methods use the joint distribution of the correlated contrasts to calibrate a shared maximum statistic and may be less conservative when that distribution is available. Whichever method is used, the comparison family should be specified before interpreting the results.

Key takeaway: Form the \(p(p-1)/2\) oriented differences, calculate each standard error using the covariance matrix, and adjust for the full family. A pair is distinguishable under the procedure when its simultaneous interval excludes zero; that conclusion is neither a claim of practical importance nor proof of inequality.

Check Your Understanding

Use the definitions and interval logic above to answer the following questions.

How many distinct pairwise comparisons are there among five coordinates, and what Bonferroni tail probability is used for each side of each interval at familywise level \(\alpha\)?
For \(p=3\), explain why the intervals for pairs \(1,2\), \(1,3\), and \(2,3\) are dependent, and why that dependence does not invalidate the Bonferroni coverage argument.
If \(S_{ii}=4\), \(S_{jj}=9\), and \(S_{ij}=2\), calculate \(c_{ij}'Sc_{ij}\). What would the contrast variance be if the covariance term were incorrectly omitted?
An interval for \(\mu_i-\mu_j\) is entirely negative. State the direction of the supported difference and explain what the familywise guarantee adds to that conclusion.
Why does an interval containing zero fail to establish equality of the two population means?