Confidence Ellipsoids for a Difference of Mean Vectors

From a Mean-Difference Estimate to a Confidence Region

When each observation is a vector, uncertainty about a difference in population means is multivariate too. A confidence region should account not only for the variability of each component, but also for their covariance. The pooled Hotelling procedure does this by using the common covariance estimate to shape an ellipsoid around the observed difference of sample means.

Let $X_{11},\ldots,X_{1n_1}$ and $X_{21},\ldots,X_{2n_2}$ be two independent samples of $p$-dimensional random vectors. Assume that all observations in group $i$ are independent and identically distributed as $N_p(\mu_i,\Sigma)$, where $\Sigma$ is positive definite and common to both groups. Write $N=n_1+n_2$, $\nu=N-2$, and define the estimand as $\delta=\mu_1-\mu_2$. The order matters: reversing the groups reverses the sign of the difference.

Definition: For group $i$, let $\bar X_i$ be its sample mean vector and let $S_i$ be its sample covariance matrix, calculated with divisor $n_i-1$. The pooled covariance estimate and estimated mean difference are $$ S_p=\frac{(n_1-1)S_1+(n_2-1)S_2}{N-2}, \qquad \hat\delta=\bar X_1-\bar X_2. $$ The pooled Hotelling confidence ellipsoid for $\delta$ uses the squared distance $$ T^2(\delta)=\frac{n_1n_2}{N}(\hat\delta-\delta)'S_p^{-1}(\hat\delta-\delta). $$

The procedure requires $n_1,n_2\geq 2$ and $N\geq p+2$. Under the stated Gaussian model, these conditions ensure that $S_p$ is nonsingular with probability one and that the denominator degrees of freedom below are positive. The common-covariance and Gaussian assumptions are substantive: the exact calibration is not asserted for unequal covariance matrices or arbitrary nonnormal populations.

The Exact Confidence Ellipsoid

Let $F_{p,\nu-p+1;1-\alpha}$ denote the $1-\alpha$ quantile of an $F$ distribution with numerator degrees of freedom $p$ and denominator degrees of freedom $\nu-p+1=N-p-1$. Define

q_\alpha=\frac{p\nu}{\nu-p+1}F_{p,\nu-p+1;1-\alpha}.

Theorem: Under the independent two-sample Gaussian model with common positive-definite covariance $\Sigma$, the region $$ \mathcal C_{1-\alpha} = \left\{\delta\in\mathbb R^p: \frac{n_1n_2}{N}(\hat\delta-\delta)'S_p^{-1}(\hat\delta-\delta) \leq q_\alpha \right\} $$ has exact finite-sample coverage $P(\delta\in\mathcal C_{1-\alpha})=1-\alpha$.

The exact pooled Hotelling calibration gives $$ \frac{\nu-p+1}{p\nu}T^2(\delta)\sim F_{p,\nu-p+1}. $$ To see why it applies, $\hat\delta$ has distribution $N_p(\delta,(1/n_1+1/n_2)\Sigma)$, and it is independent of $S_p$. Also, $\nu S_p$ has a Wishart distribution with scale matrix $\Sigma$ and $\nu$ degrees of freedom. These Gaussian sampling facts are the basis of the exact $F$ calibration.

Proof of coverage. The event that the true difference belongs to the region is exactly $$ \{\delta\in\mathcal C_{1-\alpha}\}=\{T^2(\delta)\leq q_\alpha\}. $$ Multiplying the inequality by the positive factor $(\nu-p+1)/(p\nu)$ gives $$ \left\{\frac{\nu-p+1}{p\nu}T^2(\delta) \leq \frac{\nu-p+1}{p\nu}q_\alpha\right\} = \left\{\frac{\nu-p+1}{p\nu}T^2(\delta) \leq F_{p,\nu-p+1;1-\alpha}\right\}. $$ The random variable on the left has the stated $F$ distribution, so the probability of this event is $1-\alpha$, by the definition of the quantile. Thus the region has exact coverage. $\square$

Since $1/n_1+1/n_2=N/(n_1n_2)$, the same ellipsoid can be written in the form $$ \mathcal C_{1-\alpha} = \left\{\delta: (\hat\delta-\delta)' \left[\left(\frac{1}{n_1}+\frac{1}{n_2}\right)S_p\right]^{-1} (\hat\delta-\delta) \leq q_\alpha \right\}. $$ The matrix inside the brackets estimates the covariance of the difference of sample means. This form makes clear that the region is centered at $\hat\delta$, with its shape determined by the estimated covariance of that difference.

Worked Example: Constructing a Two-Dimensional Ellipsoid

Suppose $n_1=12$, $n_2=10$, $p=2$, and the observed estimates are $$ \hat\delta=\begin{pmatrix}2\\1\end{pmatrix}, \qquad S_p=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$ Construct the 95% ellipsoid and check whether $\delta=(1,0)'$ belongs to it.

Here $N=22$, $\nu=20$, and the denominator degrees of freedom are $N-p-1=19$. For two numerator degrees of freedom, the $F_{2,19}$ cumulative distribution function is $1-(19/(19+2x))^{19/2}$. Setting this equal to $0.95$ gives $$ F_{2,19;0.95}=\frac{19}{2}\left(0.05^{-2/19}-1\right)\approx 3.523. $$ Therefore $$ q_{0.05}=\frac{2(20)}{19}(3.523)\approx 7.42. $$ The determinant of $S_p$ is $4(2)-1(1)=7>0$, and its leading diagonal entry is positive, so $S_p$ is positive definite. Its inverse is $$ S_p^{-1}=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}. $$ Thus the region is $$ \frac{60}{11}(\hat\delta-\delta)'S_p^{-1}(\hat\delta-\delta)\leq 7.42, $$ because $n_1n_2/N=120/22=60/11$.

For the candidate $\delta=(1,0)'$, the difference $\hat\delta-\delta=(1,1)'$. Its quadratic form is $$ (1,1)S_p^{-1}\begin{pmatrix}1\\1\end{pmatrix} =(1,1)\begin{pmatrix}1/7\\3/7\end{pmatrix} =\frac{4}{7}. $$ The left side of the ellipsoid inequality is therefore $(60/11)(4/7)=240/77\approx3.12$, which is less than $7.42$. The candidate lies inside the 95% region.

By contrast, the candidate $\delta=(-1,1)'$ gives $\hat\delta-\delta=(3,0)'$. Its quadratic form is $18/7$, so the ellipsoid’s left side is $(60/11)(18/7)=1080/77\approx14.03$, greater than $7.42$. That candidate is outside the region.

Shape and Simultaneous Interpretation

An ellipsoid is not generally a circle: directions with greater estimated variability receive more room. If $\lambda_1,\ldots,\lambda_p$ are the eigenvalues of $S_p$, the principal axes point along its eigenvectors. The corresponding semiaxis lengths are $\sqrt{q_\alpha(1/n_1+1/n_2)\lambda_j}$. The factor $1/n_1+1/n_2$ shrinks as the sample sizes grow, while the $F$ critical value adjusts for the dimension and finite sample size.

The region also gives simultaneous statements about every linear contrast of the mean difference. For a fixed nonzero vector $a$, its projection onto the contrast $a'\delta$ is an interval centered at $a'\hat\delta$. The next result derives that interval and explains the word “simultaneous.”

Result: The projection of $\mathcal C_{1-\alpha}$ onto $a'\delta$ is $$ a'\hat\delta \ \pm\ \sqrt{q_\alpha\left(\frac{1}{n_1}+\frac{1}{n_2}\right)a'S_pa}. $$ Consequently, with probability at least $1-\alpha$, this interval contains the true contrast $a'\delta$ for every choice of $a$ at once.

Derivation. Set $A=(1/n_1+1/n_2)S_p$, which is positive definite, and write $u=\delta-\hat\delta$. The ellipsoid condition is $u'A^{-1}u\leq q_\alpha$. For any $a$, the Cauchy–Schwarz inequality applied to $A^{1/2}a$ and $A^{-1/2}u$ yields $$ |a'u|^2 = |(A^{1/2}a)'(A^{-1/2}u)|^2 \leq (a'Aa)(u'A^{-1}u) \leq q_\alpha a'Aa. $$ Thus every point in the ellipsoid satisfies the displayed contrast bound. The bound is attained in both directions by taking $u=\pm\sqrt{q_\alpha/(a'Aa)}\,Aa$; these vectors satisfy $u'A^{-1}u=q_\alpha$. The projection endpoints are therefore exactly the stated endpoints. If the true $\delta$ is in the ellipsoid, every one of its contrasts is in the corresponding projected interval. The ellipsoid has probability $1-\alpha$ of containing $\delta$, so all these intervals cover their true contrasts simultaneously with probability at least $1-\alpha$. $\square$

Worked Example: Projecting the Region onto a Contrast

Use the estimates from the two-dimensional example and consider the contrast $a'\delta$ with $a=(1,-1)'$. Its center is $a'\hat\delta=2-1=1$. The estimated contrast variance factor is $$ \left(\frac{1}{12}+\frac{1}{10}\right)a'S_pa = \frac{11}{60}(1,-1) \begin{pmatrix}4&1\\1&2\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix}. $$ The matrix-vector product is $(3,-1)'$, so $a'S_pa=3-(-1)=4$. The variance factor is therefore $(11/60)(4)=11/15$. Using $q_{0.05}\approx7.42$, the projected half-width is $$ \sqrt{7.42(11/15)}\approx\sqrt{5.44}\approx2.33. $$ The projected interval for $\delta_1-\delta_2$ is approximately $1\pm2.33$, or $[-1.33,3.33]$. This interval follows from the 95% joint ellipsoid, so its contrast coverage is at least 95%, not necessarily exactly 95%.

The One-Dimensional Case

When $p=1$, the ellipsoid is an interval and the pooled Hotelling procedure reduces to the familiar pooled two-sample $t$ interval. In fact, $F_{1,\nu;1-\alpha}=t_{\nu;1-\alpha/2}^2$. Since $q_\alpha=F_{1,\nu;1-\alpha}$ in this case, taking the square root of the ellipsoid inequality gives the usual two-sided interval. This reduction is a useful check on the degrees of freedom and scaling.

Worked Example: Checking the One-Dimensional Reduction

Suppose $n_1=8$, $n_2=7$, the sample means are $\bar X_1=5.2$ and $\bar X_2=3.9$, and the sample variances are $S_1=1.44$ and $S_2=1.00$. Here $p=1$, $N=15$, and $\nu=13$. The estimated difference is $\hat\delta=5.2-3.9=1.3$, and $$ S_p=\frac{7(1.44)+6(1.00)}{13} =\frac{16.08}{13}\approx1.237. $$ The estimated variance of the difference is $$ S_p\left(\frac{1}{8}+\frac{1}{7}\right) \approx1.237\left(\frac{15}{56}\right) \approx0.3313, $$ so its estimated standard error is $\sqrt{0.3313}\approx0.5756$. The 95% critical value is $t_{13;0.975}\approx2.160$, equivalently $F_{1,13;0.95}\approx2.160^2\approx4.666$. The interval is $$ 1.3\pm2.160(0.5756)=1.3\pm1.243, $$ or approximately $[0.057,2.543]$. It is precisely the pooled two-sample $t$ interval written as the one-dimensional Hotelling region.

Common Pitfalls and Practical Meaning

A frequent mistake is to calculate separate 95% intervals for each coordinate and call their Cartesian product a 95% confidence region. Separate coordinate intervals do not generally provide 95% joint coverage: the chance that all coordinates cover at once can be lower. The pooled Hotelling ellipsoid instead uses a single multivariate critical value and gives an exact joint region under its assumptions. Its projections produce contrast intervals that are simultaneous, but a collection of ordinary coordinate intervals need not match those projections.

A second mistake is to use the exact pooled calibration when the groups have different population covariance matrices. The formula relies on a common $\Sigma$, not just on using a pooled estimate as a convenient calculation. Nor should an exact Gaussian claim be transferred automatically to nonnormal data. Check the model assumptions and whether the pooled covariance matrix is invertible before interpreting the region.

Finally, the ellipsoid describes plausible values of the full mean difference, not a probability distribution over those values. If the origin lies inside it, the data do not exclude a zero mean difference at the corresponding two-sided confidence level; if it lies outside, zero is excluded by that joint region. The region can also show which combinations of component differences are plausible, information that separate one-coordinate summaries can obscure.

Key takeaway: Under independent Gaussian sampling with a common positive-definite covariance, the pooled Hotelling $F$ calibration gives an exact finite-sample ellipsoid for $\mu_1-\mu_2$. Its center is the observed mean difference, its shape reflects the pooled covariance, and it supports simultaneous inference for all linear contrasts.

Check Your Understanding

Use the assumptions and formulas above to check your understanding.

Why must the difference be defined with a group order, and how would reversing that order affect the center of the ellipsoid?
For $p=3$ and $N=20$, what are the numerator and denominator degrees of freedom for the critical $F$ value?
Which part of the ellipsoid formula determines its orientation, and what determines the lengths of its principal semiaxes?
Why does projecting the ellipsoid onto a linear contrast yield simultaneous coverage, while separate coordinate intervals do not generally do so?
What model assumption is violated if the two populations have different covariance matrices, and why does that matter for the exact calibration?