Duality Between Tests and Confidence Regions

From a Test to a Confidence Set

A test asks whether a proposed parameter value is compatible with the data at a specified significance level. A confidence set reverses that question: it collects every proposed value that the test does not reject. This connection is useful in multivariate linear models because the same test statistic can describe a region of plausible scalar or vector contrasts, not just produce a reject-or-do-not-reject decision.

The connection is exact when the test is calibrated at level $\alpha$ for every value of the parameter being tested. Inverting a family of such tests then gives a confidence set with coverage at least $1-\alpha$; if each test has exact size $\alpha$, the coverage is exactly $1-\alpha$, subject to the usual treatment of boundary values.

Definition: Let $\theta$ be a parameter and, for each candidate value $\theta_0$, let $\phi_{\theta_0}(Y)$ be a test rule, where $\phi_{\theta_0}(Y)=1$ means reject $H_0:\theta=\theta_0$. The inverted confidence set is $$ C(Y)=\{\theta_0:\phi_{\theta_0}(Y)=0\}. $$ If $P_\theta\{\phi_\theta(Y)=1\}\leq\alpha$ for every $\theta$, then $P_\theta\{\theta\in C(Y)\}\geq1-\alpha$ for every $\theta$. If the rejection probability is exactly $\alpha$ for every $\theta$, the coverage is exactly $1-\alpha$.

The coverage statement follows directly from the definition: the true value is in $C(Y)$ precisely when the test of that true value does not reject. The key is to test each candidate value with the same specified rule and level. A confidence set is not obtained by choosing a candidate after seeing the data and then treating its test as if it had been specified in advance.

Exact Inversion for a Vector Contrast

Consider the fixed-design multivariate normal linear model $Y=XB+U$, where $Y$ is $n\times p$, $X$ is a fixed $n\times k$ matrix of full column rank, and the rows of $U$ are independent $N_p(0,\Sigma)$, with $\Sigma$ positive definite. Write $\nu=n-k$, assume $\nu\geq s$, and let $M$ be a $p\times s$ matrix of full column rank. For a nonzero $c\in\mathbb R^k$, the vector contrast of interest is $g=c'BM$, a $1\times s$ row vector. Its estimate is $\hat g=c'\hat B M$, where $\hat B=(X'X)^{-1}X'Y$.

Define $a=c'(X'X)^{-1}c$, $E=Y'(I-P_X)Y$, $P_X=X(X'X)^{-1}X'$, and $S_M=M'(E/\nu)M$. The matrix $S_M$ estimates the covariance of the response contrast, while $a$ accounts for the design variance of the coefficient contrast. Under the stated assumptions, $a>0$, and $S_M$ is positive definite with probability one.

Theorem: For testing $H_0:g=g_0$, use $$ T^2(g_0)=(\hat g-g_0)(aS_M)^{-1}(\hat g-g_0)'. $$ Under $H_0$, this statistic has distribution $$ T^2(g_0)\ \sim\ \frac{\nu s}{\nu-s+1}F_{s,\nu-s+1}. $$ Consequently, a level-$\alpha$ test rejects when $T^2(g_0)>\frac{\nu s}{\nu-s+1}F_{s,\nu-s+1;1-\alpha}$, where $F_{d_1,d_2;1-\alpha}$ is the $1-\alpha$ quantile of an $F$ distribution. Inverting this test gives a $1-\alpha$ confidence region for $g$: $$ \mathcal C(Y)=\left\{g_0:(\hat g-g_0)(aS_M)^{-1}(\hat g-g_0)'\leq \frac{\nu s}{\nu-s+1}F_{s,\nu-s+1;1-\alpha}\right\}. $$

Here is a derivation of the distribution and coverage. Under the null, $\hat g-g_0$ is normal with mean zero and covariance $a\Sigma_M$, where $\Sigma_M=M'\Sigma M$. Also, $M'EM$ has a Wishart distribution with scale $\Sigma_M$ and $\nu$ degrees of freedom, and it is independent of $\hat g$. These facts follow from the normal linear model: the fitted and residual projections are orthogonal, and their joint normal distribution makes them independent.

After dividing $\hat g-g_0$ by $\sqrt a$ and transforming by $\Sigma_M^{-1/2}$, obtain a standard normal vector $z$ of dimension $s$. Transforming $M'EM$ by the same covariance factor gives an independent Wishart matrix $W$ with identity scale and $\nu$ degrees of freedom. Since $S_M=M'EM/\nu$, the statistic has the same distribution as $\nu z'W^{-1}z$.

To identify this distribution, condition on the direction of $z$ and rotate coordinates so that this direction is the first coordinate. The distribution of $W$ is unchanged by this orthogonal rotation. Also, $z'z\sim\chi^2_s$. Write $W=Z'Z$, where $Z$ is a $\nu\times s$ matrix of independent standard normal entries. The reciprocal of $(W^{-1})_{11}$ is the Schur complement for the first column of $Z$: it is the squared length of that column after projection off the other $s-1$ columns. Conditional on those columns, this residual projection has $\nu-s+1$ dimensions, so its squared length has a $\chi^2_{\nu-s+1}$ distribution. Thus $z'W^{-1}z$ has the distribution of the ratio of independent $\chi^2_s$ and $\chi^2_{\nu-s+1}$ variables. By the definition of the $F$ distribution, this ratio equals $sF_{s,\nu-s+1}/(\nu-s+1)$ in distribution. Multiplying by $\nu$ gives the stated pivot.

Finally, the true $g$ belongs to the inverted region exactly when its null test does not reject. The pivot has a continuous distribution, so the rejection probability at the stated quantile is $\alpha$, and the region has coverage $1-\alpha$. $\square$

The Scalar Case and Worked Inversions

When $s=1$, the vector region reduces to an interval. The statistic is the square of a $t$ statistic with $\nu$ degrees of freedom, because $F_{1,\nu}$ has the distribution of $t_\nu^2$. Thus testing each candidate scalar value at level $\alpha$ and collecting the nonrejected values gives the usual two-sided $t$ confidence interval.

Worked Example: Inverting a Scalar Contrast Test

Suppose a scalar contrast estimate is $\hat g=2.4$, its estimated standard error is $0.5$, and the residual degrees of freedom are $\nu=12$. At the 5% level, the two-sided test of $H_0:g=g_0$ rejects when $|(\hat g-g_0)/0.5|>t_{12;0.975}$, with $t_{12;0.975}\approx2.179$.

The set of values not rejected satisfies $|2.4-g_0|\leq2.179(0.5)=1.0895$. Therefore the inverted confidence interval is $[2.4-1.0895,\ 2.4+1.0895]=[1.3105,\ 3.4895]$.

For the candidate $g_0=3$, the test statistic is $|2.4-3|/0.5=1.2$, which is below 2.179, so 3 is included. For $g_0=1$, it is $|2.4-1|/0.5=2.8$, which exceeds 2.179, so 1 is excluded. The endpoints are exactly the candidate values at the test's critical boundary.

Worked Example: Checking Membership in a Vector Confidence Ellipsoid

Suppose $s=2$, $\nu=12$, $\alpha=0.05$, and the estimated covariance matrix $aS_M$ is $A=\begin{pmatrix}1&0.25\\0.25&0.5\end{pmatrix}$. Its determinant is $1(0.5)-(0.25)^2=0.4375=7/16$, so $$ A^{-1}=\frac{1}{7}\begin{pmatrix}8&-4\\-4&16\end{pmatrix} =\begin{pmatrix}1.1429&-0.5714\\-0.5714&2.2857\end{pmatrix}. $$ Assume $\hat g=(1,-1)$.

The ellipsoid cutoff is $\frac{12(2)}{12-2+1}F_{2,11;0.95}=\frac{24}{11}F_{2,11;0.95}$. For two numerator degrees of freedom, $F_{2,11;0.95}=\frac{11}{2}(20^{2/11}-1)$, so the cutoff is $12(20^{2/11}-1)\approx8.69$.

For candidate $g_0=(2,0)$, the difference is $d=\hat g-g_0=(-1,-1)$. Its quadratic form is $d'A^{-1}d=8/7-8/7+16/7=16/7\approx2.286$, below 8.69, so this candidate lies in the confidence region. For $g_0=(4,-1)$, $d=(-3,0)$, giving $d'A^{-1}d=9(8/7)=72/7\approx10.286$, above 8.69. That candidate is rejected and lies outside the region.

Worked Example: Pointwise Intervals Versus Joint Coverage

Suppose two independent standardized estimation errors are standard normal. For each coordinate, the usual 95% interval uses the error bounds $-1.96$ and $1.96$, and hence has coverage approximately $0.95$. The probability that both intervals cover their respective coordinates is, by independence, $0.95^2=0.9025$, not 0.95.

Thus, having 95% coverage for each coordinate separately does not give 95% coverage for the pair. In this example, intervals using the central 97.5% normal quantile, approximately 2.241, have joint coverage at least $1-2(0.025)=0.95$ by the union bound. This is a Bonferroni guarantee; it need not be exact. A joint confidence region, such as the ellipsoid above under its model assumptions, instead calibrates coverage for the entire vector at once.

Pointwise and Simultaneous Coverage

A pointwise confidence statement concerns one specified scalar contrast or one specified coordinate at a time. Repeating a procedure with 95% pointwise coverage for several coordinates does not, by itself, ensure that all coordinates are covered with probability 95%. The probability of missing at least one can increase as the number of coordinates increases, and the amount depends on their dependence.

A confidence region for a vector contrast makes a joint statement: the whole true vector belongs to the region with the stated probability. Every coordinate of a vector inside the region has the corresponding coordinate value inside the region's coordinate projection. Therefore those projected intervals provide simultaneous coverage of all coordinates at least as large as the joint region's coverage. They may be wider than separate pointwise intervals.

The distinction is about the event whose probability is controlled. “This contrast is covered with probability 0.95” describes one event. “All these contrasts are covered together with probability 0.95” describes an intersection of events. Tests must be calibrated for the relevant event if the intended conclusion is simultaneous.

Common Pitfall

A common mistake is to test each component of a vector at level 0.05 and then report the resulting collection of intervals as a 95% simultaneous confidence region. Each interval may have 95% marginal coverage, but the intersection need not. The correction is to decide whether the inferential target is each coordinate separately or the vector jointly. For joint inference, invert a test calibrated for the vector, or use a procedure with a stated simultaneous guarantee, such as Bonferroni adjustment. Also keep the candidate parameter in the test: inversion collects all $g_0$ not rejected, rather than using a single test result to construct an interval by inspection.

Key takeaway: A confidence set is the collection of parameter values not rejected by their corresponding tests. Inverting a correctly calibrated vector test gives joint coverage for the vector; separate pointwise tests do not automatically provide that same simultaneous guarantee.

Check Your Understanding

Use the definitions and derivations above to answer the following questions.

Why does the event that the true parameter belongs to an inverted confidence set equal the event that its null test is not rejected?
In the vector contrast theorem, what roles do $a$ and $S_M$ play in the quadratic statistic?
For $s=1$, why is the inverted confidence set a $t$ interval?
What does a joint confidence region guarantee that separate pointwise intervals do not necessarily guarantee?
If each of two independent intervals has 95% coverage, what is their joint coverage, and why?