Confidence Regions by Inverting Rank Tests

From a Test to a Confidence Region

A test asks whether a particular value of a location parameter is compatible with the data. A confidence region reverses that question: it collects the candidate locations that the test does not reject. This connection is especially useful for multivariate rank tests, whose rejection regions need not translate into familiar ellipses and may instead have irregular shapes.

Let $X_1,\ldots,X_n$ be independent observations in $\mathbb{R}^p$ from a location model $X_i=\theta+\varepsilon_i$, where $\theta\in\mathbb{R}^p$ is the location of interest. For each proposed location $\vartheta$, the residuals are $X_i-\vartheta$. A test of $H_0:\theta=\vartheta$ produces a decision or a p-value for that candidate. Inverting the test means repeating this calculation over the possible values of $\vartheta$.

Definition: Suppose $p_{\vartheta}$ is a p-value for testing $H_0:\theta=\vartheta$, calibrated so that under that null hypothesis $P(p_{\vartheta}\leq u)\leq u$ for every $u\in[0,1]$. The nonrandomized $1-\alpha$ confidence region obtained by inversion is $$ C_{1-\alpha}=\{\vartheta\in\mathbb{R}^p:p_{\vartheta}>\alpha\}. $$ Equivalently, it is the set of candidate locations not rejected at level $\alpha$.

The strict inequality in this definition matches the convention that a candidate is rejected when its p-value is at most $\alpha$. Another convention can assign boundary points differently, but the stated convention makes the inversion explicit.

Why the Region Has Its Claimed Coverage

The coverage guarantee follows directly from the test’s size control. It does not require the region to be convex, connected, or even nonempty for every sample. The statement concerns repeated samples under the model: the true location is included with high probability.

Theorem: If, for every candidate $\vartheta$, the test p-value satisfies $P_{\theta=\vartheta}(p_{\vartheta}\leq\alpha)\leq\alpha$, then the inverted region satisfies $$ P_{\theta}(\theta\in C_{1-\alpha})\geq 1-\alpha. $$ If the test has rejection probability exactly $\alpha$ under every null, the coverage is exactly $1-\alpha$.

Proof. When the true location is $\theta$, the event that the region misses it is $$ \{\theta\notin C_{1-\alpha}\}=\{p_{\theta}\leq\alpha\}. $$ By the assumed size control, $P_{\theta}(p_{\theta}\leq\alpha)\leq\alpha$. Taking complements gives $P_{\theta}(\theta\in C_{1-\alpha})\geq1-\alpha$. If the rejection probability equals $\alpha$, the same calculation gives equality. $\square$

This result is general: any valid family of tests can be inverted. For a useful finite-sample illustration, consider a spatial rank score test under central symmetry. Assume the errors $\varepsilon_i$ are independent and identically distributed and centrally symmetric about zero, meaning that $\varepsilon_i$ and $-\varepsilon_i$ have the same distribution. No particular radial distribution is specified.

Rank score test used below: For a proposed location $\vartheta$, set $r_i=X_i-\vartheta$. Rank the lengths $\|r_i\|$ from smallest to largest; use midranks for ties. Write $R_i(\vartheta)$ for the rank and $u_i(\vartheta)=r_i/\|r_i\|$ when $r_i\ne0$. For a zero residual, set its score contribution to zero. With rank score $a(r)=r$, use $$ T(\vartheta)=\left\|\sum_{i=1}^n a(R_i(\vartheta))u_i(\vartheta)\right\|^2. $$

The statistic is large when the ranked residual directions combine to give a long vector. For its exact conditional calibration, hold the residual lengths and their unoriented directions fixed, and independently flip the sign of each nonzero direction. If there are $m$ nonzero residuals, this gives $2^m$ sign configurations. The upper-tail p-value is the fraction of configurations whose statistic is at least the observed statistic. Ties in this tail count as at least as extreme.

Exact Sign-Flip Calibration and Coverage

At the true location, central symmetry makes the signs of the residual directions exchangeable under independent sign flips, conditional on their unoriented values. The ranks and score magnitudes are unchanged by those flips because they depend only on residual lengths. Thus the sign-flip distribution calibrates the rank score statistic without specifying the common distribution of the error lengths.

Result: Under independent, identically distributed errors centrally symmetric about zero, the nonrandomized sign-flip p-value for the rank score statistic is super-uniform under $H_0:\theta=\vartheta$: $P(p_{\vartheta}\leq u)\leq u$ for every $u\in[0,1]$. Consequently, its inverted region has coverage at least $1-\alpha$.

Justification. Under the null, condition on the unoriented residual vectors and their lengths. Each of the $2^m$ sign configurations is equally likely; zero residuals contribute zero and require no sign. For any fixed collection of $2^m$ statistic values, order the configurations from largest statistic to smallest, grouping ties. The p-value assigned to a configuration is the number of configurations in its upper tail, including its tied group, divided by $2^m$. Therefore, if that p-value is at most $u$, the total number of configurations in the qualifying upper-tail groups is at most $u2^m$. The conditional probability of such a p-value is at most $u$. Averaging over the conditioned residual information gives the unconditional inequality. The coverage claim then follows from the theorem. $\square$

The test is generally conservative at finite sample sizes because the p-value can take only discrete values. Randomizing at a boundary can attain exact size, but the nonrandomized region is often preferred because its membership rule is deterministic.

Finite-Sample Calculations

Worked Example: A Candidate Location and Its Exact P-Value

Consider four bivariate observations $X_1=(1,0)$, $X_2=(0,2)$, $X_3=(-3,0)$, and $X_4=(0,-4)$. Test the candidate $\vartheta=(0,0)$ with the rank score $a(r)=r$.

The residual lengths are $1,2,3,4$, so the ranks are $1,2,3,4$. The weighted direction vectors are respectively $(1,0)$, $(0,2)$, $(-3,0)$, and $(0,-4)$. Their sum is $(-2,-2)$, giving the observed statistic $$ T(0,0)=(-2)^2+(-2)^2=8. $$

To calculate the sign-flip p-value, let $s_i\in\{-1,1\}$ be the sign applied to the $i$th weighted vector. The resulting statistic is $$ (s_1-3s_3)^2+(2s_2-4s_4)^2. $$ The absolute value of the first coordinate is either $2$ or $4$, and that of the second is either $2$ or $6$. The statistic is at least the observed value $8$ in all 16 configurations. Only when both absolute values are $2$ does it equal $8$; the other configurations give larger values. Thus $p_{(0,0)}=16/16=1$, and this candidate is retained for any conventional significance level below 1.

Worked Example: Recomputing the Test at Another Candidate

Use the same observations, but test $\vartheta=(1,0)$. The first residual is zero. The other residuals are $(-1,2)$, $(-4,0)$, and $(-1,-4)$, with lengths $\sqrt{5}$, $4$, and $\sqrt{17}$. Including the zero residual in the length ranking gives ranks $1,2,3,4$; its contribution is zero.

The three nonzero weighted direction vectors are $$ A=2\frac{(-1,2)}{\sqrt{5}},\qquad B=3(-1,0),\qquad C=4\frac{(-1,-4)}{\sqrt{17}}. $$ Their sum is approximately $(-4.865,-2.092)$, so the observed statistic is approximately $28.04$. There are $2^3=8$ sign configurations. Up to an overall sign, the four distinct sums are $A+B+C$, $A+B-C$, $A-B+C$, and $A-B-C$; their squared lengths are approximately $28.04$, $40.70$, $5.66$, and $41.60$. Overall sign reversal does not change squared length, so six of the eight configurations are at least as extreme as observed. Hence $p_{(1,0)}=6/8=0.75$.

At significance level $\alpha=0.30$, the candidate $(1,0)$ is retained, while the candidate $(0,0)$, whose p-value is 1, is also retained. The calculations illustrate that the confidence region is assembled by recomputing ranks and the sign-flip reference distribution at each candidate; one test statistic evaluated at a single location does not determine the entire region.

Worked Example: Tied Lengths and Midranks

Suppose $n=2$, the proposed location is $(0,0)$, and both observed residuals are $(1,0)$. Their lengths tie, so each receives midrank $(1+2)/2=1.5$. The weighted vectors are both $(1.5,0)$, and the observed statistic is $$ T(0,0)=\|(3,0)\|^2=9. $$

The four sign configurations produce sums $(3,0)$, $(0,0)$, $(0,0)$, and $(-3,0)$, with statistics $9,0,0,9$. The upper-tail p-value is $2/4=0.5$. At level $\alpha=0.4$, the candidate is retained. The tied ranks are not broken arbitrarily: using midranks preserves the rank rule and ensures that sign flips leave the score magnitudes unchanged.

Assumptions, Interpretation, and Common Pitfalls

The exact guarantee above depends on the test’s assumptions and calibration. Independent central symmetry supports the sign-flip reference distribution; it does not assert that the errors are normally distributed or that the region must have an elliptical shape. If the errors are not centrally symmetric, or if observations are dependent in a way that invalidates independent sign flips, the stated exact calibration need not hold. A different rank test may instead use a justified permutation or asymptotic reference distribution. In every case, the coverage statement follows from that test’s actual size control, not merely from calling the result a confidence region.

A common mistake is to calculate the rank statistic at a point estimate and then label a contour around that estimate as a confidence region. That construction is justified only if it is equivalent to inverting a calibrated test. The correction is to test each candidate location using the prescribed statistic and null calibration, and retain exactly those candidates that are not rejected. Another mistake is to treat “not rejected” as proof that a candidate is true. It means only that the test does not exclude it at the chosen level.

The resulting set may be irregular, disconnected, or empty for a particular sample; the coverage guarantee is a repeated-sampling property, not a promise about its shape or nonemptiness. Its boundary can also be sensitive to discrete p-values and ties. These features are consequences of finite-sample rank-test inversion, not errors to be repaired by drawing a smoother contour.

Key takeaway: A multivariate location confidence region is obtained by inverting a test over candidate locations. Its coverage is at least the nominal level only when the tests are correctly calibrated under the model assumptions.

Check Your Understanding

Use the definitions and calculations above to check your understanding.

Why does a level-$\alpha$ test at the true location imply coverage of at least $1-\alpha$ after inversion?
Which quantities remain unchanged when residual direction signs are flipped in the rank score test?
Why can the nonrandomized sign-flip p-value be conservative at finite sample sizes?
For the tied-length example, what would be wrong with assigning the two tied observations different ranks arbitrarily?
Does a confidence region formed by inversion have to be convex? Explain what property its coverage guarantee actually concerns.