Null Limits of Multivariate Rank Statistics

Why Null Limits Need More Than a Rank Definition

A rank statistic is often distribution-free in a particular finite-sample setting, but that does not mean every statistic built from multivariate ranks has the same null distribution. The limiting law depends on what is being summed, how the ranks are estimated, and whether the first-order variation vanishes under the null. In particular, a vector of rank scores usually leads to a quadratic form in normal variables, while a pairwise statistic can have a normal limit or, in a degenerate case, a nonnormal limit.

The goal here is to identify the assumptions that support these conclusions. The key distinction is between a score whose first-order projection is nonzero and one whose first-order projection vanishes. The former is governed by a central limit theorem; the latter requires a different analysis.

Spatial-Rank Scores and Quadratic-Form Limits

For a distribution \(F\) on \(\mathbb{R}^p\), define its population spatial-rank function by \(\mathbf R_F(x)=E[S(x-Y)]\), where \(Y\sim F\) and \(S(z)=z/\|z\|\) for \(z\ne0\), with \(S(0)=0\). A spatial-rank score can be this vector itself or a transformed score such as \(a(\|\mathbf R_F(x)\|)\mathbf R_F(x)\). Under a centrally symmetric null distribution about the specified center, an odd score has mean zero, provided its expectation exists.

Proof. The multivariate central limit theorem gives \(\sqrt n\,\bar\psi_n\xrightarrow{d}Z\), where \(Z\sim N_p(0,\Sigma)\). By the continuous mapping theorem, \(T_n\xrightarrow{d}Z^\mathsf{T}Z\). Write \(\Sigma=Q\Lambda Q^\mathsf{T}\), where \(Q\) is orthogonal and \(\Lambda\) is diagonal with entries \(\lambda_1,\ldots,\lambda_p\). The distribution of \(Q^\mathsf{T}Z\) is that of \(\Lambda^{1/2}W\), with \(W\sim N_p(0,I_p)\). Therefore \(Z^\mathsf{T}Z\) has the distribution of \(\sum_j\lambda_j W_j^2\). If \(\Sigma\) is positive definite, \(n\bar\psi_n^\mathsf{T}\Sigma^{-1}\bar\psi_n\) converges to \(W^\mathsf{T}W\), which has a \(\chi_p^2\) distribution. \(\square\)

In an application, the score may use an estimated center, an empirical spatial-rank map, or estimated score parameters. The theorem still applies if the resulting score sum has the same first-order limit as a sum of independent, mean-zero influence scores. One precise sufficient condition is an expansion \(\sqrt n\,\bar\psi_n^{\,\mathrm{est}}=n^{-1/2}\sum_{i=1}^n\psi_{\mathrm{eff}}(X_i)+o_p(1)\), where \(\psi_{\mathrm{eff}}\) has mean zero and finite covariance. The covariance in the limit is then that of \(\psi_{\mathrm{eff}}\), not necessarily that of the unestimated score. This expansion must be justified for the estimator and score in use; empirical ranks do not automatically have negligible estimation error.

Depth Scores in Two-Sample Comparisons

A depth function \(D_F(x)\) assigns a centrality value to \(x\) relative to \(F\). A depth-rank score is a chosen function of that value, for example \(q_F(x)=J(D_F(x))\), where \(J\) is a specified score function. Consider two independent samples of sizes \(n_1\) and \(n_2\), drawn from the same continuous distribution \(F\) under the null. If \(q_F(X)\) has finite variance \(\sigma_q^2>0\), the difference of sample mean scores has a direct null limit.

Proof. Under the null, both group means have expectation \(\mu_q=E[q_F(X)]\). For each group, the ordinary central limit theorem applies to its independent scores. Independence of the samples makes the two limiting normal variables independent. The variance of \(\bar q_1-\bar q_2\) is exactly \(\sigma_q^2(1/n_1+1/n_2)\). Dividing by its square root therefore gives a sum of two independent, asymptotically normal terms with total limiting variance one, proving the result. \(\square\)

This statement uses the population depth score. If the depth is computed from a pooled empirical distribution, the scores share the estimated reference distribution and are dependent. A valid asymptotic calculation must account for that estimation, or calibration must reproduce it, as a permutation procedure does under exchangeability.

Pairwise Rank Statistics and the U-Statistic Projection

Many multivariate rank procedures are averages of pairwise comparisons. Their null limits can be derived from a U-statistic decomposition. Let \(h(x,y)\) be a symmetric kernel, let \(X\) and \(Y\) be independent with common distribution \(F\), and define \(\theta=E[h(X,Y)]\). Assume \(E[h(X,Y)^2]<\infty\). Define the first projection \(h_1(x)=E[h(x,Y)]-\theta\), and the remainder \(h_2(x,y)=h(x,y)-\theta-h_1(x)-h_1(y)\). By construction, \(E[h_2(X,Y)\mid X]=0\).

Proof. Substituting the definition of \(h_2\) into the pairwise average gives \(U_n-\theta=(2/n)\sum_i h_1(X_i)+{\binom n2}^{-1}\sum_{i<j}h_2(X_i,X_j)\). The first term, multiplied by \(\sqrt n\), converges by the ordinary central limit theorem to a normal distribution with variance \(4\operatorname{Var}(h_1(X))\). For the remainder, two terms \(h_2(X_i,X_j)\) and \(h_2(X_k,X_l)\) have zero covariance if their index pairs are disjoint, by independence and zero mean. They also have zero covariance if the pairs share one index: conditioning on the shared observation, the other observations are independent and each conditional expectation is zero. Consequently the variance of the average remainder is \(\operatorname{Var}(h_2(X,Y))/\binom n2=O(n^{-2})\). Its product by \(\sqrt n\) converges to zero in probability by Chebyshev's inequality. Slutsky's theorem gives the stated limit. \(\square\)

Degeneracy, Calibration, and a Common Pitfall

The U-statistic theorem requires \(\operatorname{Var}(h_1(X))>0\). If \(h_1(X)=0\) almost surely, the first-order normal term disappears, so the theorem does not apply. The remainder then determines the limit; for a square-integrable degenerate kernel, that limit can involve a weighted sum of centered chi-square variables rather than a normal distribution. A test calibrated with a normal approximation in this case can have the wrong size.

In practice, choose calibration to match the statistic and its null assumptions:

• Covariance-standardized Gaussian limits: estimate the score covariance consistently and use a chi-square reference distribution when the score vector has a nondegenerate normal limit.
• Unstandardized quadratic forms: use the estimated eigenvalues of the score covariance to approximate the corresponding chi-square mixture, or simulate its normal quadratic form.
• Permutation calibration: permute group labels when the null makes observations exchangeable, reproducing the empirical ranks or depths in each permutation. For blocked designs, permute only within the allowed blocks.
• Degenerate or complicated statistics: use a null bootstrap or simulation that preserves the null structure, after checking that the method is valid for the statistic’s degeneracy and sampling design.

Common mistake: “The statistic is based on ranks, so its null limit must be standard normal or chi-square.” Rank construction alone implies neither conclusion. A vector score with covariance \(\Sigma\) gives a weighted chi-square for its unstandardized squared norm, and an estimated rank map can change the first-order covariance. A degenerate pairwise kernel may not have a Gaussian limit at all. The correction is to identify the score or kernel, verify its null expectation and regularity conditions, determine whether its first projection is nonzero, and calibrate the resulting limit rather than choosing a reference distribution by appearance.