Why Standardize Before Ranking?
In one dimension, ordering observations is unchanged if every value is shifted or multiplied by a positive constant. In several dimensions, coordinate scales and correlations complicate the idea of distance from a center. Ranking raw Euclidean distances can therefore make a procedure depend on the units in which each coordinate is recorded. Scatter standardization addresses this problem by transforming deviations from a center into a common scale before ranking them.
The key distinction is between affine invariance and distribution-free calibration. The first is an algebraic property: a procedure gives the same result after a nonsingular linear change of coordinates and a translation. The second concerns the null distribution of a statistic. Invariance alone does not imply that this null distribution is independent of the underlying population distribution, especially when scatter is estimated from the data.
Standardized Residuals and Radial Ranks
The symmetric inverse square root is a convenient convention, not a claim that it is the only possible whitening transformation. Any whitening matrix \(C\) satisfying \(C\hat S C^T=I_p\) gives standardized coordinates that differ from the symmetric-root version by an orthogonal transformation. Euclidean radii, and hence radial ranks, are the same.
A common rank-score statistic uses a score \(a(r)\) assigned to rank \(r\), then combines scores with directions. One example is \(T=\|\sum_{i=1}^n a(R_i)U_i\|^2\). The squared norm makes the statistic unchanged by a common orthogonal rotation. Its exact null calibration, however, requires additional assumptions or a valid randomization scheme.
Worked Example: Standardizing Before Assigning Radial Ranks
Suppose the center is known to be \((0,0)\) and the scatter matrix is \(\Sigma=\operatorname{diag}(4,1)\). Consider observations \(x_1=(2,0)\), \(x_2=(0,3)\), and \(x_3=(-4,0)\). Find their standardized residuals, radii, ranks, and directions.
The inverse square root is \(\Sigma^{-1/2}=\operatorname{diag}(1/2,1)\). Therefore \(z_1=(1,0)\), \(z_2=(0,3)\), and \(z_3=(-2,0)\). Their radii are \(1,3,2\), respectively, so the ranks are \(R_1=1\), \(R_2=3\), and \(R_3=2\). The directions are \(u_1=(1,0)\), \(u_2=(0,1)\), and \(u_3=(-1,0)\).
For scores \(a(1)=1\), \(a(2)=2\), and \(a(3)=3\), the score-vector sum is \(1u_1+2u_3+3u_2=(1,0)+(-2,0)+(0,3)=(-1,3)\). Thus \(T=(-1)^2+3^2=10\).
Affine Invariance: The Main Algebra
Use column vectors and transform the data by \(Y_i=BX_i+b\), where \(B\) is a nonsingular \(p\times p\) matrix and \(b\in\mathbb R^p\). A location estimator is affine-equivariant if \(\hat\mu_Y=B\hat\mu_X+b\); a scatter estimator is affine-equivariant if \(\hat S_Y=B\hat S_XB^T\). The same conditions apply when the center and scatter are known population parameters.
Proof. Write \(Z_i=\hat S_X^{-1/2}(X_i-\hat\mu_X)\) and \(Z_i^Y=\hat S_Y^{-1/2}(Y_i-\hat\mu_Y)\). By equivariance, \(\hat S_Y=B\hat S_XB^T\) and \(Y_i-\hat\mu_Y=B(X_i-\hat\mu_X)\). Hence \(Z_i^Y=QZ_i\), where \(Q=\hat S_Y^{-1/2}B\hat S_X^{1/2}\). Direct multiplication gives \(QQ^T=\hat S_Y^{-1/2}B\hat S_XB^T\hat S_Y^{-1/2}=I_p\). Since \(Q\) is square, this makes \(Q\) orthogonal. Orthogonal transformations preserve lengths, so \(\|Z_i^Y\|=\|Z_i\|\), and all radial ranks agree. When the radii are nonzero, \(U_i^Y=QU_i\); therefore the score-vector sum is transformed by \(Q\), and its squared norm is preserved. \(\square\)
Worked Example: A Nonspherical Coordinate Change
Let the original scatter be \(I_2\) and transform coordinates by \(B=\begin{pmatrix}1&1\\0&1\end{pmatrix}\). The transformed scatter is \(BB^T=\begin{pmatrix}2&1\\1&1\end{pmatrix}\), whose inverse is \(\begin{pmatrix}1&-1\\-1&2\end{pmatrix}\). Compare the squared standardized radii of \(x_1=(1,0)\) and \(x_2=(0,2)\) before and after transformation.
Before transformation, the squared radii are \(x_1^Tx_1=1\) and \(x_2^Tx_2=4\). The transformed observations are \(y_1=(1,0)\) and \(y_2=(2,2)\). For \(y_1\), the squared radius using the transformed scatter is \((1,0)\begin{pmatrix}1&-1\\-1&2\end{pmatrix}(1,0)^T=1\). For \(y_2\), first multiply the inverse scatter by the column vector: \(\begin{pmatrix}1&-1\\-1&2\end{pmatrix}(2,2)^T=(0,2)^T\). Thus the squared radius is \((2,2)(0,2)^T=4\). The ordering, as well as both squared radii, is unchanged despite the shear transformation.
When Are Rank Scores Distribution-Free?
A useful exact distribution-free statement is available when the center and scatter are known and standardization produces a spherical sample. Suppose \(Z_i=R_iU_i\), where the observations are independent and identically distributed, \(U_i\) is uniform on the unit sphere in \(\mathbb R^p\), and \(R_i=\|Z_i\|\) has a continuous distribution. Under spherical symmetry, each direction is independent of its radius, and the directions are independent across observations. Continuity ensures there are no tied radii with positive probability.
Derivation. Because the radii are independent and identically distributed with a continuous distribution, each of their \(n!\) strict orderings has probability \(1/n!\): permuting the observation labels leaves their joint distribution unchanged, and the \(n!\) ordering events are disjoint and exhaust probability one. The vector of radius ranks is therefore a uniform random permutation. Independence of each radius from its direction implies that this rank permutation is independent of the full collection of directions. Consequently the statistic is a function of a uniform permutation and independent uniform directions, with no other feature of the radial distribution entering its law. This proves the assertion. \(\square\)
This result explains a role of rank scores: they use the ordering of radii while avoiding dependence on their numerical scale or distributional shape. It does not automatically apply if the center and scatter are estimated from the same observations. Those estimates use the data, so the standardized radii and directions need not have the independence structure in the derivation. An affine-equivariant estimator can preserve the coordinate invariance proved above while the finite-sample null distribution still depends on the underlying radial law.
Worked Example: What Changes When Scatter Is Estimated?
Suppose \(p=2\), the true center and scatter are known, and the standardized observations follow a continuous spherical distribution. A rank score based on the ordering of the standardized radii has the distribution-free property just derived. Now replace the known scatter by an estimate computed from the same observations.
The affine-invariance theorem still applies if that scatter estimate is positive definite and affine-equivariant: after a nonsingular coordinate change, all standardized residuals are rotated by a common orthogonal matrix. But the finite-sample distribution-free derivation no longer follows. The estimated scatter depends on the observations’ radii and directions, so the estimated standardized directions and radius ranks can be dependent in a way that varies with the radial distribution. Thus “affine-invariant” is justified by equivariance, while “exactly distribution-free” needs a separate calibration argument.
For a different calibration route, consider a test based on exchangeable observations with group labels. If the entire procedure is symmetric in the observations and the null makes every label assignment equally likely, a conditional permutation test can calibrate the statistic given the observed data. Any scatter estimate used by the statistic must be handled according to the design: if it depends on the assignment, recompute it for each assignment; if it is a symmetric pooled estimate, it can remain fixed. This conditional randomization argument is distinct from the spherical rank argument.
Ties, Singularity, and Practical Calibration
Ties can arise from rounded measurements, discrete distributions, or identical standardized radii. A rank convention, such as assigning average ranks, defines the statistic, but it does not make a continuous-rank null approximation exact. In a permutation test, random tie-breaking can produce a discrete-uniform rank among the \(|\mathcal A|\) assignments when all assignments are equally likely and ties are broken symmetrically. Rejecting for rank at most \(k\) then has exact conditional size \(k/|\mathcal A|\), an attainable level. At an arbitrary level \(\alpha\), this rule need not have size \(\alpha\); exact size at that level requires randomizing rejection at the boundary or using an equivalent continuous randomized p-value.
Scatter standardization also requires an invertible scatter matrix. If \(p\) is large relative to the sample size, or if the observations lie in a lower-dimensional subspace, the sample covariance may be singular. Then its inverse square root does not exist, so the procedure as defined cannot be applied. One must specify a different method, such as dimension reduction justified by the scientific design, or a suitable positive-definite scatter estimator. Any change must be analyzed on its own terms.
Worked Example: Singularity and the Cost of Ridge Regularization
Take the three centered observations \((-1,0)\), \((0,0)\), and \((1,0)\). Their sample covariance, using divisor \(n-1=2\), is \(\hat S=\operatorname{diag}(1,0)\), which is singular. The second coordinate has zero sample variation, so \(\hat S^{-1/2}\) is undefined.
One possible numerical modification is ridge regularization, replacing \(\hat S\) by \(\hat S+\lambda I\) for \(\lambda>0\). This is positive definite here, but is not affine-equivariant under general coordinate rescaling. To verify the issue, rescale the first coordinate by \(B=\operatorname{diag}(2,1)\). The transformed sample covariance is \(\operatorname{diag}(4,0)\). A residual with first coordinate \(1\) is standardized in that coordinate to \(1/\sqrt{1+\lambda}\) before rescaling and to \(2/\sqrt{4+\lambda}\) after rescaling. Their squares are equal only if \(1/(1+\lambda)=4/(4+\lambda)\), which implies \(4+\lambda=4+4\lambda\), hence \(\lambda=0\). For every positive \(\lambda\), the standardized values differ. Ridge regularization may be useful, but it changes the invariance property and must not be presented as an affine-invariant fix without further adjustment.
Common Mistakes and a Practical Checklist
- Equating invariance with distribution-freeness. Equivariance of the location and scatter estimates proves that the statistic is unchanged by affine coordinate transformations. It does not establish that its null law is independent of the population distribution.
- Ranking raw Euclidean distances. Unless coordinates are already on a common spherical scale, these ranks can change under a change of units. Standardize first, using a specified center and scatter.
- Ignoring the source of scatter. The known-scatter spherical argument relies on independence between radii and directions. Estimating scatter from the same observations can invalidate that finite-sample argument.
- Calling a discretized permutation test exact at every level. A discrete-uniform rank gives exact size at attainable grid points. At other levels, boundary randomization is needed for exact size.
- Inverting a singular scatter estimate or silently adding a ridge. Singularity prevents the stated standardization, and adding \(\lambda I\) generally sacrifices affine equivariance.
A reliable analysis states the location and scatter procedures, checks positive definiteness, establishes affine equivariance if claimed, and names the calibration argument separately. Exact distribution-free rank calibration, asymptotic calibration, and conditional permutation calibration are different justifications; one should not be substituted for another without checking its assumptions.
Check Your Understanding
Use the assumptions in each question explicitly.
- Why do affine-equivariant location and scatter estimates make standardized residuals differ by an orthogonal transformation under a nonsingular affine change of coordinates?
- Under what conditions does a rank-score statistic based on radial ranks have a null law independent of the radial distribution?
- Why does estimating scatter from the same observations prevent the known-scatter distribution-free argument from applying automatically?
- With 12 equally likely assignments, what conditional sizes are attained by a nonrandomized rank test that rejects for rank at most \(k\)? How could a size of \(0.10\) be obtained exactly?
- Why is adding \(\lambda I\), for positive \(\lambda\), generally not an affine-equivariant solution to singularity?