Comparing Distance and Kernel Tests Across Alternatives

Why Compare the Tests Through Alternatives?

Distance covariance and the Hilbert–Schmidt independence criterion (HSIC) are both general tools for detecting dependence, but they do not weight every feature of a joint distribution in the same way. Their behavior depends on the distance used to construct one statistic and on the kernel and bandwidth used to construct the other. A test that is effective for a smooth, monotone association need not be the most sensitive choice for a rapidly oscillating relationship or for rare, extreme observations.

The comparison is about which aspects of dependence receive emphasis, not a universal ranking. For characteristic choices, both population statistics vanish exactly under independence. That shared independence characterization does not mean they have equal finite-sample power, equal response to all alternatives, or interchangeable numerical scales.

Frequency Weights and the Choices They Represent

Definition: Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) have finite first moments. Write \(\varphi_{XY}(t,s)\) for their joint characteristic function and \(\varphi_X(t)\), \(\varphi_Y(s)\) for their marginal characteristic functions. Define the dependence discrepancy \(\Delta(t,s)=\varphi_{XY}(t,s)-\varphi_X(t)\varphi_Y(s)\). For Euclidean distance, the squared population distance covariance has the representation \[ \mathcal{V}^2(X,Y)=\frac{1}{c_p c_q}\int_{\mathbb{R}^p}\int_{\mathbb{R}^q} \frac{|\Delta(t,s)|^2}{\|t\|^{p+1}\|s\|^{q+1}}\,ds\,dt, \qquad c_d=\frac{\pi^{(d+1)/2}}{\Gamma((d+1)/2)}. \] The integrand is understood away from the zero-frequency sets, which have Lebesgue measure zero.

This representation shows that distance covariance measures a weighted discrepancy over frequencies. Its Euclidean-distance weight is fixed once the coordinate scale and Euclidean metric are fixed. Rescaling a coordinate changes the distances and can therefore change the statistic’s emphasis. Other distance constructions can change that emphasis too; comparisons must specify the metric rather than treating “distance covariance” as independent of all such choices.

For HSIC, suppose the kernels are bounded, continuous, and stationary, with representations \(k_X(x,x')=\int e^{it^\top(x-x')}\,\Lambda_X(dt)\) and \(k_Y(y,y')=\int e^{is^\top(y-y')}\,\Lambda_Y(ds)\), where the spectral measures are finite and symmetric. Then \[ \operatorname{HSIC}(X,Y)=\int |\Delta(t,s)|^2\,\Lambda_X(dt)\Lambda_Y(ds). \] Thus the kernels select the frequency weights. For Gaussian kernels \(k_\sigma(x,x')=\exp(-\|x-x'\|^2/(2\sigma^2))\), the spectral measure is Gaussian with covariance matrix \(\sigma^{-2}I\). A characteristic kernel gives positive spectral weight throughout frequency space, so a nonzero dependence discrepancy is not ignored altogether; nevertheless, the weights can make some discrepancies contribute much less than others.

Worked Examples Across Three Alternatives

Worked Example: Monotone Gaussian Dependence

Let \(X,Z\) be independent standard normal random variables and set \(Y=\rho X+\sqrt{1-\rho^2}\,Z\), where \(0<|\rho|<1\). Both marginal variances are one, and \(\operatorname{Cov}(X,Y)=\rho\). This is a smooth monotone association: for positive \(\rho\), larger \(X\) tends to accompany larger \(Y\), and for negative \(\rho\), the direction is reversed.

The joint characteristic function and the product of the marginal characteristic functions are \(\varphi_{XY}(t,s)=\exp[-(t^2+s^2+2\rho ts)/2]\) and \(\varphi_X(t)\varphi_Y(s)=\exp[-(t^2+s^2)/2]\). At \(t=s=u\ne 0\), their difference is \(e^{-u^2(1+\rho)}-e^{-u^2}\), which is nonzero because \(\rho\ne0\). By continuity, the discrepancy is nonzero on a region around that frequency pair. The Euclidean distance-covariance weight is positive there, so distance covariance is positive.

The same reasoning applies to Gaussian-kernel HSIC: its spectral measures give positive weight to every frequency region, so its integral is positive as well. For comparison, HSIC with linear kernels \(k_X(x,x')=xx'\) and \(k_Y(y,y')=yy'\) equals \(\operatorname{Cov}(X,Y)^2=\rho^2\). That linear-kernel version directly captures this association, but it is not a general independence test: zero covariance does not rule out nonlinear dependence.

Worked Example: Oscillatory Dependence With Zero Covariance

Let \(X\) be uniform on \([-\pi,\pi]\) and set \(Y=\cos(2X)\). Since \(X\) has mean zero and \(x\cos(2x)\) is odd, \(\operatorname{Cov}(X,Y)=E[X\cos(2X)]=0\). Nevertheless, \(Y\) is not independent of \(X\): conditional on \(X=x\), \(Y\) is exactly \(\cos(2x)\), while its marginal variance is positive. In particular, \(E[Y^2]=1/2\), since the average of \(\cos^2(2x)\) over this interval is \(1/2\).

Linear-kernel HSIC is therefore zero, despite the dependence. This is a limitation of that kernel for this alternative, not a failure of HSIC with every kernel. Both variables are bounded, so the finite-first-moment condition for Euclidean distance covariance holds; because the variables are dependent, its characteristic-function discrepancy is nonzero somewhere and its integral is positive. Gaussian kernels on both variables are characteristic as well, so their population HSIC is positive.

The association oscillates rather than following a single increasing or decreasing trend. A kernel with a very broad Gaussian bandwidth can initially emphasize its linear covariance, as the derivation below makes precise. That does not imply its HSIC is exactly zero at every finite bandwidth: higher-order terms can still register the oscillatory dependence.

Worked Example: Rare, Large Concordant Values

Let \(B\) be Bernoulli with \(P(B=1)=\varepsilon\), where \(0<\varepsilon<1\), and let \(X=Y=MB\) for \(M>0\). The pair takes the value \((M,M)\) only with probability \(\varepsilon\); otherwise it is \((0,0)\). This simple model isolates the effect of a rare, large observation.

For independent copies \(X',Y'\), the Euclidean distance-covariance formula in one dimension is \(\mathcal{V}^2=E[|X-X'||Y-Y'|]+E|X-X'|E|Y-Y'|-2E[|X-X'||Y-Y''|]\), where the primed variables in the last term are independent copies and \(X=Y\) in each pair. The distance \(|X-X'|\) is \(M\) with probability \(2\varepsilon(1-\varepsilon)\) and zero otherwise. Hence the first term is \(2\varepsilon(1-\varepsilon)M^2\), and the second is \(4\varepsilon^2(1-\varepsilon)^2M^2\).

For the last expectation, condition on \(X\). The conditional mean distance to an independent copy is \(\varepsilon M\) when \(X=0\) and \((1-\varepsilon)M\) when \(X=M\). Because \(X'\) and \(Y''\) are conditionally independent given \(X\), the expectation is \((1-\varepsilon)(\varepsilon M)^2+\varepsilon((1-\varepsilon)M)^2=\varepsilon(1-\varepsilon)M^2\). Substitution gives \(\mathcal{V}^2=4\varepsilon^2(1-\varepsilon)^2M^2\).

Now use Gaussian kernels with the same bandwidth \(\sigma\) for \(X\) and \(Y\). Put \(r=\exp[-M^2/(2\sigma^2)]\), the kernel value between 0 and \(M\). In the kernel feature space, the two possible feature vectors differ by a vector \(\delta\) with squared norm \(\|\delta\|^2=2(1-r)\). The covariance operator of the feature vector with itself is \(\varepsilon(1-\varepsilon)\delta\otimes\delta\); its squared Hilbert–Schmidt norm is therefore \(\operatorname{HSIC}=4\varepsilon^2(1-\varepsilon)^2(1-r)^2\).

For fixed \(\sigma\), this HSIC is bounded as \(M\) grows, while \(\mathcal{V}^2\) grows proportionally to \(M^2\). This compares how the two population formulas respond to the amplitude, not which statistic has greater power: their scales differ, and finite-sample calibration and variability also matter. The example does show why a bounded kernel can limit the direct influence of extreme pairwise distances, while Euclidean distance retains their magnitude.

Why a Broad Gaussian Bandwidth Can Miss the Leading Signal

A broad Gaussian kernel varies only slightly across the observed range. On bounded scalar data, its centered version has a leading term proportional to the product of centered observations. Consequently, the leading term of HSIC is determined by squared covariance. The derivation explains why a large bandwidth can be poorly matched to a nonlinear alternative whose covariance is zero.

Result: Suppose \(X\) and \(Y\) are bounded scalar random variables and use Gaussian kernels with the same bandwidth \(\sigma\). As \(\sigma\to\infty\), \[ \operatorname{HSIC}_\sigma(X,Y)=\frac{\operatorname{Cov}(X,Y)^2}{\sigma^4}+O(\sigma^{-6}). \]

Proof. Write \(\tilde k_\sigma(x,x')\) for the kernel centered with respect to the distribution of \(X\). Boundedness of \(X\) makes the Taylor remainder in \(e^{-(x-x')^2/(2\sigma^2)}=1-(x-x')^2/(2\sigma^2)+O(\sigma^{-4})\) uniform over possible \(x,x'\). Centering removes the constant term. Centering the quadratic term gives \(\widetilde{-(x-x')^2/2}=(x-E[X])(x'-E[X])\). Thus \(\tilde k_\sigma(X,X')=(X-E[X])(X'-E[X])/\sigma^2+O(\sigma^{-4})\), uniformly; the corresponding expansion for \(Y\) holds too.

HSIC is \(E[\tilde k_\sigma(X,X')\tilde k_\sigma(Y,Y')]\), where \((X',Y')\) is an independent copy of \((X,Y)\). Multiplying the leading terms and using independence of the copies gives \(E[(X-E[X])(Y-E[Y])]^2/\sigma^4=\operatorname{Cov}(X,Y)^2/\sigma^4\). The products of a leading term of order \(\sigma^{-2}\) and a remainder of order \(\sigma^{-4}\) are uniformly of order \(\sigma^{-6}\); the product of both remainders is of smaller order. This proves the expansion. \(\square\)

For the oscillatory example, the leading term is zero because the covariance is zero. This is a statement about the large-bandwidth expansion, not a claim that the kernel test cannot detect the dependence at a suitable finite bandwidth. With multivariate data, coordinate scaling and separate bandwidths by coordinate add further choices: they change which directions and scales receive appreciable weight.

Choosing a Comparison Without Declaring a Winner

The examples suggest a practical way to compare methods before analysis. First describe the alternative of interest: smooth monotone association, oscillation, localized nonlinear structure, or rare extreme co-occurrence. Then ask which distances or frequency ranges are emphasized by the chosen metric and kernel. For distance covariance, the Euclidean metric’s coordinate scales matter. For HSIC, bandwidth and kernel family govern the spectral weighting; a characteristic choice ensures broad population coverage but does not make all patterns equally easy to detect in a finite sample.

Tail behavior also affects which definitions and computations are available. Standard Euclidean distance covariance requires finite first moments. A bounded kernel can define HSIC without those moment conditions, although that fact alone does not guarantee high power for a particular tail alternative. For heavy-tailed data, check the conditions for the statistic and its calibration rather than inferring validity from a population formula alone.

Common pitfall: “Both tests are consistent, so their power must be similar for every dependence pattern.” This confuses an independence characterization with finite-sample sensitivity. Characteristic choices can detect every fixed dependent alternative at the population level, yet their weights, bandwidths, and sampling variability can differ substantially. Compare specified procedures under specified alternatives; do not infer a universal ordering from consistency.

The central lesson is that test choice is partly a modeling choice about scale and structure. Distance covariance and HSIC provide general dependence measures, but a useful comparison names the metric, kernel, bandwidth, alternative, and calibration. A result under one such choice should not be treated as a property of every version of that method.

Check Your Understanding

Use the formulas and examples above to test your understanding.

What is the dependence discrepancy \(\Delta(t,s)\), and how do the two statistics weight it differently?
Why does zero covariance make linear-kernel HSIC zero for the oscillatory example without implying independence?
In the rare-extreme example, what happens to squared distance covariance as \(M\) increases with \(\varepsilon\) fixed? What happens to Gaussian-kernel HSIC at fixed \(\sigma\)?
What does the large-bandwidth expansion say when \(\operatorname{Cov}(X,Y)=0\), and what does it not establish about every finite bandwidth?
Why does a characteristic kernel not imply that a test has equal power against all alternatives?