Sample-Size and Dimension Diagnostics

Why Sample Size and Dimension Must Be Diagnosed Together

A multivariate independence test can be correctly implemented and still have little ability to detect dependence. The problem may be limited sample size, a high-dimensional representation in which informative structure is diluted, or a tuning choice that makes pairwise similarities nearly indistinguishable. A fourth possibility is that the observations are dependent in a way the calibration does not accommodate. These problems can all produce few rejections, but they require different remedies.

The central diagnostic task is therefore not simply to ask whether \(n\) is “large” or whether the dimension is “high.” Instead, examine the geometry of the observed data, the scale used by the statistic, the amount of independent information, and the test’s behavior under the null. The diagnostics below apply to distance-covariance and kernel-based procedures without replacing their specific test statistics or calibration schemes.

Distance Concentration as Dimension Grows

In high dimensions, pairwise Euclidean distances can become similar in relative terms. This does not mean every distance is identical, nor does it imply that every distance-based procedure fails. It does mean that a statistic relying on variation among distances may have less geometric contrast to work with, especially when informative coordinates are mixed with many irrelevant ones.

A simple reference calculation uses independent \(X,X'\sim N(0,I_p)\). Their difference has distribution \(X-X'\sim N(0,2I_p)\), so the squared distance is a sum of \(p\) independent squared normal coordinates. Its relative variability decreases with \(p\).

Proof. Write \(X-X'=(Z_1,\ldots,Z_p)\), where the \(Z_j\) are independent \(N(0,2)\) variables. For each coordinate, \(E[Z_j^2]=2\) and \(\operatorname{Var}(Z_j^2)=2(2^2)=8\), using the variance \(2v^2\) of the square of a \(N(0,v)\) variable. Independence gives \[ E\left[\sum_{j=1}^p Z_j^2\right]=2p,\qquad \operatorname{Var}\left(\sum_{j=1}^p Z_j^2\right)=8p. \] The coefficient of variation is the standard deviation divided by the mean, hence \(\sqrt{8p}/(2p)=\sqrt{2/p}\). \(\square\)

This is a diagnostic, not a universal theorem about test power. Scaling coordinates differently, using a non-Euclidean metric, or concentrating dependence in a small subset of coordinates changes the relevant geometry. In practice, inspect pairwise-distance distributions within each margin, their spread relative to their typical scale, and whether a small number of variables dominate the distances.

Kernel Bandwidth and the Scale of Pairwise Distances

For a Gaussian radial kernel, use the convention \(k_{\sigma}(x,x')=\exp\{-\|x-x'\|^2/(2\sigma^2)\}\). If \(\sigma^2\) is very small relative to typical squared distances, most distinct pairs receive similarities near zero. If it is very large, similarities are near one and may vary little across pairs. Either extreme can reduce useful contrast. The relevant scale depends on the data’s units and dimension, so a bandwidth value has no meaning in isolation.

Derivation. The coordinate differences \(Z_j=X_j-X'_j\) are independent \(N(0,2)\). For \(Z\sim N(0,2)\), direct integration of its density gives \(E[\exp\{-Z^2/(2\sigma^2)\}]=(1+2/\sigma^2)^{-1/2}\). The kernel factorizes across coordinates: \[ k_{\sigma}(X,X')=\prod_{j=1}^p \exp\{-Z_j^2/(2\sigma^2)\}. \] Independence therefore makes the expectation the product of \(p\) identical expectations, giving the stated expression. \(\square\)

A practical bandwidth diagnostic reports the distribution of off-diagonal kernel values, not just the selected bandwidth. If nearly all values are indistinguishable near zero or near one, investigate rescaling, robust bandwidth choices, or a prespecified range of scales. Any tuning performed using the data must also be reflected in the inferential procedure; a calibration that treats a data-selected bandwidth as fixed may not have the claimed null behavior.

Effective Sample Size and Dependence Among Observations

The nominal sample size counts rows; the effective sample size describes how much independent information a particular estimator receives. It depends on the dependence structure and the estimand, so there is no single universally correct replacement for \(n\) in every multivariate test. Two useful calculations are available for common designs.

For \(G\) independent clusters, each containing \(m\) observations, suppose a scalar observation has variance \(\tau^2\), and within each cluster every distinct pair has covariance \(\rho\tau^2\). The variance of the overall sample mean follows directly from summing all variances and covariances.

Proof. The variance of the sum within one cluster is \(m\tau^2+m(m-1)\rho\tau^2\): there are \(m\) individual variances and \(m(m-1)\) ordered covariance terms. Independent clusters add their variances, so \[ \operatorname{Var}(\bar Z) =\frac{Gm\tau^2+Gm(m-1)\rho\tau^2}{(Gm)^2} =\frac{\tau^2}{Gm}\{1+(m-1)\rho\}. \] Equating this variance to \(\tau^2/n_{\mathrm{eff}}\) yields the stated effective sample size. \(\square\)

For a stationary scalar sequence with variance \(\tau^2\) and autocorrelation \(\rho_k\) at lag \(k\), direct expansion gives \(\operatorname{Var}(\bar Z)=\tau^2 n^{-1}[1+2\sum_{k=1}^{n-1}(1-k/n)\rho_k]\). When the autocorrelations are summable and \(n\) is sufficiently large, this motivates the approximation \(n_{\mathrm{eff}}\approx n/[1+2\sum_{k\ge1}\rho_k]\). It is an approximation for the mean under those dependence conditions, not a generic effective sample size for every multivariate statistic.

Separate Weak Signal From Poor Calibration

A low rejection rate under an alternative can reflect low power, but first establish that the null calibration is trustworthy. For a composite null, size is the worst-case rejection probability across null parameter values. In a simulation at one chosen null configuration, report the rejection rate there and its Monte Carlo uncertainty; do not label it the size unless the relevant supremum has been controlled.

For \(B\) independent simulation replicates at a fixed configuration, if \(R\) replicates reject, the estimated rejection rate is \(\hat r=R/B\). Its Monte Carlo standard error is approximately \(\sqrt{\hat r(1-\hat r)/B}\), from the binomial variance. A rate near the target level at selected null configurations is evidence about calibration, but does not prove uniform size control. For permutation procedures, verify that the permutations preserve the design’s exchangeability structure; unrestricted permutation of clustered or serially dependent observations may invalidate the reference distribution.

A disciplined investigation changes one feature at a time where possible. Compare power across sample sizes while holding the alternative and analysis fixed; inspect distances and kernel values as dimension changes; evaluate more than one scientifically reasonable bandwidth; and repeat null calibration under the actual sampling design. This separates a test with little sensitivity to a particular signal from a test whose rejection threshold is not behaving as intended.

Common Pitfall and Practical Interpretation

A common mistake is to see few rejections in a high-dimensional study and conclude immediately that the variables are nearly independent. The reasoning is incomplete: weak dependence is one possibility, but concentrated distances, a poorly scaled bandwidth, correlated observations, or a miscalibrated reference distribution can lead to the same pattern. The correction is to diagnose geometry, tuning, independent information, and null behavior separately before interpreting power.

These checks matter because a dependence test’s result is conditional on how the data were represented, how its scale was chosen, and how its reference distribution was obtained. Reporting those details makes a negative result interpretable: readers can see whether the study had a meaningful chance to detect the dependence pattern of interest, rather than treating nonrejection as evidence that dependence is absent.