Worked Independence-Test Selection and Analysis

From a Scientific Question to a Test

A test of independence asks whether two random vectors \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) are independent. Choosing a statistic is not just a computational decision: different statistics respond to different forms of association. A test can be correctly calibrated and still have little power against the dependence pattern that matters in an application.

A defensible analysis therefore connects four decisions: the alternatives that matter, the statistic’s sensitivity to those alternatives, the calibration justified by the design, and the limits of the conclusion. Cross-covariance, distance covariance, and HSIC offer different approaches. Their established properties provide a starting point, but no single statistic is automatically best for every problem.

The population cross-covariance matrix is \(\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])^T]\), when the required moments exist. Distance covariance and HSIC instead use distances or kernels to capture broader distributional features. Under their respective regularity conditions, distance covariance and HSIC with characteristic kernels can detect general forms of dependence; the earlier material on distance covariance and HSIC gives their definitions and consistency conditions.

What Cross-Covariance Can and Cannot Establish

A basic implication helps explain the limitation of cross-covariance. If \(X\) and \(Y\) are independent and have finite second moments, their population cross-covariance is zero. The converse is generally false.

Proof. Write \(\mu_X=E[X]\) and \(\mu_Y=E[Y]\). Finite second moments ensure that each coordinate product below is integrable, by the Cauchy–Schwarz inequality. For every coordinate pair \(j,k\), independence gives \(E[(X_j-\mu_{X,j})(Y_k-\mu_{Y,k})]=E[X_j-\mu_{X,j}]E[Y_k-\mu_{Y,k}]=0\). Thus every entry of the cross-covariance matrix is zero. \(\square\)

Calculate the Statistic Before Calibrating It

For paired observations, an empirical cross-covariance matrix can be defined with divisor \(n\) as \(\widehat C_{XY}=n^{-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)^T\). This convention is useful for defining a test statistic; a sample covariance using divisor \(n-1\) is a different scaling. In either case, centering matters. The uncentered quantity \(n^{-1}\sum_i X_iY_i^T\) is a cross-moment, not generally a cross-covariance.

For a matrix-valued statistic, a scalar test statistic might be a norm such as \(\|\widehat C_{XY}\|_F\). But scaling variables differently can change that norm, and a few large-scale coordinates can dominate it. Standardization may help when scientifically appropriate, but estimating scale parameters can also affect calibration. The choice should be specified and justified rather than made after inspecting which version gives the smallest p-value.

Calibration: Match the Reference Distribution to the Design

A statistic’s null distribution is not determined by its formula alone. For example, under independence, appropriately exchangeable paired observations permit a permutation test: keep the \(X\) observations fixed, permute the \(Y\) observations among them, and recalculate the statistic. This tests whether the observed pairings are unusually associated relative to pairings allowed under the null. It is not valid to permute freely when the study design restricts which observations can be exchanged; in an exchangeable design, the appropriate restricted permutations must respect those restrictions.

Proof. Under the null, the joint distribution of all observed \(X\) and \(Y\) values is invariant to permuting the \(Y\) labels: independence separates the \(X\) sample from the \(Y\) sample, and the identically distributed \(Y_i\) are exchangeable. Conditional on the observed values, the observed matching therefore has the same distribution as a uniformly selected matching from its permutation orbit. In the absence of ties, the rank of the observed statistic among the \(n!\) permuted statistics is uniform, so the fraction of statistics at least as large as the observed one is a valid upper-tail p-value. With ties, counting all statistics equal to the observed value in the upper tail can only increase this fraction, making the test conservative. Hence \(P(p\leq\alpha\mid\text{observed values})\leq\alpha\), and averaging over the observed values gives the unconditional level bound. \(\square\)

When enumerating all permutations is impractical, a Monte Carlo permutation procedure approximates the reference distribution; its p-value calculation must account for the finite number of sampled permutations. Other calibrations may be appropriate for other settings. For instance, parametric bootstrap calibration requires a credible parametric model and must account for estimated margins when those are part of the procedure. Distance and kernel statistics can have degenerate null limits, so using an ordinary normal or chi-square reference distribution without justification can give incorrect type-I error. The earlier material on degenerate null calibration addresses this issue.

Sensitivity, Selection, and the Reported Conclusion

A practical selection process begins with the scientific alternative, not with a search for the smallest p-value. If linear cross-association is the target, a cross-covariance statistic may give a direct and interpretable test. If nonlinear or more general dependence is important, distance covariance or HSIC may be better aligned with the question. Kernel choice and bandwidth can influence HSIC’s finite-sample behavior, so prespecified alternatives or a justified sensitivity analysis are more informative than silently reporting whichever choice rejects.

Sensitivity analysis asks whether the substantive conclusion changes under reasonable, defensible choices: for example, alternative kernel bandwidths, distance specifications, or scaling conventions. It does not make an invalid calibration valid. If the analysis selects a statistic, bandwidth, or preprocessing rule after examining the same data, the nominal p-value for the selected procedure may no longer control the type-I error. Prespecification, an appropriate multiplicity adjustment, or genuinely separate data for selection and testing can address this concern.

The report should name the test statistic, explain why its sensitivity fits the scientific question, identify the calibration and the design assumptions that justify it, and give the p-value and chosen significance level. A rejection is evidence against independence under the stated procedure; it does not identify a causal direction or establish the form or practical importance of the dependence. A nonrejection means the data did not provide sufficient evidence against independence with this test, sample size, and calibration. It does not prove independence, especially when the test may be insensitive to relevant alternatives.

Common Mistakes

• Equating zero cross-covariance with independence. Zero cross-covariance is necessary under independence with finite second moments, but the worked nonlinear example shows it is not sufficient.
• Using an uncentered cross-moment as though it were a covariance. Subtract the sample means in the empirical covariance, or state explicitly that the uncentered quantity is being used.
• Permuting observations that the design does not permit exchanging. Unrestricted permutations require exchangeability under the null; use design-respecting permutations when exchangeability is restricted.
• Treating every calibrated test as equally sensitive. Calibration controls false positives under its assumptions; it does not guarantee power against every dependence pattern.
• Interpreting nonrejection as proof of independence. State the evidence and its limits, including the statistic, calibration, and relevant sensitivity checks.