Common Calibration and Interpretation Errors

Why Calibration and Interpretation Fail

A dependence test is not just a statistic followed by a threshold. Its conclusion relies on a particular null hypothesis, a sampling or randomization structure, and a reference distribution that is valid under that structure. A calculation can be flawless and still answer the wrong question if the pairs were broken, the permutations were not allowed by the design, or the p-value was treated as an effect-size measure.

The central distinction is between evidence against a null hypothesis and the magnitude or form of dependence. A small p-value can be compelling evidence against independence without telling us whether the dependence is practically important, linear, nonlinear, or useful for prediction. Conversely, zero correlation does not establish independence.

Definition: Random variables \(X\) and \(Y\) are independent if their joint distribution is the product of their marginal distributions; equivalently, \(P(X\in A,Y\in B)=P(X\in A)P(Y\in B)\) for all measurable sets \(A\) and \(B\). If \(X\) and \(Y\) have finite second moments, they are uncorrelated when \(\operatorname{Cov}(X,Y)=0\). Independence implies zero covariance under these moment conditions, but the converse is not generally true.

Keep the Observation Pairs Intact

For paired observations \((X_i,Y_i)\), the pairing is part of the data. An independence statistic compares features of \(X\) with features of \(Y\) across the same observational units. Reordering the rows of a data table changes nothing, because both measurements move together. Reordering just one variable generally changes the observed association and is a randomization operation that requires justification; it is not a harmless data-cleaning step.

Worked Example: Breaking Pairs Changes the Association

Suppose the three observed pairs are \((0,0)\), \((1,1)\), and \((2,2)\). Calculate the sample covariance, then keep the \(X\)-values fixed and reorder the \(Y\)-values as \(1,2,0\).

In the original data, both sample means are 1. The sum of centered cross-products is \( (-1)(-1)+(0)(0)+(1)(1)=2\), so the sample covariance, using denominator \(n-1=2\), is \(2/2=1\).

After reordering only \(Y\), its mean is still 1, but the centered \(Y\)-values are \(0,1,-1\). The cross-product sum is \( (-1)(0)+(0)(1)+(1)(-1)=-1\), giving sample covariance \(-1/2\).

The marginal values and their means have not changed, but the paired association has changed from positive to negative. Thus, independently sorting or shuffling one column destroys the observed pairing. If both columns are moved together by the same row permutation, every pair is preserved and the covariance remains 1.

When Is a Permutation Calibration Valid?

An unrestricted permutation test for independence uses the idea that, under the null, the observed matching between the \(X\)-values and \(Y\)-values is no more special than the other matchings. That symmetry needs assumptions. In particular, iid paired observations under independence give iid \(X_i\)'s and iid \(Y_i\)'s, with the two sample vectors independent. Design features such as clusters, time trends, repeated measures, or strata can remove the needed exchangeability. A permutation scheme must reflect the null hypothesis and the design; “shuffle the data” is not a general calibration rule.

Theorem: Suppose \((X_i,Y_i)\), \(i=1,\ldots,n\), are iid pairs and \(X_i\) is independent of \(Y_i\). Suppose also that the marginal distribution of \(Y_i\) is atomless, so the \(Y_i\)'s have no ties almost surely. Conditional on the \(X\)-values and the unordered \(Y\)-values, each of the \(n!\) matchings of the \(Y\)-values to the \(X\)-values is equally likely. Consequently, for any statistic \(T\), the permutation upper-tail p-value \(p=(1/n!)\sum_{\pi} \mathbf{1}\{T(X_1,\ldots,X_n;Y_{\pi(1)},\ldots,Y_{\pi(n)})\ge T_{\mathrm{obs}}\}\) is valid under this null: \(P(p\leq\alpha)\leq\alpha\) for every \(\alpha\in[0,1]\). No density assumption is required.

Proof. Under the null, the \(X\)-sample vector is independent of the \(Y\)-sample vector. Because the \(Y_i\)'s are iid, their joint distribution is invariant under every permutation. Since they have no ties almost surely, each ordering of their unordered values corresponds to one distinct ordering. Therefore, conditional on the \(X\)-values and unordered \(Y\)-values, every ordering is equally likely. In particular, the observed matching is uniform over the \(n!\) candidates used to calculate the permutation distribution. This conclusion follows from symmetry and absence of ties; it does not require a joint or marginal density.

To see the p-value bound, hold the conditioned data fixed and list the \(M=n!\) permutation statistics, counting each labeled permutation, including repetitions of statistic values. If an observed statistic has upper-tail count \(r\), its p-value is \(r/M\). At most \(\lfloor \alpha M\rfloor\) of the \(M\) positions can have upper-tail count at most \(\alpha M\): positions tied at the same statistic value have the same tail count, and ties cannot increase the number of positions passing the cutoff. The observed position is uniform, so its conditional probability of satisfying \(p\leq\alpha\) is at most \(\lfloor\alpha M\rfloor/M\leq\alpha\). Averaging over the conditioned data gives \(P(p\leq\alpha)\leq\alpha\). \(\square\)

Atomless means that individual values have probability zero; it does not mean that a density exists. The symmetry argument above therefore applies to continuous distributions that may be singular. With tied observations, one can still permute labeled indices and use the corresponding permutation distribution, with ties counted consistently; the p-value is generally conservative. The important point is that the exchangeability argument, not a density calculation, justifies the calibration.

Worked Example: A Stratified Null Is Not an Unrestricted Null

Let \(Z\) indicate one of two equally common strata, with \(P(Z=1)=P(Z=0)=1/2\). Within each stratum, let \(X=10Z+\varepsilon_X\) and \(Y=10Z+\varepsilon_Y\), where \(\varepsilon_X\) and \(\varepsilon_Y\) are independent \(N(0,1)\) variables, also independent of \(Z\). Within either stratum, \(X\) and \(Y\) are independent: the only random parts are independent errors.

Pooled across strata, however, \(\operatorname{Cov}(X,Y)=\operatorname{Cov}(10Z+\varepsilon_X,10Z+\varepsilon_Y)=100\operatorname{Var}(Z)=100(1/4)=25\). The error terms contribute zero covariance, and their covariances with \(Z\) are zero. Thus the pooled variables are not independent, despite independence within each stratum.

An unrestricted test of pooled independence is addressing a different null from a test of independence within strata. Permuting \(Y\)-values across the two groups also disregards their different group distributions. If the scientific target is conditional independence given \(Z\), the analysis must preserve or condition on the strata—for example, by using a justified within-stratum procedure. The correct restriction depends on the design and the null being tested; it cannot be chosen merely because it produces a convenient reference distribution.

State the null precisely.
Distinguish pooled independence, independence conditional on covariates or strata, and a design-based null such as no treatment effect.

Identify what is exchangeable.
Check which observations could legitimately be relabeled under that null, accounting for clustering, repeated measurements, and the sampling scheme.

Match the calibration.
Use a reference distribution justified for the statistic and design. An unrestricted permutation, restricted permutation, parametric bootstrap, or asymptotic calibration is not interchangeable without assumptions.

Zero Correlation Does Not Mean Independence

Correlation summarizes a particular aspect of linear association. With finite second moments, independence implies zero covariance, because \(E[XY]=E[X]E[Y]\). But covariance can cancel across a nonlinear relationship. Dependence tests based on distance or kernels can detect forms of association that correlation does not capture, subject to the assumptions of the chosen method.

Worked Example: A Dependent Pair With Zero Covariance

Let \(X\sim\operatorname{Uniform}(-1,1)\) and set \(Y=X^2\). Since the distribution of \(X\) is symmetric, \(E[X]=0\); direct integration gives \(E[Y]=E[X^2]=1/3\) and \(E[XY]=E[X^3]=0\). Therefore \(\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]=0-(0)(1/3)=0\).

Nevertheless, \(Y\) is completely determined by \(X\), so the variables are dependent. For an explicit probability check, define \(A=\{|X|\leq1/2\}\) and \(B=\{Y\leq1/4\}\). These events are identical because \(Y=X^2\). Hence \(P(A)=P(B)=1/2\) and \(P(A\cap B)=1/2\), whereas \(P(A)P(B)=1/4\). The defining factorization for independence fails.

Common mistake: “The estimated correlation is near zero, so there is no dependence.” This treats a linear summary as a complete description of the joint distribution. The correction is to state only what the statistic supports: near-zero correlation is evidence of little linear association in the measured sample, not proof of independence. A suitable independence test can assess a broader null, but its own power, calibration, and target must still be considered.

A Small P-Value Is Not a Strength Estimate

A p-value is a probability calculated under a specified null and calibration: it measures how often the test would produce a statistic at least as extreme as the observed one if that null and its assumptions held. It is not the probability that the null is true, and it is not a numerical estimate of dependence strength. Small p-values can result from modest departures detected with large samples; large p-values can occur with substantial departures when samples are small or the test has little power against that pattern.

Worked Example: The Same Perfect Rank Association, Different P-Values

Consider the two-sided exact permutation test based on Spearman’s rank correlation, using the absolute rank correlation as the test statistic. In both samples below, the ranks of \(X\) and \(Y\) agree perfectly: the observed rank correlation is \(1\), its maximum possible absolute value.

For \(n=4\) distinct observations, there are \(4!=24\) matchings of the \(Y\)-ranks to the \(X\)-ranks. Only the identity matching gives rank correlation \(1\), and only the reverse matching gives rank correlation \(-1\). Thus the exact two-sided p-value is \(2/24=1/12\), approximately \(0.0833\).

For \(n=8\), the same perfect rank association has \(8!=40{,}320\) possible matchings, and again only the identity and reverse matchings attain absolute rank correlation 1. The exact p-value is \(2/40{,}320=1/20{,}160\), approximately \(0.0000496\).

The observed rank association is equally strong in both samples, but the p-values differ because the null reference distributions and sample sizes differ. This calculation does not say that p-values are uninformative: the smaller value is stronger evidence against the permutation null. It does show why a p-value cannot serve as an effect-size estimate. Report an appropriate measure of dependence and uncertainty alongside the test result, and describe what the test can detect.

Practical Checks Before Reporting

A careful analysis separates four questions. First, what is the scientific null: pooled independence, conditional independence, or something else? Second, are the observational pairs and any design restrictions represented correctly? Third, is the reference distribution valid for the statistic under that null? Fourth, what does the observed statistic measure, and what does it leave out?

A test result should be reported with the test statistic, calibration method, p-value, and relevant sample or design details. Where useful, add a dependence summary suited to the scientific question and explain its limitations. Do not describe a non-significant result as proof of independence, or a significant result as evidence of a large or important effect. Those conclusions require information beyond the p-value.

Key takeaway: Preserve the pairs, permute only what the null and design make exchangeable, distinguish zero correlation from independence, and interpret a p-value as calibrated evidence against a stated null—not as a dependence-strength estimate.

Check Your Understanding

For each question, identify the assumption or interpretation that needs attention.

Why does shuffling the rows of a paired data table preserve the observed association, while shuffling one column alone changes it?
In the permutation-test theorem, why does symmetry of an iid sample suffice even when the marginal distribution has no density?
In the stratified example, what null is tested by pooled independence, and why is it different from independence within strata?
Give the event-based argument showing that \(Y=X^2\) is dependent on \(X\), even though their covariance is zero.
Two studies have the same estimated dependence summary but different sample sizes and p-values. Why is it incorrect to call the study with the smaller p-value “more strongly dependent” on that basis alone?