Nonlinear Alternatives Hidden From Correlation

When Correlation Misses Dependence

Covariance summarizes one particular feature of a joint distribution: whether large values of one variable tend to accompany large or small values of the other in a linear way. Dependence is broader. Two variables can be linked by a curved relationship, or by a relationship whose positive and negative contributions cancel in the covariance. In either case, a zero correlation does not establish independence.

The Gaussian case is a special exception: for jointly Gaussian variables, zero covariance implies independence. Outside that setting, correlation is a limited diagnostic. Here we examine a basic nonlinear pattern, show how transformed covariances can expose it, and identify why even a collection of such summaries must be interpreted carefully.

Definition: If \(X\) and \(Y\) have finite second moments, their cross-covariance is \(\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])]\). If both variances are positive, their correlation is \(\operatorname{Corr}(X,Y)=\operatorname{Cov}(X,Y)/\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}\). Zero covariance means that this particular centered product has expectation zero; it does not, by itself, mean that \(X\) and \(Y\) are independent.

The cancellation is easiest to see when \(X\) is symmetric about zero and \(Y=X^2\). Positive and negative values of \(X\) have the same square. Consequently, contributions to the cross-covariance from opposite signs cancel, even though the value of \(Y\) is determined exactly by \(X\).

A Symmetric Variable Paired With Its Square

Theorem: Suppose \(X\) is symmetric about zero, \(E[|X|^3]<\infty\), and \(X^2\) is not almost surely constant. Set \(Y=X^2\). Then \(\operatorname{Cov}(X,Y)=0\), but \(X\) and \(Y\) are dependent.

Proof. Symmetry and the moment condition give \(E[X]=0\) and \(E[X^3]=0\). Since \(Y=X^2\),

\operatorname{Cov}(X,Y) =E[X^3]-E[X]E[X^2] =0.

For dependence, \(Y\) is a function of \(X\), and it is not almost surely constant. Therefore there is a Borel set \(C\) for which \(0<P(Y\in C)<1\). The event \(A=\{Y\in C\}\) belongs both to the sigma-field generated by \(X\) and to the sigma-field generated by \(Y\). If \(X\) and \(Y\) were independent, these sigma-fields would be independent, so \(A\) would be independent of itself. That would imply \(P(A)=P(A)^2\), which is impossible when \(0<P(A)<1\). Thus \(X\) and \(Y\) are dependent. \(\square\)

Worked Example: A Standard Normal Variable and Its Square

Let \(X\sim N(0,1)\) and \(Y=X^2\). Calculate the ordinary covariance and a covariance involving the squared variable.

For a standard normal variable, \(E[X]=0\), \(E[X^2]=1\), and \(E[X^3]=0\). Hence

\operatorname{Cov}(X,Y) =E[X^3]-E[X]E[X^2] =0.

This zero is not evidence of independence: \(Y\) is determined by \(X\), and is not constant. To see how a nonlinear summary responds, use \(E[X^4]=3\):

\operatorname{Cov}(X^2,Y) =\operatorname{Cov}(X^2,X^2) =E[X^4]-(E[X^2])^2 =3-1^2 =2.

The ordinary covariance misses the symmetric, U-shaped relationship; the covariance of the squared variables detects it.

Transformed Covariances as Targeted Summaries

A general way to look for a specified nonlinear pattern is to transform one or both variables and then calculate a covariance. For functions \(\phi\) and \(\psi\) for which the transformed variables have finite second moments, this summary is

\operatorname{Cov}(\phi(X),\psi(Y)).

Under independence, this covariance is zero whenever it exists. Indeed, independence gives \(E[\phi(X)\psi(Y)]=E[\phi(X)]E[\psi(Y)]\). But a nonzero transformed covariance detects only some departures from independence, and a zero value does not rule out others. Choosing \(\phi(x)=x^2\) is useful when a quadratic pattern is plausible; it does not make the resulting summary a universal dependence measure.

Worked Example: A Four-Point Symmetric Distribution

Suppose \(X\) takes each of \(-2,-1,1,2\) with probability \(1/4\), and let \(Y=X^2\). Compute the covariance, verify dependence directly, and calculate a transformed covariance.

Symmetry gives \(E[X]=0\), and direct calculation gives

E[X^2]=\frac{4+1+1+4}{4}=\frac{5}{2}, \qquad E[X^3]=\frac{-8-1+1+8}{4}=0.

Therefore

\operatorname{Cov}(X,Y) =E[X^3]-E[X]E[X^2] =0.

Yet \(Y\) and \(X\) are dependent. For example, \(P(X=1,Y=1)=1/4\), whereas \(P(X=1)P(Y=1)=(1/4)(1/2)=1/8\). These probabilities differ, so the variables are not independent. Finally, since \(Y=X^2\),

\begin{aligned} E[X^4]&=\frac{16+1+1+16}{4}=\frac{17}{2},\\ \operatorname{Cov}(X^2,Y) &=E[X^4]-(E[X^2])^2 =\frac{17}{2}-\left(\frac{5}{2}\right)^2 =\frac{9}{4}. \end{aligned}

The ordinary cross-covariance is zero, but the transformed covariance is positive.

The same strategy can be applied to transformations chosen for a scientific question: a squared exposure, a threshold indicator, or a nonlinear score. The choice determines which patterns are emphasized. It is important to distinguish this targeted search from a general test of independence.

Worked Example: Quadratic Dependence With Added Noise

Let \(X\sim N(0,1)\), let \(\varepsilon\sim N(0,1)\) be independent of \(X\), and define \(Y=X^2+\varepsilon\). Calculate two covariances and establish that the variables remain dependent despite the noise.

Independence of \(X\) and \(\varepsilon\) gives \(\operatorname{Cov}(X,\varepsilon)=0\). Thus

\operatorname{Cov}(X,Y) =\operatorname{Cov}(X,X^2)+\operatorname{Cov}(X,\varepsilon) =0+0=0.

For the squared transformation, independence also gives \(\operatorname{Cov}(X^2,\varepsilon)=0\), so

\operatorname{Cov}(X^2,Y) =\operatorname{Cov}(X^2,X^2)+\operatorname{Cov}(X^2,\varepsilon) =\operatorname{Var}(X^2) =3-1^2 =2.

To verify dependence, note that \(E[Y\mid X]=X^2\), while \(E[Y]=E[X^2]+E[\varepsilon]=1\). If \(X\) and \(Y\) were independent, then the conditional expectation \(E[Y\mid X]\) would equal \(E[Y]\) almost surely, since \(Y\) is integrable. This would require \(X^2=1\) almost surely, which is false for a standard normal variable. Therefore they are dependent. The independent noise changes the variability of \(Y\), but it does not erase the nonlinear association.

Why No Single Transformed Covariance Is Enough

The examples show that a carefully chosen transformation can reveal structure hidden from ordinary correlation. They do not imply that one can always find dependence by trying a small number of transformations. A chosen transformation may simply be insensitive to the pattern present in the joint distribution.

For real-valued variables, independence has an exact characterization using all pairs of Borel sets. This gives a precise contrast between a universal condition and a finite set of moment checks.

Proposition: Real-valued random variables \(X\) and \(Y\) are independent if and only if, for every pair of Borel sets \(A\) and \(B\),

\operatorname{Cov}\bigl(\mathbf{1}_{\{X\in A\}},\mathbf{1}_{\{Y\in B\}}\bigr)=0.

Proof. The indicators are bounded, so their covariance exists. By the definition of covariance, the displayed condition is equivalent to

P(X\in A,Y\in B)-P(X\in A)P(Y\in B)=0

for every pair \(A,B\). This factorization for all Borel sets is the definition of independence of real-valued random variables. Therefore the condition is both necessary and sufficient. \(\square\)

This characterization involves every pair of sets, not merely one or a few covariances. In applications, a finite feature set can be useful when it represents meaningful alternatives, but it cannot generally replace the full condition without additional assumptions about the model or the alternatives of interest.

From a Nonlinear Summary to a Test

With independent and identically distributed observations \((X_i,Y_i)\), a targeted nonlinear statistic can be formed by computing the sample covariance between transformed observations. For example, when fourth moments are finite, one may calculate

\widehat{\operatorname{Cov}}(X^2,Y) =\frac{1}{n}\sum_{i=1}^n \left(X_i^2-\overline{X^2}\right)(Y_i-\overline{Y}),

where \(\overline{X^2}=n^{-1}\sum_{i=1}^n X_i^2\) and \(\overline{Y}=n^{-1}\sum_{i=1}^n Y_i\). A large magnitude can be evidence against independence when the scientific alternative predicts association between \(X^2\) and \(Y\). The statistic is a targeted diagnostic, not a complete test procedure by itself: a testing rule also needs a null calibration, and its validity depends on the sampling assumptions and moment conditions.

A useful conceptual distinction is that independence implies zero covariance for every admissible pair of transformations, but zero covariance for a selected pair says only that one feature has vanished. A test based on that feature may have power against a particular nonlinear alternative and little power against another. Feature selection should therefore reflect the alternative being investigated, or be supplemented by a method designed to assess broader forms of dependence.

Common Pitfall and Practical Interpretation

Wrong reasoning: “The estimated correlation is near zero, so the variables are independent.” Correction: Near-zero correlation is evidence only about linear association; the square examples have exactly zero population covariance and are dependent.
Wrong reasoning: “A significant covariance between squared variables proves all forms of dependence are present in a known way.” Correction: It identifies association for that chosen transformation. It does not determine the full joint distribution.
Wrong reasoning: “If several transformed covariances are zero, independence follows.” Correction: This conclusion requires a family rich enough to characterize independence, or further model assumptions. A finite list of zero moments is not, in general, that guarantee.
Wrong reasoning: “A sample transformed covariance is automatically a calibrated test.” Correction: A statistic needs a reference distribution or another justified calibration under the null, along with appropriate sampling and moment assumptions.

Key takeaway: Correlation is one moment-based view of association, not a general measure of dependence. Nonlinear transformations can expose specific hidden patterns, but a zero result for any selected finite set of summaries does not by itself establish independence.

Check Your Understanding

Use the definitions and calculations above to answer the following.

For a symmetric \(X\) with a finite third absolute moment, why does \(\operatorname{Cov}(X,X^2)=0\)? Which additional condition in the theorem ensures dependence?
For \(X\sim N(0,1)\) and \(Y=X^2\), calculate \(\operatorname{Cov}(X^2,Y)\). What feature does this reveal that \(\operatorname{Cov}(X,Y)\) misses?
In the four-point example, verify dependence by comparing one joint probability with the product of its marginal probabilities.
Why does adding independent noise in the third example not make \(X\) and \(Y\) independent?
What extra ingredient is needed before a sample transformed covariance can be used as a formal test?