Jacobian Covariance Transformations

Why the Jacobian Transforms Covariance

A vector estimate can have uncertainty in several directions at once. When we report smooth functions of its coordinates, the uncertainty in those reported quantities depends not only on each input variance, but also on how the transformation locally stretches, shrinks, and mixes directions. The Jacobian records precisely this local linear behavior.

The central rule is that a first-order covariance matrix transforms as \(J\Sigma J^{\mathsf T}\), where \(\Sigma\) is the input covariance matrix and \(J\) is the Jacobian of the transformation. The order matters: the Jacobian acts on the input vector on the left, and its transpose acts on the right. This tutorial derives that rule for exact linear transformations and for asymptotic normal limits.

Definition: For a random vector \(X=(X_1,\ldots,X_p)^{\mathsf T}\) with finite second moments, its covariance matrix is the \(p\) by \(p\) matrix \(\operatorname{Cov}(X)=E[(X-E[X])(X-E[X])^{\mathsf T}]\). Its \((i,j)\) entry is \(\operatorname{Cov}(X_i,X_j)\). For a differentiable function \(g:\mathbb R^p\to\mathbb R^q\), the Jacobian at \(\theta\) is the \(q\) by \(p\) matrix \(J_g(\theta)\) whose \((i,j)\) entry is \(\partial g_i(\theta)/\partial\theta_j\).

With these conventions, if \(X\) has covariance matrix \(\Sigma\), then \(JX\) has covariance matrix of size \(q\) by \(q\), while \(J\) is \(q\) by \(p\) and \(\Sigma\) is \(p\) by \(p\). Thus \(J\Sigma J^{\mathsf T}\) has the required dimensions. Each off-diagonal entry describes covariance between a pair of transformed coordinates; it should not be discarded unless there is a specific reason it is zero.

Exact Covariance Under a Linear Transformation

For a fixed matrix \(A\) and vector \(b\), the transformation \(Y=AX+b\) is linear up to a shift. Its covariance transformation is exact, not an approximation.

Theorem: If \(X\) is a random \(p\)-vector with finite second moments, \(A\) is a fixed \(q\) by \(p\) matrix, and \(b\) is a fixed \(q\)-vector, then \(\operatorname{Cov}(AX+b)=A\operatorname{Cov}(X)A^{\mathsf T}\).

Proof. Write \(\mu=E[X]\). Since \(A\) and \(b\) are fixed, \(E[AX+b]=A\mu+b\). Therefore

\begin{aligned} \operatorname{Cov}(AX+b) &=E\left[\{AX+b-(A\mu+b)\}\{AX+b-(A\mu+b)\}^{\mathsf T}\right]\\ &=E\left[A(X-\mu)(X-\mu)^{\mathsf T}A^{\mathsf T}\right]\\ &=A\,E[(X-\mu)(X-\mu)^{\mathsf T}]\,A^{\mathsf T}\\ &=A\operatorname{Cov}(X)A^{\mathsf T}. \end{aligned}

The fixed matrices can be moved outside the expectation because every entry of the displayed matrix is a finite linear combination of entries of \((X-\mu)(X-\mu)^{\mathsf T}\), which are integrable under the finite-second-moment assumption. The shift cancels when the vector is centered. \(\square\)

Worked Example: Sum and Difference of Two Correlated Quantities

Suppose \(X=(X_1,X_2)^{\mathsf T}\) has covariance matrix \(\Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}\). Define \(Y_1=X_1+X_2\) and \(Y_2=X_1-X_2\). Find the covariance matrix of \(Y=(Y_1,Y_2)^{\mathsf T}\).

The transformation matrix and its transpose are \(A=\begin{pmatrix}1&1\\1&-1\end{pmatrix}\) and \(A^{\mathsf T}=\begin{pmatrix}1&1\\1&-1\end{pmatrix}\). Applying the theorem,

\begin{aligned} A\Sigma &=\begin{pmatrix}1&1\\1&-1\end{pmatrix} \begin{pmatrix}4&1\\1&9\end{pmatrix} =\begin{pmatrix}5&10\\3&-8\end{pmatrix},\\ \operatorname{Cov}(Y) &=A\Sigma A^{\mathsf T} =\begin{pmatrix}5&10\\3&-8\end{pmatrix} \begin{pmatrix}1&1\\1&-1\end{pmatrix} =\begin{pmatrix}15&-5\\-5&11\end{pmatrix}. \end{aligned}

As a check, \(\operatorname{Var}(Y_1)=4+9+2(1)=15\), and \(\operatorname{Var}(Y_2)=4+9-2(1)=11\). The cross-covariance is \(\operatorname{Cov}(X_1+X_2,X_1-X_2)=4-9=-5\): the two input cross-terms cancel. This check agrees with the matrix product.

The First-Order Covariance Transformation

A smooth nonlinear function is not globally a matrix multiplication, but near a fixed point \(\theta\), its first-order change is described by its Jacobian. The multivariate delta method turns this local linear approximation into a limiting distribution and its covariance matrix.

Theorem: Let \(T_n\) be a random vector in \(\mathbb R^p\), let \(r_n>0\) with \(r_n\to\infty\), and suppose \(r_n(T_n-\theta)\xrightarrow{d}Z\), where \(Z\sim N_p(0,\Sigma)\). Let \(g:\mathbb R^p\to\mathbb R^q\) be differentiable at \(\theta\), with \(\theta\) an interior point of its domain, and let \(J=J_g(\theta)\). Then \(r_n\{g(T_n)-g(\theta)\}\xrightarrow{d}N_q(0,J\Sigma J^{\mathsf T})\).

The covariance matrix \(J\Sigma J^{\mathsf T}\) is the covariance of the limiting normal vector. The theorem does not, by itself, claim that the finite-sample covariance of \(r_n\{g(T_n)-g(\theta)\}\) converges to this matrix; that stronger claim requires additional control of moments.

Proof. Convergence in distribution of \(r_n(T_n-\theta)\) implies that this sequence is bounded in probability. Since \(r_n\to\infty\), it follows that \(T_n-\theta\xrightarrow{p}0\). Differentiability at \(\theta\) gives \(g(\theta+h)=g(\theta)+Jh+\rho(h)\), where \(\|\rho(h)\|/\|h\|\to0\) as \(h\to0\), with the ratio taken as zero at \(h=0\).

Consequently, \(\|\rho(T_n-\theta)\|/\|T_n-\theta\|\xrightarrow{p}0\), with the same zero convention. Also \(r_n\|T_n-\theta\|=O_p(1)\), since \(r_n(T_n-\theta)\) converges in distribution. The product of a term converging to zero in probability and a term bounded in probability converges to zero in probability. Hence \(r_n\rho(T_n-\theta)=o_p(1)\), and

r_n\{g(T_n)-g(\theta)\} =J\,r_n(T_n-\theta)+o_p(1) \xrightarrow{d}JZ.

The last convergence follows from the continuous mapping theorem and Slutsky's theorem. A linear transformation of a multivariate normal vector is multivariate normal, and the exact linear covariance result gives \(\operatorname{Cov}(JZ)=J\Sigma J^{\mathsf T}\). Thus \(JZ\sim N_q(0,J\Sigma J^{\mathsf T})\), as asserted. \(\square\)

Worked Covariance Calculations

Worked Example: Scaling One Coordinate

Suppose an input vector has covariance matrix \(\Sigma=\begin{pmatrix}0.040625&0.0325\\0.0325&0.53\end{pmatrix}\), and define \(g(x_1,x_2)=(x_1/2,x_2)\). For example, this may describe the first-order transformation of an estimator with that covariance matrix. Find the transformed covariance matrix.

The Jacobian is constant: \(J=\begin{pmatrix}1/2&0\\0&1\end{pmatrix}\). First multiply on the left, which scales the first row of \(\Sigma\):

J\Sigma =\begin{pmatrix}1/2&0\\0&1\end{pmatrix} \begin{pmatrix}0.040625&0.0325\\0.0325&0.53\end{pmatrix} =\begin{pmatrix}0.0203125&0.01625\\0.0325&0.53\end{pmatrix}.

Multiplying on the right by \(J^{\mathsf T}=J\) scales the first column by \(1/2\). Thus

J\Sigma J^{\mathsf T} =\begin{pmatrix}0.01015625&0.01625\\0.01625&0.53\end{pmatrix}.

The off-diagonal entry is \(0.5(0.0325)(1)=0.01625\), not a sum of terms with incorrectly paired derivatives or covariance entries. The diagonal entries also check directly: \((0.5)^2(0.040625)=0.01015625\) and \(1^2(0.53)=0.53\). The resulting matrix is symmetric, as a covariance matrix must be.

Worked Example: Product and Ratio of Two Estimates

Suppose \(T_n=(T_{n1},T_{n2})^{\mathsf T}\) satisfies \(\sqrt n(T_n-(2,4)^{\mathsf T})\xrightarrow{d}N_2(0,\Sigma)\), where \(\Sigma=\begin{pmatrix}0.04&0.01\\0.01&0.09\end{pmatrix}\). Consider the transformed estimates \(g_1(x_1,x_2)=x_1x_2\) and \(g_2(x_1,x_2)=x_1/x_2\). Find their joint limiting covariance matrix.

The point \((2,4)\) is in the region \(x_2\ne0\), where both functions are differentiable. Their Jacobian there is

J_g(2,4) =\begin{pmatrix} x_2&x_1\\ 1/x_2&-x_1/x_2^2 \end{pmatrix}_{(x_1,x_2)=(2,4)} =\begin{pmatrix}4&2\\0.25&-0.125\end{pmatrix}.

For clarity, the first diagonal entry is the variance formed with row \((4,2)\): \(4^2(0.04)+2(4)(2)(0.01)+2^2(0.09)=0.64+0.16+0.36=1.16\). The second diagonal entry, from row \((0.25,-0.125)\), is \(0.25^2(0.04)+2(0.25)(-0.125)(0.01)+(-0.125)^2(0.09)=0.0025-0.000625+0.00140625=0.00328125\).

For the off-diagonal entry, multiply the first Jacobian row, the input covariance matrix, and the transpose of the second row. Expanded term by term, this is \(4(0.04)(0.25)+4(0.01)(-0.125)+2(0.01)(0.25)+2(0.09)(-0.125)=0.04-0.005+0.005-0.0225=0.0175\). Therefore the joint limit is

\sqrt n\left[ \begin{pmatrix}T_{n1}T_{n2}\\T_{n1}/T_{n2}\end{pmatrix} -\begin{pmatrix}8\\0.5\end{pmatrix} \right] \xrightarrow{d} N_2\left( \begin{pmatrix}0\\0\end{pmatrix}, \begin{pmatrix}1.16&0.0175\\0.0175&0.00328125\end{pmatrix} \right).

The off-diagonal value is part of the joint approximation: the two transformed estimates are not treated as asymptotically independent merely because they are reported separately.

Interpretation and a Common Pitfall

The Jacobian maps small input deviations into first-order output deviations. The left multiplication \(J\Sigma\) combines the input covariance with the derivatives of the output coordinates; multiplication by \(J^{\mathsf T}\) then accounts for the derivatives in the second output coordinate of each covariance entry. This is why off-diagonal terms of the input covariance can contribute to both transformed variances and transformed covariances.

A frequent mistake is to treat the diagonal variances independently, transform each one separately, and set the output covariances to zero. That is incorrect unless the matrix calculation or other assumptions establish zero covariance. For example, even when input coordinates are uncorrelated, two transformed coordinates can share the same input coordinates and therefore be correlated. Another common error is reversing the Jacobian orientation: with \(g:\mathbb R^p\to\mathbb R^q\) defined above, the correct product is \(J\Sigma J^{\mathsf T}\), not \(J^{\mathsf T}\Sigma J\).

For a sample mean vector with covariance \(\Sigma/n\), the first-order covariance approximation for a smooth transformed estimate is \(J\Sigma J^{\mathsf T}/n\), evaluated at the relevant parameter value. This is an approximation for a nonlinear transformation, while the exact identity applies only to affine transformations. Neither formula requires \(\Sigma\) to be invertible; a singular input covariance can simply produce a singular transformed covariance.

Key takeaway: For a differentiable map from \(p\) inputs to \(q\) outputs, form the \(q\) by \(p\) Jacobian using output coordinates as rows. The first-order transformed covariance is \(J\Sigma J^{\mathsf T}\); calculate the full matrix, including every off-diagonal entry.

Check Your Understanding

Use the Jacobian orientation and covariance rule to answer these questions.

If \(g:\mathbb R^3\to\mathbb R^2\), what are the dimensions of \(J_g(\theta)\), \(\Sigma\), and \(J_g(\theta)\Sigma J_g(\theta)^{\mathsf T}\)?
Why does adding a fixed vector \(b\) to \(AX\) have no effect on its covariance?
For a differentiable vector function, what additional conclusion does the multivariate delta method give beyond the covariance formula for a fixed linear transformation?
When can the off-diagonal entries of the input covariance matrix contribute to a transformed variance?
Does convergence to a limiting normal distribution alone guarantee convergence of the finite-sample covariance matrices? Explain briefly.