The Multivariate Delta Method for Transformed Estimates

Why Transform a Vector Estimate?

Many statistical targets are functions of several estimated parameters. A study may estimate two rates and then report their ratio; a model may estimate several group means and then compare them through linear contrasts. If the original estimates have a joint asymptotic distribution, the multivariate delta method describes the approximate joint distribution of these transformed targets.

The essential calculation is a matrix version of the scalar delta method: differentiate each target with respect to each estimated parameter, then use the resulting Jacobian to propagate the covariance. This keeps the dependence among the original estimates in the calculation. Ignoring that dependence can give incorrect variances and covariances.

Definition: Let \(\hat{\theta}\) be a \(p\)-dimensional estimator of a parameter \(\theta\in\mathbb{R}^p\). Suppose \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N_p(0,\Sigma)\), where \(\Sigma\) is a \(p\times p\) covariance matrix. For a function \(g:\mathbb{R}^p\to\mathbb{R}^k\), its Jacobian at \(\theta\) is the \(k\times p\) matrix \[ J_g(\theta)=\left[\frac{\partial g_i(\theta)}{\partial\theta_j}\right]_{i=1,\ldots,k;\,j=1,\ldots,p}. \]

The Multivariate Delta Method

Assume that \(\theta\) is an interior point of an open subset of \(\mathbb{R}^p\), that \(g\) is differentiable at \(\theta\), and that the stated joint asymptotic normal limit for \(\hat{\theta}\) holds. The covariance matrix \(\Sigma\) may be singular; in that case the limiting normal distribution is understood to be possibly degenerate.

Theorem (Multivariate Delta Method): Under these conditions, \[ \sqrt{n}\{g(\hat{\theta})-g(\theta)\} \xrightarrow{d} N_k\!\left(0,\ J_g(\theta)\Sigma J_g(\theta)^{\mathsf T}\right). \] Thus the first-order asymptotic covariance matrix of \(g(\hat{\theta})\) is \[ \frac{1}{n}J_g(\theta)\Sigma J_g(\theta)^{\mathsf T}. \]

The covariance formula has compatible dimensions: \(J_g(\theta)\) is \(k\times p\), \(\Sigma\) is \(p\times p\), and \(J_g(\theta)^{\mathsf T}\) is \(p\times k\). The result is a \(k\times k\) matrix because the transformation has \(k\) outputs. Its diagonal entries give the asymptotic variances on the \(1/n\) scale; its off-diagonal entries give the corresponding covariances.

Find the original limit.
Write the joint limit of \(\sqrt{n}(\hat{\theta}-\theta)\) and identify \(\Sigma\).

Differentiate the transformation.
Compute the Jacobian \(J_g(\theta)\), evaluating it at the true parameter.

Propagate the covariance.
Multiply in the order \(J_g(\theta)\Sigma J_g(\theta)^{\mathsf T}\).

Interpret the result.
Read the diagonal as marginal asymptotic variances and the off-diagonal as asymptotic covariances of the transformed estimates.

Why the Jacobian Formula Holds

Differentiability gives a first-order linear approximation to \(g\) near \(\theta\). The proof must also show that its remainder is negligible at the \(\sqrt{n}\) scale. The assumed limit implies \(\sqrt{n}(\hat{\theta}-\theta)=O_p(1)\), and hence \(\hat{\theta}\xrightarrow{p}\theta\).

Proof: By differentiability at \(\theta\), there is a remainder \(r(h)\) such that \[ g(\theta+h)-g(\theta)=J_g(\theta)h+r(h), \qquad \frac{\|r(h)\|}{\|h\|}\longrightarrow 0 \quad\text{as }h\longrightarrow 0. \] Set \(h=\hat{\theta}-\theta\). Consistency implies that \(\hat{\theta}-\theta\xrightarrow{p}0\), so the remainder ratio converges to zero in probability (with the ratio defined as zero when \(h=0\)). Since \(\sqrt{n}\|\hat{\theta}-\theta\|=O_p(1)\), it follows that \(\sqrt{n}\,r(\hat{\theta}-\theta)=o_p(1)\). Therefore, \[ \sqrt{n}\{g(\hat{\theta})-g(\theta)\} = J_g(\theta)\sqrt{n}(\hat{\theta}-\theta)+o_p(1). \] The continuous mapping theorem gives a normal limit for the first term, with covariance \(J_g(\theta)\Sigma J_g(\theta)^{\mathsf T}\). Slutsky’s theorem shows that adding the \(o_p(1)\) remainder does not change that limit. \(\square\)

Worked Examples

Worked Example: Two Nonlinear Targets and Their Covariance

Suppose \(\theta=(\theta_1,\theta_2)^{\mathsf T}=(2,4)^{\mathsf T}\), and \[ \sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N_2(0,\Sigma), \qquad \Sigma=\begin{pmatrix}1&0.3\\0.3&2\end{pmatrix}. \] Consider the two targets \(g_1(\theta)=\theta_1^2+2\theta_2\) and \(g_2(\theta)=\theta_1/\theta_2\).

Their values at the true parameter are \(g_1(\theta)=2^2+2(4)=12\) and \(g_2(\theta)=2/4=0.5\). The gradient of the first target is \((2\theta_1,2)\), while that of the second is \((1/\theta_2,-\theta_1/\theta_2^2)\). At \((2,4)\), the Jacobian is \[ J_g(\theta)=\begin{pmatrix}4&2\\0.25&-0.125\end{pmatrix}. \]

For the first row, \(\Sigma(4,2)^{\mathsf T}=(4.6,5.2)^{\mathsf T}\), so its variance is \(4(4.6)+2(5.2)=28.8\). For the second row, \[ \Sigma\begin{pmatrix}0.25\\-0.125\end{pmatrix} = \begin{pmatrix}0.25+0.3(-0.125)\\0.3(0.25)+2(-0.125)\end{pmatrix} = \begin{pmatrix}0.2125\\-0.175\end{pmatrix}. \] Consequently, the covariance between the two transformed targets is \[ (4,2)\begin{pmatrix}0.2125\\-0.175\end{pmatrix} =4(0.2125)+2(-0.175)=0.85-0.35=0.50. \] The variance of the second target is \(0.25(0.2125)+(-0.125)(-0.175)=0.053125+0.021875=0.075\). Thus the transformed limiting covariance is \[ J_g(\theta)\Sigma J_g(\theta)^{\mathsf T} = \begin{pmatrix}28.8&0.50\\0.50&0.075\end{pmatrix}. \]

In particular, \(\sqrt{n}\{(g_1(\hat{\theta}),g_2(\hat{\theta}))^{\mathsf T}-(12,0.5)^{\mathsf T}\}\) converges to a centered bivariate normal distribution with this covariance matrix. The equal off-diagonal entries are required because a covariance matrix is symmetric.

Worked Example: A Linear Contrast of Three Estimates

Suppose \(\hat{\theta}=(\hat{\theta}_1,\hat{\theta}_2,\hat{\theta}_3)^{\mathsf T}\) satisfies \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N_3(0,\Sigma)\), where \[ \Sigma=\begin{pmatrix}4&1&0\\1&9&2\\0&2&16\end{pmatrix}. \] We want the contrast \(g(\theta)=\theta_1-2\theta_2+\theta_3\).

This is a linear transformation with row Jacobian \(c^{\mathsf T}=(1,-2,1)\). Its asymptotic variance on the \(\sqrt n\) scale is \[ c^{\mathsf T}\Sigma c = (1,-2,1) \begin{pmatrix}4&1&0\\1&9&2\\0&2&16\end{pmatrix} \begin{pmatrix}1\\-2\\1\end{pmatrix}. \] First, \(\Sigma c=(2,-15,12)^{\mathsf T}\): its entries are \(4-2+0=2\), \(1-18+2=-15\), and \(0-4+16=12\). Therefore, \[ c^{\mathsf T}\Sigma c=2+30+12=44. \] The asymptotic variance of the estimated contrast is \(44/n\), and its asymptotic standard error is \(\sqrt{44/n}\). The covariance terms in \(\Sigma\) contribute to this result; using only the three diagonal entries would not give the correct variance.

Worked Example: The Ratio of Two Estimated Means

Suppose \(\hat{\theta}=(\hat{\mu}_1,\hat{\mu}_2)^{\mathsf T}\), with true value \(\theta=(3,2)^{\mathsf T}\), and \[ \sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N_2\!\left(0,\begin{pmatrix}4&1\\1&9\end{pmatrix}\right). \] For the ratio \(g(\theta)=\theta_1/\theta_2\), the target is \(3/2\) and its gradient is \((1/\theta_2,-\theta_1/\theta_2^2)\). Thus \(J_g(\theta)=(1/2,-3/4)\).

Multiplying the covariance matrix by the gradient column gives \[ \begin{pmatrix}4&1\\1&9\end{pmatrix} \begin{pmatrix}1/2\\-3/4\end{pmatrix} = \begin{pmatrix}2-3/4\\1/2-27/4\end{pmatrix} = \begin{pmatrix}5/4\\-25/4\end{pmatrix}. \] The transformed asymptotic variance is then \[ (1/2,-3/4)\begin{pmatrix}5/4\\-25/4\end{pmatrix} = \frac{5}{8}+\frac{75}{16} = \frac{10}{16}+\frac{75}{16} = \frac{85}{16}. \] Hence \(\sqrt{n}\{\hat{\mu}_1/\hat{\mu}_2-3/2\}\) converges to a centered normal distribution with variance \(85/16\), and the ratio’s first-order asymptotic variance is \(85/(16n)\).

Linear Transformations and Multiple Targets

For a linear transformation \(g(\theta)=A\theta\), the Jacobian is \(A\) everywhere. Therefore, if the original estimator has asymptotic covariance \(\Sigma\), the transformed estimator has asymptotic covariance \(A\Sigma A^{\mathsf T}\). This also shows why separate transformed estimates generally remain dependent: the off-diagonal entries of this matrix need not be zero.

Derivation for linear transformations: Since \(A\hat{\theta}-A\theta=A(\hat{\theta}-\theta)\), we have \[ \sqrt{n}(A\hat{\theta}-A\theta)=A\sqrt{n}(\hat{\theta}-\theta). \] A fixed linear map sends a normal limit with covariance \(\Sigma\) to a normal limit with covariance \(A\Sigma A^{\mathsf T}\). This is exactly the Jacobian formula with \(J_g(\theta)=A\).

For nonlinear transformations, the same covariance propagation is local: the Jacobian gives the best first-order linear approximation around the true parameter. The approximation is most informative when the estimator is concentrated near that parameter and the derivatives are not changing too sharply nearby. It is a first-order asymptotic statement, not an exact finite-sample variance identity.

Common Pitfalls and Practical Interpretation

Dropping dependence. Replacing \(\Sigma\) with a diagonal matrix discards covariances. This is valid only when the relevant covariances are actually zero or when the model justifies that simplification.
Using the wrong Jacobian orientation. With \(k\) outputs and \(p\) inputs, this convention uses a \(k\times p\) Jacobian. The order \(J_g\Sigma J_g^{\mathsf T}\) then produces a \(k\times k\) covariance matrix.
Evaluating derivatives at the estimate without explanation. The theorem’s limit uses \(J_g(\theta)\), evaluated at the true parameter. In applications, replacing \(\theta\) by a consistent estimator often yields a plug-in estimate of the asymptotic covariance, but that is an estimation step, not the theorem’s limiting formula.
Calling the approximation exact. The formula describes the leading asymptotic covariance, with scale \(1/n\). It does not assert that the finite-sample distribution is exactly normal or that its exact variance equals this expression.
Ignoring where the function is defined. A ratio requires a nonzero denominator at the target parameter, and differentiability must hold at the point where the estimator converges. A transformation that is not differentiable there may require a different argument.

Key takeaway: For a smooth \(k\)-dimensional target formed from a \(p\)-dimensional estimator, differentiate the target to get a \(k\times p\) Jacobian, then calculate \(J_g(\theta)\Sigma J_g(\theta)^{\mathsf T}\). Retaining the full covariance matrix is essential for correct marginal variances and joint inference.

Check Your Understanding

For each question, identify the relevant Jacobian or covariance calculation before interpreting the result.

If an estimator has a three-dimensional asymptotic covariance matrix and the target has two components, what are the dimensions of its Jacobian and transformed covariance matrix?
Why does the multivariate delta method require consistency of \(\hat{\theta}\) near the point where \(g\) is differentiated?
For a linear contrast \(c^{\mathsf T}\hat{\theta}\), write its asymptotic variance in terms of \(c\) and \(\Sigma\).
What information is lost if the off-diagonal entries of the original covariance matrix are set to zero without justification?
For \(g(\theta_1,\theta_2)=\theta_1/\theta_2\), state the condition on the target parameter needed for the usual differentiable delta method.