Multivariate Delta Method for Smooth Functionals

Why Smooth Transformations Have Normal Limits

An estimator may have a tractable limiting distribution while the quantity of scientific interest is a transformation of that estimator. Examples include a ratio of two estimated means, a log odds, or several derived quantities considered jointly. The multivariate delta method gives a first-order approximation to the distribution of such transformations: locally, a smooth function behaves like its linear approximation, and a linear transformation of a normal limit is again normal.

The key calculation is not simply to apply a function to the limiting normal random vector. The estimator itself concentrates near its limiting parameter, so the function is approximated near that parameter. Its derivative determines how uncertainty is stretched, compressed, and correlated.

The First-Order Multivariate Delta Method

Definition: A function \(g:U\to\mathbb R^q\), where \(U\) is an open subset of \(\mathbb R^p\), is differentiable at \(\theta\in U\) if there is a \(q\times p\) matrix \(J\) such that \[ g(\theta+h)=g(\theta)+Jh+r(h), \qquad \frac{\|r(h)\|}{\|h\|}\longrightarrow 0 \quad\text{as }h\longrightarrow 0. \] The matrix \(J\) is the Jacobian \(Dg(\theta)\). Its entry in row \(j\), column \(k\) is the partial derivative of the \(j\)th component of \(g\) with respect to the \(k\)th coordinate.

Here and below, \(\|\cdot\|\) is the Euclidean norm. Differentiability at \(\theta\) means that the remainder is negligible relative to the size of a small change in the input. A function with continuous first partial derivatives on a neighborhood of \(\theta\) is differentiable there, but continuity of the derivatives is a sufficient condition, not a requirement of the theorem.

Theorem (First-order multivariate delta method): Let \(T_n\) be a random vector in \(\mathbb R^p\), let \(\theta\in U\) for an open set \(U\), and suppose \(P(T_n\in U)\to 1\). Suppose \[ \sqrt n(T_n-\theta)\xrightarrow{d}N_p(0,\Sigma), \] where \(\Sigma\) is a positive semidefinite \(p\times p\) matrix. If \(g:U\to\mathbb R^q\) is differentiable at \(\theta\), with Jacobian \(J=Dg(\theta)\), then \[ \sqrt n\{g(T_n)-g(\theta)\} \xrightarrow{d} N_q(0,J\Sigma J^\mathsf{T}). \] If \(g(T_n)\) is not defined when \(T_n\notin U\), it may be assigned any value on that event; since its probability tends to zero, this does not change the limit.

The covariance may be singular. In that case the limiting normal distribution is supported on a lower-dimensional set, or can be concentrated at a point. The matrix \(J\Sigma J^\mathsf{T}\) records how the input uncertainty is carried through the transformation.

Proof by Taylor Expansion

We prove the theorem using differentiability, tightness of the scaled estimator, and Slutsky’s theorem. Write \(Z_n=\sqrt n(T_n-\theta)\). The assumed convergence in distribution implies \(Z_n=O_p(1)\), meaning that the sequence is bounded in probability. In particular, \(T_n-\theta=Z_n/\sqrt n\xrightarrow{p}0\), so \(T_n\) approaches \(\theta\) in probability.

Expand at the target point.
On the event \(T_n\in U\), differentiability gives \(g(T_n)-g(\theta)=J(T_n-\theta)+r(T_n-\theta)\), with \(\|r(h)\|/\|h\|\to 0\).

Control the scaled remainder.
Define \(\varepsilon(h)=\|r(h)\|/\|h\|\) for \(h\ne0\), and set \(\varepsilon(0)=0\). Then \(\varepsilon(h)\to0\) as \(h\to0\), so \(\varepsilon(T_n-\theta)\xrightarrow{p}0\). Also, \(\sqrt n\|T_n-\theta\|=\|Z_n\|=O_p(1)\). Therefore \(\sqrt n\|r(T_n-\theta)\|\leq \|Z_n\|\varepsilon(T_n-\theta)\xrightarrow{p}0\).

Take the limit.
The expansion becomes \(\sqrt n\{g(T_n)-g(\theta)\}=JZ_n+o_p(1)\). By the continuous mapping theorem, \(JZ_n\xrightarrow{d}N_q(0,J\Sigma J^\mathsf{T})\); Slutsky’s theorem then gives the stated limit. Since \(P(T_n\notin U)\to0\), the choice of values outside \(U\) cannot alter convergence in distribution. \(\square\)

The remainder argument is essential: differentiability gives an error small compared with \(\|T_n-\theta\|\), while the central limit theorem says that this difference is typically of order \(n^{-1/2}\). Together they make the error negligible on the \(\sqrt n\) scale.

Scalar Contrasts of a Vector Transformation

Corollary: Under the theorem’s assumptions, for any fixed \(a\in\mathbb R^q\), \[ \sqrt n\,a^\mathsf{T}\{g(T_n)-g(\theta)\} \xrightarrow{d} N\!\left(0,\ a^\mathsf{T}J\Sigma J^\mathsf{T}a\right). \]

To see this, apply the linear map \(x\mapsto a^\mathsf{T}x\) to the theorem’s limiting vector. The continuous mapping theorem gives a univariate normal limit. Its variance is \(a^\mathsf{T}(J\Sigma J^\mathsf{T})a\), by the formula for the variance of a linear combination. If this variance is zero, the limiting distribution is concentrated at zero; a nondegenerate normal approximation requires positive variance. \(\square\)

This result is useful when the transformed estimator is vector-valued but a particular contrast is of interest. Its uncertainty can be calculated directly from the transformed covariance matrix, without treating the coordinates of the transformed vector as independent.

Worked Calculations

Worked Example: A Ratio of Two Poisson Means

Let \(X_1,\ldots,X_n\) be independent Poisson random variables with mean \(4\), and let \(Y_1,\ldots,Y_n\) be an independent sample of Poisson random variables with mean \(9\). Consider the ratio \(g(T_n)=\bar X/\bar Y\), where \(T_n=(\bar X,\bar Y)^\mathsf{T}\).

The two sample means have limiting covariance matrix \(\Sigma=\begin{pmatrix}4&0\\0&9\end{pmatrix}\) on the \(\sqrt n\) scale: each sample mean has variance equal to its Poisson mean divided by \(n\), and the two samples are independent. Thus \(\sqrt n\{T_n-(4,9)^\mathsf{T}\}\xrightarrow{d}N_2(0,\Sigma)\). For \(g(x,y)=x/y\), the Jacobian at \((4,9)\) is the row vector \[ J=\begin{pmatrix}1/9&-4/81\end{pmatrix}, \] because \(\partial g/\partial x=1/y\) and \(\partial g/\partial y=-x/y^2\).

The transformed limit has mean \(g(4,9)=4/9\). Its variance is \[ J\Sigma J^\mathsf{T} =4(1/9)^2+9(-4/81)^2 =\frac{4}{81}+\frac{144}{6561} =\frac{324+144}{6561} =\frac{52}{729}. \] Consequently, \(\sqrt n(\bar X/\bar Y-4/9)\xrightarrow{d}N(0,52/729)\). The ratio is defined when \(\bar Y>0\); here \(P(\bar Y=0)=P(Y_1=\cdots=Y_n=0)=e^{-9n}\to0\), so assigning any value when the denominator is zero does not affect the limit.

Worked Example: Two Jointly Transformed Sample Means

Suppose \(W_1,\ldots,W_n\) are independent observations from a bivariate normal distribution with mean \((1,2)^\mathsf{T}\) and covariance matrix \(\Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}\). This covariance matrix is valid: its diagonal entries are positive and its determinant is \(4(9)-1=35>0\). Let \(T_n\) be the sample mean vector and define \(g(x,y)=(x+y,xy)^\mathsf{T}\).

The vector central limit theorem gives \(\sqrt n\{T_n-(1,2)^\mathsf{T}\}\xrightarrow{d}N_2(0,\Sigma)\). The transformed value is \(g(1,2)=(3,2)^\mathsf{T}\), and the Jacobian is \[ J=\begin{pmatrix}1&1\\2&1\end{pmatrix}, \] since the component derivatives are \((1,1)\) for \(x+y\) and \((y,x)\) for \(xy\).

Calculate the transformed covariance in two steps: \[ \Sigma J^\mathsf{T} =\begin{pmatrix}4&1\\1&9\end{pmatrix} \begin{pmatrix}1&2\\1&1\end{pmatrix} =\begin{pmatrix}5&9\\10&11\end{pmatrix}, \qquad J\Sigma J^\mathsf{T} =\begin{pmatrix}1&1\\2&1\end{pmatrix} \begin{pmatrix}5&9\\10&11\end{pmatrix} =\begin{pmatrix}15&20\\20&29\end{pmatrix}. \] The off-diagonal value \(20\) is the limiting covariance between the two transformed coordinates, not zero. The result is \[ \sqrt n\{g(T_n)-(3,2)^\mathsf{T}\} \xrightarrow{d} N_2\!\left(0,\begin{pmatrix}15&20\\20&29\end{pmatrix}\right). \] The transformed covariance is positive definite: its determinant is \(15(29)-20^2=35>0\).

Worked Example: The Logarithm of an Exponential Mean

Let \(X_1,\ldots,X_n\) be independent exponential observations with mean \(2\), so each has variance \(4\). For their sample mean \(T_n=\bar X\), the central limit theorem gives \(\sqrt n(\bar X-2)\xrightarrow{d}N(0,4)\). The target is the log mean, estimated by \(g(\bar X)=\log\bar X\).

The function \(g(x)=\log x\) is differentiable on the open set \((0,\infty)\), and \(g'(2)=1/2\). Also, \(\bar X>0\) with probability one. Therefore, \[ \sqrt n\{\log\bar X-\log 2\} \xrightarrow{d} N\!\left(0,(1/2)^2(4)\right) =N(0,1). \] The derivative halves small changes in the sample mean near \(2\); the variance on the \(\sqrt n\) scale is consequently \(1\), not \(4\).

Interpretation and Common Pitfalls

The Jacobian acts on deviations in the estimator, and the covariance transforms on both sides: \(J\Sigma J^\mathsf{T}\). One side accounts for how the function maps input deviations to output deviations; the transpose accounts for the corresponding covariance calculation. This formula automatically retains dependence among the input coordinates and among the transformed coordinates.

Using only the limiting normal vector inside the function. The theorem concerns the local expansion around \(\theta\), not the distribution of \(g(Z)\) for a normal vector \(Z\). In general, a nonlinear transformation of a normal vector is not normal.
Dropping a Jacobian transpose or ignoring covariance terms. For a vector-valued function, compute \(J\Sigma J^\mathsf{T}\). Do not discard off-diagonal entries unless independence or another calculation establishes that they are zero.
Confusing the scaled limit with an unscaled variance. The theorem gives a limit for \(\sqrt n\{g(T_n)-g(\theta)\}\). It corresponds to the approximation \(\operatorname{Cov}\{g(T_n)\}\approx J\Sigma J^\mathsf{T}/n\), when that covariance approximation is appropriate.
Applying the first-order result where differentiability fails. For example, \(g(x)=|x|\) is not differentiable at \(0\). A zero derivative can also make the first-order limit degenerate. Such cases require separate analysis rather than an automatic use of this theorem.

The method matters because it converts a known joint asymptotic distribution into uncertainty statements for quantities that are often easier to interpret scientifically. Its scope is precise: a differentiable transformation, an estimator with a joint normal limit at the stated rate, and the Jacobian evaluated at the limiting parameter.

Check Your Understanding

Use the theorem and its assumptions to answer these questions.

If \(T_n\) has a two-dimensional asymptotic covariance matrix \(\Sigma\) and \(g\) maps into three dimensions, what are the dimensions of \(J\) and \(J\Sigma J^\mathsf{T}\)?
Why does convergence of \(\sqrt n(T_n-\theta)\) to a normal distribution imply \(T_n\xrightarrow{p}\theta\)?
For a scalar transformation \(g\), what formula gives its limiting variance in terms of \(g'(\theta)\) and the limiting variance of \(T_n\)?
Why does a zero off-diagonal entry in the transformed covariance matrix not follow merely from the components of \(g\) being written separately?
What regularity condition fails for \(g(x)=|x|\) at \(\theta=0\), and why can the first-order theorem not be invoked there?