Gradient Formulas for Scalar Functionals

From a Vector Limit to a Scalar Variance

Many estimators are naturally collected into a vector, but the quantity of scientific interest is often a single number: a contrast, a ratio, or another smooth function of the vector. The multivariate delta method turns the vector’s asymptotic covariance into an approximate variance for that scalar quantity. The key calculation is local: evaluate the gradient of the function at the limiting parameter, then use it to weight the covariance matrix.

This tutorial focuses on carrying out and interpreting that calculation. In particular, covariance terms matter: changing one coordinate of a function can be associated with changing another coordinate, so the off-diagonal entries of the covariance matrix cannot generally be ignored.

We use a column-vector gradient throughout. Thus the variance calculation is written as \(\nabla g(\theta)^\mathsf{T}\Sigma\nabla g(\theta)\), a scalar. The covariance matrix \(\Sigma\) is symmetric and nonnegative definite; it may be singular.

For a conventional differentiability condition, it is enough that the domain contain an open neighborhood of \(\theta\) and that \(g\) be differentiable there at \(\theta\). More generally, the proof only requires the first-order remainder condition at \(\theta\). A boundary point or a function that is not differentiable there requires separate care; the theorem does not simply apply by substituting a boundary value into a gradient formula.

Why the Gradient Produces the Variance

Differentiability says that for a small vector \(h\), the function changes by its linear approximation \(\nabla g(\theta)^\mathsf{T}h\), plus a remainder smaller than \(\|h\|\). Asymptotic normality makes \(T_n-\theta\) small at rate \(1/r_n\). After multiplying by \(r_n\), the remainder disappears, leaving a linear combination of the coordinates of the limiting normal vector.

Here is the full justification of those steps. Write \(R(h)=\rho(h)\|h\|\), where \(\rho(h)\to0\) as \(h\to0\), taking \(R(0)=0\). Since \(r_n(T_n-\theta)\) converges in distribution, it is bounded in probability; consequently \(T_n-\theta\xrightarrow{p}0\). Continuity of \(\rho\) at zero gives \(\rho(T_n-\theta)\xrightarrow{p}0\), while \(r_n\|T_n-\theta\|=O_p(1)\). Hence $$ r_nR(T_n-\theta) =\rho(T_n-\theta)\,r_n\|T_n-\theta\| \xrightarrow{p}0. $$ The linear term is $$ \nabla g(\theta)^\mathsf{T}\{r_n(T_n-\theta)\} \xrightarrow{d} \nabla g(\theta)^\mathsf{T}Z, \qquad Z\sim N_p(0,\Sigma), $$ by the continuous mapping theorem. A linear combination of a multivariate normal vector is normal, with mean zero and variance \(\nabla g(\theta)^\mathsf{T}\Sigma\nabla g(\theta)\). Slutsky’s theorem then gives the stated limit. \(\square\)

To verify the expansion, multiply out the quadratic form: $$ a^\mathsf{T}\Sigma a =\sum_{i=1}^p\sum_{j=1}^p a_i\Sigma_{ij}a_j. $$ The diagonal terms are \(a_i^2\Sigma_{ii}\). For each pair \(i<j\), symmetry gives \(\Sigma_{ij}=\Sigma_{ji}\), so the two terms for that pair sum to \(2a_i a_j\Sigma_{ij}\). This proves the formula. \(\square\)

The diagonal entries of \(\Sigma\) are the coordinate variances in the limiting vector, and its off-diagonal entries are the coordinate covariances. A positive or negative covariance can increase or decrease the scalar variance depending on the signs of the corresponding gradient components.

Worked Gradient Calculations

Plug-In Variances and Interpretation

In applications, \(\theta\) and often \(\Sigma\) are unknown. A standard estimated first-order variance replaces them with consistent estimates. Specifically, suppose \(\hat\theta\xrightarrow{p}\theta\), \(g\) has a continuous gradient near \(\theta\), and a covariance estimator \(\hat\Sigma\xrightarrow{p}\Sigma\). Then $$ \widehat V_n= \frac{1}{n}\nabla g(\hat\theta)^\mathsf{T}\hat\Sigma\nabla g(\hat\theta) \xrightarrow{p} \frac{1}{n}\nabla g(\theta)^\mathsf{T}\Sigma\nabla g(\theta) $$ in the sense that \(n\widehat V_n\xrightarrow{p}\nabla g(\theta)^\mathsf{T}\Sigma\nabla g(\theta)\). Indeed, continuity gives \(\nabla g(\hat\theta)\xrightarrow{p}\nabla g(\theta)\); the entries of the quadratic form are sums and products of quantities converging in probability, so the continuous mapping theorem gives the claimed convergence. Slutsky’s theorem then supports use of the estimated standard error when the limiting variance is positive.

The estimated covariance matrix must match the scaling convention. If \(\sqrt n(T_n-\theta)\) has limiting covariance \(\Sigma\), then the covariance of \(T_n\) is approximately \(\Sigma/n\). If a reported matrix already estimates the covariance of \(T_n\), dividing the resulting quadratic form by \(n\) again would be an error.

Common Mistakes

• Dropping the covariance terms. Using only squared gradient components times marginal variances is correct only when the relevant covariances are zero. Use the full quadratic form, or the expanded formula including every off-diagonal contribution.
• Using the gradient at the estimate in the theoretical limit. The limiting variance uses \(\nabla g(\theta)\). A gradient at \(\hat\theta\) is appropriate for a plug-in estimate, not for defining the population limit.
• Confusing variance of the scaled limit with variance of the estimator. Under a \(\sqrt n\) limit, the quadratic form is the variance of the limiting distribution of the scaled error. The estimator’s first-order variance is that quantity divided by \(n\).
• Assuming a zero gradient gives an ordinary nonzero normal approximation. If \(\nabla g(\theta)=0\), the first-order limit is degenerate at zero. This does not establish a useful nondegenerate approximation; a different order of analysis may be needed.
• Using a derivative where the function is not defined or differentiable. A ratio requires a nonzero limiting denominator, and logarithms require positive arguments. Check the domain and the limiting point before applying the formula.