Directional Derivatives and the Extended Delta Method

Why a Directional Derivative Is Needed

The ordinary delta method turns a small estimation error into an approximately linear change in a function of the estimator. At a kink, however, a single linear approximation may not exist. The correct first-order change can still be well defined in each direction, and that directional change can determine the limiting distribution.

The extended delta method formalizes this idea. Its key condition, Hadamard directional differentiability, controls not only a perturbation along a fixed direction but also nearby directions. That stability is important because the direction in an asymptotic argument is itself random.

Hadamard Directional Differentiability

The allowance for \(h_m\) to vary is the defining stability feature: convergence must hold even when the directions approach \(h\) rather than being exactly equal to \(h\). The derivative need not be linear. In contrast, ordinary differentiability gives a linear derivative and is therefore a special, more restrictive case.

For a function on an open set, every fixed direction is locally feasible because \(\theta\) is an interior point. Directional derivatives are positively homogeneous: for \(c>0\), substituting \(ct_m\) for \(t_m\) shows that \(g'_\theta(ch)=c g'_\theta(h)\). This does not imply additivity, which is why the derivative can be nonlinear.

The Extended Delta Method

The almost-sure domain condition ensures that \(g(T_n)\) is defined. A useful alternative is to define the statistic arbitrarily whenever \(T_n\notin U\), provided \(P(T_n\notin U)\to0\). The probability of the exceptional event then tends to zero, so that choice does not affect convergence in distribution.

Proof of the Extended Delta Method

Set \(Z_n=r_n(T_n-\theta)\) and \(t_n=1/r_n\). Then \(Z_n\xrightarrow{d}Z\), so \((Z_n)\) is tight. We first show that the nonlinear transformation has the expansion

Fix a compact set \(K\subset\mathbb{R}^d\). Since \(U\) is open and contains \(\theta\), all points \(\theta+t_nz\), \(z\in K\), belong to \(U\) for sufficiently large \(n\). Define, for such \(n\),

We claim that \(\sup_{z\in K}\|R_n(z)\|\to0\). If not, some subsequence would have points \(z_n\in K\) and a constant \(\varepsilon>0\) with \(\|R_n(z_n)\|\ge\varepsilon\). Compactness gives a further subsequence with \(z_n\to z\in K\). Hadamard directional differentiability, applied to \(t_n\) and \(z_n\), gives

Continuity of \(g'_\theta\) also gives \(g'_\theta(z_n)\to g'_\theta(z)\). Their difference must therefore tend to zero, contradicting \(\|R_n(z_n)\|\ge\varepsilon\). This proves the claim.

Given any \(\varepsilon>0\), tightness allows a compact \(K\) such that \(P(Z_n\notin K)<\varepsilon\) for all sufficiently large \(n\). On the event \(Z_n\in K\), the uniform bound just proved makes \(\|R_n(Z_n)\|\) tend to zero. Since \(\varepsilon\) is arbitrary, \(R_n(Z_n)=o_p(1)\), establishing the expansion. By continuity of \(g'_\theta\), the continuous mapping theorem gives \(g'_\theta(Z_n)\xrightarrow{d}g'_\theta(Z)\). Slutsky’s theorem now yields the stated result. \(\square\)

Derivatives of Maxima and Positive Parts

To verify the formula, write \(M=\max_j\theta_j\) and \(t_m\downarrow0\) with \(h_m\to h\). The sequence \(h_m\) is bounded. If there are coordinates outside \(I(\theta)\), each has a strictly positive gap \(M-\theta_j\); because there are finitely many coordinates, the smallest such gap is positive. For all sufficiently large \(m\), the perturbation \(t_mh_m\) is too small to make any of these inactive coordinates attain the maximum. Thus

for all sufficiently large \(m\). If every coordinate is active, the same equality holds without needing an inactive-coordinate argument. Subtracting \(M\), dividing by \(t_m\), and taking limits gives \(\max_{j\in I(\theta)}h_j\). Finally, the maximum over a fixed finite set is continuous, proving continuity of the derivative.

The scalar positive-part function is \(g(x)=x_+=\max(x,0)\). Its directional derivative is

For \(\theta>0\), sufficiently small perturbations remain positive, so the difference quotient is \(h_m\) and tends to \(h\). For \(\theta<0\), sufficiently small perturbations remain negative, so the quotient is zero. At \(\theta=0\), positive homogeneity of the positive-part function gives \((t_mh_m)_+/t_m=(h_m)_+\to h_+\). Each displayed derivative is continuous as a function of \(h\).

Worked Examples

Common Pitfall and Practical Interpretation

Common mistake: applying the ordinary delta method at a kink by selecting one slope and concluding that the limit is a linear transformation of the original Gaussian limit. At \(\theta=0\), the positive-part function has different changes for positive and negative perturbations; there is no single ordinary derivative. The correct transformation is \(Z\mapsto Z_+\), which can have an atom at zero. For a tied maximum, the correct map is the maximum over the active coordinates, not an arbitrarily chosen coordinate.

The extended delta method is useful whenever the estimator has a regular first-order limit but the final quantity is a nonsmooth function of it. The limiting law is obtained by applying the directional derivative to the entire limiting random vector. Because that derivative may be nonlinear, familiar normal approximations need not apply even when the estimator’s limit is normal.

Before using the theorem, check that the statistic is defined, that the derivative is Hadamard directional (not merely a pointwise one-sided limit), and that the derivative map is continuous for the stated version of the theorem. Then identify the active directions at the point of interest and apply the derivative to the joint limit.