Inverse Functions and the Delta Method

Why Inverse Transformations Need Care

Many estimators are obtained by solving an equation rather than by estimating the target directly. If a parameter $\theta$ determines a quantity $\eta=h(\theta)$, an estimate $\hat{\eta}$ can be converted back to the parameter by applying the inverse function: $\hat{\theta}=h^{-1}(\hat{\eta})$. This occurs in parameter transformations, calibration, and quantile estimation.

The first-order delta method tells us how a smooth transformation changes an asymptotic limit. For an inverse transformation, its derivative is the reciprocal of the forward derivative. That reciprocal determines the scale of the limit, and its square multiplies the variance. But the inverse and its derivative are valid locally only when the function really has a suitable local inverse. Differentiability at one point alone is not enough.

When the Local Inverse Exists

Definition: Let $h$ be continuously differentiable on an open interval containing $\theta$, and suppose $h'(\theta)\ne 0$. There is a smaller open interval around $\theta$ on which $h$ is one-to-one. Its image is an open interval around $\eta=h(\theta)$, and the inverse $g=h^{-1}$ is differentiable there, with $$ g'(\eta)=\frac{1}{h'(\theta)}. $$

The neighborhood conditions matter. Since $h'$ is continuous and nonzero at $\theta$, it has a constant sign on a sufficiently small interval around $\theta$. The mean value theorem then implies that $h$ is strictly monotone on that interval, hence one-to-one. Continuity and strict monotonicity ensure that its image is an open interval around $h(\theta)$, so the inverse is defined on a genuine neighborhood of $\eta$.

Theorem: Under the conditions in the definition, the inverse derivative is $g'(\eta)=1/h'(\theta)$.

To verify the formula, take $y\ne\eta$ in the local image and set $x=g(y)$. Then $x\ne\theta$, $y=h(x)$, and $x\to\theta$ as $y\to\eta$. Therefore $$ \frac{g(y)-g(\eta)}{y-\eta} = \frac{x-\theta}{h(x)-h(\theta)} \longrightarrow \frac{1}{h'(\theta)}. $$ The denominator limit is legitimate because $h$ is differentiable at $\theta$ and $h'(\theta)\ne0$. This proves differentiability of the inverse and the stated derivative. $\square$

A weaker-looking claim such as “$h$ is differentiable at $\theta$ and $h'(\theta)\ne0$” does not by itself guarantee that the range of $h$ contains a neighborhood of $h(\theta)$. The theorem uses continuity and one-to-one behavior on a neighborhood; the stated continuously differentiable condition ensures both locally.

The Inverse Delta Method

Theorem: Suppose $h$ is continuously differentiable near $\theta$, $h'(\theta)\ne0$, $\eta=h(\theta)$, and $\hat{\eta}_n$ satisfies $$ \sqrt{n}(\hat{\eta}_n-\eta)\xrightarrow{d}N(0,\tau^2), $$ where $0\le \tau^2<\infty$. Let $g$ be the local inverse of $h$. Define $\hat{\theta}_n=g(\hat{\eta}_n)$ whenever $\hat{\eta}_n$ is in the local image (and define it arbitrarily otherwise). Then $$ \sqrt{n}(\hat{\theta}_n-\theta)\xrightarrow{d} N\left(0,\frac{\tau^2}{[h'(\theta)]^2}\right). $$

Here is the justification. Convergence in distribution of $\sqrt n(\hat{\eta}_n-\eta)$ implies that this sequence is bounded in probability. In particular, $\hat{\eta}_n\xrightarrow{p}\eta$, so the probability that $\hat{\eta}_n$ lies outside the local image tends to zero. Differentiability of $g$ at $\eta$ gives $$ g(\hat{\eta}_n)-g(\eta) = g'(\eta)(\hat{\eta}_n-\eta)+r_n, \qquad \frac{r_n}{|\hat{\eta}_n-\eta|}\xrightarrow{p}0, $$ with the remainder taken as zero when $\hat{\eta}_n=\eta$. Since $\sqrt n(\hat{\eta}_n-\eta)$ is bounded in probability, it follows that $\sqrt n r_n\xrightarrow{p}0$. Also $g(\eta)=\theta$ and $g'(\eta)=1/h'(\theta)$. Thus $$ \sqrt n(\hat{\theta}_n-\theta) = \frac{1}{h'(\theta)}\sqrt n(\hat{\eta}_n-\eta)+o_p(1). $$ Slutsky’s theorem gives the stated normal limit; multiplying a centered normal variable by $1/h'(\theta)$ multiplies its variance by $1/[h'(\theta)]^2$. The arbitrary definition outside the image does not affect the limit because that event has probability tending to zero. $\square$

The sign of the derivative reverses the direction of the fluctuation if $h$ is decreasing, but it disappears from the variance. A small $|h'(\theta)|$, however, produces a large inverse derivative and can greatly amplify uncertainty.

Worked Example: Inverting a Square

Suppose $\theta=2$, $h(\theta)=\theta^2$ on the positive half-line, and an estimator of $\eta=h(\theta)=4$ satisfies $$ \sqrt n(\hat{\eta}_n-4)\xrightarrow{d}N(0,1). $$ Use the positive local inverse to find the limit for $\hat{\theta}_n=\sqrt{\hat{\eta}_n}$.

The inverse is $g(\eta)=\sqrt{\eta}$. Its derivative at $\eta=4$ is $$ g'(4)=\frac{1}{2\sqrt{4}}=\frac14. $$ Equivalently, $h'(2)=4$, so the inverse derivative is $1/4$. The inverse delta method gives $$ \sqrt n(\hat{\theta}_n-2)\xrightarrow{d} N\left(0,\left(\frac14\right)^2\right) = N\left(0,\frac{1}{16}\right). $$ The factor $1/16$ is the variance multiplier, not the standard-deviation multiplier; the standard deviation is multiplied by $1/4$.

Worked Example: The Reciprocal of an Exponential Sample Mean

Let $X_1,\ldots,X_n$ be independent and identically distributed exponential random variables with rate $\lambda>0$. Their mean is $1/\lambda$, and their variance is $1/\lambda^2$. The central limit theorem gives $$ \sqrt n\left(\bar X-\frac{1}{\lambda}\right)\xrightarrow{d} N\left(0,\frac{1}{\lambda^2}\right). $$ Consider the estimator $\hat{\lambda}=1/\bar X$.

The forward map from rate to mean is $h(\lambda)=1/\lambda$, with inverse $g(m)=1/m$. Both are continuously differentiable near the true values, and $h'(\lambda)=-1/\lambda^2\ne0$. The inverse derivative at $m=1/\lambda$ is $$ g'\left(\frac{1}{\lambda}\right) = -\frac{1}{(1/\lambda)^2} = -\lambda^2. $$ Consequently, $$ \sqrt n(\hat{\lambda}-\lambda)\xrightarrow{d} N\left(0,(-\lambda^2)^2\frac{1}{\lambda^2}\right) = N(0,\lambda^2). $$ For example, if $\lambda=2$, the limiting variance is $4$. The negative derivative indicates that a sample mean above its target corresponds to an estimated rate below its target; it does not make the variance negative.

Quantiles as Inverse Functions

For a distribution function $F$, a $p$-quantile $q_p$ is often characterized by $F(q_p)=p$, where $0<p<1$. If $F$ is differentiable near $q_p$ with density $f$ continuous there and $f(q_p)>0$, the local inverse $Q=F^{-1}$ has derivative $$ Q'(p)=\frac{1}{f(q_p)}. $$ Thus a small change in probability level corresponds locally to a change in quantile about $1/f(q_p)$ times as large. A low density near the quantile means the inverse is steep, so quantile estimates can be variable.

For a sample quantile, applying the scalar delta method directly to $F_n(q_p)$ is not a complete argument: the sample quantile is obtained by inverting the entire empirical distribution function, and its argument is random. The following derivation uses the defining order statistic and a binomial count to justify the inverse-variance formula.

Theorem: Let $X_1,\ldots,X_n$ be independent and identically distributed with distribution function $F$. Fix $0<p<1$, and suppose there is a unique $q_p$ locally satisfying $F(q_p)=p$. Assume $F$ is differentiable at $q_p$, with derivative $f(q_p)>0$, and is continuous and strictly increasing in a neighborhood of $q_p$. Define $\hat q_{p,n}=X_{(\lceil np\rceil)}$, the order statistic with rank $\lceil np\rceil$. Then $$ \sqrt n(\hat q_{p,n}-q_p)\xrightarrow{d} N\left(0,\frac{p(1-p)}{[f(q_p)]^2}\right). $$

To prove the result, fix any real $t$ and put $x_n=q_p+t/\sqrt n$. Let $B_n$ be the number of observations not exceeding $x_n$. Then $B_n$ has a binomial distribution with parameters $n$ and $r_n=F(x_n)$. By the definition of the order statistic, $$ P\left(\sqrt n(\hat q_{p,n}-q_p)\le t\right) = P(\hat q_{p,n}\le x_n) = P(B_n\ge k_n), \qquad k_n=\lceil np\rceil. $$ Differentiability of $F$ at $q_p$ yields $$ r_n=p+\frac{f(q_p)t}{\sqrt n}+o(n^{-1/2}). $$ In particular, $r_n\to p$, so $r_n(1-r_n)\to p(1-p)>0$. The binomial central limit theorem applies to this sequence of binomial probabilities: centering by $nr_n$ and scaling by $\sqrt{nr_n(1-r_n)}$ gives a standard normal limit. This also follows from the triangular-array Lindeberg condition, since the centered Bernoulli summands are bounded while their total variance grows proportionally to $n$.

The standardized threshold satisfies $$ \frac{k_n-nr_n}{\sqrt{nr_n(1-r_n)}} \longrightarrow -\frac{f(q_p)t}{\sqrt{p(1-p)}}, $$ because $k_n=np+O(1)$. The limiting normal distribution is continuous, so taking the upper-tail probability gives $$ P(B_n\ge k_n)\longrightarrow \Phi\left(\frac{f(q_p)t}{\sqrt{p(1-p)}}\right), $$ where $\Phi$ is the standard normal distribution function. This is the distribution function at $t$ of a centered normal variable with variance $p(1-p)/[f(q_p)]^2$, proving the theorem. $\square$

Worked Example: The Sample Median of a Standard Normal Distribution

For a standard normal distribution, the median is $q_{0.5}=0$, and its density at zero is $f(0)=1/\sqrt{2\pi}$. Substituting $p=0.5$ into the sample-quantile result gives $$ \sqrt n(\hat q_{0.5,n}-0)\xrightarrow{d} N\left(0,\frac{(0.5)(0.5)}{(1/\sqrt{2\pi})^2}\right). $$ Since $(1/\sqrt{2\pi})^2=1/(2\pi)$, the variance is $$ \frac{1/4}{1/(2\pi)}=\frac{\pi}{2}. $$ Thus the asymptotic variance under the $\sqrt n$ scaling is $\pi/2$, and the approximate standard error for a large sample is $\sqrt{\pi/(2n)}$.

Common Mistakes and Practical Interpretation

Assuming a local inverse from a derivative at one point. A nonzero derivative at a point alone does not ensure that the function maps a neighborhood onto an open interval. Check local continuity and one-to-one behavior; continuous differentiability with nonzero derivative on a neighborhood is a convenient sufficient condition.
Using the derivative of the forward map instead of its inverse. If the estimated quantity is $h(\theta)$, the derivative multiplying its fluctuation is $1/h'(\theta)$, not $h'(\theta)$.
Forgetting to square the multiplier for a variance. A derivative $c$ multiplies a standard deviation by $|c|$, but multiplies a variance by $c^2$.
Ignoring a small density at a quantile. The quantile variance contains $1/[f(q_p)]^2$. A small density means a larger asymptotic variance, not a smaller one.
Applying the scalar delta method to an empirical quantile without justification. The sample quantile inverts $F_n$, not just the single random quantity $F_n(q_p)$. The binomial-count argument establishes its limit under the stated conditions.

Inverse transformations are useful because they turn limits for an easier-to-estimate quantity into limits for the target parameter. The key checks are local invertibility, a nonzero derivative, and the correct reciprocal derivative. For quantiles, the density at the target quantile plays the role of the forward derivative.

Check Your Understanding

Answer each question using the local inverse conditions and variance formulas.

If $h'(\theta)=-3$ and the estimator of $h(\theta)$ has asymptotic variance $9$ under $\sqrt n$ scaling, what is the inverse estimator’s asymptotic variance?
Why does a nonzero derivative at a single point not, by itself, establish the usual local inverse result?
For a quantile $q_p$, how does the asymptotic variance change when $f(q_p)$ becomes smaller, with $p$ fixed?
For the exponential-rate example, explain why the negative derivative gives a positive asymptotic variance.
What binomial random variable is used to derive the sample-quantile limit, and what event involving it corresponds to the sample quantile being at most $x$?