Rates of Convergence from Local Criterion Curvature

Why Local Curvature Determines a Rate

Consistency says that an estimator approaches the parameter it targets, but it does not say how quickly. That speed matters: it determines the scale on which estimation error remains visible as sample size grows. For an extremum estimator, the rate is governed locally by a competition between two forces. The population criterion pulls the estimator toward its optimum; sampling variation perturbs the criterion and can move its sample optimum away.

The key calculation compares how fast the population criterion rises with the size of its random fluctuations near the optimum. Quadratic growth paired with a smooth, approximately linear stochastic perturbation usually yields the familiar root-\(n\) rate. Different local growth or fluctuation patterns can yield different rates.

Definition: Let \(Q(\theta)\) be a population criterion minimized at \(\theta_0\), and let \(Q_n(\theta)\) be its sample counterpart. The estimator \(\hat{\theta}_n\) minimizes, or approximately minimizes, \(Q_n\). A local convergence rate \(a_n\) means \(a_n\|\hat{\theta}_n-\theta_0\|=O_p(1)\). Here \(O_p(1)\) means bounded in probability. The rate describes the scale of the estimation error; it does not by itself guarantee a particular limiting distribution.

We use a minimization convention. A maximization problem can be put in this form by changing the sign of the criterion. Write \(\delta=\theta-\theta_0\), and use \(\|\cdot\|\) for the Euclidean norm. Local power-law growth means that, for some \(c>0\) and \(r>0\),

Q(\theta_0+\delta)-Q(\theta_0)\ \geq\ c\|\delta\|^r

for \(\delta\) sufficiently close to zero. The exponent \(r\) describes the shape of the population criterion near its minimum. For a smooth criterion with a positive-definite Hessian, a Taylor expansion typically gives \(r=2\). A higher exponent, such as \(r=4\), describes a flatter minimum.

The Basic Rate Balance

Suppose that the centered sample criterion increment has stochastic size \(n^{-1/2}\|\delta\|^s\) locally, for some \(s<r\). Informally, this means its magnitude at distance \(\|\delta\|\) from \(\theta_0\) is of that order. For a regular smooth criterion, the leading random increment is often a random slope times \(\delta\), so \(s=1\). The estimate is displaced until the population increase and random perturbation are comparable:

\|\delta\|^r\ \asymp\ n^{-1/2}\|\delta\|^s \quad\Longrightarrow\quad \|\delta\|\ \asymp\ n^{-1/[2(r-s)]}.

This balance identifies a candidate rate, not a complete theorem on its own. To justify a rate rigorously, the local stochastic bound must hold uniformly over the relevant parameter neighborhoods, the estimator must be localized near \(\theta_0\), and the criterion must have sufficient separation outside those neighborhoods. These conditions turn the scale comparison into a bound in probability.

Rate principle: If local population growth is of order \(\|\delta\|^r\) and the local centered criterion fluctuation is \(O_p(n^{-1/2}\|\delta\|^s)\), with \(r>s\), balancing the two gives the candidate rate \(n^{-1/[2(r-s)]}\). The common smooth case has \(s=1\), giving \(n^{-1/[2(r-1)]}\). Local curvature alone does not establish consistency or the stochastic increment bound.

A short localization argument explains why this balance matters. Suppose the estimator is an exact minimizer and has already been shown to lie in a neighborhood where the local bounds apply. Its sample criterion cannot exceed the criterion at \(\theta_0\). Decomposing the sample criterion into population and centered parts gives

0\ \geq\ Q_n(\hat{\theta}_n)-Q_n(\theta_0) = Q(\hat{\theta}_n)-Q(\theta_0) + \bigl[(Q_n-Q)(\hat{\theta}_n)-(Q_n-Q)(\theta_0)\bigr].

If \(d_n=\|\hat{\theta}_n-\theta_0\|\), population growth contributes at least \(cd_n^r\), while the stochastic increment is bounded on the relevant local neighborhoods by a random multiple of \(n^{-1/2}d_n^s\). Thus, schematically, \(cd_n^r\) cannot be larger than that stochastic term. Solving the resulting balance yields the candidate rate. A full proof controls the random bound across neighborhoods, rather than treating a pointwise order statement as uniform.

Quadratic Growth and the Root-\(n\) Rate

In the regular case, quadratic population growth (\(r=2\)) and a stochastic increment linear in distance (\(s=1\)) give \(n^{-1/2}\). The same result can be seen in a simple sample-mean criterion.

Worked Example: A Quadratic Criterion for a Mean

Suppose \(Y_1,\ldots,Y_n\) are independent and identically distributed, with mean \(\mu\) and finite variance \(\sigma^2\). Consider \(Q_n(\theta)=n^{-1}\sum_{i=1}^n(Y_i-\theta)^2\), whose population criterion is \(Q(\theta)=E[(Y_i-\theta)^2]\). The population optimum is \(\theta_0=\mu\). For \(\delta=\theta-\mu\), expansion gives

Q(\mu+\delta)-Q(\mu)=\delta^2, \qquad Q_n(\mu+\delta)-Q_n(\mu)=\delta^2-2\delta(\bar{Y}-\mu).

The population increase is exactly quadratic. By the central limit theorem, \(\bar{Y}-\mu=O_p(n^{-1/2})\), so the random increment \(-2\delta(\bar{Y}-\mu)\) is \(O_p(n^{-1/2}|\delta|)\). Equating the two magnitudes gives \(\delta^2\asymp n^{-1/2}|\delta|\), hence \(|\delta|\asymp n^{-1/2}\). In fact, direct minimization gives \(\hat{\theta}_n=\bar{Y}\), confirming the rate.

For a concrete hypothetical sample of \(n=100\) observations, suppose \(\bar{Y}=2.4\), \(\mu=2.0\), and the observations have sample standard deviation \(s=1.5\). Then \(\hat{\theta}_n-\mu=2.4-2.0=0.4\) units, and \(\sqrt{n}(\hat{\theta}_n-\mu)=10(0.4)=4.0\) units. The standard error scale estimated from the sample is \(s/\sqrt{n}=1.5/10=0.15\) units. This single realized error illustrates the root-\(n\) scaling; it does not imply that every sample error equals its standard error.

Flatter Population Criteria

When the stochastic perturbation remains locally linear but population growth is flatter than quadratic, the estimator generally converges more slowly than at the root-\(n\) rate. For example, with \(r=4\) and \(s=1\), the balance gives \(n^{-1/6}\), not \(n^{-1/2}\). The sample size must grow substantially before the parameter error shrinks by a given factor.

Worked Example: Quartic Growth with a Random Linear Perturbation

Let \(Z_1,\ldots,Z_n\) be independent and identically distributed with \(E[Z_i]=0\) and \(\operatorname{Var}(Z_i)=1\). Consider the hypothetical criterion \(Q_n(\theta)=\theta^4-2\theta\bar{Z}\), where \(\bar{Z}=n^{-1}\sum_i Z_i\). Its population criterion is \(Q(\theta)=\theta^4\), minimized at \(\theta_0=0\). The criterion grows as \(|\theta|^4\), and the centered random increment is \(-2\theta\bar{Z}=O_p(n^{-1/2}|\theta|)\).

Differentiating the sample criterion and setting the derivative to zero gives \(4\hat{\theta}_n^3-2\bar{Z}=0\). The second derivative is \(12\theta^2\geq 0\), and the quartic criterion grows at both tails, so this solution is a global minimizer. Therefore,

\hat{\theta}_n=\left(\frac{\bar{Z}}{2}\right)^{1/3}, \qquad |\hat{\theta}_n|=O_p(n^{-1/6}).

The rate also follows by balancing \(|\theta|^4\) against \(n^{-1/2}|\theta|\): for nonzero \(|\theta|\), this gives \(|\theta|^3\asymp n^{-1/2}\), or \(|\theta|\asymp n^{-1/6}\). For a hypothetical sample with \(n=100\) and \(\bar{Z}=0.08\), the minimizer is \((0.08/2)^{1/3}=0.04^{1/3}\approx 0.342\). Here \(n^{1/6}|\hat{\theta}_n|\approx 2.154(0.342)=0.737\), a finite scaled error consistent with the stated rate.

When Local Stochastic Fluctuations Are Not Linear

The stochastic exponent \(s\) need not equal one. Some criteria have rough local fluctuations: their random increments over a distance \(d\) are of order \(n^{-1/2}d^{1/2}\). If the population criterion is quadratic, balancing \(d^2\) with \(n^{-1/2}d^{1/2}\) gives a rate of \(n^{-1/3}\). Such a rate cannot be inferred from population curvature alone; the local structure of the random criterion matters equally.

Worked Example: Brownian-Type Local Fluctuations

Consider a hypothetical scalar criterion on \([-1,1]\), \(Q_n(\theta)=\theta^2-n^{-1/2}B(\theta)\), where \(B\) is a two-sided standard Brownian motion with \(B(0)=0\). The population part is \(Q(\theta)=\theta^2\), with a unique minimum at zero. Brownian increments over a distance \(d\) have size \(d^{1/2}\), so the stochastic criterion increment has size \(n^{-1/2}d^{1/2}\). The balance is

d^2\asymp n^{-1/2}d^{1/2} \quad\Longrightarrow\quad d^{3/2}\asymp n^{-1/2} \quad\Longrightarrow\quad d\asymp n^{-1/3}.

Brownian scaling makes this balance especially transparent. Set \(\theta=n^{-1/3}t\). Then, using \(B(n^{-1/3}t)\) distributed as \(n^{-1/6}B(t)\),

n^{2/3}Q_n(n^{-1/3}t) \ \overset{d}{=}\ t^2-B(t).

The rescaled criterion has a nondegenerate form on the expanding rescaled domain, consistent with an estimator of order \(n^{-1/3}\). For \(n=1000\), the balance scale is \(1000^{-1/3}=0.1\) parameter units. This is a scale calculation, not a claim that a particular sample minimizer must equal 0.1.

Approximate Minimization and Practical Implications

Numerical optimizers may return an approximate, rather than exact, minimizer. Suppose the returned value satisfies \(Q_n(\hat{\theta}_n)\leq Q_n(\theta_0)+\varepsilon_n\), where \(\varepsilon_n\geq0\) is the optimization error on the criterion scale. In the localization inequality, this adds \(\varepsilon_n\) to the stochastic term. Under quadratic growth and a root-\(n\) target rate, the criterion gap at that rate is of order \(n^{-1}\); an optimization error of that order can affect the rate calculation, while an error that is \(o_p(n^{-1})\) is negligible at that scale. More generally, compare the optimization error with the population criterion increase at the candidate rate.

These calculations matter when assessing how much precision an estimation algorithm needs, comparing regular and nonregular estimators, and deciding which asymptotic scale is relevant. But a local rate argument is not a substitute for consistency. The earlier material on identification and criterion separation addresses why the estimator approaches the target; local curvature and stochastic increments then describe how quickly it does so.

Common Pitfalls

Using curvature alone. A quadratic population criterion does not automatically imply a root-\(n\) rate. The stochastic increment must also have the appropriate local size.
Balancing the wrong quantities. Compare criterion increments, not the parameter distance directly with a criterion fluctuation. For example, with quadratic growth and a linear random increment, compare \(d^2\) with \(n^{-1/2}d\).
Treating a candidate balance as a complete proof. A pointwise order calculation does not control the random criterion uniformly around the estimator. Localization and suitable uniform stochastic bounds are also needed.
Ignoring optimization error. An approximate minimizer can have a slower rate if its criterion error is too large relative to the criterion gap at the candidate rate.

Key takeaway: Derive a candidate estimator rate by matching the local growth of the population criterion to the local size of the centered sample-criterion increment. For growth \(d^r\) and fluctuations \(n^{-1/2}d^s\), the balance gives \(d\asymp n^{-1/[2(r-s)]}\) when \(r>s\). Justifying that rate also requires appropriate localization and uniform control.

Check Your Understanding

For each question, identify the population-growth and stochastic-fluctuation orders before giving a rate.

A criterion has quadratic population growth and a stochastic increment of order \(n^{-1/2}d\). What candidate rate follows?
With linear stochastic increments, how does quartic population growth change the rate compared with quadratic growth?
A criterion has population growth \(d^2\) and stochastic increments of order \(n^{-1/2}d^{1/2}\). Derive the candidate rate.
Why does a local rate calculation not, by itself, establish consistency?
For a quadratic-growth criterion with a root-\(n\) candidate rate, what criterion-error scale should an approximate optimizer make negligible?