Consistency of Extremum Estimators

Why Maximizing a Sample Criterion Can Work

Many estimators are defined by optimizing a criterion computed from data: a likelihood, a loss with its sign reversed, or an empirical average of scores. The criterion changes with the sample, so the sample maximizer need not equal the parameter that optimizes the population criterion. Consistency asks whether the sample optimizer nevertheless approaches that population target as the sample size grows.

The central argument is geometric. If the population criterion has a unique maximizer, parameters a fixed distance away must have a strictly worse criterion value under suitable compactness and continuity conditions. Uniform convergence then makes the empirical criterion accurate enough to preserve that gap. Exact maximization is not essential: an estimator whose criterion value is sufficiently close to the empirical maximum has the same consistency guarantee.

Population and Empirical Criteria

Definition: Let $\Theta$ be a parameter space. A population criterion is a function $Q:\Theta\to\mathbb{R}$. A sample criterion is a random function $Q_n:\Theta\to\mathbb{R}$ based on a sample of size $n$. A population maximizer $\theta_0$ satisfies $Q(\theta_0)=\sup_{\theta\in\Theta}Q(\theta)$. An estimator $\hat{\theta}_n$ is an approximate maximizer with tolerance $r_n\geq 0$ if $$ Q_n(\hat{\theta}_n)\geq \sup_{\theta\in\Theta}Q_n(\theta)-r_n. $$

An exact empirical maximizer has $r_n=0$. The approximate definition accommodates numerical optimization and estimators for which an exact maximizer is inconvenient to define. We assume the relevant suprema and random quantities are measurable, and that $\hat{\theta}_n$ takes values in $\Theta$.

Uniform convergence in probability: The sample criteria converge uniformly in probability to $Q$ on $\Theta$ if $$ \sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|\xrightarrow{p}0. $$

Uniform convergence controls the largest criterion error over the whole parameter space, rather than the error at one fixed parameter. That distinction is essential: the estimator itself is data-dependent, so its selected parameter can move as $n$ changes.

Separation of the Population Optimum

The first ingredient in the consistency argument is a positive criterion gap away from the target. For any $\varepsilon>0$, consider parameters at least $\varepsilon$ away from $\theta_0$. If $\Theta$ is a compact subset of a metric space, $Q$ is continuous, and $\theta_0$ is its unique maximizer, then $$ \Delta_\varepsilon = Q(\theta_0)- \sup_{\{\theta\in\Theta:\ d(\theta,\theta_0)\geq\varepsilon\}}Q(\theta) >0, $$ provided the set in the supremum is nonempty. Indeed, that set is closed in a compact space and hence compact. Continuity means $Q$ attains its maximum there. Uniqueness of $\theta_0$ means that attained value is strictly less than $Q(\theta_0)$.

This is a global separation statement: it concerns every parameter outside a fixed neighborhood, not just parameters infinitesimally close to $\theta_0$. It is the reason the argument establishes consistency without needing a local Taylor expansion.

Consistency Theorem for Approximate Extremum Estimators

Theorem: Suppose $\Theta$ is a compact subset of a metric space, $Q$ is continuous on $\Theta$, and $Q$ has a unique maximizer $\theta_0$. Suppose $$ \sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|\xrightarrow{p}0 \quad\text{and}\quad r_n\xrightarrow{p}0. $$ If $\hat{\theta}_n\in\Theta$ satisfies $Q_n(\hat{\theta}_n)\geq\sup_{\theta\in\Theta}Q_n(\theta)-r_n$, then $\hat{\theta}_n\xrightarrow{p}\theta_0$.

Proof. Fix $\varepsilon>0$. If no parameter in $\Theta$ is at least $\varepsilon$ from $\theta_0$, then $P(d(\hat{\theta}_n,\theta_0)\geq\varepsilon)=0$. Otherwise, the separation argument gives $\Delta_\varepsilon>0$. Write $U_n=\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|$. On the event $d(\hat{\theta}_n,\theta_0)\geq\varepsilon$, the definition of $\Delta_\varepsilon$ gives $Q(\hat{\theta}_n)\leq Q(\theta_0)-\Delta_\varepsilon$. Therefore $$ Q_n(\theta_0)-Q_n(\hat{\theta}_n) \geq Q(\theta_0)-U_n-\{Q(\theta_0)-\Delta_\varepsilon+U_n\} =\Delta_\varepsilon-2U_n. $$ Also, approximate maximization implies $\sup_{\theta\in\Theta}Q_n(\theta)-Q_n(\hat{\theta}_n)\leq r_n$, while $\sup_{\theta\in\Theta}Q_n(\theta)\geq Q_n(\theta_0)$. Hence $Q_n(\theta_0)-Q_n(\hat{\theta}_n)\leq r_n$. If $U_n<\Delta_\varepsilon/3$ and $r_n<\Delta_\varepsilon/3$, the first bound is strictly greater than $\Delta_\varepsilon/3$, while the second is strictly less than $\Delta_\varepsilon/3$, a contradiction. Thus $$ P\{d(\hat{\theta}_n,\theta_0)\geq\varepsilon\} \leq P(U_n\geq\Delta_\varepsilon/3)+P(r_n\geq\Delta_\varepsilon/3) \longrightarrow 0. $$ This holds for every $\varepsilon>0$, proving $\hat{\theta}_n\xrightarrow{p}\theta_0$. $\square$

The proof uses the gap in population criterion values to rule out parameters far from the target. Uniform convergence transfers that gap to the sample criterion, and the vanishing tolerance prevents approximate maximization from absorbing it. An analogous almost-sure conclusion follows if uniform convergence and the tolerance convergence both hold almost surely.

Worked Applications

Worked Example: A Bernoulli Likelihood on a Compact Parameter Space

Let $X_1,\ldots,X_n$ be independent Bernoulli random variables with success probability $p=0.6$. Take $\Theta=[0.2,0.8]$ and use the average log-likelihood $Q_n(\theta)=\bar X_n\log\theta+(1-\bar X_n)\log(1-\theta)$, where $\bar X_n=n^{-1}\sum_{i=1}^nX_i$. The population criterion is $Q(\theta)=0.6\log\theta+0.4\log(1-\theta)$.

For uniform convergence, subtract the criteria and collect terms: $$ \sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)| = |\bar X_n-0.6|\sup_{\theta\in[0.2,0.8]} \left|\log\frac{\theta}{1-\theta}\right| \leq |\bar X_n-0.6|\log 4. $$ The bound uses $1/4\leq\theta/(1-\theta)\leq4$ on the stated interval. By the law of large numbers, $\bar X_n\xrightarrow{p}0.6$, so the displayed supremum converges to zero in probability.

The derivative of the population criterion is $Q'(\theta)=0.6/\theta-0.4/(1-\theta)=(0.6-\theta)/\{\theta(1-\theta)\}$. It is positive for $\theta<0.6$, zero at $0.6$, and negative for $\theta>0.6$. Thus $Q$ has the unique maximizer $\theta_0=0.6$ on $\Theta$. The theorem applies to any approximate maximizer whose tolerance converges to zero in probability.

For a numerical check, suppose $n=10$ observations include 7 successes. Then $\bar X_n=0.7$, and the empirical maximizer is $0.7$, which belongs to $\Theta$. Its criterion is $0.7\log(0.7)+0.3\log(0.3)\approx-0.6109$. At the true parameter, $Q_n(0.6)=0.7\log(0.6)+0.3\log(0.4)\approx-0.6325$. The sample criterion is higher at $0.7$ for this sample, but the uniform-convergence and separation argument shows that the maximizer approaches $0.6$ as the sample grows.

Worked Example: Normal Location With a Quadratic Criterion

Let $X_1,\ldots,X_n$ be independent $N(1,1)$ observations, and let $\Theta=[-2,2]$. Consider $Q_n(\theta)=-\frac{1}{2n}\sum_{i=1}^n(X_i-\theta)^2$. Its population version is $$ Q(\theta)=-\frac12 E[(X_1-\theta)^2] =-\frac12\{1+(1-\theta)^2\}. $$ This criterion has the unique maximizer $\theta_0=1$, because the squared term is nonnegative and equals zero only at $1$.

Expanding the sample criterion and its expectation gives $$ \sup_{\theta\in[-2,2]}|Q_n(\theta)-Q(\theta)| \leq \frac12\left( \left|\frac1n\sum_{i=1}^nX_i^2-E[X_1^2]\right| +4|\bar X_n-E[X_1]| \right). $$ Here the coefficient $4$ is $2\sup_{\theta\in[-2,2]}|\theta|$, from the term $2\theta\bar X_n$. Since $E[X_1]=1$, $E[X_1^2]=2$, and both sample moments converge to their expectations by the law of large numbers, the bound converges to zero in probability. The theorem therefore gives consistency of any approximate maximizer with $r_n\xrightarrow{p}0$.

For example, take the observed values $0,1,2,1,1$. Their mean is $1$, and their mean square is $7/5=1.4$. Thus $Q_5(\theta)=-\frac12(1.4-2\theta+\theta^2)$, whose maximum on $[-2,2]$ is at $\theta=1$, with value $-0.2$. At $\theta=1.1$, the value is $-\frac12(1.4-2.2+1.21)=-0.205$, a criterion loss of $0.005$. This illustrates that a nearby parameter can be an approximate maximizer; the theorem requires its tolerance to vanish as the sample size increases.

Worked Example: Choosing Between Two Treatments

Suppose each sampled subject has binary outcomes $Y_{iA}$ and $Y_{iB}$ under two treatments, with the subject-level pairs independent and identically distributed across $i$. Assume $E[Y_{iA}]=0.7$ and $E[Y_{iB}]=0.5$. Let $\Theta=\{A,B\}$, set $Q_n(j)=n^{-1}\sum_{i=1}^nY_{ij}$, and define $Q(j)=E[Y_{ij}]$. The population criterion has the unique maximizer $A$, with gap $Q(A)-Q(B)=0.2$.

There are only two parameter values, so $$ \sup_{j\in\{A,B\}}|Q_n(j)-Q(j)| =\max\{|Q_n(A)-0.7|,\ |Q_n(B)-0.5|\} \xrightarrow{p}0 $$ by the law of large numbers for each sample mean. The maximum of these two errors also converges to zero: the probability it exceeds any $\eta>0$ is at most the sum of the two probabilities that an individual error exceeds $\eta$. The parameter space is finite, hence compact, and the theorem applies.

To see the separation directly, if an approximate maximizer selects $B$, then $Q_n(A)-Q_n(B)\leq r_n$, since $A$ is one of the available choices in the empirical supremum. But $Q_n(A)-Q_n(B)\xrightarrow{p}0.2$, whereas $r_n\xrightarrow{p}0$. The probability of selecting $B$ must therefore tend to zero. For a sample check, suppose five subjects have outcomes $Y_A=(1,1,0,1,0)$ and $Y_B=(0,1,0,0,1)$. The empirical means are $0.6$ and $0.4$, respectively, so treatment $A$ is the exact empirical maximizer for these observations.

Common Pitfall: Pointwise Convergence Is Not Enough

A tempting but incorrect argument is: “For each fixed $\theta$, $Q_n(\theta)$ approaches $Q(\theta)$, so the empirical maximizer must approach the population maximizer.” The parameter selected by maximization may itself vary with $n$. Pointwise convergence controls each fixed parameter, not necessarily the criterion at a moving sequence of parameters.

For a concrete illustration, take $\Theta=\mathbb{R}$, $Q(\theta)=-\theta^2$, and $Q_n(\theta)=-\theta^2+(n^2+1)\mathbf{1}\{\theta=n\}$. For every fixed $\theta$, eventually $\theta\ne n$, so $Q_n(\theta)=Q(\theta)$ eventually and pointwise convergence holds. Yet $Q_n(n)=1$, while $Q_n(\theta)\leq0$ for every $\theta\ne n$. Thus the unique empirical maximizer is $n$, which does not converge to the unique population maximizer $0$. Uniform convergence fails. In applications, verify uniform control (or a suitable alternative), rather than inferring it from pointwise convergence.

Compactness and continuity are convenient sufficient conditions for separation, not the only possible ones. Other arguments can establish a positive gap directly, including on a noncompact parameter space. The essential requirements are uniform accuracy of the criterion and enough separation to rule out parameters far from the target.

Key takeaway: Consistency follows when the population criterion has a uniquely separated optimum, the empirical criterion converges uniformly to it, and any error in maximization vanishes in probability. The theorem concerns criterion values; it does not require an exact optimizer.

Check Your Understanding

Use the theorem and its proof to assess each situation.

Why does continuity on a compact parameter space help establish a positive gap outside an $\varepsilon$-neighborhood of a unique maximizer?
In the consistency proof, which inequality uses the assumption that $\hat{\theta}_n$ is an approximate maximizer?
For the Bernoulli criterion on $[0.2,0.8]$, why is the bound involving $\log 4$ uniform over the parameter space?
What changes in the theorem if the empirical maximizer is exact rather than approximate?
Explain why pointwise convergence does not rule out the moving maximizers in the counterexample.