Consistency from Identification and Criterion Separation

Why Uniform Convergence Needs Identification

An extremum estimator chooses a parameter value that maximizes or minimizes a sample criterion. If the sample criterion is uniformly close to a population criterion, it is tempting to conclude that their optimizers must be close too. That conclusion needs another condition: the population criterion must meaningfully distinguish its optimum from parameter values far away. Otherwise, a small change in the criterion can make a distant parameter value look just as good.

This tutorial develops that condition, called criterion separation, and shows how it combines with uniform convergence to establish consistency. The argument is for maximization; changing signs gives the corresponding result for minimization.

Definition: Let $\Theta$ be a parameter space and let $Q:\Theta\to\mathbb{R}$ be a deterministic population criterion. A value $\theta_0\in\Theta$ is the uniquely identified maximizer if $Q(\theta_0)>Q(\theta)$ for every $\theta\ne\theta_0$. The criterion is separated at $\theta_0$ if, for every $\varepsilon>0$ for which the set is nonempty, $$ \sup_{\theta\in\Theta:\,\|\theta-\theta_0\|\ge\varepsilon} Q(\theta)<Q(\theta_0). $$ Equivalently, the gap $\delta_\varepsilon=Q(\theta_0)-\sup_{\|\theta-\theta_0\|\ge\varepsilon}Q(\theta)$ is strictly positive.

Unique maximization is a pointwise comparison: each individual $\theta\ne\theta_0$ has a lower criterion value. Separation is stronger because it requires a positive gap from the best criterion value among all parameters at least $\varepsilon$ away. This distinction matters when that set contains infinitely many values.

Write $Q_n:\Theta\to\mathbb{R}$ for the random sample criterion based on $n$ observations, and let $\hat{\theta}_n$ be a maximizer of $Q_n$. Uniform convergence in probability means that the largest absolute discrepancy over the entire parameter space becomes small in probability:

\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|\xrightarrow{p}0.

This is stronger than convergence at each fixed $\theta$: it controls the criterion even at parameter values that might depend on the sample.

The Consistency Result

Theorem: Suppose that (1) $\hat{\theta}_n$ is a measurable exact maximizer of $Q_n$ on $\Theta$, so $Q_n(\hat{\theta}_n)\ge Q_n(\theta)$ for every $\theta\in\Theta$; (2) $Q_n$ converges uniformly in probability to $Q$; and (3) $Q$ is separated at its unique maximizer $\theta_0$. Then $\hat{\theta}_n$ is consistent for $\theta_0$: $$ \hat{\theta}_n\xrightarrow{p}\theta_0. $$

The theorem assumes that the sample maximizer exists. Compactness of $\Theta$ and continuity of each sample criterion are common sufficient conditions for existence, but the proof uses only the stated exact-maximizer property. Later results can relax exact maximization; here it is part of the setup.

Proof: The Gap Beats the Uniform Error

Fix any $\varepsilon>0$ for which parameters at least $\varepsilon$ from $\theta_0$ exist, and let $\delta_\varepsilon>0$ be the separation gap. Define the uniform error $R_n=\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|$. On the event $R_n<\delta_\varepsilon/3$, every parameter $\theta$ with $\|\theta-\theta_0\|\ge\varepsilon$ satisfies

Q_n(\theta_0)-Q_n(\theta) \ge Q(\theta_0)-Q(\theta)-2R_n \ge \delta_\varepsilon-2R_n > \frac{\delta_\varepsilon}{3}>0.

The first inequality follows by bounding $Q_n(\theta_0)$ below by $Q(\theta_0)-R_n$ and $Q_n(\theta)$ above by $Q(\theta)+R_n$. Thus, on this event, no parameter at least $\varepsilon$ away can maximize $Q_n$: its sample criterion is strictly below the criterion at $\theta_0$. Therefore,

P(\|\hat{\theta}_n-\theta_0\|\ge\varepsilon) \le P(R_n\ge\delta_\varepsilon/3)\longrightarrow 0.

This holds for every $\varepsilon>0$, which is precisely convergence in probability of $\hat{\theta}_n$ to $\theta_0$. If there are no parameters outside the $\varepsilon$-neighborhood, the event on the left is empty and the conclusion for that $\varepsilon$ is immediate.

When Compactness and Continuity Give Separation

A useful way to verify separation is to check compactness and continuity. Suppose $\Theta$ is compact, $Q$ is continuous, and $\theta_0$ is its unique maximizer. For a fixed $\varepsilon>0$, the set $\{\theta\in\Theta:\|\theta-\theta_0\|\ge\varepsilon\}$ is closed in a compact set, hence compact. If it is nonempty, continuity means $Q$ attains a maximum on it. That maximum cannot equal $Q(\theta_0)$, since that would give another maximizer. It must therefore be strictly smaller, giving a positive gap.

Compactness and continuity are sufficient, not necessary. In applications, separation may be verified directly even when the parameter space is not compact. Conversely, a unique maximizer without separation is not enough.

Worked Example: A Bernoulli Criterion on a Compact Parameter Space

Suppose $Y_1,\ldots,Y_n$ are independent Bernoulli observations with success probability $p_0\in[0,1]$. Use the criterion $Q_n(p)=-n^{-1}\sum_{i=1}^n(Y_i-p)^2$ on $\Theta=[0,1]$. For a hypothetical sample of eight observations with five successes, find the maximizer and verify the ingredients for consistency.

Write $\bar{Y}=n^{-1}\sum_iY_i$. Since $Y_i^2=Y_i$, $Q_n(p)=-\bar{Y}+2p\bar{Y}-p^2$, a strictly concave quadratic maximized at $\hat p_n=\bar{Y}$. In the stated sample, $\bar{Y}=5/8=0.6250$. Its criterion value is $Q_n(0.625)=-[5(0.375)^2+3(0.625)^2]/8=-1.875/8=-0.2344$. At $p=0.5$, the value is $-[8(0.5)^2]/8=-0.2500$, so the reported sample maximizer has the higher criterion.

The population criterion is $Q(p)=-E[(Y_i-p)^2]=-p_0+2pp_0-p^2$. Completing the square gives $Q(p)=-(p-p_0)^2-p_0(1-p_0)$, so its unique maximizer is $p_0$, and its gap at distance at least $\varepsilon$ is at least $\varepsilon^2$, whenever that set is nonempty.

For uniform convergence, subtract the criteria: $Q_n(p)-Q(p)=(2p-1)(\bar{Y}-p_0)$. Since $|2p-1|\le1$ on $[0,1]$, $\sup_{p\in[0,1]}|Q_n(p)-Q(p)|\le|\bar{Y}-p_0|\xrightarrow{p}0$ by the law of large numbers for Bernoulli observations. The parameter space is compact and both criteria are continuous; the sample maximizer exists. The theorem therefore gives $\hat p_n\xrightarrow{p}p_0$. The eight-observation calculation illustrates the estimator; consistency follows from the stated sampling assumptions, not from that single sample.

Worked Example: A Linear-Regression Criterion

Suppose $(X_i,Y_i)$ are independent and identically distributed, $E[X_i^2]$ and $E[Y_i^2]$ are finite, and $E[X_i^2]>0$. Assume the population linear-projection coefficient is $\beta_0=E[X_iY_i]/E[X_i^2]$. On a compact interval $B$ containing $\beta_0$, consider $Q_n(\beta)=-n^{-1}\sum_i(Y_i-X_i\beta)^2$ and $Q(\beta)=-E[(Y_i-X_i\beta)^2]$.

Expanding the population criterion shows that its $\beta$-dependent terms are $2\beta E[X_iY_i]-\beta^2E[X_i^2]$. Completing the square, or differentiating this quadratic, gives the unique maximizer $\beta_0$. More precisely, $Q(\beta_0)-Q(\beta)=E[X_i^2](\beta-\beta_0)^2$. Thus the gap at distance at least $\varepsilon$ is at least $E[X_i^2]\varepsilon^2>0$, if such values occur in $B$.

For a hypothetical sample $(X_i,Y_i)=(1,2),(1,3),(2,4),(2,5)$, the unconstrained sample maximizer is $\hat\beta=(\sum_iX_iY_i)/(\sum_iX_i^2)=23/10=2.300$. The residuals are $-0.3,0.7,-0.6,0.4$, so the sample criterion at this value is $Q_n(2.3)=-(0.09+0.49+0.36+0.16)/4=-0.2750$.

To check uniform convergence, expand the sample criterion: $Q_n(\beta)=-\overline{Y^2}+2\beta\overline{XY}-\beta^2\overline{X^2}$, where each bar denotes the corresponding sample average. The population criterion has the same expansion with expectations replacing averages. The law of large numbers gives convergence of all three averages. Because $\beta$ is restricted to a bounded interval, the absolute difference between the criteria is bounded by $|\overline{Y^2}-E[Y_i^2]|+2\sup_{\beta\in B}|\beta|\,|\overline{XY}-E[X_iY_i]|+\sup_{\beta\in B}\beta^2\,|\overline{X^2}-E[X_i^2]|$, which converges in probability to zero. Compactness gives an attained sample maximum, and the theorem yields consistency for $\beta_0$. The assumptions describe the hypothetical data-generating process; the four listed observations alone cannot verify them.

Why Separation Cannot Be Omitted

The next constructed example shows that even uniform convergence and a unique population maximizer do not suffice when the parameter space is not compact and separation fails. It is a criterion-sequence example rather than an empirical study.

Worked Example: Uniform Convergence but a Distant Maximizer

Let $\Theta=[0,\infty)$ and $Q(\theta)=-\theta^2/(1+\theta^4)$. This population criterion is zero at $\theta_0=0$ and negative at every $\theta>0$, so its unique maximizer is zero. However, for every $\varepsilon>0$, $Q(\theta)$ approaches zero as $\theta$ grows without bound while $\theta\ge\varepsilon$. Hence the supremum outside the $\varepsilon$-neighborhood is zero, and the separation gap is zero.

For integers $n\ge2$, define $\phi(u)=\max(1-4|u|,0)$ and $Q_n(\theta)=Q(\theta)+(2/n)\phi(\theta-n)$. The added triangular bump is supported on $[n-1/4,n+1/4]$, equals one at $\theta=n$, and satisfies $\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|=2/n\to0$. Thus convergence is uniform, deterministically.

At the center of the bump, $Q_n(n)=-n^2/(1+n^4)+2/n>0$ for $n\ge2$. Outside the bump, $Q_n(\theta)=Q(\theta)\le0$. The continuous criterion therefore attains a global maximum inside the bump, and every maximizer satisfies $\hat\theta_n\ge n-1/4$. In particular, $\hat\theta_n$ does not converge to zero. The small uniform perturbation can defeat the population criterion arbitrarily far away precisely because the population gap there is not positive.

Common Mistake and Practical Checklist

A common mistake is to reason, “The sample criterion is uniformly close to the population criterion, and the population optimum is unique, so the sample optimizer must be consistent.” The constructed example shows the missing step: uniqueness compares each fixed alternative with the optimum, while consistency must rule out the best alternative across an entire region that can extend arbitrarily far away.

For a maximizing estimator, check the argument in this order:

Specify the parameter space and the sample and population criteria, including whether the estimator maximizes or minimizes.
Establish that the population criterion has a unique optimum at the target parameter.
Verify a positive criterion gap outside every fixed neighborhood; compactness and continuity are one standard route.
Show uniform convergence over the relevant parameter space, not only convergence at each fixed parameter value.
Ensure that the sample extremum is attained, as required by the theorem’s exact-maximizer condition.

For minimization, apply the same logic to $-Q_n$ and $-Q$, or reverse the inequalities: values at least $\varepsilon$ from the target must have a population criterion at least a positive amount larger than the minimum. The essential structure is unchanged: uniform error smaller than a fraction of the population gap prevents an extremum from lying outside the target neighborhood.

Check Your Understanding

Use the definitions and proof above to answer the following.

In the consistency proof, why is the criterion difference bounded using $2R_n$, rather than $R_n$?
For a maximizing criterion on a compact parameter space, explain how continuity and a unique maximizer imply a positive gap outside a fixed neighborhood.
In the Bernoulli example, what is the population criterion gap at a parameter $p$, relative to $p_0$?
In the linear-regression example, which moment condition ensures that distinct coefficient values have distinct population criterion values?
Why does the distant-bump example not contradict the consistency theorem?