Why Convergence of Criteria Can Locate an Optimum
An extremum estimator chooses a parameter value that makes a sample criterion as small or as large as possible. To show that this estimator is consistent, it is not enough to know that the sample criterion is close to its population counterpart at a few parameter values. The key is to control the discrepancy uniformly over the parameter space: then a sample optimum cannot remain far from the population optimum without producing a detectable criterion difference.
The argmin theorem makes this reasoning precise. Its central argument is short: if the population criterion is separated from its minimum outside any neighborhood of the target, then a sample minimizer far away would require the sample criterion to have a correspondingly large error. Uniform convergence makes that error unlikely.
Conditions for the Argmin Theorem
Use the usual Euclidean distance on \(\Theta\subseteq\mathbb{R}^k\). The population criterion must have a unique minimizer \(\theta_0\), and the sample criterion must have a minimizer. A convenient set of sufficient conditions ensures these properties and provides the separation needed for the proof.
- \(\Theta\) is compact and \(\theta_0\in\Theta\).
- \(Q\) is continuous on \(\Theta\), and \(\theta_0\) is its unique minimizer.
- \(Q_n\) converges uniformly in probability to \(Q\), meaning that for every \(\varepsilon>0\), \[ P\left(\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|>\varepsilon\right)\to 0. \]
- For each \(n\), \(\hat{\theta}_n\) is a measurable minimizer of \(Q_n\) on \(\Theta\).
The compactness and continuity conditions are a useful way to obtain separation. For a fixed \(\varepsilon>0\), the set of parameter values at least \(\varepsilon\) from \(\theta_0\) is closed within the compact space \(\Theta\), and is therefore compact. If it is nonempty, continuity ensures that \(Q(\theta)-Q(\theta_0)\) attains its minimum on this set. Uniqueness of \(\theta_0\) means that the difference is strictly positive at every point in the set, so its attained minimum is a positive number, \(\delta_\varepsilon\). If the set is empty, no parameter value in \(\Theta\) is that far from \(\theta_0\), and the corresponding probability bound is automatic.
Compactness also helps ensure existence of optima: a continuous real-valued function on a compact parameter space attains its minimum and maximum. In the theorem, the sample-minimizer condition is stated explicitly so that the conclusion does not depend on an implicit existence or selection assumption. The direct separation version can also be used when the compactness-and-continuity route is unavailable.
Why Uniform Convergence Forces the Minimizer to Be Close
Here is the central inequality. Write the uniform criterion error as \(D_n=\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|\). On the event that \(\hat{\theta}_n\) is at least \(\varepsilon\) from \(\theta_0\), separation gives a population criterion gap of at least \(\delta_\varepsilon\). But \(\hat{\theta}_n\) minimizes the sample criterion, so \(Q_n(\hat{\theta}_n)-Q_n(\theta_0)\leq 0\). Adding and subtracting those sample criterion values yields
Thus, if \(\|\hat{\theta}_n-\theta_0\|\geq\varepsilon\), then \(\delta_\varepsilon\leq 2D_n\). Taking probabilities gives
The last convergence follows from uniform convergence in probability. This is precisely \(\hat{\theta}_n\to_p\theta_0\). The bound also shows what each assumption contributes: separation supplies a nonzero criterion gap, while uniform convergence prevents sample error from concealing that gap. The argument applies to every sample minimizer, not just a specially chosen one.
Worked Example: A Sample Least-Squares Criterion
Suppose hypothetical observations \(Y_1,\ldots,Y_n\) are independent and identically distributed, with \(P(Y_i=2)=P(Y_i=6)=1/2\). Consider estimating a scalar \(\theta\in\Theta=[0,8]\) by minimizing the average squared error \(Q_n(\theta)=n^{-1}\sum_{i=1}^n(Y_i-\theta)^2\). Verify the conditions and identify the population optimum.
Let \(p_n\) be the fraction of observations equal to 2. Then \(1-p_n\) is the fraction equal to 6, so \[ Q_n(\theta)=p_n(2-\theta)^2+(1-p_n)(6-\theta)^2. \] The population criterion is \[ Q(\theta)=\tfrac12(2-\theta)^2+\tfrac12(6-\theta)^2=(\theta-4)^2+4. \] It is continuous on the compact interval \([0,8]\) and has the unique minimizer \(\theta_0=4\).
To verify uniform convergence, subtract the two criteria: \[ Q_n(\theta)-Q(\theta)=(p_n-\tfrac12)\{(2-\theta)^2-(6-\theta)^2\} =(p_n-\tfrac12)(8\theta-32). \] For \(\theta\in[0,8]\), the largest possible value of \(|8\theta-32|\) is 32. Therefore \[ \sup_{\theta\in[0,8]}|Q_n(\theta)-Q(\theta)|=32|p_n-\tfrac12|\to_p 0, \] because the law of large numbers gives \(p_n\to_p1/2\). The sample criterion is continuous, so it attains a minimum on \([0,8]\). In a hypothetical sample of 10 observations with six values equal to 2 and four equal to 6, \(p_n=0.6\); minimizing the displayed quadratic gives \(\hat{\theta}_n=6-4p_n=6-4(0.6)=3.6\). This particular estimate need not equal 4, but the theorem shows that the sample minimizer converges in probability to 4 as the sample size grows.
Maximization and a Likelihood Example
Maximization requires no new proof. If \(\hat{\theta}_n\) maximizes \(Q_n\) and \(\theta_0\) uniquely maximizes \(Q\), then \(-\hat{\theta}_n\) is not the relevant transformation; rather, minimizing \(-Q_n\) is exactly equivalent to maximizing \(Q_n\). Apply the argmin theorem to the criteria \(-Q_n\) and \(-Q\). Compactness, continuity, uniform convergence, and separation from the unique maximum then imply \(\hat{\theta}_n\to_p\theta_0\).
Worked Example: Maximizing a Bernoulli Log-Likelihood
Suppose hypothetical outcomes \(Y_i\) are independent Bernoulli random variables with success probability \(p_0=0.6\), and the parameter space is \(\Theta=[0.1,0.9]\). The normalized sample log-likelihood is \[ Q_n(p)=\hat{p}_n\log p+(1-\hat{p}_n)\log(1-p), \] where \(\hat{p}_n=n^{-1}\sum_{i=1}^nY_i\). The population criterion replaces \(\hat{p}_n\) by \(0.6\): \[ Q(p)=0.6\log p+0.4\log(1-p). \]
On \([0.1,0.9]\), both criteria are continuous. The derivative of \(Q\) is \(0.6/p-0.4/(1-p)\), which is zero at \(p=0.6\); its second derivative is \(-0.6/p^2-0.4/(1-p)^2<0\). Thus \(Q\) is strictly concave and has a unique maximum at \(p_0=0.6\).
For every \(p\in[0.1,0.9]\), \[ |Q_n(p)-Q(p)|=|\hat{p}_n-0.6|\left|\log\frac{p}{1-p}\right| \leq |\hat{p}_n-0.6|\log 9. \] Since \(\hat{p}_n\to_p0.6\) by the law of large numbers, this upper bound shows uniform convergence in probability. The maximizing version of the theorem therefore gives consistency of the sample maximizer. For example, if a hypothetical sample of 100 outcomes contains 58 successes, then \(\hat{p}_n=58/100=0.58\). Differentiating \(Q_n\) gives \(\hat{p}_n/p-(1-\hat{p}_n)/(1-p)=0\), whose solution is \(p=0.58\), inside the parameter space. This finite-sample estimate is 0.58, while the theorem concerns its convergence in probability to 0.6 as \(n\) grows.
Why Pointwise Convergence Is Not Enough
A common mistake is to replace uniform convergence with pointwise convergence. Pointwise convergence controls \(Q_n(\theta)\) for each fixed \(\theta\), but the sample minimizer is itself selected using the entire criterion and can move with \(n\). It can therefore follow a narrow region where the sample criterion is unusually low, even though that region eventually avoids every fixed parameter value.
Worked Example: Pointwise Convergence With a Misleading Minimizer
Let \(\Theta=[0,1]\), let \(Q(\theta)=\theta\), and set \(\theta_n=1/2+1/(n+4)\) and \(w_n=1/(n+4)^2\). Define the continuous sample criterion \[ Q_n(\theta)=\theta-2\max\left\{1-\frac{|\theta-\theta_n|}{w_n},\,0\right\}. \] The second term is a narrow downward spike centered at \(\theta_n\). At the center, \(Q_n(\theta_n)=\theta_n-2<0\). Outside the spike, \(Q_n(\theta)=\theta\geq0\); within the spike, the criterion decreases toward the center from the left and increases away from it on the right. Hence its unique minimizer is \(\theta_n\), which converges to \(1/2\), not to the unique minimizer \(0\) of \(Q\).
Nevertheless, \(Q_n\) converges pointwise to \(Q\). For any fixed \(\theta\neq1/2\), the spike eventually moves away from \(\theta\). At \(\theta=1/2\), its distance from the spike center is \(1/(n+4)\), which eventually exceeds the spike width \(1/(n+4)^2\); the spike term is then zero there as well. But uniform convergence fails: at the moving center, \[ |Q_n(\theta_n)-Q(\theta_n)|=2 \] for every \(n\). This example isolates the role of uniform control: even continuous criteria on a compact space can have misleading minimizers when convergence is only pointwise.
Common Pitfall and Practical Use
Do not conclude that an extremum estimator is consistent merely because the sample criterion converges to the population criterion at each fixed parameter value. Check uniform convergence, or use another result that establishes the needed control. Also check that the population optimum is unique and separated: if several parameter values minimize \(Q\), a sample minimizer may converge to one of them without converging to a designated single value. For maximization, state the criterion orientation clearly and apply the result to the negative criterion or use the equivalent maximum version.
In applications, the theorem is a bridge rather than a complete proof of consistency. One must still establish that the criterion is well-defined, that an optimizer exists, that the population target is uniquely identified, and that the required uniform convergence holds under the data-generating assumptions. The next step in analyzing an estimator may concern optimization that is only approximate; the exact-minimizer argument here relies on the sample optimum inequality \(Q_n(\hat{\theta}_n)\leq Q_n(\theta_0)\).
Check Your Understanding
Use the theorem and its proof to answer the following questions.
- Which assumption provides a positive population criterion gap outside an \(\varepsilon\)-neighborhood of the unique minimizer?
- In the proof, why is the sample criterion difference \(Q_n(\hat{\theta}_n)-Q_n(\theta_0)\) nonpositive?
- If the uniform criterion error is at most \(\delta_\varepsilon/3\), can a sample minimizer be at least \(\varepsilon\) from \(\theta_0\)? Explain using the inequality in the proof.
- How can the maximization result be derived from the minimization result?
- In the downward-spike example, why does pointwise convergence hold while the minimizers fail to approach the population minimizer?