Compactness and Continuity Conditions for ERM Consistency

From Uniform Risk Convergence to Parameter Consistency

An estimator can have nearly optimal empirical risk without being close to the parameter that minimizes population risk. To conclude that the estimated parameter itself converges, we need a way to translate small risk differences into small parameter differences. Compactness, continuity, and a unique population minimizer provide that link: parameters a fixed distance from the minimizer must have a strictly larger population risk.

Let \(Z_1,\ldots,Z_n\) be independent observations from a common distribution \(P\), and let \(\Theta\) be a parameter space. For a loss \(\ell_\theta(z)\), define the population and empirical risks by

An empirical risk minimizer (ERM) is a parameter \(\hat\theta_n\) that minimizes \(R_n\) over \(\Theta\). The argument below also permits approximate minimizers. Uniform convergence means that the largest risk discrepancy over the whole parameter space tends to zero:

Why Compactness and Continuity Matter

Assume \(\Theta\) is a compact metric space and \(R\) is continuous. The extreme value theorem ensures that \(R\) attains its minimum. Suppose the minimizer \(\theta^\star\) is unique. For any fixed \(\varepsilon>0\), the set of parameters at least \(\varepsilon\) away from \(\theta^\star\) is closed and hence compact. If that set is nonempty, continuity ensures that \(R\) attains a minimum on it. Uniqueness then forces that minimum to be strictly larger than \(R(\theta^\star)\).

This gives a positive risk gap outside every fixed neighborhood of \(\theta^\star\). It is the key deterministic fact: a parameter that is far from \(\theta^\star\) cannot have population risk arbitrarily close to the minimum. Uniform convergence then ensures that empirical risks preserve this separation with high probability.

Proof. Fix \(\varepsilon>0\). If no parameter lies at least \(\varepsilon\) from \(\theta^\star\), then the event \(d(\hat\theta_n,\theta^\star)\geq\varepsilon\) is empty. Otherwise, let \(C_\varepsilon=\{\theta\in\Theta:d(\theta,\theta^\star)\geq\varepsilon\}\). This is a nonempty compact set. By continuity, \(R\) attains its minimum on \(C_\varepsilon\); by uniqueness of \(\theta^\star\), that minimum exceeds \(R(\theta^\star)\). Thus

Write \(U_n=\sup_{\theta\in\Theta}|R_n(\theta)-R(\theta)|\). On the event \(d(\hat\theta_n,\theta^\star)\geq\varepsilon\), the gap definition and the uniform error bound give

Meanwhile, the approximate-minimizer property and the fact that \(\theta^\star\in\Theta\) give

Combining the inequalities shows that the event \(d(\hat\theta_n,\theta^\star)\geq\varepsilon\) can occur only if \(g_\varepsilon\leq 2U_n+\delta_n\). Because \(U_n\xrightarrow{p}0\) and \(\delta_n\xrightarrow{p}0\), the probability of this last event tends to zero. This holds for every \(\varepsilon>0\), proving \(\hat\theta_n\xrightarrow{p}\theta^\star\). Continuity on compact \(\Theta\) also ensures that each \(R_n\) attains its minimum, whenever \(R_n\) is continuous. \(\square\)

Worked Examples

A Lipschitz Condition That Gives Uniform Convergence

Uniform convergence is an assumption in the consistency theorem, but a useful sufficient condition can be checked directly. A common approach is to control how rapidly the loss can change as the parameter changes, then cover the compact parameter space with finitely many nearby points.

Proof. Compactness gives a finite diameter \(D=\sup_{\theta,\eta\in\Theta}\|\theta-\eta\|\). The Lipschitz condition implies

so every loss is integrable. Taking expectations in the Lipschitz inequality gives \(|R(\theta)-R(\eta)|\leq E[L(Z)]\|\theta-\eta\|\), hence \(R\) is continuous.

Fix a radius \(a>0\), and choose a finite \(a\)-net \(\theta_1,\ldots,\theta_m\) for \(\Theta\): every \(\theta\in\Theta\) is within distance \(a\) of at least one net point. For such a point \(\theta_j\), the triangle inequality and the Lipschitz condition give

Taking the supremum over \(\Theta\) bounds it by the maximum discrepancy over the finite net plus the second term. For fixed \(a\), the finite maximum converges to zero in probability by the weak law of large numbers at each net point and a union bound. The weak law also gives \(n^{-1}\sum_i L(Z_i)\xrightarrow{p}E[L(Z)]\). Given any target error \(b>0\), first choose \(a\) small enough that \(2aE[L(Z)]<b/2\) (if \(E[L(Z)]=0\), this term is zero for every \(a\)); then, for that fixed net, the probability that the displayed bound exceeds \(b\) tends to zero. Therefore the supremum discrepancy converges to zero in probability. \(\square\)

What the Conditions Do—and Do Not—Guarantee

The compactness and continuity assumptions serve different purposes. Compactness and continuity ensure that a minimum is attained; uniqueness gives a positive risk gap outside each neighborhood; uniform convergence ensures empirical risk cannot erase that gap. The finite-net proposition supplies one route to uniform convergence, not a consequence of compactness alone.

A common mistake is to argue that convergence of \(R_n(\theta)\) to \(R(\theta)\) for each fixed \(\theta\) automatically gives consistency of the minimizer. Pointwise convergence controls one parameter at a time, whereas an empirical minimizer is selected by comparing risks across the entire parameter space. The correction is to establish uniform convergence, or another condition strong enough to control those comparisons.

A second distinction is between parameter consistency and risk consistency. Uniform convergence and the ERM property can make the excess population risk small, but a small risk difference need not imply a small parameter difference unless the population risk separates distant parameters from its minimizer. The compactness, continuity, and uniqueness argument supplies that separation here. These conditions establish consistency, not a rate of convergence or an asymptotic distribution.