Uniform Convergence for Criteria and Scores

Why Uniform Convergence Matters

An empirical criterion or estimating function is calculated from a sample, so it changes when the sample changes. For a fixed parameter value, a law of large numbers can often show that this sample quantity approaches its population counterpart. But estimation compares values across many parameter values. To control those comparisons, we need convergence that holds uniformly over the parameter space.

For example, an estimator that maximizes an empirical criterion may be inconsistent with pointwise convergence alone: the empirical and population criteria could be close at each fixed parameter while still differing substantially at parameter values that move with the sample size. Uniform convergence rules out this kind of discrepancy. It supplies an important ingredient for consistency, but identification and appropriate behavior of the estimator are also needed.

Definitions and Sufficient Conditions

Let $Z_1,\ldots,Z_n$ be independent and identically distributed observations with the same distribution as $Z$, and let $\Theta$ be a parameter space. For a real-valued function $m_\theta(z)$, define the empirical and population criteria by

\mathbb P_n m_\theta = \frac{1}{n}\sum_{i=1}^n m_\theta(Z_i), \qquad P m_\theta = E[m_\theta(Z)].

Here the index $\theta$ ranges over $\Theta$. For an estimating function $\psi_\theta(z)$ taking values in $\mathbb R^q$, define $\mathbb P_n\psi_\theta$ and $P\psi_\theta$ in the same way, component by component.

Definition: The empirical criteria converge uniformly almost surely to their population criteria on $\Theta$ if $$ \sup_{\theta\in\Theta} \left|\mathbb P_n m_\theta-Pm_\theta\right| \xrightarrow{a.s.}0. $$ For a vector-valued estimating function, uniform convergence means $$ \sup_{\theta\in\Theta} \left\|\mathbb P_n\psi_\theta-P\psi_\theta\right\| \xrightarrow{a.s.}0, $$ where $\|\cdot\|$ is a norm on $\mathbb R^q$.

A standard sufficient condition for this uniform law of large numbers is compactness of the parameter space, continuity in the parameter, and an integrable envelope. The next result states one version for criteria. Measurability of the suprema below is assumed; it holds, for example, when the parameter space is a compact subset of $\mathbb R^d$, the sample functions are continuous, and the relevant suprema can be represented using a countable dense subset.

Theorem (Uniform strong law under an integrable envelope): Suppose $\Theta$ is a compact subset of $\mathbb R^d$, $Z_1,Z_2,\ldots$ are iid, and, with probability one, $\theta\mapsto m_\theta(Z)$ is continuous on $\Theta$. Suppose there is a measurable function $M$ such that $|m_\theta(Z)|\leq M(Z)$ for every $\theta\in\Theta$ almost surely and $E[M(Z)]<\infty$. Then $$ \sup_{\theta\in\Theta} \left|\mathbb P_n m_\theta-Pm_\theta\right| \xrightarrow{a.s.}0. $$

The continuity condition concerns the parameter index for almost every observation. The envelope controls all parameter values at once and makes the population criterion finite. Compactness permits approximation of the entire parameter space by a finite set of nearby parameter values.

Proof by Finite Nets

We prove the theorem. For $\delta>0$, define the modulus

\omega_\delta(z) = \sup_{\substack{\theta,\eta\in\Theta\\ \|\theta-\eta\|\leq\delta}} |m_\theta(z)-m_\eta(z)|.

By the envelope assumption, $0\leq\omega_\delta(Z)\leq 2M(Z)$, so it is integrable. For almost every $Z$, continuity on compact $\Theta$ implies uniform continuity; therefore $\omega_\delta(Z)\to 0$ as $\delta\downarrow0$. The dominated convergence theorem gives $E[\omega_\delta(Z)]\to0$.

Fix $\delta>0$. Compactness gives a finite $\delta$-net $\{\theta_1,\ldots,\theta_J\}\subset\Theta$: for every $\theta\in\Theta$, some $\theta_j$ satisfies $\|\theta-\theta_j\|\leq\delta$. For such a pair,

\left|\mathbb P_n m_\theta-Pm_\theta\right| \leq \left|\mathbb P_n m_{\theta_j}-Pm_{\theta_j}\right| +\mathbb P_n\omega_\delta+P\omega_\delta.

The ordinary strong law of large numbers applies to each of the finitely many $m_{\theta_j}(Z_i)$, which are integrable by the envelope assumption, and to $\omega_\delta(Z_i)$. Hence, almost surely,

\limsup_{n\to\infty} \sup_{\theta\in\Theta} \left|\mathbb P_n m_\theta-Pm_\theta\right| \leq 2E[\omega_\delta(Z)].

Apply this argument to a sequence $\delta_k\downarrow0$ and intersect the corresponding probability-one events. The left side is bounded by $2E[\omega_{\delta_k}(Z)]$ for every $k$, and these bounds tend to zero. The left side is nonnegative, so it equals zero almost surely. This proves the result. $\square$

Worked Examples

Worked Example: A Bernoulli Empirical Criterion

Let $X_1,\ldots,X_n$ be iid Bernoulli with success probability $p\in[0,1]$. Use the compact parameter space $\Theta=[0,1]$ and criterion $m_\theta(x)=-(x-\theta)^2$. Verify uniform convergence and identify the population maximizer.

Since $x,\theta\in[0,1]$, $|m_\theta(x)|\leq1$, so $M(x)=1$ is an integrable envelope. For every $x$, the criterion is continuous in $\theta$. The theorem therefore applies. We can also calculate the uniform error exactly. Write $\overline X_n=n^{-1}\sum_{i=1}^n X_i$. Because $X_i^2=X_i$,

\mathbb P_n m_\theta = -\overline X_n+2\theta\overline X_n-\theta^2, \qquad Pm_\theta = -p+2\theta p-\theta^2.

Subtracting gives $\mathbb P_n m_\theta-Pm_\theta=(2\theta-1)(\overline X_n-p)$. Since $\sup_{\theta\in[0,1]}|2\theta-1|=1$,

\sup_{\theta\in[0,1]} |\mathbb P_n m_\theta-Pm_\theta| = |\overline X_n-p| \xrightarrow{a.s.}0.

The population criterion is $Pm_\theta=-p(1-p)-(\theta-p)^2$, so it has the unique maximizer $\theta=p$. This example exhibits the ingredient needed for criterion-based consistency: the empirical criterion is uniformly close to a population criterion whose preferred parameter is identified.

Worked Example: Uniform Convergence of a Poisson Score

Suppose $X_1,\ldots,X_n$ are iid Poisson with mean $\lambda>0$. On a compact parameter interval $\Theta=[a,b]$, where $0<a<b<\infty$, consider the score contribution $\psi_\theta(x)=x/\theta-1$. Verify uniform convergence to its expectation.

For every $\theta\in[a,b]$, $|\psi_\theta(x)|\leq x/a+1$. This envelope has expectation $\lambda/a+1<\infty$, and $\psi_\theta(x)$ is continuous in $\theta$ for each $x$. The theorem applies. Directly, $P\psi_\theta=\lambda/\theta-1$, and

\sup_{\theta\in[a,b]} |\mathbb P_n\psi_\theta-P\psi_\theta| = \sup_{\theta\in[a,b]} \frac{|\overline X_n-\lambda|}{\theta} = \frac{|\overline X_n-\lambda|}{a} \xrightarrow{a.s.}0.

The last convergence follows from the strong law and $E[X_i]=\lambda$. Thus the empirical score is uniformly close to its population score on this interval. Uniform convergence alone does not establish consistency of a root: one must also address whether the population equation identifies the target and whether the selected sample root behaves appropriately.

Worked Example: Pointwise Convergence Can Fail to Be Uniform

Let $X_i$ be iid Bernoulli with $p=1/2$, but now take $\Theta=\mathbb R$ and $m_\theta(x)=-(x-\theta)^2$. At each fixed $\theta$, the strong law gives $\mathbb P_n m_\theta\to Pm_\theta$ almost surely. Show why that does not imply uniform convergence.

As in the first example,

\mathbb P_n m_\theta-Pm_\theta = (2\theta-1)(\overline X_n-1/2).

For every odd $n$, $\overline X_n$ is an integer divided by an odd integer and therefore cannot equal $1/2$. Consequently, for every odd $n$,

\sup_{\theta\in\mathbb R} |\mathbb P_n m_\theta-Pm_\theta| = \sup_{\theta\in\mathbb R} |2\theta-1|\,|\overline X_n-1/2| = \infty.

The failure is not a failure of the pointwise strong law. Rather, the parameter space is unbounded, and there is no finite integrable envelope controlling the criterion over all real $\theta$. Uniform convergence needs conditions that control the whole parameter space; pointwise convergence by itself provides no such control.

From Uniform Convergence to Consistency

For an empirical criterion, uniform convergence means that, with high probability for large $n$, no parameter value has an empirical criterion far from its population value. If the population criterion is uniquely optimized at the target and is separated from that optimum outside every fixed neighborhood, this uniform control prevents a parameter well away from the target from appearing empirically preferable merely because of sample fluctuations. The separation is an identification property; uniform convergence controls the empirical approximation.

For estimating functions, the parallel idea is that the empirical equation is uniformly close to its population equation. If the population equation has an isolated zero, this helps control where sample roots can occur. It does not, by itself, guarantee that a chosen root exists, is well behaved, or converges to that zero. Those additional questions are part of establishing consistency.

Common Pitfalls

Replacing uniform convergence with pointwise convergence. Proving convergence for each fixed $\theta$ does not control a supremum over $\theta$. A finite-net argument requires conditions that control changes between nearby parameter values.
Ignoring the envelope. Continuity alone does not bound the size of $m_\theta(Z)$ as $\theta$ varies. The integrable envelope in the theorem makes the needed averages finite and supports the dominated-convergence step.
Assuming uniform convergence proves consistency. Uniform convergence controls empirical approximation, not identification or the behavior of the estimator. These ingredients must also be established.
Assuming noncompactness always rules out a uniform law. The unbounded example shows that pointwise convergence need not be uniform; it does not show that uniform laws are impossible on noncompact spaces. Other conditions can sometimes provide the needed control.

Key takeaway: Compactness, continuity in the parameter, and an integrable envelope provide a useful sufficient route to a uniform law of large numbers. Uniform convergence controls the empirical approximation over the parameter space; identification and appropriate estimator behavior are still needed for consistency.

Check Your Understanding

Try these questions before moving on.

State the difference between pointwise convergence of empirical criteria and uniform convergence over $\Theta$.
In the uniform strong law, what role does the integrable envelope play in the proof?
Why does compactness allow a finite-net argument, and where does continuity enter that argument?
For the Poisson score example, calculate the population score and the supremum of the empirical score error.
What additional properties, beyond uniform convergence, are needed to use it as part of a consistency argument?