Uniform Convergence of Sample Criteria

Why Uniform Convergence Matters

A sample criterion $Q_n(\theta)$ is a random function of a parameter $\theta$. In many estimation problems, the criterion is computed from a sample and compared with a deterministic population criterion $Q(\theta)$. Pointwise convergence says that for each fixed parameter value, the sample criterion gets close to its population counterpart. Uniform convergence asks for something stronger: does it get close at once across the whole parameter space?

That distinction matters when the parameter value of interest may vary with the sample. A bound that holds only at each fixed $\theta$ does not automatically control the criterion at every value in a continuum. This tutorial develops a finite-net argument for obtaining uniform convergence and shows how regularity bounds can supply the needed control between nearby parameter values.

Definition: Let $Q_n(\theta)$ be a random criterion and $Q(\theta)$ a deterministic function on a parameter space $\Theta$. They converge uniformly in probability on $\Theta$ if, for every $\varepsilon>0$, $$ P\!\left(\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|>\varepsilon\right)\longrightarrow 0. $$ The supremum measures the largest discrepancy anywhere in $\Theta$. By contrast, pointwise convergence in probability requires $P(|Q_n(\theta)-Q(\theta)|>\varepsilon)\to 0$ separately for each fixed $\theta$.

The uniform statement is stronger: if it holds, pointwise convergence follows at each $\theta$, since $|Q_n(\theta)-Q(\theta)|\leq\sup_{\vartheta\in\Theta}|Q_n(\vartheta)-Q(\vartheta)|$. The reverse implication does not hold in general. Uniform convergence concerns the whole criterion, not just its values at any fixed finite collection of parameter values.

Finite Parameter Spaces

For a finite parameter space, pointwise convergence is enough. If $\Theta=\{\theta_1,\ldots,\theta_K\}$, then $$ \sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)| =\max_{1\leq j\leq K}|Q_n(\theta_j)-Q(\theta_j)|. $$ For any $\varepsilon>0$, the probability that this maximum exceeds $\varepsilon$ is at most the sum of the probabilities that each individual error exceeds $\varepsilon$. Each probability tends to zero by pointwise convergence, and there are only $K$ of them. Their sum therefore tends to zero.

Worked Example: A Criterion on Two Parameter Values

Suppose $Z_1,\ldots,Z_n$ are independent Bernoulli random variables with $P(Z_i=1)=0.5$, and let $\Theta=\{0,1\}$. Consider the sample and population criteria $$ Q_n(\theta)=\frac{1}{n}\sum_{i=1}^n(Z_i-\theta)^2, \qquad Q(\theta)=E[(Z_i-\theta)^2]. $$ For either fixed $\theta$, the summands are independent and bounded, so the Weak Law of Large Numbers gives $Q_n(\theta)\to Q(\theta)$ in probability.

Because $Z_i$ takes only 0 and 1, $Q(0)=E[Z_i^2]=0.5$, while $Q(1)=E[(Z_i-1)^2]=0.5$. Suppose a hypothetical sample of $n=100$ has sample mean $\bar Z=0.52$. Since $Z_i^2=Z_i$, $$ Q_n(0)=\frac{1}{100}\sum_{i=1}^{100}Z_i^2=0.52, \qquad Q_n(1)=\frac{1}{100}\sum_{i=1}^{100}(1-Z_i)^2=1-0.52=0.48. $$

For this sample, the largest discrepancy over the two possible parameter values is $$ \max\{|0.52-0.50|,\ |0.48-0.50|\}=0.02. $$ The example illustrates the finite-space argument: convergence at both values implies uniform convergence on this particular $\Theta$. A single sample calculation illustrates the criterion’s discrepancy; the convergence claim comes from the probability argument, not from this one sample.

Finite Nets and Stochastic Equicontinuity

A compact subset of a finite-dimensional Euclidean space can be covered by finitely many balls of any positive radius. The centers of such a cover form a finite net. If a criterion is well behaved between nearby parameters, checking its error at the net points can then control the error everywhere.

Write $G_n(\theta)=Q_n(\theta)-Q(\theta)$. The relevant regularity condition is stochastic equicontinuity: for every $\varepsilon>0$, $$ \lim_{\delta\downarrow 0}\ \limsup_{n\to\infty} P\!\left( \sup_{\substack{\theta,\theta'\in\Theta\\ \|\theta-\theta'\|<\delta}} |G_n(\theta)-G_n(\theta')|>\varepsilon \right)=0. $$ In words, at sufficiently small parameter distances, the increments of the error process are small with high probability for large samples. This condition concerns increments of $Q_n-Q$, not just increments of $Q_n$.

Theorem: Suppose $\Theta$ is a compact subset of a finite-dimensional Euclidean space, $Q_n(\theta)\to Q(\theta)$ in probability for each fixed $\theta$, and $G_n=Q_n-Q$ is stochastically equicontinuous on $\Theta$. Assuming the suprema in question are measurable (or the usual conditions ensuring their measurability), $Q_n$ converges uniformly in probability to $Q$ on $\Theta$.

The proof separates error at net points from error between net points. Fix a tolerance $\varepsilon>0$ and an arbitrary probability tolerance $\eta>0$. By stochastic equicontinuity, choose $\delta>0$ so that, for all sufficiently large $n$, the probability that the error process changes by more than $\varepsilon/2$ over a distance less than $\delta$ is at most $\eta$. Compactness supplies a finite $\delta$-net $\{\theta_1,\ldots,\theta_K\}$. For every $\theta$, some net point lies within $\delta$, so $$ \sup_{\theta\in\Theta}|G_n(\theta)| \leq \max_{1\leq j\leq K}|G_n(\theta_j)| +\sup_{\substack{\theta,\theta'\in\Theta\\ \|\theta-\theta'\|<\delta}}|G_n(\theta)-G_n(\theta')|. $$

The second term exceeds $\varepsilon/2$ with probability at most $\eta$ for large $n$. For the first term, the union bound gives $$ P\!\left(\max_{1\leq j\leq K}|G_n(\theta_j)|>\varepsilon/2\right) \leq \sum_{j=1}^{K}P(|G_n(\theta_j)|>\varepsilon/2)\longrightarrow 0, $$ because $K$ is finite and convergence holds at each fixed net point. Thus the limit superior of the probability that the supremum error exceeds $\varepsilon$ is at most $\eta$. Since $\eta$ can be made arbitrarily small, that probability tends to zero. This establishes uniform convergence.

Finite-dimensionality enters through the finite net: compact subsets of finite-dimensional Euclidean space have finite covers at any fixed positive radius. Compactness and finite-dimensionality do not, by themselves, give convergence; the pointwise convergence and control of increments are also essential.

Envelope Bounds as a Route to Equicontinuity

A common setting is an average criterion $Q_n(\theta)=n^{-1}\sum_{i=1}^n m(Z_i,\theta)$, where $Z_1,\ldots,Z_n$ are independent and identically distributed observations and $Q(\theta)=E[m(Z_i,\theta)]$. Suppose that, for all $\theta,\theta'\in\Theta$, $$ |m(z,\theta)-m(z,\theta')|\leq M(z)\|\theta-\theta'\|, \qquad E[M(Z_i)]<\infty. $$ The nonnegative function $M$ is an integrable envelope for changes in the criterion as the parameter varies. The bound implies that $Q$ is finite and Lipschitz whenever the expectations defining it exist.

Let $P_nM=n^{-1}\sum_{i=1}^n M(Z_i)$ and $PM=E[M(Z_i)]$. The triangle inequality gives $$ |G_n(\theta)-G_n(\theta')| \leq (P_nM+PM)\|\theta-\theta'\|. $$ By the Weak Law of Large Numbers with finite mean, $P_nM\to PM$ in probability, so $P_nM+PM$ is bounded in probability. Consequently, for any fixed $\varepsilon>0$, the probability that the increment bound exceeds $\varepsilon$ over distances smaller than $\delta$ can be made arbitrarily small by taking $\delta$ sufficiently small. This verifies stochastic equicontinuity. If $m(Z_i,\theta)$ is integrable at each fixed $\theta$, the Weak Law also gives pointwise convergence; on a compact finite-dimensional $\Theta$, the finite-net result then yields uniform convergence.

Worked Example: Uniform Convergence for a Squared-Error Criterion

Let $Z_1,\ldots,Z_n$ be independent and identically distributed standard normal random variables, and let $\Theta=[-1,1]$. Define $$ Q_n(\theta)=\frac{1}{n}\sum_{i=1}^n(Z_i-\theta)^2, \qquad Q(\theta)=E[(Z_i-\theta)^2]=1+\theta^2. $$ The population formula uses $E[Z_i]=0$ and $E[Z_i^2]=1$. For each fixed $\theta$, the Weak Law applies to the squared-error summands, which have finite expectation.

To find an envelope, factor the difference of two squared errors: $$ |(z-\theta)^2-(z-\theta')^2| =|\theta-\theta'||2z-\theta-\theta'| \leq (2|z|+2)|\theta-\theta'|, $$ since $\theta,\theta'\in[-1,1]$. Thus $M(z)=2|z|+2$ is an envelope and $E[M(Z_i)]<\infty$. The envelope argument establishes stochastic equicontinuity, and compactness of $[-1,1]$ gives uniform convergence in probability.

The calculation can also be seen directly. For a hypothetical sample of $n=100$, suppose the sample summaries are $\bar Z=0.10$ and $n^{-1}\sum_i Z_i^2=1.08$. Expanding the square, $$ Q_n(\theta)-Q(\theta) =\left(\frac{1}{n}\sum_{i=1}^n Z_i^2-1\right)-2\theta\bar Z =0.08-0.20\theta. $$ On $[-1,1]$, this linear discrepancy ranges from $0.28$ at $\theta=-1$ to $-0.12$ at $\theta=1$. Therefore, for this sample, $$ \sup_{\theta\in[-1,1]}|Q_n(\theta)-Q(\theta)| =\max\{|0.28|,|-0.12|\}=0.28. $$ The sample’s largest discrepancy is dimensionless, as are $Z_i$ and $\theta$. Uniform convergence is an asymptotic probability statement: the Weak Law and envelope argument show that this supremum tends to zero in probability as $n$ grows, not that every finite-sample supremum must be small.

Why Pointwise Convergence Alone Is Not Enough

A compact, finite-dimensional parameter space does not make pointwise convergence uniform automatically. The following example shows how a discrepancy can move with $n$, avoiding every fixed parameter eventually while remaining large somewhere in the parameter space.

Worked Example: A Moving Spike

Take $\Theta=[0,1]$, let $Q(\theta)=0$, and define the deterministic sequence $$ Q_n(\theta)=\mathbf{1}\{\theta=1/n\}. $$ For any fixed $\theta\in[0,1]$, the equality $\theta=1/n$ can hold for at most one positive integer $n$. Thus $Q_n(\theta)\to 0=Q(\theta)$ for every fixed $\theta$.

Nevertheless, for each $n$, choosing $\theta=1/n$ gives $Q_n(\theta)=1$. Hence $$ \sup_{\theta\in[0,1]}|Q_n(\theta)-Q(\theta)|=1 $$ for every $n$, so uniform convergence fails.

The increment condition fails as well. For any fixed $\delta>0$, take $n$ large enough that $1/n+\delta/2\leq 1$. The two parameters $1/n$ and $1/n+\delta/2$ are less than $\delta$ apart, but the criterion changes by 1 between them. Thus arbitrarily small neighborhoods do not control the increments, even though the parameter space is one-dimensional and compact.

Common Pitfall

Incorrect reasoning: “The criterion converges in probability at each parameter value, and the parameter space is finite-dimensional and compact, so convergence must be uniform.”
Correction: Compactness gives finite nets, but pointwise convergence controls only the values at a fixed finite set of net points. A separate condition, such as stochastic equicontinuity, is needed to control the criterion between those points.
Another distinction: Uniform convergence in probability is not a statement that the criterion is close for every possible sample. It says that the probability of a supremum discrepancy larger than any fixed positive tolerance tends to zero.

Key takeaway: A standard route to uniform convergence in probability combines pointwise convergence with control over nearby parameter values. On a compact finite-dimensional parameter space, an integrable envelope can provide that control by bounding criterion increments and establishing stochastic equicontinuity.

Check Your Understanding

Try these questions before moving on.

State the difference between pointwise convergence in probability and uniform convergence in probability for $Q_n(\theta)$.
Why does pointwise convergence imply uniform convergence when the parameter space has only finitely many points?
In the finite-net argument, what role does stochastic equicontinuity play after the criterion has been controlled at the net points?
Suppose $|m(z,\theta)-m(z,\theta')|\leq M(z)\|\theta-\theta'\|$. What moment condition on $M(Z_i)$ supports the envelope argument described here?
For the moving-spike example, why does compactness of $[0,1]$ fail to establish uniform convergence?