Pseudo-True Limits Under Model Misspecification

Why a Misspecified Model Still Has a Target

A model can be useful even when none of its parameter values describes the data-generating distribution exactly. In that case, estimation does not recover a parameter that makes the model true. Instead, the chosen loss determines which available parameter is best for the population. The central question is whether minimizing the empirical loss identifies that population target as the sample grows.

This tutorial gives a compact-parameter consistency result for that setting. The conclusion is convergence to the set of population-risk minimizers; convergence to one particular parameter requires an additional uniqueness condition. The result concerns a fixed parameter family and does not assert that the family contains the truth.

Definition: Let $Z_1,\ldots,Z_n$ be independent, identically distributed observations with the same distribution as $Z$, and let $m_\theta(Z)$ be the loss at parameter $\theta\in\Theta$. The population risk and empirical risk are $$ R(\theta)=E[m_\theta(Z)],\qquad R_n(\theta)=\frac{1}{n}\sum_{i=1}^n m_\theta(Z_i). $$ A population-risk minimizer is any $\theta^*\in\Theta$ attaining $R^*=\inf_{\theta\in\Theta}R(\theta)$. When the model is misspecified, such a minimizer is often called a pseudo-true parameter: it is optimal for the specified family and loss, not necessarily a parameter that makes the model true.

The target is inherently loss-dependent. Squared error, absolute error, and negative log-likelihood answer different optimization questions, even when applied to the same observations and parameter values. The consistency argument below identifies the target for the loss actually used; it does not declare that target to be a universally correct description of the data.

A Uniform Convergence Result

Write $\|\cdot\|$ for Euclidean distance and $\operatorname{dist}(\theta,A)=\inf_{\eta\in A}\|\theta-\eta\|$. We use a strong but convenient set of sufficient conditions: the parameter space is compact, the loss is continuous in the parameter for almost every observation, and a single integrable envelope bounds the loss. These conditions ensure that the empirical risks converge uniformly, not merely at each fixed parameter.

Theorem: Suppose $\Theta$ is a compact subset of $\mathbb{R}^d$, $Z_1,Z_2,\ldots$ are iid, and $m_\theta(z)$ is measurable in $z$ for each $\theta$. Suppose also that, with probability one, $\theta\mapsto m_\theta(Z)$ is continuous on $\Theta$, and that there is a measurable $M$ with $E[M(Z)]<\infty$ such that $\sup_{\theta\in\Theta}|m_\theta(Z)|\le M(Z)$ almost surely. Then $R$ is finite and continuous, has a nonempty compact minimizer set $\Theta^*=\arg\min_{\theta\in\Theta}R(\theta)$, and $$ \sup_{\theta\in\Theta}|R_n(\theta)-R(\theta)|\xrightarrow{a.s.}0. $$ If $\hat\theta_n$ is a measurable empirical minimizer, so $R_n(\hat\theta_n)=\inf_{\theta\in\Theta}R_n(\theta)$, then $$ \operatorname{dist}(\hat\theta_n,\Theta^*)\xrightarrow{a.s.}0. $$ If $\Theta^*=\{\theta^*\}$, this implies $\hat\theta_n\xrightarrow{a.s.}\theta^*$.

Here is the uniform convergence argument. For $\delta>0$, define the modulus $\omega_\delta(z)=\sup\{|m_\theta(z)-m_\eta(z)|:\theta,\eta\in\Theta,\ \|\theta-\eta\|\le\delta\}$. Continuity on compact $\Theta$ implies $\omega_\delta(Z)\to0$ almost surely as $\delta\downarrow0$, and the envelope gives $0\le\omega_\delta(Z)\le2M(Z)$. Thus dominated convergence yields $E[\omega_\delta(Z)]\to0$. The relevant suprema are measurable: for a continuous function on a compact subset of Euclidean space, they can be computed using a countable dense subset.

Approximate the parameter space.
For fixed $\delta>0$, compactness gives a finite $\delta$-net $\theta_1,\ldots,\theta_J$ in $\Theta$: every $\theta$ is within $\delta$ of some net point.

Control deviations between net points.
For any $\theta$, choose a net point $\theta_j$ within $\delta$. The triangle inequality gives $|R_n(\theta)-R(\theta)|\le \max_j|R_n(\theta_j)-R(\theta_j)|+P_n\omega_\delta+E[\omega_\delta]$, where $P_n\omega_\delta=n^{-1}\sum_{i=1}^n\omega_\delta(Z_i)$.

Apply the strong law.
For fixed $\delta$, the strong law makes the finite maximum tend to zero and $P_n\omega_\delta$ tend to $E[\omega_\delta]$, almost surely. Hence the limsup of the uniform deviation is at most $2E[\omega_\delta]$. Apply this on a countable sequence $\delta\downarrow0$, and use $E[\omega_\delta]\to0$, to obtain uniform convergence.

For each fixed $\theta$, the envelope also permits dominated convergence to show that $R(\theta)=E[m_\theta(Z)]$ is continuous in $\theta$. Continuity and compactness then ensure that $R$ attains its minimum and that its minimizer set $\Theta^*$ is nonempty and compact.

It remains to connect uniform convergence to minimizers. Let $A_\varepsilon=\{\theta\in\Theta:\operatorname{dist}(\theta,\Theta^*)\ge\varepsilon\}$. If this set is nonempty, it is compact and contains no population minimizer. Continuity therefore gives a strictly positive gap $\Delta_\varepsilon=\min_{\theta\in A_\varepsilon}R(\theta)-R^*>0$. For any $\theta^*\in\Theta^*$, empirical minimization implies $$ R(\hat\theta_n)\le R_n(\hat\theta_n)+\sup_{\theta\in\Theta}|R_n(\theta)-R(\theta)| \le R_n(\theta^*)+\sup_{\theta\in\Theta}|R_n(\theta)-R(\theta)| \le R^*+2\sup_{\theta\in\Theta}|R_n(\theta)-R(\theta)|. $$ Almost surely, the last term is eventually smaller than $R^*+\Delta_\varepsilon$. Thus $\hat\theta_n\notin A_\varepsilon$ eventually. This holds for every $\varepsilon>0$, proving convergence in distance to $\Theta^*$. If the set is the singleton $\{\theta^*\}$, distance to it is $\|\hat\theta_n-\theta^*\|$, giving the stated parameter convergence. $\square$

Worked Examples: The Loss Determines the Target

Worked Example: A Constrained Bernoulli Likelihood

Suppose $Z_i\sim\operatorname{Bernoulli}(0.8)$, but fit the Bernoulli family with success probability $q\in\Theta=[0.2,0.6]$. The family cannot represent the true success probability, so this is a misspecified, constrained fit. Its negative log-likelihood loss is $m_q(z)=-z\log q-(1-z)\log(1-q)$.

Taking expectation under the actual distribution gives $$ R(q)=-0.8\log q-0.2\log(1-q),\qquad R'(q)=-\frac{0.8}{q}+\frac{0.2}{1-q} =\frac{q-0.8}{q(1-q)}. $$ For every $q\in[0.2,0.6]$, the denominator is positive and $q-0.8<0$, so $R'(q)<0$. The risk decreases throughout the allowed interval, and its unique minimum is the boundary point $q^*=0.6$, not the true value $0.8$.

The empirical minimizer is the sample proportion $\bar Z_n=n^{-1}\sum_i Z_i$ clipped to $[0.2,0.6]$. Indeed, the unconstrained Bernoulli negative log-likelihood is minimized at $\bar Z_n$, and restricting the interval moves that minimizer to the nearest endpoint when it lies outside the interval. By the strong law, $\bar Z_n\to0.8$ almost surely, so eventually $\bar Z_n>0.6$ and the constrained estimate equals $0.6$. The limit is the best available value under this likelihood and parameter restriction.

Worked Example: Squared Error and Absolute Error Give Different Targets

Let $Z\sim\operatorname{Exp}(1)$, so $E[Z]=1$, $\operatorname{Var}(Z)=1$, and $F(z)=1-e^{-z}$ for $z\ge0$. Consider constant predictions $\theta\in[0,2]$, first under squared loss $m_\theta(z)=(z-\theta)^2$, then under absolute loss $m_\theta(z)=|z-\theta|$. Neither family of constant predictions describes the full, nondegenerate distribution.

For squared loss, expanding around the mean gives $$ R_2(\theta)=E[(Z-\theta)^2] =\operatorname{Var}(Z)+(E[Z]-\theta)^2 =1+(1-\theta)^2. $$ This is uniquely minimized over $[0,2]$ at $\theta=1$.

For absolute loss and an interior $\theta$, differentiating the two integrals on either side of $\theta$ gives $R_1'(\theta)=P(Z<\theta)-P(Z>\theta)=2F(\theta)-1$; the distribution has no atom at $\theta$. Here $R_1'(\theta)=1-2e^{-\theta}$. It is negative for $\theta<\log 2$, zero at $\theta=\log 2$, and positive for $\theta>\log 2$. Thus the unique absolute-loss target is the median $\log 2$, approximately $0.693$. Both empirical minimizers converge to their respective targets under the stated consistency conditions, but the targets differ because the losses differ.

Worked Example: Tied Population Minimizers

Let $Z$ take values 0 and 1 with probability $1/2$ each. Consider the finite, hence compact, parameter space $\Theta=\{0,1,2\}$ and the following losses: $m_0(z)=\mathbf{1}\{z=1\}$, $m_1(z)=\mathbf{1}\{z=0\}$, and $m_2(z)=1$. The risks are $R(0)=1/2$, $R(1)=1/2$, and $R(2)=1$, so $\Theta^*=\{0,1\}$. There is no unique pseudo-true parameter.

For a sample with $N_0$ zeros and $N_1$ ones, the empirical risks are $R_n(0)=N_1/n$, $R_n(1)=N_0/n$, and $R_n(2)=1$. Since $N_0+N_1=n$, at least one of the first two risks is at most $1/2$, so parameter 2 is never an empirical minimizer. If $N_1<N_0$, parameter 0 is the unique minimizer; if $N_0<N_1$, parameter 1 is the unique minimizer; if the counts tie, both 0 and 1 minimize.

Thus the empirical minimizer is always in the population minimizer set, but the data do not single out one member of that set. The consistency conclusion is correctly stated as convergence in distance to $\{0,1\}$, not convergence to a predetermined one of its elements.

Interpretation and a Common Pitfall

The pseudo-true parameter is a property of the combination of the data distribution, the model family, and the loss. Changing any of these can change the target. In likelihood problems, for example, the population target is the parameter minimizing expected negative log-likelihood over the specified family. In prediction problems, squared and absolute losses can favor different summaries of the same distribution, as the exponential example demonstrates.

Pitfall: treating the limit as the true parameter. It is wrong to argue, “The empirical minimizer is consistent, so it must converge to the parameter that generated the data.” Under misspecification, that parameter may not even belong to the family. The theorem instead identifies a minimizer of the population risk within the chosen family.
Pitfall: claiming convergence to one value without uniqueness. Uniform convergence guarantees that empirical minimizers approach the population minimizer set. If that set has several elements, the theorem alone does not say which one a particular selection will approach.
Pitfall: assuming compactness alone is enough. The argument also uses control of the loss and continuity in the parameter to establish uniform convergence. Without appropriate conditions, pointwise convergence of empirical risks need not control their minimizers uniformly.

Key takeaway: Under compactness, almost-sure continuity, and an integrable envelope, empirical risk minimizers converge almost surely in distance to the population-risk minimizer set. A unique minimizer gives convergence to one pseudo-true parameter, whose value depends on the loss and model family.

Check Your Understanding

Use the definitions and assumptions above to answer the following.

What distinguishes a pseudo-true parameter from a parameter that correctly represents the data-generating distribution?
Which assumptions in the theorem ensure that the population risk attains its minimum on $\Theta$?
Why does convergence of empirical risk to population risk at each fixed parameter not by itself establish the minimizer conclusion?
In the exponential example, why do squared loss and absolute loss select different targets?
If the population minimizer set contains two parameters, what does the theorem guarantee about an empirical minimizer, and what does it not guarantee?

Pseudo-True Limits Under Model Misspecification

What You'll Learn