Oracle Bounds Under Model Misspecification

Why Misspecification Changes the Comparison

An oracle inequality compares a data-driven predictor with a benchmark chosen from candidate models. It does not require one of those models to contain the true conditional target. In fact, when every candidate is misspecified, the benchmark must include the error that remains even after choosing the best predictor available in each model. Otherwise, a bound can appear to promise accuracy no candidate could attain.

We will use squared-error prediction to make this distinction precise. The same oracle-comparison logic applies to other risks, but the exact decomposition into noise and squared distance relies on squared error. Here the target is the conditional mean, and the central question is how well the candidate models can approximate it.

Definition: Let $(X,Y)$ have a fixed distribution, and let $f$ be a measurable predictor with finite squared-error risk $R(f)=E[(Y-f(X))^2]$. Write $m_0(X)=E[Y\mid X]$, assuming $E[Y^2]<\infty$. For a nonempty candidate model $\mathcal F_m$, its approximation error relative to the conditional mean is $$ a_m=\inf_{f\in\mathcal F_m}E[(f(X)-m_0(X))^2]. $$ The model is misspecified relative to $m_0$ if no $f\in\mathcal F_m$ satisfies $f(X)=m_0(X)$ almost surely. The infimum defining $a_m$ need not be attained. The collection of candidate models is misspecified if none of its members contains a predictor equal to $m_0$ almost surely.

There is an important distinction between not containing the target and having positive approximation error. A model may fail to contain $m_0$ while its approximation error is zero, if it contains predictors that approach $m_0$ arbitrarily closely in squared mean. Positive approximation error requires that even the infimum distance from the target is positive.

The Squared-Error Benchmark

The unrestricted risk benchmark is $R_*=R(m_0)$, the risk of the conditional mean. Under the stated finite-second-moment condition, it is finite. The next identity explains exactly how a candidate’s excess risk relates to its distance from the target.

Theorem: For every measurable $f$ with finite squared-error risk, $$ R(f)=R_*+E[(f(X)-m_0(X))^2]. $$ Consequently, $\inf_{f\in\mathcal F_m}R(f)=R_*+a_m$, whether or not the infimum is attained.

Proof. Write $Y-f(X)=(Y-m_0(X))+(m_0(X)-f(X))$ and expand the square:

$$ R(f)=E[(Y-m_0(X))^2]+E[(f(X)-m_0(X))^2] +2E[(Y-m_0(X))(m_0(X)-f(X))]. $$

The cross term is zero. Indeed, $m_0(X)-f(X)$ is a function of $X$, and the defining conditional-mean property gives $E[Y-m_0(X)\mid X]=0$. Therefore,

E[(Y-m_0(X))(m_0(X)-f(X))] =E\!\left[(m_0(X)-f(X))E[Y-m_0(X)\mid X]\right]=0.

All terms are integrable: $Y$, $m_0(X)$, and $f(X)$ have finite second moments under the assumptions, so their relevant products are integrable by the Cauchy–Schwarz inequality. This proves the identity. Taking the infimum over $f\in\mathcal F_m$ gives $\inf_{f\in\mathcal F_m}R(f)=R_*+a_m$. $\square$

Worked Example: A Misspecified Constant Model

Let $X\sim\mathrm{Uniform}[-1,1]$, let $Y=X+\varepsilon$, and assume $\varepsilon$ is independent of $X$, with $E[\varepsilon]=0$ and $\operatorname{Var}(\varepsilon)=1$. Consider the model of all constant predictors, $\mathcal F=\{f_c:f_c(x)=c,\ c\in\mathbb R\}$.

The conditional mean is $m_0(x)=x$, and $R_*=E[\varepsilon^2]=1$. Since $E[X]=0$ and $E[X^2]=1/3$, the approximation error is

a=\inf_{c\in\mathbb R}E[(c-X)^2] =\inf_{c\in\mathbb R}\left(c^2+\frac{1}{3}\right) =\frac{1}{3}.

The infimum is attained at $c=0$. The best constant predictor therefore has risk $R_*+a=1+1/3=4/3$. No constant equals $X$ almost surely, so the model is misspecified and its positive approximation error remains even if its best predictor is known exactly.

The Penalized Oracle Under Simultaneous Deviations

A model-selection procedure uses empirical risks and penalties, while the oracle benchmark uses population risks. To connect them, suppose that all candidate models satisfy a simultaneous deviation bound. The next result makes the comparison explicit and allows the fitted predictor to be only an approximate minimizer of the penalized empirical criterion.

Theorem: Let $\{\mathcal F_m:m\in\mathcal M\}$ be nonempty candidate models, and suppose $R(f)$ is finite for every candidate under consideration. Let $R_n(f)$ be an empirical risk, and let $\operatorname{pen}_m$ be a deterministic penalty for model $m$. Suppose there are deterministic numbers $\delta_m\geq 0$ such that, on an event $\mathcal E$, $$ |R_n(f)-R(f)|\leq\delta_m \quad\text{for every }m\in\mathcal M\text{ and }f\in\mathcal F_m. $$ Assume $\operatorname{pen}_m\geq\delta_m$ for each $m$. If a selected pair $(\hat m,\hat f)$, with $\hat f\in\mathcal F_{\hat m}$, satisfies $$ R_n(\hat f)+\operatorname{pen}_{\hat m} \leq \inf_{m\in\mathcal M,\ g\in\mathcal F_m} \{R_n(g)+\operatorname{pen}_m\}+\varepsilon $$ for some $\varepsilon\geq0$, then on $\mathcal E$, $$ R(\hat f)-R_* \leq \inf_{m\in\mathcal M}\{a_m+2\operatorname{pen}_m\}+\varepsilon, $$ where $R_*$ is the unrestricted benchmark and $a_m=\inf_{g\in\mathcal F_m}\{R(g)-R_*\}$. For squared-error prediction with conditional-mean target, this $a_m$ equals the approximation error defined above.

Proof. Fix a model $m$ and any $g\in\mathcal F_m$. On $\mathcal E$, the deviation inequality for $\hat f$, the approximate-minimization condition, and the deviation inequality for $g$, in that order, give

\begin{aligned} R(\hat f) &\leq R_n(\hat f)+\delta_{\hat m}\\ &\leq R_n(g)+\operatorname{pen}_m-\operatorname{pen}_{\hat m} +\varepsilon+\delta_{\hat m}\\ &\leq R(g)+\delta_m+\operatorname{pen}_m-\operatorname{pen}_{\hat m} +\varepsilon+\delta_{\hat m}. \end{aligned}

Because $\operatorname{pen}_{\hat m}\geq\delta_{\hat m}$, the two terms $\delta_{\hat m}-\operatorname{pen}_{\hat m}$ have a nonpositive sum. Because $\delta_m\leq\operatorname{pen}_m$, it follows that

R(\hat f)\leq R(g)+2\operatorname{pen}_m+\varepsilon.

Taking the infimum over $g\in\mathcal F_m$ yields $R(\hat f)\leq\inf_{g\in\mathcal F_m}R(g)+2\operatorname{pen}_m+\varepsilon$. Since the infimum need not be attained, this step can be justified by choosing, for each $\eta>0$, a $g$ whose risk is within $\eta$ of the infimum, obtaining the inequality with an additional $\eta$, and then letting $\eta$ decrease to zero. Subtract $R_*$ to get

R(\hat f)-R_*\leq a_m+2\operatorname{pen}_m+\varepsilon.

This holds for every $m$, so taking the infimum over $m$ proves the claim. $\square$

The bound compares the selected predictor with the best available approximation in each model, plus a penalty cost. It does not assert that the selected predictor recovers $m_0$, or that any model is correctly specified. The guarantee is relative to $R_*$; an absolute-risk statement adds that baseline back.

Worked Example: Comparing Two Misspecified Candidates

Let $X\sim\mathrm{Uniform}[-1,1]$ and $Y=1+2X+\varepsilon$, where $\varepsilon$ is independent of $X$, has mean zero, and has variance one. Thus $m_0(x)=1+2x$ and $R_*=1$. Compare two singleton models: $\mathcal F_A=\{f_A:f_A(x)=0\}$ and $\mathcal F_B=\{f_B:f_B(x)=x\}$.

For model A, using $E[X]=0$ and $E[X^2]=1/3$,

a_A=E[(0-(1+2X))^2] =E[1+4X+4X^2] =1+0+\frac{4}{3} =\frac{7}{3}.

For model B,

a_B=E[(X-(1+2X))^2] =E[(-1-X)^2] =E[1+2X+X^2] =1+0+\frac{1}{3} =\frac{4}{3}.

Both models are misspecified: neither predictor equals $1+2X$ almost surely. Their best population risks are $R_*+a_A=10/3$ and $R_*+a_B=7/3$. If their illustrative penalties are $\operatorname{pen}_A=0$ and $\operatorname{pen}_B=1/4$, the oracle values $a_m+\operatorname{pen}_m$ are $7/3$ for A and $4/3+1/4=19/12$ for B. The penalized oracle prefers B. This calculation is a comparison of benchmarks, not a claim that the empirical procedure must choose B on every data set.

When the Approximation Infimum Is Not Attained

Writing “the best predictor in the model” can be misleading when a model is not closed in the relevant risk metric. The oracle benchmark is properly expressed using an infimum. The following example gives a positive approximation error that has no minimizing predictor in the stated class.

Worked Example: A Positive Error With No Best Predictor

Let $X\sim\mathrm{Uniform}(0,1)$, $Y=X+\varepsilon$, and assume $\varepsilon$ is independent of $X$, has mean zero, and has variance one. Let $m_0(x)=x$, and use the model $\mathcal F=\{f_c:f_c(x)=c,\ 0<c<1/2\}$.

For any allowed $c$, the squared approximation error is

E[(c-X)^2] =\int_0^1(c-x)^2\,dx =c^2-c+\frac{1}{3} =\left(c-\frac{1}{2}\right)^2+\frac{1}{12}.

Every allowed $c$ is strictly less than $1/2$, so the displayed value is strictly greater than $1/12$. But the sequence $c_k=1/2-1/k$, for integers $k>2$, belongs to the model and satisfies

E[(c_k-X)^2] =\frac{1}{12}+\frac{1}{k^2} \longrightarrow\frac{1}{12}.

Therefore $a=1/12$, but no $f_c$ in this model attains that value. The model is misspecified, and its best achievable risk is an infimum, $R_*+a=1+1/12=13/12$, not the risk of a predictor in the model.

Interpreting the Bound and Avoiding a Common Error

The approximation term records what the candidate models cannot represent; the penalty term pays for selecting among them using data. Under misspecification, a small penalty cannot erase a large approximation error. Conversely, a model can have a large penalty and still be the oracle choice if its reduction in approximation error is sufficiently large. This is the balance that the model-specific benchmark captures.

A common mistake is to state an oracle bound relative to the risk of a correctly specified candidate, even when no such candidate exists. For example, claiming that a procedure’s excess risk tends to zero merely because estimation error becomes small ignores the residual distance from $m_0$. The correction is to state the bound relative to $R_*$ and retain $a_m$ in the oracle comparison. If every $a_m$ is bounded away from zero, the bound itself should not suggest that candidate-model risk can reach $R_*$.

Another mistake is to replace an infimum by “the smallest achievable error” without checking attainment. The third example shows why this matters: the oracle value is well-defined, but there is no predictor in that model achieving it. Oracle inequalities compare against the value of the infimum, and their proof must allow approximating sequences rather than assume a minimizer exists.

Key takeaway: Under misspecification, the oracle benchmark is the best model-specific approximation error plus its complexity cost. In squared-error prediction, the unrestricted benchmark is $R_*$, and each model’s residual excess risk is $a_m$; neither the oracle comparison nor its proof requires that the infimum be attained.

Check Your Understanding

Use the definitions and arguments above to answer the following questions.

Can a model fail to contain $m_0$ while its approximation error is zero? Explain how this can happen.
For squared-error prediction, what condition makes the cross term in the risk decomposition vanish?
Why does the penalized oracle inequality compare $R(\hat f)-R_*$ with $a_m$, rather than claiming that this excess risk must be zero?
In the uniform-$(0,1)$ example, why is $1/12$ an infimum but not an achieved approximation error?
If a model-specific risk infimum is not attained, how can the oracle proof still compare the selected predictor with it?