Identifiability Conditions for Parameter Recovery

Why Risk Convergence May Not Recover a Parameter

An estimator can achieve nearly the smallest possible population risk without being close to the parameter we hope to recover. The reason is that risk measures predictive or decision performance, while parameter distance measures where the estimator lies in the parameter space. These are different notions of accuracy. To connect them, we need a condition ensuring that parameters far from the target pay a definite price in risk.

This distinction matters when empirical risk minimization (ERM) is used to estimate a parameter. Risk consistency says that the fitted parameter's population risk approaches the minimum. Parameter consistency says that the fitted parameter itself approaches the target. The first does not imply the second automatically: the risk must distinguish the target from parameters far away.

Definition: Let $(\Theta,d)$ be a metric parameter space, and let $R:\Theta\to\mathbb{R}$ be a population risk with a minimizer $\theta^*$, so $R(\theta^*)=\inf_{\theta\in\Theta}R(\theta)$. The risk is separated at $\theta^*$ if, for every $\varepsilon>0$ for which the set is nonempty, $$ \inf_{\theta:\,d(\theta,\theta^*)\geq\varepsilon} \bigl\{R(\theta)-R(\theta^*)\bigr\}>0. $$ The positive infimum is the risk gap outside the $\varepsilon$-neighborhood of the target. An estimator $\hat{\theta}_n$ is risk consistent if $R(\hat{\theta}_n)-R(\theta^*)\xrightarrow{p}0$, and parameter consistent if $d(\hat{\theta}_n,\theta^*)\xrightarrow{p}0$.

The definition requires a gap for each fixed positive distance, not one universal gap that works for all distances. A quantitative version is often useful: if $R(\theta)-R(\theta^*)\geq c\,d(\theta,\theta^*)^q$ for some constants $c>0$ and $q>0$, then the risk gap is at least $c\varepsilon^q$ whenever $d(\theta,\theta^*)\geq\varepsilon$.

The Risk Gap Gives Parameter Consistency

Theorem: Suppose $\theta^*$ minimizes $R$, the separation condition holds at $\theta^*$, and $\hat{\theta}_n$ is a measurable estimator for which $R(\hat{\theta}_n)-R(\theta^*)\xrightarrow{p}0$. Then $d(\hat{\theta}_n,\theta^*)\xrightarrow{p}0$.

The result is a direct conversion of one error measure into another. If an estimate is at least $\varepsilon$ away from the target, separation forces its excess risk to be at least a fixed positive amount. A risk-consistent estimate is unlikely to have excess risk that large.

Fix a parameter tolerance.
Choose $\varepsilon>0$. If no parameter lies at distance at least $\varepsilon$, the event $d(\hat{\theta}_n,\theta^*)\geq\varepsilon$ is empty.

Use separation.
Otherwise let $\delta_\varepsilon=\inf_{d(\theta,\theta^*)\geq\varepsilon}\{R(\theta)-R(\theta^*)\}>0$.

Compare the events.
Whenever $d(\hat{\theta}_n,\theta^*)\geq\varepsilon$, the excess risk is at least $\delta_\varepsilon$. Thus the probability of that parameter event is bounded by the probability that excess risk is at least $\delta_\varepsilon$.

Formally, in the nonempty case,

P\bigl(d(\hat{\theta}_n,\theta^*)\geq\varepsilon\bigr) \leq P\bigl(R(\hat{\theta}_n)-R(\theta^*)\geq\delta_\varepsilon\bigr) \longrightarrow 0.

The last convergence follows from risk consistency in probability: for any fixed positive threshold, the probability that the nonnegative excess risk exceeds that threshold tends to zero. This holds for every $\varepsilon>0$, which is precisely parameter consistency in probability. $\square$

Worked Example: Squared-Error Risk for a Mean

Let $Y_1,\ldots,Y_n$ be independent and identically distributed with $E[Y_i]=\mu$ and $\operatorname{Var}(Y_i)=\sigma^2<\infty$. Estimate a real parameter $\theta$ using squared loss $\ell_\theta(Y)=(Y-\theta)^2$. Find the population risk, identify its separation, and verify parameter consistency for the empirical risk minimizer.

For any real $\theta$, expand $Y-\theta=(Y-\mu)+(\mu-\theta)$. Since $E[Y-\mu]=0$,

R(\theta)=E[(Y-\theta)^2] =E[(Y-\mu)^2]+(\mu-\theta)^2 =\sigma^2+(\theta-\mu)^2.

Thus the unique minimizer is $\theta^*=\mu$, and the excess risk is exactly $R(\theta)-R(\mu)=(\theta-\mu)^2$. If $|\theta-\mu|\geq\varepsilon$, the excess risk is at least $\varepsilon^2$. Separation therefore holds, with gap $\delta_\varepsilon=\varepsilon^2$.

The empirical risk is $n^{-1}\sum_{i=1}^n(Y_i-\theta)^2$, minimized at $\hat{\theta}_n=\bar{Y}_n$. Its population excess risk is $(\bar{Y}_n-\mu)^2$. Independence and finite variance give

E\bigl[(\bar{Y}_n-\mu)^2\bigr]=\frac{\sigma^2}{n}\longrightarrow 0.

Markov's inequality yields $P(|\bar{Y}_n-\mu|\geq\varepsilon)\leq \sigma^2/(n\varepsilon^2)\to0$. Here the risk gap is not merely qualitative: it is exactly the square of parameter error, so the risk and parameter errors can be related directly.

When Does Separation Follow Automatically?

A useful sufficient condition is compactness of the parameter space, continuity of the risk, and uniqueness of its minimizer. Compactness prevents parameters from escaping indefinitely; continuity ensures that a sequence with risks approaching the minimum has a limit point that also attains the minimum.

Theorem: Let $(\Theta,d)$ be a compact metric space, and let $R:\Theta\to\mathbb{R}$ be continuous with a unique minimizer $\theta^*$. Then $R$ is separated at $\theta^*$.

Proof. Fix $\varepsilon>0$. If $A_\varepsilon=\{\theta\in\Theta:d(\theta,\theta^*)\geq\varepsilon\}$ is empty, there is nothing to prove. Otherwise $A_\varepsilon$ is a closed subset of a compact space, hence compact. Continuity implies that $R$ attains its minimum on $A_\varepsilon$, say at $\theta_\varepsilon\in A_\varepsilon$. Since $\theta_\varepsilon\neq\theta^*$ and $\theta^*$ is the unique minimizer on $\Theta$, $R(\theta_\varepsilon)>R(\theta^*)$. Therefore

\inf_{\theta\in A_\varepsilon}\{R(\theta)-R(\theta^*)\} =R(\theta_\varepsilon)-R(\theta^*)>0.

This is separation for the chosen $\varepsilon$; because $\varepsilon$ was arbitrary, it holds for every positive $\varepsilon$. $\square$

Worked Example: A Compact Parameter Space

Let $\Theta=[-1,1]$ and $R(\theta)=(\theta-0.3)^2+0.2(\theta-0.3)^4$. Verify separation at the minimizer.

Both terms are nonnegative, and the sum is zero only at $\theta^*=0.3$. Thus the minimizer is unique, and $R$ is continuous on the compact interval $[-1,1]$. The theorem already guarantees separation. In this example, there is also an explicit bound: since the quartic term is nonnegative,

R(\theta)-R(0.3) =(\theta-0.3)^2+0.2(\theta-0.3)^4 \geq(\theta-0.3)^2.

Consequently, for every $\varepsilon>0$, all $\theta\in\Theta$ with $|\theta-0.3|\geq\varepsilon$ have excess risk at least $\varepsilon^2$. If this set is empty, the separation requirement is vacuous; otherwise the displayed lower bound gives a positive gap.

The compactness theorem is a sufficient condition, not a necessary one. The squared-error example has a separated risk on the noncompact space $\mathbb{R}$. Conversely, uniqueness alone on a noncompact space does not guarantee separation: risk can approach its minimum along parameters that move farther and farther away.

Two Ways Parameter Recovery Can Fail

Worked Example: A Unique Minimizer With Vanishing Gaps at Infinity

Take $\Theta=\mathbb{R}$ with its usual distance and define the nonnegative risk $R(\theta)=\theta^2/(1+\theta^4)$. Show that the minimizer is unique but that a risk-consistent sequence need not be parameter consistent.

The denominator is positive, and $R(\theta)=0$ exactly when $\theta=0$. Thus $\theta^*=0$ is the unique minimizer. But for the sequence $\theta_n=n$,

R(\theta_n)-R(0)=\frac{n^2}{1+n^4} \leq\frac{1}{n^2}\longrightarrow 0, \qquad |\theta_n-\theta^*|=n\longrightarrow\infty.

So this sequence has excess risk tending to zero but does not approach the target. For any fixed $\varepsilon>0$, the set $\{|\theta|\geq\varepsilon\}$ contains arbitrarily large values of $|\theta|$, along which $R(\theta)\to0$; its infimum excess risk is therefore zero. Separation fails. This example shows that uniqueness of a minimizer does not by itself turn risk consistency into parameter consistency.

The sequence here illustrates the failure of the implication from risk consistency to parameter consistency; it is not claimed to be an ERM sequence for a particular sampling problem. The theorem applies to an ERM estimator once its population risk consistency has been established, but a separate argument is needed to rule out this kind of risk geometry.

Worked Example: Two Parameters With the Same Data Distribution

Suppose $Y$ is Bernoulli with success probability $1/4$, and a model indexed by $\theta\in[-1,1]$ assigns success probability $q_\theta=\theta^2$. Use negative log-likelihood as the loss. Can risk consistency recover a true parameter $\theta^*=-1/2$?

The population risk depends on $\theta$ only through $q_\theta$:

R(\theta)=-\frac14\log(q_\theta)-\frac34\log(1-q_\theta), \qquad 0<q_\theta<1.

As a function of $q$, the derivative is $-1/(4q)+3/(4(1-q))$, which is zero exactly when $3q=1-q$, or $q=1/4$. The risk is minimized at that probability, corresponding to both $\theta=-1/2$ and $\theta=1/2$. These two parameters generate exactly the same distribution. In particular, relative to the target $-1/2$, the parameter $1/2$ has zero excess risk but distance $1$. Separation fails for every $\varepsilon\leq1$.

For $n$ observations, let $\bar{Y}_n$ be their sample proportion and select the likelihood maximizer $\hat{\theta}_n=\sqrt{\bar{Y}_n}$. This is a maximizer because the likelihood depends on $\theta$ only through $q_\theta$; the nonnegative square root is a tie-breaking choice. By the law of large numbers, $\bar{Y}_n\xrightarrow{p}1/4$, so $\hat{\theta}_n\xrightarrow{p}1/2$. Continuity of the Bernoulli population risk at $q=1/4$ gives $R(\hat{\theta}_n)-R(-1/2)\xrightarrow{p}0$, yet $|\hat{\theta}_n-(-1/2)|\xrightarrow{p}1$. The data identify the success probability here, but not the sign of the parameter.

Rates, Interpretation, and a Common Mistake

A quantitative separation bound transfers rates as well as consistency. Suppose, for all parameters under consideration,

R(\theta)-R(\theta^*)\geq c\,d(\theta,\theta^*)^q

for constants $c>0$ and $q>0$. Substituting $\hat{\theta}_n$ and rearranging gives

d(\hat{\theta}_n,\theta^*) \leq \left(\frac{R(\hat{\theta}_n)-R(\theta^*)}{c}\right)^{1/q}.

Thus, if the excess risk is $O_p(a_n)$ for a positive sequence $a_n$, the parameter error is $O_p(a_n^{1/q})$. For squared-error risk, $q=2$, so an excess-risk rate of $O_p(n^{-1})$ yields a parameter-error rate of $O_p(n^{-1/2})$. This conversion requires the stated lower bound; it cannot be inferred from a risk rate alone.

Common mistake: “The risk has a unique minimizer, and the estimator's risk approaches the minimum, so the estimator must approach that minimizer.” The reasoning omits a uniform gap away from the target. On a noncompact parameter space, parameters can escape while their excess risks shrink, as the $R(\theta)=\theta^2/(1+\theta^4)$ example shows. With multiple minimizers, the data may determine an optimal distribution or prediction but not a particular parameter representation, as in the Bernoulli example. Check separation—or a sufficient condition that implies it—before drawing a parameter-recovery conclusion.

In applications, the risk is often a function of the distribution or predictions induced by a parameter. Different parameter values can induce identical distributions, and distinct distributions can have risks that become arbitrarily close. A parameter claim therefore needs more than good predictive risk: it needs the parameterization and risk to distinguish the target at the distance scale being studied.

Check Your Understanding

Use the definitions and arguments above to answer these questions.

State the separation condition at $\theta^*$ in terms of the infimum excess risk outside an $\varepsilon$-neighborhood.
In the risk-consistency theorem, why must the risk gap be positive for each fixed $\varepsilon$?
Which three conditions in the compactness theorem guarantee separation, and where is uniqueness used in its proof?
For the bound $R(\theta)-R(\theta^*)\geq c\,d(\theta,\theta^*)^q$, what parameter-error rate follows from an excess-risk rate $O_p(n^{-1})$?
How can a model have a unique risk minimizer but fail separation? How is that failure different from having two parameters with identical distributions?