M-Estimators as Optimizers of Empirical Criteria

From a Criterion to an Estimator

Many estimators can be described by the same operation: choose the parameter value that makes a data-dependent criterion as large or as small as possible. Maximum likelihood is one familiar example, but the criterion could instead be a negative loss, a squared-error measure, or another summary of how well a parameter value fits the observations. This common construction is called M-estimation.

The central distinction is between optimizing the criterion computed from the observed sample and optimizing its population expectation. The first produces an estimator; the second describes the parameter value that the criterion favors on average under the data-generating distribution. These are related, but they are not the same object.

Definition: Let $X_1,\ldots,X_n$ be observations and let $\Theta$ be a parameter space. For a measurable criterion function $m_\theta(x)$, define the empirical criterion $$ M_n(\theta)=\frac{1}{n}\sum_{i=1}^n m_\theta(X_i), \qquad \theta\in\Theta. $$ A maximizer $\hat{\theta}\in\Theta$ of $M_n(\theta)$ is an M-estimator when the criterion is to be maximized. If the chosen criterion is a loss to be minimized, an M-estimator is instead a minimizer of that criterion. The corresponding population criterion, when the expectation exists, is $M(\theta)=E[m_\theta(X)]$; a maximizer or minimizer of $M$ is called a population target for that criterion.

A maximum or minimum need not exist, and there may be more than one optimizer. The definition is therefore conditional: when an optimizer exists, an estimator is a choice of optimizer (and, for it to be a random variable, that choice must be measurable). This tutorial uses criteria for which the displayed calculations establish the needed optimizer directly.

Empirical Criteria and Maximum Likelihood

Suppose $X_1,\ldots,X_n$ are independent observations with density or mass function $f_\theta$, for $\theta\in\Theta$. Their likelihood is $L_n(\theta)=\prod_{i=1}^n f_\theta(X_i)$. Whenever the likelihood is positive, its log divided by $n$ is the empirical average of the individual log-likelihood contributions:

\frac{1}{n}\ell_n(\theta) =\frac{1}{n}\log L_n(\theta) =\frac{1}{n}\sum_{i=1}^n \log f_\theta(X_i).

Thus maximum likelihood is M-estimation with $m_\theta(x)=\log f_\theta(x)$. Equivalently, it minimizes the average negative log-likelihood. This is ordinary parametric likelihood evaluated at the observed data; the phrase here does not refer to the separate nonparametric empirical-likelihood method.

Result: If $M_n(\theta)$ is finite and $a>0$, then $M_n$ and $aM_n+b$ have exactly the same maximizers whenever $b$ is constant in $\theta$. Likewise, they have the same minimizers. In particular, replacing a sum by its average does not change its optimizers.

To prove the result, take any $\theta_1,\theta_2\in\Theta$. Because $a>0$, $M_n(\theta_1)\geq M_n(\theta_2)$ if and only if $aM_n(\theta_1)+b\geq aM_n(\theta_2)+b$. The ordering of all candidate values is unchanged, so the maximizing set is unchanged; the same comparison with $\leq$ proves the minimizing claim. A strictly increasing transformation, such as the logarithm applied to a positive likelihood, also preserves maximizers because it preserves the ordering of candidate values. $\square$

Worked Example: Bernoulli Maximum Likelihood

Suppose $X_1,\ldots,X_5$ are Bernoulli observations with success probability $p\in[0,1]$, and the observed values are $1,1,0,1,0$. There are three successes, so $\bar X=3/5$. For $0<p<1$, the average log-likelihood is

M_5(p)=\frac{3}{5}\log p+\frac{2}{5}\log(1-p).

Its derivative is $M_5'(p)=\frac{3}{5p}-\frac{2}{5(1-p)} =\frac{3-5p}{5p(1-p)}$. It is positive for $p<3/5$ and negative for $p>3/5$. Also, $M_5''(p)=-\frac{3}{5p^2}-\frac{2}{5(1-p)^2}<0$, so the interior critical point is the unique maximizer. At either endpoint the likelihood is zero because the sample contains both outcomes. Therefore $\hat p=3/5$.

If the observations instead come from a Bernoulli distribution with success probability $q$, the population criterion for candidate $p$ is $M(p)=q\log p+(1-q)\log(1-p)$. For $0<q<1$, the same derivative argument gives population target $p=q$. In this sample, the empirical optimizer is $3/5$; the population target is determined by $q$, which need not be $3/5$.

Loss Functions and Their Targets

A loss function measures the cost of using a candidate parameter value. Minimizing the average loss is M-estimation directly; alternatively, one can maximize its negative. This simple change of sign makes loss-based estimation fit the same criterion framework as likelihood.

Worked Example: Squared-Error Loss Estimates a Mean

For a location parameter $a\in\mathbb{R}$, take the squared-error loss $m_a(x)=(x-a)^2$. For observations $2,4,7,7$, the empirical mean is $\bar X=5$. Expanding around that mean gives

\sum_{i=1}^4(X_i-a)^2 =\sum_{i=1}^4(X_i-\bar X)^2+4(a-\bar X)^2.

Indeed, write $X_i-a=(X_i-\bar X)+(\bar X-a)$; the cross term sums to zero because $\sum_i(X_i-\bar X)=0$. Here $\sum_i(X_i-\bar X)^2=9+1+4+4=18$, so the total squared error is $18+4(a-5)^2$. It is uniquely minimized at $\hat a=5$, with average loss $18/4=4.5$.

For a random variable $X$ with finite second moment, the population squared-error criterion is minimized at $a=E[X]$. The same expansion, replacing the empirical average by expectation, yields $E[(X-a)^2]=E[(X-E[X])^2]+(a-E[X])^2$. The first term does not depend on $a$, and the second is uniquely minimized at $a=E[X]$.

The population target depends on the chosen criterion, not just on the name of the parameter. Squared-error loss targets the mean when that mean exists. Other loss functions can target different features of the distribution. The defining operation remains the same: specify the criterion first, then identify its optimizer.

Population Log-Likelihood and the True Distribution

A useful connection arises when the candidate likelihood family contains the true data-generating distribution. Let the true density or mass function be $f_{\theta_0}$, with respect to a common dominating measure, and consider the population log-likelihood $M(\theta)=E_{\theta_0}[\log f_\theta(X)]$. Under suitable integrability conditions, its value cannot exceed the value at the truth. This follows from the nonnegativity of Kullback–Leibler divergence.

Result: Suppose $f_{\theta_0}$ and $f_\theta$ are densities or mass functions with respect to the same measure. Assume $E_{\theta_0}|\log f_{\theta_0}(X)|<\infty$, and that each candidate expectation $E_{\theta_0}[\log f_\theta(X)]$ is defined, possibly as $-\infty$. Then $$ E_{\theta_0}[\log f_\theta(X)] \leq E_{\theta_0}[\log f_{\theta_0}(X)]. $$ Equality holds only if the candidate distribution equals the true distribution (up to sets of measure zero). Thus the true distribution is a population maximizer; if distinct parameter values represent distinct distributions, $\theta_0$ is the unique population target.

Proof. If $f_\theta=0$ on a set to which the true distribution assigns positive probability, the candidate expected log-likelihood is $-\infty$, and the inequality is strict. Otherwise, on the support where $f_{\theta_0}>0$, set $r=f_\theta/f_{\theta_0}$. The elementary inequality $\log r\leq r-1$ gives

E_{\theta_0}\!\left[\log\frac{f_\theta(X)}{f_{\theta_0}(X)}\right] \leq \int_{\{f_{\theta_0}>0\}} f_{\theta_0}(x) \left(\frac{f_\theta(x)}{f_{\theta_0}(x)}-1\right)\,dx =\int_{\{f_{\theta_0}>0\}}f_\theta(x)\,dx-1 \leq 0.

The same argument uses a sum instead of an integral for a discrete distribution. The final inequality holds because a density or mass function has total integral or sum one, while its mass on the support of $f_{\theta_0}$ cannot exceed one. Equality requires equality in $\log r\leq r-1$ wherever $f_{\theta_0}>0$, hence $r=1$ there, and also requires no candidate mass outside that support. These conditions say the two distributions agree. Adding the finite quantity $E_{\theta_0}[\log f_{\theta_0}(X)]$ proves the claim. $\square$

Worked Example: Exponential Likelihood and Its Population Target

Let $X_1,\ldots,X_n$ have an exponential distribution with rate $\lambda>0$, whose density is $f_\lambda(x)=\lambda e^{-\lambda x}$ for $x\geq0$. For observations $1,2,3$, the sample mean is $\bar X=2$, and the average log-likelihood is

M_3(\lambda)=\log\lambda-2\lambda.

Differentiating gives $M_3'(\lambda)=1/\lambda-2$, which is zero at $\hat\lambda=1/2$. Since $M_3''(\lambda)=-1/\lambda^2<0$, this is the unique maximizer. It is also the global maximizer: $M_3(\lambda)$ tends to $-\infty$ as $\lambda$ approaches zero or infinity.

If the true rate is $\lambda_0$, then $E_{\lambda_0}[X]=1/\lambda_0$, so the population log-likelihood is $M(\lambda)=\log\lambda-\lambda/\lambda_0$. Its derivative is $1/\lambda-1/\lambda_0$, and its second derivative is negative. The unique population target is therefore $\lambda_0$. The observed-sample optimizer $1/2$ is calculated from the observed mean; the population target is set by the true distribution.

Why the Distinction Matters

The population criterion describes which parameter the chosen scoring rule favors under the data-generating distribution. The empirical criterion describes the score computed from one particular sample. Even when the population target is scientifically meaningful, a finite-sample optimizer can differ from it, as the Bernoulli example illustrates. Defining both objects precisely prevents an estimate from being mistaken for a guaranteed truth.

A common mistake is to reason, “The population criterion is maximized at $\theta_0$, so the sample criterion must also be maximized at $\theta_0$.” The conclusion does not follow: $M_n(\theta)$ is calculated from the observed values, whereas $M(\theta)$ averages over the data-generating distribution. For the Bernoulli sample above, the likelihood optimizer is $3/5$, even if the true success probability is, for example, $q=1/2$. The correction is to identify the optimizer of the actual criterion in the problem at hand, and to keep that sample optimizer distinct from the optimizer of its expectation.

Another error is to assume that changing a criterion’s scale changes its optimizer. Multiplication by a positive constant, addition of a parameter-independent constant, or taking a strictly increasing transformation preserves the ordering of candidates. But a transformation that is not increasing, or an added term that depends on the parameter, can change the optimizer. Always check how a proposed rewrite affects comparisons across $\theta$.

Key takeaway: An M-estimator optimizes a criterion computed from the sample. Its population target optimizes the expectation of the corresponding criterion. Likelihood and loss functions are important examples, but the criterion determines what is being targeted.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

Why does replacing a summed log-likelihood by its average leave the maximizing parameter unchanged?
For squared-error loss, what population quantity is the target when $E[X^2]$ is finite, and what identity establishes it?
In the Bernoulli example, distinguish the empirical optimizer from the population target.
What support condition is involved in the equality case for the population log-likelihood result?
Give an example of a transformation of a criterion that could change its optimizer, and explain why the optimizer-preservation result does not apply.