Estimation Rules As Decisions Under Quadratic Loss

From Estimation to Decision

An estimator is not judged by its numerical answer alone. Its performance depends on the parameter value and on the possible data the model could generate. Decision theory makes these ingredients explicit: a rule maps data to an action, a loss measures the cost of that action, and risk averages the loss over repeated sampling at a specified parameter value.

This distinction also matters when a distribution is placed on the parameter space. The risk at a fixed parameter is a function of that parameter. Averaging this function under a parameter distribution produces one overall number. That average is useful, but it answers a different question from risk at any particular parameter value.

Definition: A statistical decision problem specifies a parameter space $\Theta$, an observation $X$ with distribution $P_\theta$ for each $\theta\in\Theta$, an action space $\mathcal{A}$, and a loss function $L(\theta,a)$. An (nonrandomized) decision rule is a measurable function $\delta:\mathcal{X}\to\mathcal{A}$. In a scalar estimation problem, the action is an estimate of $\theta$, so the rule may be written $\hat{\theta}=\delta(X)$.

Measurability ensures that the rule and its loss can be treated as random variables under each sampling distribution. We focus on scalar estimation with $\mathcal{A}\subseteq\mathbb{R}$. Under squared-error loss, the penalty for choosing action $a$ when the true parameter is $\theta$ is the squared distance between them.

Definition: The squared-error loss is $L(\theta,a)=(a-\theta)^2$. The risk of a rule $\delta$ at a fixed parameter value is $$ R(\theta,\delta)=E_\theta\!\left[(\delta(X)-\theta)^2\right], $$ where $E_\theta$ denotes expectation when $X\sim P_\theta$. Risk is a function of $\theta$; it may be finite or $+\infty$.

For a discrete observation, risk is a probability-weighted sum of the losses at the possible outcomes. If the model has density or mass function $f_\theta$ relative to a common measure $\mu$, then

R(\theta,\delta) = \int_{\mathcal{X}}(\delta(x)-\theta)^2 f_\theta(x)\,d\mu(x).

A loss for one realized data set is not the same thing as risk. The realized loss depends on the particular observed $X$; risk averages that loss over all possible observations under $P_\theta$, keeping $\theta$ fixed. This averaging is why risk is a property of the rule and the model, not a single observed outcome.

Worked Example: Risk in a One-Trial Bernoulli Model

Let $X\sim\operatorname{Bernoulli}(p)$, where $p\in[0,1]$, and use the rule $\delta(X)=X$ to estimate $p$. Under squared-error loss, calculate its risk at $p=0.2$, and distinguish that risk from the loss after observing $X=0$.

If $X=0$, the rule reports 0, so the loss is $(0-p)^2=p^2$. If $X=1$, it reports 1, so the loss is $(1-p)^2$. The sampling probabilities of these outcomes are $1-p$ and $p$, respectively. Therefore,

R(p,\delta)=(1-p)p^2+p(1-p)^2 =p(1-p)\{p+(1-p)\} =p(1-p).

At $p=0.2$, the risk is $0.2(0.8)=0.16$. If the observed outcome is $X=0$, the realized loss is instead $0.2^2=0.04$. The two numbers differ because the realized loss concerns one outcome, while the risk includes both possible outcomes: $0.8(0.04)+0.2(0.64)=0.032+0.128=0.16$.

Risk Is Evaluated at a Fixed Parameter

The parameter in $R(\theta,\delta)$ is held fixed while the expectation is taken over the data. Consequently, two rules can have different risk profiles across $\Theta$, and a rule with low risk at one parameter value need not have low risk elsewhere. The risk function records this dependence rather than hiding it behind a single summary.

Worked Example: A Normal-Model Risk Function

Suppose $X\sim N(\theta,1)$, with $\theta\in\mathbb{R}$, and consider the rule $\delta(X)=X/2$. Derive its risk and evaluate it at $\theta=0$ and $\theta=1$.

Write $X=\theta+Z$, where $Z\sim N(0,1)$. Then $\delta(X)-\theta=(Z-\theta)/2$, so

R(\theta,\delta) =E_\theta\!\left[\frac{(Z-\theta)^2}{4}\right] =\frac{1}{4}\left(E[Z^2]-2\theta E[Z]+\theta^2\right) =\frac{1+\theta^2}{4}.

Here $E[Z]=0$ and $E[Z^2]=1$. Thus $R(0,\delta)=1/4$, while $R(1,\delta)=2/4=1/2$. These are values of the same risk function at two different fixed parameters, not two estimates obtained from the observed data.

Integrated Risk Under a Parameter Distribution

A probability distribution $\pi$ on $\Theta$ can be used to average risk across parameter values. This distribution is sometimes called a prior distribution, but the definition of the average does not require a Bayesian decision rule. It only requires a specified probability measure on the parameter space and a well-defined risk function.

Definition: For a probability measure $\pi$ on $\Theta$, the integrated risk (also called the Bayes risk of $\delta$ under $\pi$) is $$ r_\pi(\delta)=\int_\Theta R(\theta,\delta)\,\pi(d\theta). $$ This is an average of fixed-parameter risks. It is a single nonnegative number, possibly $+\infty$.

When the model and loss are jointly measurable, the integrated risk can also be calculated by averaging loss under the joint experiment in which $\theta$ is drawn from $\pi$ and then data are drawn from $P_\theta$. The nonnegativity of squared-error loss permits the order of integration to be exchanged by Tonelli’s theorem, even if the resulting value is infinite.

Theorem: Suppose $\pi$ is a probability measure on $\Theta$, the sampling model is a measurable probability kernel $\theta\mapsto P_\theta$, and $(\theta,x)\mapsto(\delta(x)-\theta)^2$ is jointly measurable. Then $$ r_\pi(\delta) = \int_\Theta\int_{\mathcal{X}}(\delta(x)-\theta)^2\,P_\theta(dx)\,\pi(d\theta). $$ Equivalently, this is the expected squared-error loss in the joint experiment that first draws $\theta\sim\pi$ and then draws $X\sim P_\theta$.

Proof. By the definition of risk,

r_\pi(\delta) =\int_\Theta R(\theta,\delta)\,\pi(d\theta) =\int_\Theta\left\{\int_{\mathcal{X}}(\delta(x)-\theta)^2\,P_\theta(dx)\right\}\pi(d\theta).

The integrand $(\delta(x)-\theta)^2$ is nonnegative and jointly measurable. Tonelli’s theorem therefore identifies this iterated integral with the integral of the same loss under the joint distribution of $(\theta,X)$. The equality holds for nonnegative extended integrals, including when either side is infinite. $\square$

Worked Example: Averaging Bernoulli Risk Under a Beta Distribution

Continue with $X\sim\operatorname{Bernoulli}(p)$ and $\delta(X)=X$, whose fixed-parameter risk is $R(p,\delta)=p(1-p)$. Let $p$ have the proper distribution $\operatorname{Beta}(2,2)$, with density $\pi(p)=6p(1-p)$ for $0<p<1$. Find the integrated risk.

By definition, average the risk function—not a single realized loss—against the density:

r_\pi(\delta) =\int_0^1 p(1-p)\,6p(1-p)\,dp =6\int_0^1p^2(1-p)^2\,dp.

Expanding the integrand gives

6\int_0^1(p^2-2p^3+p^4)\,dp =6\left(\frac{1}{3}-\frac{2}{4}+\frac{1}{5}\right) =6\left(\frac{1}{30}\right) =\frac{1}{5}.

Thus the integrated risk is $0.2$. At the particular parameter value $p=1/2$, the risk is $1/4=0.25$. There is no contradiction: $1/4$ is the risk at one fixed value, whereas $1/5$ averages risk over the full Beta distribution.

What Risk Comparisons Do—and Do Not—Say

Integrated risk summarizes performance relative to a chosen parameter distribution. A comparison of integrated risks is therefore distribution-dependent: a region of parameter space that receives more probability contributes more to the average. It does not, by itself, establish that one rule has lower risk at every parameter value.

Theorem: Let $\delta_1$ and $\delta_2$ be two rules with measurable risk functions. If $R(\theta,\delta_1)\leq R(\theta,\delta_2)$ for $\pi$-almost every $\theta$, then $r_\pi(\delta_1)\leq r_\pi(\delta_2)$. If both risks are integrable and the inequality is strict on a set of positive $\pi$-measure, then $r_\pi(\delta_1)<r_\pi(\delta_2)$.

Proof. The difference $R(\theta,\delta_2)-R(\theta,\delta_1)$ is nonnegative $\pi$-almost everywhere under the first condition. Integrating this inequality gives $r_\pi(\delta_2)-r_\pi(\delta_1)\geq0$, so $r_\pi(\delta_1)\leq r_\pi(\delta_2)$. If both risks are integrable and the difference is strictly positive on a set of positive measure, its integral is strictly positive. Therefore $r_\pi(\delta_2)-r_\pi(\delta_1)>0$. $\square$

The converse need not hold: one rule can have lower integrated risk even though its risk is higher at some parameter values. The average only aggregates the comparison using $\pi$; it does not remove the variation in the underlying risk functions.

Common Pitfall

Wrong reasoning: “The risk is $0.16$ because the observed loss was $0.16$,” or “an integrated risk of $0.2$ means the risk is $0.2$ for each parameter.” Both statements conflate quantities that are defined differently.

Correction: Compute realized loss from the observed action and the true parameter. Compute $R(\theta,\delta)$ by averaging loss over the data distribution at that fixed $\theta$. Compute $r_\pi(\delta)$ by averaging the resulting risk function over $\pi$. Keep the conditioning and the distribution being averaged explicit at each step.

Key takeaway: Under squared-error loss, an estimator is a measurable rule, its fixed-parameter risk is expected loss under $P_\theta$, and its integrated risk is an average of that risk under a specified distribution on $\Theta$. These are distinct performance summaries.

Check Your Understanding

Use the definitions to answer each question.

In a scalar estimation problem, what makes a function of the data a decision rule, and what does its squared-error loss measure?
For a fixed $\theta$, which distribution is used to calculate $R(\theta,\delta)$?
In the one-trial Bernoulli example, what is the realized loss when $X=1$, and how does it differ from the risk?
What additional ingredient is needed to turn a risk function into an integrated risk?
Why does lower integrated risk not imply lower risk at every parameter value?