Unbounded Losses and Infinite Risk

Why Unbounded Losses Need Extra Care

A loss can be finite for every individual observation and still have infinite population risk. The reason is that risk averages loss over the outcome distribution: rare, very large penalties can make that expectation diverge. With squared loss, the relevant issue is the size of the outcome’s tail. With binary logarithmic loss, the critical issue is whether the forecast assigns a probability near zero to an outcome that can actually occur.

Throughout, risk is the expectation of a nonnegative loss, understood as an extended expectation that may equal \(+\infty\). This convention makes finiteness a property to check rather than an assumption. The earlier material on population risk defines risk as expected loss; here we examine what happens when the loss is unbounded.

Squared Loss and the Outcome’s Second Moment

Let \(Y\) be a real-valued outcome and let \(a\in\mathbb{R}\) be a fixed, finite prediction. The squared loss is \((Y-a)^2\), and its population risk is

$$ R(a)=E[(Y-a)^2]. $$

Even though \((Y-a)^2\) is finite whenever \(Y\) is a finite real number, its expectation need not be finite. The precise condition is a finite second moment.

Theorem: For any real-valued random variable \(Y\) and any finite \(a\), \(E[(Y-a)^2]<\infty\) if and only if \(E[Y^2]<\infty\). Thus, either every finite prediction has finite squared risk, or every finite prediction has infinite squared risk.

Proof. For real numbers \(u,v\), \((u+v)^2\leq 2u^2+2v^2\). Applying this with \(u=Y\) and \(v=-a\) gives

(Y-a)^2\leq 2Y^2+2a^2.

If \(E[Y^2]<\infty\), taking expectations shows \(E[(Y-a)^2]\leq 2E[Y^2]+2a^2<\infty\). Conversely, write \(Y=(Y-a)+a\), and apply the same inequality:

Y^2\leq 2(Y-a)^2+2a^2.

If \(E[(Y-a)^2]<\infty\), this gives \(E[Y^2]\leq 2E[(Y-a)^2]+2a^2<\infty\). Both directions hold for each finite \(a\), proving the claim. \(\square\)

Worked Example: A Finite Squared Risk

Suppose \(Y\sim N(3,4)\), so \(E[Y]=3\) and \(\operatorname{Var}(Y)=4\). Find the squared risk of predicting \(a=1\), and check whether it is finite.

Since \(Y\) has a finite second moment, its squared risk is finite. Expanding around its mean gives

E[(Y-a)^2] =E[((Y-E[Y])+(E[Y]-a))^2] =\operatorname{Var}(Y)+(E[Y]-a)^2.

The cross term has expectation zero because \(E[Y-E[Y]]=0\). Substituting the values,

$$ R(1)=4+(3-1)^2=4+4=8. $$

The risk is \(8\), a finite value. More generally, whenever \(E[Y^2]<\infty\), this variance-plus-squared-bias expression holds for every finite \(a\).

The identity also explains the effect of an extreme but finite prediction. When \(E[Y^2]<\infty\), it implies \(R(a)=\operatorname{Var}(Y)+(E[Y]-a)^2\), which grows without bound as \(|a|\) grows. A large finite prediction does not by itself make risk infinite, but it can make risk arbitrarily large. If \(E[Y^2]=\infty\), the theorem shows that changing the finite prediction cannot repair the infinite risk.

Worked Example: Infinite Risk Under a Heavy Tail

Let \(Y\) have a Pareto distribution with minimum value \(1\) and tail probability \(P(Y>y)=y^{-3/2}\) for \(y\geq 1\). Determine whether predicting any finite \(a\) gives finite squared risk.

The density is \(f(y)=\tfrac{3}{2}y^{-5/2}\) for \(y\geq1\). Its second moment is

E[Y^2]=\int_1^\infty y^2\frac{3}{2}y^{-5/2}\,dy =\frac{3}{2}\int_1^\infty y^{-1/2}\,dy =+\infty.

Indeed, the integral up to \(M\) equals \(\tfrac{3}{2}\cdot 2(\sqrt{M}-1)=3(\sqrt{M}-1)\), which tends to infinity. By the theorem, \(E[(Y-a)^2]=+\infty\) for every finite \(a\). This is true even though each realized \(Y\) is finite with probability one: the tail is heavy enough that the average squared penalty diverges.

Binary Logarithmic Loss and Forecast Boundaries

For a binary outcome \(Y\in\{0,1\}\), let \(p\in[0,1]\) be the forecast probability of \(Y=1\). The logarithmic loss is the negative log probability assigned to the outcome that occurs. At the endpoints, assigning probability zero to the realized outcome gives infinite loss.

Definition: The binary logarithmic loss at outcome \(y\) and forecast \(p\) is

\ell(y,p)=-y\log p-(1-y)\log(1-p),

with the conventions that \(-\log 0=+\infty\) and that a term with coefficient zero contributes zero. In particular, \(\ell(1,0)=+\infty\), \(\ell(0,1)=+\infty\), and the correctly matched endpoint losses satisfy \(\ell(0,0)=\ell(1,1)=0\).

Write \(q=P(Y=1)\). If \(0<p<1\), both possible outcomes have finite loss, so the risk is finite and equals

R(p)=E[\ell(Y,p)]=-q\log p-(1-q)\log(1-p).

At a boundary forecast, risk depends on whether the outcome to which probability zero was assigned can occur. At \(p=0\), risk is infinite if \(q>0\), and is zero if \(q=0\). At \(p=1\), risk is infinite if \(q<1\), and is zero if \(q=1\). These statements include degenerate outcome distributions: a boundary forecast can have finite risk when it matches the outcome that occurs with probability one.

Worked Example: A Boundary Forecast With Infinite Risk

Suppose \(P(Y=1)=0.2\), and the forecast is \(p=0\). Find the risk.

When \(Y=0\), which occurs with probability \(0.8\), the loss is \(-\log(1)=0\). When \(Y=1\), which occurs with probability \(0.2\), the forecast assigns probability zero to the outcome and the loss is \(+\infty\). Therefore,

R(0)=0.8(0)+0.2(+\infty)=+\infty.

Any positive probability of the excluded outcome is enough to make the expectation infinite; it does not matter that this outcome is less common.

Approaching a Boundary

A forecast need not equal zero or one for its log loss to be large. For fixed \(q\) strictly between zero and one, the risk diverges as \(p\) approaches either boundary. For instance, as \(p\) approaches zero, the term \(-q\log p=q\log(1/p)\) grows without bound, while \(-(1-q)\log(1-p)\) tends to zero. The analogous divergence near one is driven by \(-(1-q)\log(1-p)\).

Worked Example: A Small Forecast Probability

Let \(q=0.2\), and forecast \(p=10^{-6}\). Calculate the risk approximately and identify which outcome drives it.

Using \(\log(10^{-6})\approx-13.8155\) and \(\log(1-10^{-6})\approx-0.0000010000005\),

\begin{aligned} R(10^{-6}) &=-0.2\log(10^{-6})-0.8\log(1-10^{-6})\\ &\approx 0.2(13.8155)+0.8(0.0000010000005)\\ &\approx 2.76310. \end{aligned}

The forecast is very small but positive, so the risk is finite. Most of it comes from the \(Y=1\) outcome, whose probability under the true distribution is \(0.2\) but whose forecast probability is only \(10^{-6}\). As \(p\) decreases further toward zero, this contribution increases without bound.

The limiting behavior changes when the outcome distribution is degenerate. If \(q=0\), then for \(0<p<1\), \(R(p)=-\log(1-p)\), which tends to zero as \(p\) decreases to zero. In that case the outcome is always zero, and the forecast is approaching the matching boundary. If instead \(q>0\), the term \(q\log(1/p)\) diverges. Thus, “a forecast approaches zero” alone does not determine whether risk diverges; the probability of the outcome excluded by that forecast matters.

Common Mistakes and Practical Checks

Checking only whether each loss is finite. A finite observed squared loss does not establish finite population risk. Check \(E[Y^2]\); if it is infinite, squared risk is infinite for every finite prediction.
Assuming a very small probability is the same as zero. For a binary outcome, an interior forecast has finite loss for either outcome, however large that loss is. Infinite loss occurs at a boundary when the outcome assigned probability zero can occur.
Ignoring the outcome distribution near a log boundary. With \(q=0\), risk can approach zero as \(p\to0\); with any \(q>0\), it diverges. State the distributional condition before drawing a conclusion.
Confusing large risk with infinite risk. A finite but extreme squared prediction can produce arbitrarily large risk when the second moment exists. It remains finite for each finite prediction.

Key takeaway: For squared loss, finite risk at a finite prediction is equivalent to a finite second moment of the outcome. For binary logarithmic loss, interior forecasts have finite risk, while boundary forecasts have infinite risk precisely when they assign zero probability to an outcome that can occur.

Check Your Understanding

For each question, state the condition that determines whether risk is finite.

If \(E[Y^2]=+\infty\), can choosing a different finite squared-loss prediction make risk finite? Why?
For a binary outcome with \(P(Y=1)=0.4\), what is the risk at \(p=1\)?
If \(P(Y=1)=0\), what happens to logarithmic risk as \(p\) approaches zero from above?
Why can squared risk be infinite even when every observed outcome is a finite real number?
For fixed \(0<q<1\), which term in the binary log-risk formula forces divergence as \(p\) approaches zero?