A Worked Audit of Loss Properties

Why Audit a Loss?

A loss function is often introduced by name, but its name alone does not tell us everything needed for an analysis. “Squared loss,” for example, is unbounded when predictions and outcomes range over all real numbers, but bounded if both are restricted to a fixed interval. A probability score can be convex as a function of the forecast without being proper. A useful audit therefore begins by fixing the loss and its domain, then checks each property separately.

This tutorial audits common regression losses and two probability-forecast losses. The goal is not to choose a universally best loss. It is to calculate risks in concrete cases and identify exactly which mathematical properties hold under the stated assumptions.

Definition: A loss is a function \(\ell(y,a)\) of an outcome \(y\) and a prediction or action \(a\). For a random outcome \(Y\), its risk at \(a\) is \(R(a)=E[\ell(Y,a)]\), when this expectation is defined; it may be infinite. A loss is bounded on a specified domain if there is a finite constant \(M\) such that \(|\ell(y,a)|\leq M\) everywhere on that domain. It is convex in the prediction if, for fixed \(y\), \(a\mapsto\ell(y,a)\) is convex.

The domain is essential. In the regression audit below, outcomes and predictions range over \(\mathbb{R}\), unless a restriction is stated. For probability forecasts, the outcome is binary, \(Y\in\{0,1\}\), and the forecast is \(p\in[0,1]\). Convexity and continuity refer to the prediction argument with the outcome held fixed. Propriety, in contrast, is a property of expected score under a probability distribution for the outcome.

Three Regression Losses

For a real outcome \(y\), prediction \(a\), and residual \(r=y-a\), the squared and absolute losses are \(\ell_2(y,a)=r^2\) and \(\ell_1(y,a)=|r|\). For a fixed threshold \(\delta>0\), the Huber loss is

\ell_{\delta}(y,a)= \begin{cases} \frac{1}{2}(y-a)^2, & |y-a|\leq\delta,\\ \delta|y-a|-\frac{1}{2}\delta^2, & |y-a|>\delta. \end{cases}

All three losses are continuous and convex in \(a\), and all are unbounded above over \((y,a)\in\mathbb{R}^2\). Their curvature differs. Squared loss has second derivative \(2\) with respect to \(a\), so it has uniform positive curvature. Absolute loss is linear on either side of \(a=y\) and is not differentiable there; it has no positive curvature lower bound. Huber loss is quadratic when \(|y-a|<\delta\) and linear in the tails. It is differentiable at the joining points, but its second derivative changes there and is not defined in the classical sense at those points. Its linear tails mean it is not strongly convex on all of \(\mathbb{R}\).

Worked Example: Risks Under a Two-Point Outcome Distribution

Let \(P(Y=0)=P(Y=2)=1/2\), and use Huber loss with \(\delta=1\). Calculate the risks at predictions \(a=1\) and \(a=0.5\) for all three regression losses.

At \(a=1\), the residuals \(Y-a\) are \(-1\) and \(1\). Squared loss is \(1\) for either outcome, so \(R_2(1)=(1+1)/2=1\). Absolute loss is also \(1\) for either outcome, so \(R_1(1)=1\). For Huber loss, each residual has magnitude \(1=\delta\), and the quadratic branch gives loss \(\frac12(1)^2=0.5\). Thus \(R_{\delta}(1)=(0.5+0.5)/2=0.5\).

At \(a=0.5\), the residuals are \(-0.5\) and \(1.5\). The squared losses are \(0.25\) and \(2.25\), giving \(R_2(0.5)=(0.25+2.25)/2=1.25\). The absolute losses are \(0.5\) and \(1.5\), giving \(R_1(0.5)=(0.5+1.5)/2=1\). For Huber loss, the first residual gives \(\frac12(0.5)^2=0.125\); the second is in the linear branch and gives \(1(1.5)-\frac12(1)^2=1\). Therefore \(R_{\delta}(0.5)=(0.125+1)/2=0.5625\).

These calculations illustrate why loss properties and risk values should not be conflated. The Huber loss is unbounded as a function of residual, even though its risk in this example is finite at both predictions.

Auditing Probability-Forecast Losses

For a binary outcome and forecast \(p\in[0,1]\), the Brier loss is \(\ell_{\mathrm{B}}(y,p)=(y-p)^2\). The logarithmic loss is

\ell_{\log}(y,p)=-y\log p-(1-y)\log(1-p).

At the boundaries, define the logarithmic loss by its limiting values: it is zero for a correctly certain forecast, such as \(y=1,p=1\), and \(+\infty\) for an incorrectly certain forecast, such as \(y=1,p=0\). This convention makes the loss extended-valued on \([0,1]\).

Worked Example: Comparing Brier and Logarithmic Risks

Suppose \(Y\) is Bernoulli with \(P(Y=1)=q=0.7\). Compare forecasts \(p=0.6\) and \(p=0.8\). The risk of any binary-outcome loss is \(q\ell(1,p)+(1-q)\ell(0,p)\).

For Brier loss, \(\ell_{\mathrm{B}}(1,p)=(1-p)^2\) and \(\ell_{\mathrm{B}}(0,p)=p^2\). At \(p=0.6\), the risk is \(0.7(0.4)^2+0.3(0.6)^2=0.112+0.108=0.22\). At \(p=0.8\), it is \(0.7(0.2)^2+0.3(0.8)^2=0.028+0.192=0.22\).

For logarithmic loss, the risk is \(-0.7\log p-0.3\log(1-p)\), with natural logarithms. At \(p=0.6\), this is \(-0.7\log(0.6)-0.3\log(0.4)\approx0.357578+0.274887=0.6325\). At \(p=0.8\), it is \(-0.7\log(0.8)-0.3\log(0.2)\approx0.156200+0.482831=0.6390\). The two scores rank these forecasts differently in this particular comparison: Brier risks tie, while logarithmic risk is lower at \(0.6\).

The Brier loss is bounded between \(0\) and \(1\) on the stated domain, continuous, and strictly convex in \(p\), with second derivative \(2\). Logarithmic loss is convex and continuous on the open interval \(0<p<1\). For either fixed outcome, it diverges at the endpoint that assigns probability zero to the realized event, so it is unbounded above on the full domain and is not finite-valued or continuous on the closed interval. In the interior its curvature is

\frac{\partial^2}{\partial p^2}\ell_{\log}(y,p) =\frac{y}{p^2}+\frac{1-y}{(1-p)^2}>0, \qquad 0<p<1.

Propriety Is an Expected-Loss Property

A probability scoring rule is proper if, when the true probability of \(Y=1\) is \(q\), expected loss is minimized by reporting \(p=q\). It is strictly proper if that minimizer is unique. These definitions concern the risk as a function of the reported forecast, not convexity of the loss for a fixed outcome.

Theorem: On \(p\in[0,1]\), the Brier score and logarithmic score are strictly proper for a binary outcome, using the extended boundary convention for logarithmic loss. That is, for every \(q\in[0,1]\), their expected loss under \(Y\sim\operatorname{Bernoulli}(q)\) is uniquely minimized at \(p=q\).

Proof for the Brier score. Its risk is

R_{\mathrm{B}}(p) =q(1-p)^2+(1-q)p^2 =q-2qp+p^2 =q(1-q)+(p-q)^2.

The term \(q(1-q)\) does not depend on \(p\), and \((p-q)^2\geq0\), with equality exactly when \(p=q\). This proves unique minimization for every \(q\in[0,1]\). \(\square\)

Proof for logarithmic loss. For \(0<q<1\) and \(0<p<1\), the risk is \(R_{\log}(p)=-q\log p-(1-q)\log(1-p)\). Its derivative is

R_{\log}'(p)=-\frac{q}{p}+\frac{1-q}{1-p} =\frac{p-q}{p(1-p)}.

The denominator is positive. Hence the derivative is negative for \(p<q\) and positive for \(p>q\), so the unique interior minimum is \(p=q\). The risk tends to \(+\infty\) at either endpoint when \(0<q<1\), so neither endpoint is another minimizer. If \(q=0\), the risk is \(-\log(1-p)\), uniquely minimized at \(p=0\). If \(q=1\), it is \(-\log p\), uniquely minimized at \(p=1\). This proves strict propriety also at the endpoints. \(\square\)

A Compact Property Audit

The table summarizes the audit under the domains specified above. For the three regression losses, “unbounded” refers to allowing arbitrary real outcomes and predictions. The probability losses use binary outcomes and forecasts in \([0,1]\).

Loss	Bounded?	Convex and continuous in prediction?	Curvature and propriety
Squared regression	No	Yes, on \(\mathbb{R}\)	Second derivative \(2\); no probability-score propriety claim
Absolute regression	No	Yes; not differentiable at zero residual	Linear away from zero; no positive global curvature bound
Huber regression, fixed \(\delta\)	No	Yes; differentiable, with curvature changes at the joins	Quadratic near zero and linear in the tails; no positive global curvature bound
Brier score	Yes, between \(0\) and \(1\)	Yes, on \([0,1]\)	Second derivative \(2\); strictly proper
Logarithmic score	No, on \([0,1]\)	Convex and continuous on \((0,1)\), but extended-valued at boundary forecasts	Positive interior curvature; strictly proper

A useful practical sequence is to state the outcome space and prediction space first, then check boundedness, continuity, convexity, and curvature while holding the outcome fixed. Only after that should a probability score be checked for propriety by calculating its expected loss under a general true probability \(q\).

Common Pitfall

Wrong reasoning: “The logarithmic loss is convex in \(p\), so it must be proper.” Convexity alone does not imply that the expected loss is minimized at the true probability; the location of the risk minimum must be established. The correction is to form \(R(p)=E[\ell(Y,p)]\) under a general \(q\) and verify its minimizer, as done above. Similarly, saying that a loss is “bounded” without naming its domain is incomplete: squared loss is unbounded on \(\mathbb{R}^2\), while Brier loss is bounded on its binary-outcome, probability-forecast domain.

This audit matters because properties answer different questions. Boundedness controls the largest possible contribution of a single observation; convexity supports optimization; curvature describes how sharply the loss changes near a prediction; and propriety checks whether a probability score rewards honest probability reports in expectation. None of these labels can safely stand in for the others.

Check Your Understanding

For each question, state the relevant domain or distribution before drawing a conclusion.

For squared regression loss, explain why restricting both \(y\) and \(a\) to \([-2,2]\) changes the boundedness audit.
With \(Y\) equally likely to be \(0\) or \(2\), verify the absolute-loss risk at \(a=1\) and \(a=0.5\).
What feature of the Huber loss prevents it from having a positive curvature lower bound over all real predictions?
For a Bernoulli outcome with true probability \(q\), write the Brier risk and identify the term that depends on the forecast's distance from \(q\).
Why does convexity of a loss for each fixed outcome not, by itself, establish propriety?