Bounded Losses and Range Conventions

Why the Range of a Loss Matters

A loss assigns a numerical penalty to a prediction and the outcome it is meant to predict. Its scale affects how large a reported risk looks: a risk of 8 may be small when losses range from 0 to 100, but large when they range from 0 to 10. A bound on the loss also immediately bounds its expected value, so it provides a simple check on risk calculations.

There is no single convention for reporting loss ranges. Some losses take values in \([0,1]\); others lie in \([0,B]\), or in a general interval \([a,b]\). The endpoints and the width of the interval carry different information. In particular, a maximum value is not the same as the width of the range when the loss has a positive lower bound.

Definition and Range Conventions

Let \(Z=(X,Y)\) denote an observation with distribution \(P\), let \(h\) be a prediction rule, and let \(\mathcal{L}(h,Z)\) be its loss. Assume that, for each predictor under discussion, \(z\mapsto\mathcal{L}(h,z)\) is measurable. Its population risk is \(R_P(h)=E_P[\mathcal{L}(h,Z)]\), when this expectation exists.

A declared bounding interval need not be the smallest possible one. For example, a loss that only takes the values 0 and 1 is bounded in \([0,2]\), but \([0,2]\) is not its tightest interval. When referring to the loss's actual range, we mean the set of values it can take; when referring to a bound, we mean an interval known to contain those values. In common statistical usage, “range” may also mean interval width. To avoid ambiguity, this tutorial uses interval width for \(b-a\).

Sometimes a bound holds only almost surely under \(P\): for a fixed predictor \(h\), \(a\leq\mathcal{L}(h,Z)\leq b\) with probability one. That is enough to bound \(R_P(h)\), because changes on a probability-zero event do not change an expectation. It is weaker than a pointwise bound over the whole observation space. It may also depend on \(P\) or on \(h\); a shared bound over all predictors is a stronger and often more useful statement.

Bounds on Population Risk

The basic consequence of boundedness is that expectation stays within the same interval. This conclusion does not require a finite observation space. It only requires a measurable loss and an almost-sure bound under the distribution being used.

Proof. The random variable \(\mathcal{L}(h,Z)\) is measurable and bounded almost surely, so it is integrable. The random variables \(\mathcal{L}(h,Z)-a\) and \(b-\mathcal{L}(h,Z)\) are nonnegative almost surely. Their expectations are therefore nonnegative. By linearity of expectation, \(E_P[\mathcal{L}(h,Z)]-a\geq 0\) and \(b-E_P[\mathcal{L}(h,Z)]\geq 0\). Rearranging gives \(a\leq R_P(h)\leq b\). \(\square\)

The bound applies separately to each predictor satisfying the hypothesis. If the same \([a,b]\) bound holds for every predictor in a class, then every risk in that class lies in \([a,b]\). In particular, a loss in \([0,B]\) gives \(0\leq R_P(h)\leq B\); its risk cannot exceed its maximum possible loss.

Changing the Scale and Comparing Predictors

A common convention is to transform a loss bounded in \([a,b]\), with \(a<b\), into a unit-bounded loss by subtracting its lower bound and dividing by its interval width: \(\widetilde{\mathcal{L}}(h,z)=(\mathcal{L}(h,z)-a)/(b-a)\). This maps the bounding interval \([a,b]\) to \([0,1]\). The transformed risk can be calculated directly from the original one.

Proof. By definition, \(\widetilde{R}_P(h)=E_P[c\mathcal{L}(h,Z)+d]\). Linearity of expectation gives \(\widetilde{R}_P(h)=cE_P[\mathcal{L}(h,Z)]+d=cR_P(h)+d\). For any pair \(h_1,h_2\), subtracting the transformed risks gives \(\widetilde{R}_P(h_1)-\widetilde{R}_P(h_2)=c(R_P(h_1)-R_P(h_2))\). Since \(c>0\), this difference has the same sign as the original risk difference. Thus the ordering, including ties, is unchanged. \(\square\)

For unit-interval normalization, \(c=1/(b-a)>0\) and \(d=-a/(b-a)\). Consequently, \(\widetilde{R}_P(h)=(R_P(h)-a)/(b-a)\). The transformed risk is not generally just the original risk divided by the width: the subtraction of \(a\) is also needed if the loss has a positive lower bound. If \(a=0\), the subtraction has no effect.

The same affine calculation applies to empirical averages whenever they are defined: applying the same positive scale and common offset to each observed loss transforms every predictor's empirical average in the same way. This does not mean every change in a loss convention preserves rankings. The scale and offset must be shared across the predictors being compared.

A Common Mistake and a Practical Check

A frequent mistake is to say that any rescaling leaves the preferred predictor unchanged. That claim is too broad. For instance, multiplying every loss by the same positive constant preserves rankings, but multiplying each predictor's loss by a different constant need not. Suppose \(h_A\) has risk 2 and \(h_B\) has risk 3. The first is preferred under the original loss. If, instead, the reported losses for \(h_A\) are multiplied by 2 while those for \(h_B\) are multiplied by 0.5, the reported risks become 4 and 1.5, reversing the ranking. This is not a common rescaling of one loss; it changes the comparison between predictors.

A related error is to confuse a maximum with a width. For a loss in \([2,10]\), the maximum is 10, the minimum is 2, and the width is 8. Stating which quantity is being used prevents incorrect normalizations. Before interpreting or comparing risks, check the loss interval, whether its bounds hold for all relevant observations and predictors, and whether any transformation is shared across the comparison.