Why the Scale of a Loss Matters
A loss records the penalty for a prediction and an outcome; its risk is the expected penalty under the data distribution. Changing the units in which a loss is recorded should not ordinarily change which prediction has lower risk. For example, converting a penalty from one unit to another by multiplying by a positive constant should preserve all comparisons. Adding the same fixed amount to every loss should do so as well.
The key qualification is the same: the multiplier and offset must not vary across the predictions being compared. Moreover, a nonlinear transformation of each observed loss is not generally equivalent to applying that transformation to its expectation. Since risk is an expectation, this distinction can change the ranking of predictors.
Risk and Transformed Losses
A positive affine transformation has the form \(\tilde\ell(g,z)=a\ell(g,z)+b\), where \(a>0\) and \(b\) are constants shared by every prediction and outcome. If the original risk is finite, the transformed risk is finite and follows directly from linearity of expectation:
The positive multiplier rescales every risk difference, while the common offset cancels from every difference. Thus the transformation changes numerical risk values, but not their ordering.
Worked Example: Converting Risk Units
Suppose three predictors have risks \(R(g_1)=2\), \(R(g_2)=2.5\), and \(R(g_3)=4\). Replace the loss by \(\tilde\ell=3\ell+7\). Calculate the transformed risks and compare the predictors.
For \(g_1\), \(\tilde R(g_1)=3(2)+7=13\). For \(g_2\), \(\tilde R(g_2)=3(2.5)+7=14.5\). For \(g_3\), \(\tilde R(g_3)=3(4)+7=19\). The original order \(R(g_1)<R(g_2)<R(g_3)\) remains \(13<14.5<19\).
The comparator difference between \(g_2\) and \(g_1\) was \(2.5-2=0.5\). After transformation it is \(14.5-13=1.5\), exactly three times as large. The offset of 7 has no effect on this difference.
The Affine-Invariance Result
- \(R(g)\le R(h)\) if and only if \(\tilde R(g)\le\tilde R(h)\); and
- \(\tilde R(g)-\tilde R(h)=a\{R(g)-R(h)\}\).
Proof. Linearity of expectation gives \(\tilde R(g)=aR(g)+b\) and \(\tilde R(h)=aR(h)+b\). Subtracting,
Because \(a>0\), the right-hand side is nonpositive exactly when \(R(g)-R(h)\) is nonpositive, and it is negative exactly when that original difference is negative. The same equality shows that a zero difference remains zero. This proves both claims. \(\square\)
The result also applies to optimal predictors. If \(\inf_{g\in\mathcal G}R(g)\) is finite, then
Indeed, multiplication by \(a>0\) and addition of \(b\) preserve the order of all candidate values, including their infimum. Subtracting this equality from \(\tilde R(g)=aR(g)+b\) gives
Thus excess risk relative to the best achievable risk is also scaled by \(a\). The set of minimizers is unchanged, whether or not the infimum is attained: a predictor attains the infimum before transformation exactly when it does afterward.
Worked Example: Comparator-Relative Regret
A reference predictor \(g_0\) has risk 5, while a candidate \(g\) has risk 6.2. Apply \(\tilde\ell=0.4\ell+2\), then calculate the candidate’s risk difference relative to \(g_0\).
The transformed risks are \(\tilde R(g_0)=0.4(5)+2=4\) and \(\tilde R(g)=0.4(6.2)+2=4.48\). The original comparator difference is \(6.2-5=1.2\). The transformed difference is \(4.48-4=0.48\), and \(0.4(1.2)=0.48\).
The candidate remains worse than the reference predictor, but the numerical size of its disadvantage is now expressed on a different scale. The additive constant affects neither conclusion nor difference.
When an Offset Depends on the Observation
A useful extension allows the common offset to depend on the observed data. Suppose \(\tilde\ell(g,Z)=a\ell(g,Z)+c(Z)\), where \(a>0\), \(c(Z)\) is the same function for every \(g\), and \(E[c(Z)]\) is finite. Then
The expected offset is constant across predictors, so the same risk-ranking and comparator-difference conclusions hold. This extension is valid only when the expectation exists and is finite under the stated conditions. If the offset depends on the prediction, it need not cancel.
Worked Example: A Shared Observation-Dependent Offset
Let \(Z\) equal 0 or 1 with probability \(1/2\) each. Predictor \(g_A\) has loss 0 at \(Z=0\) and 2 at \(Z=1\); predictor \(g_B\) has loss 1 at either outcome. Transform each loss by \(\tilde\ell(g,Z)=2\ell(g,Z)+Z\).
The original risks are \(R(g_A)=\tfrac12(0)+\tfrac12(2)=1\) and \(R(g_B)=\tfrac12(1)+\tfrac12(1)=1\). The shared offset has expectation \(E[Z]=\tfrac12\). Hence the transformed risks are \(\tilde R(g_A)=2(1)+\tfrac12=2.5\) and \(\tilde R(g_B)=2(1)+\tfrac12=2.5\). Their tie remains a tie.
The calculation uses the same offset \(Z\) for both predictors at each outcome. A different offset for each predictor would not be common and would not be covered by this result.
Why Nonlinear Transformations Can Reverse Rankings
For a nonlinear function \(\phi\), the transformed risk is generally \(E[\phi(\ell(g,Z))]\), not \(\phi(R(g))\). These quantities need not agree. Even when \(\phi\) is increasing, so that it preserves the ordering of two losses at any single fixed outcome, it need not preserve the ordering of their expectations. The predictors may incur different distributions of loss across outcomes, and the transformation changes how those values contribute to the average.
Worked Example: Squaring Reverses the Risk Ranking
Predictor \(g_A\) has loss 0 with probability 0.9 and loss 10 with probability 0.1. Predictor \(g_B\) has constant loss 1.2. Compare their risks, then compare risks under the nonlinear transformation \(\phi(x)=x^2\).
Before transformation, \(R(g_A)=0.9(0)+0.1(10)=1\), while \(R(g_B)=1.2\). Thus \(g_A\) has lower risk. After squaring each loss, \(\tilde R(g_A)=0.9(0^2)+0.1(10^2)=10\), whereas \(\tilde R(g_B)=1.2^2=1.44\). Now \(g_B\) has lower risk.
The function \(x^2\) is strictly increasing on the nonnegative losses in this example, but that does not prevent reversal after expectation. The rare loss of 10 contributes 10 to the transformed risk of \(g_A\), dominating its frequent zero losses.
A nonlinear map can also reverse rankings in the opposite curvature direction. For example, a concave transformation such as the square root can reduce the influence of large losses relative to small ones. The next calculation makes the reversal explicit.
Worked Example: A Concave Transformation Also Changes the Winner
Let predictor \(g_A\) have constant loss 1.5. Let \(g_B\) have loss 0 or 4, each with probability \(1/2\). Compare risks before and after applying \(\phi(x)=\sqrt{x}\).
The original risks are \(R(g_A)=1.5\) and \(R(g_B)=\tfrac12(0)+\tfrac12(4)=2\), so \(g_A\) is preferred. The transformed risks are \(\tilde R(g_A)=\sqrt{1.5}\), approximately \(1.225\), and \(\tilde R(g_B)=\tfrac12\sqrt{0}+\tfrac12\sqrt{4}=1\). After transformation, \(g_B\) is preferred.
Although the square root is increasing on nonnegative values, its expectation weighs the loss distribution differently from the original mean. The ranking change is caused by applying the transformation before averaging.
Common Mistakes and Practical Consequences
A frequent error is to reason, “The transformation is increasing, so the predictor with lower risk must still have lower risk.” That reasoning compares transformed values of two numbers, but risk comparisons involve averages of possibly different random losses. The examples show that \(E[\phi(\ell)]\) can rank predictors differently from \(E[\ell]\), even when \(\phi\) is increasing.
Another mistake is to assume any affine-looking change preserves rankings. The multiplier must be strictly positive and common across predictors. A zero multiplier makes every transformed risk tie; a negative multiplier reverses the ordering. A predictor-specific offset can also alter the ranking because it does not cancel. The table summarizes the distinctions.
| Transformation | Effect on risk comparison | Condition |
|---|---|---|
| \(a\ell+b\) | Preserves rankings and ties; multiplies differences by \(a\) | \(a>0\), with common \(a,b\) |
| \(a\ell+c(Z)\) | Preserves rankings and ties; multiplies differences by \(a\) | \(a>0\), common integrable offset |
| \(\phi(\ell)\), nonlinear | May change rankings after expectation | Check transformed risks directly |
| \(a\ell+b_g\) | May change rankings | Offset varies across predictors |
Positive affine transformations are useful when changing the units or baseline of a loss while retaining the same decision criterion. But once the loss is changed nonlinearly, the resulting risk is a different objective, not merely a rescaled version of the old one. In applications, verify whether the scientific or decision-making goal justifies that change rather than assuming the same predictor will remain optimal.
Check Your Understanding
Answer each question using the definitions and comparisons above.
- If \(R(g)=3\), \(R(h)=4\), and \(\tilde\ell=2\ell+5\), what are the transformed risks and their difference?
- Why must the multiplier in an affine transformation be positive to guarantee preservation of risk rankings?
- Under what conditions can an observation-dependent additive term be included without changing risk rankings?
- How can a strictly increasing nonlinear function reverse the ordering of two predictors’ risks?
- What happens to comparator-relative excess risk when the loss is transformed by \(a\ell+b\) with \(a>0\)?