Surrogate Risk and Replacement Losses

Why Replace a Loss?

A prediction rule is judged through a loss: the loss assigns a penalty to each observed outcome and prediction, and risk averages that penalty over the data distribution. Sometimes an analyst replaces the loss of direct interest with another loss that is more convenient or informative for a particular task. The resulting expected loss is called a surrogate risk. “Surrogate” describes its role as a replacement; the word alone does not guarantee that it gives the same ranking of prediction rules as the target loss.

The comparison must keep the data distribution and prediction rule fixed. For a particular rule, we can calculate both its target risk and its surrogate risk. But the two numbers measure different losses, and a smaller surrogate risk for one rule does not automatically imply a smaller target risk. The distinction is important whenever conclusions about predictive performance are drawn from an objective that differs from the one ultimately of interest.

Definition: Target Risk and Surrogate Risk

Let $Z=(X,Y)$ be an observation with a fixed probability distribution. Let $f$ be a measurable prediction rule, so $f(X)$ is the prediction made when the feature value is $X$. Suppose the target loss $\ell(y,a)$ and the replacement loss $\tilde{\ell}(y,a)$ are measurable, nonnegative functions of outcome $y$ and prediction $a$. The two risks are expectations under the same distribution of $Z$.

Definition: The target risk of $f$ is $R_{\ell}(f)=E[\ell(Y,f(X))]$. The surrogate risk of $f$ is $R_{\tilde{\ell}}(f)=E[\tilde{\ell}(Y,f(X))]$, where $\tilde{\ell}$ is used in place of the target loss $\ell$. These risks may be finite or infinite; a direct subtraction or numerical comparison of the risks below will be made only when the relevant risks are finite.

A surrogate can be used with the same prediction space as the target loss, or with scores that are converted to decisions by a specified rule. In either case, a risk comparison is well-defined only after the prediction and loss have been specified. The data distribution is held fixed: changing it at the same time as changing the loss would mix two different sources of change.

There are two distinct comparisons. First, fix $f$ and compare $R_{\ell}(f)$ with $R_{\tilde{\ell}}(f)$; this asks how the replacement changes the measured risk for that prediction. Second, fix one loss and compare risks for different rules $f$ and $g$; this asks which rule that loss favors. A difference in the first comparison does not, on its own, determine the second.

Worked Example: A Weighted Classification Replacement

Let $Y$ be either category A or category B, with $P(Y=\mathrm{A})=0.7$ and $P(Y=\mathrm{B})=0.3$. There are two constant prediction rules: $f_{\mathrm{A}}$ always predicts A, and $f_{\mathrm{B}}$ always predicts B. Take the target loss to be zero-one loss: it is 0 for a correct prediction and 1 for an incorrect one. As a replacement, use a weighted loss that charges 10 for predicting A when the outcome is B, and 0 for predicting B when the outcome is A; correct predictions have loss 0.

For $f_{\mathrm{A}}$, an error occurs only when $Y=\mathrm{B}$, so its target risk is $0.3$. Its replacement loss is 10 in that same case and 0 otherwise, giving surrogate risk $0.3(10)+0.7(0)=3$.

For $f_{\mathrm{B}}$, an error occurs when $Y=\mathrm{A}$, so its target risk is $0.7$. The replacement loss assigns 0 to this error, so its surrogate risk is $0.7(0)+0.3(0)=0$.

Thus the target loss prefers $f_{\mathrm{A}}$, since $0.3<0.7$, while the replacement loss prefers $f_{\mathrm{B}}$, since $0<3$. This is an intentionally consequential weighting: the replacement encodes a different penalty for different errors. Calling it a surrogate does not make its ranking agree with the target ranking.

What Can Be Said About the Two Risks?

When both risks are finite, their difference for the same rule has an exact expression. It is the expected difference between the losses evaluated on the very same observed outcome and prediction. This identity is elementary, but it prevents a common ambiguity: subtracting risks does not compare different prediction rules.

Proposition: If $R_{\ell}(f)$ and $R_{\tilde{\ell}}(f)$ are finite, then $$ R_{\tilde{\ell}}(f)-R_{\ell}(f) = E\!\left[\tilde{\ell}(Y,f(X))-\ell(Y,f(X))\right]. $$

Proof. Write $U=\tilde{\ell}(Y,f(X))$ and $V=\ell(Y,f(X))$. Both are nonnegative and have finite expectation by assumption. Since $|U-V|\leq U+V$, the difference $U-V$ is integrable. Linearity of expectation therefore applies:

E[U-V]=E[U]-E[V] =R_{\tilde{\ell}}(f)-R_{\ell}(f).

Substituting the definitions of $U$ and $V$ gives the stated identity. $\square$

A useful sufficient condition for comparing the risks is a uniform bound on how much the replacement loss differs from the target loss. The bound must apply to every outcome and prediction that can occur, not merely to an average selected after seeing the result.

Theorem: Suppose that, for some finite $\varepsilon\geq 0$, $$ |\tilde{\ell}(y,a)-\ell(y,a)|\leq\varepsilon $$ for every relevant outcome $y$ and prediction $a$. Then every rule with finite target risk has finite surrogate risk, and $$ |R_{\tilde{\ell}}(f)-R_{\ell}(f)|\leq\varepsilon. $$ In particular, for two rules $f$ and $g$, if $R_{\ell}(g)-R_{\ell}(f)>2\varepsilon$, then $R_{\tilde{\ell}}(g)>R_{\tilde{\ell}}(f)$.

Proof. The pointwise bound implies $\tilde{\ell}(Y,f(X))\leq\ell(Y,f(X))+\varepsilon$. Taking expectations shows that finite target risk implies finite surrogate risk. Applying the same pointwise bound to the absolute difference and then taking expectations gives

|R_{\tilde{\ell}}(f)-R_{\ell}(f)| = \left|E[\tilde{\ell}(Y,f(X))-\ell(Y,f(X))]\right| \leq E[|\tilde{\ell}(Y,f(X))-\ell(Y,f(X))|] \leq\varepsilon.

For the ranking claim, the bound gives $R_{\tilde{\ell}}(g)\geq R_{\ell}(g)-\varepsilon$ and $R_{\tilde{\ell}}(f)\leq R_{\ell}(f)+\varepsilon$. Subtracting these inequalities yields

R_{\tilde{\ell}}(g)-R_{\tilde{\ell}}(f) \geq R_{\ell}(g)-R_{\ell}(f)-2\varepsilon>0.

Hence the surrogate risk ranks $g$ above $f$ in risk, as claimed. $\square$

This result gives a condition that can justify a ranking comparison; it does not say that every replacement loss satisfies such a bound. If the target-risk gap is small compared with the possible discrepancy, the theorem gives no ranking guarantee. Without an additional relationship between the losses, the exact difference identity is all that is available.

Worked Example: Absolute and Squared Error Can Rank Predictions Differently

Let $Y=0$ with probability $0.9$ and $Y=100$ with probability $0.1$. Consider two constant predictions, $f_0=0$ and $f_1=1$. Use absolute error, $\ell(y,a)=|y-a|$, as the target loss and squared error, $\tilde{\ell}(y,a)=(y-a)^2$, as the replacement.

For $f_0$, the target risk is $0.9|0-0|+0.1|100-0|=10$. Its surrogate risk is $0.9(0-0)^2+0.1(100-0)^2=1000$.

For $f_1$, the target risk is $0.9|0-1|+0.1|100-1|=0.9+9.9=10.8$. Its surrogate risk is $0.9(0-1)^2+0.1(100-1)^2=0.9+980.1=981$.

Absolute error prefers $f_0$, since $10<10.8$, whereas squared error prefers $f_1$, since $981<1000$. Both risks use the same distribution and the same two predictions; the reversal arises from replacing the loss. The example also shows why the numerical scale of one risk cannot be read as though it were the scale of the other.

Surrogate Scores and Target Decisions

In some prediction problems the rule produces a score, while the target loss assesses a decision derived from that score. The two losses can then respond differently even when the target decisions agree. For example, zero-one classification loss records whether a score gives the correct sign, while a margin-based replacement can also respond to how far the score lies from its decision boundary. This is a difference in what each loss measures, not a guarantee about which loss is preferable.

Worked Example: Same Classification Risk, Different Surrogate Risk

Let $Y\in\{-1,+1\}$, with $P(Y=+1)=0.75$ and $P(Y=-1)=0.25$. A score $a$ predicts the sign of $a$; both scores below are positive, so both predict $+1$. Define target loss as 1 when the sign is incorrect and 0 otherwise. Use the margin replacement $\tilde{\ell}(y,a)=\max(0,1-ya)$. Compare constant scores $f_{0.2}=0.2$ and $f_2=2$.

Each score predicts $+1$, so each has target risk $P(Y=-1)=0.25$. For score $0.2$, the replacement loss is $1-(+1)(0.2)=0.8$ when $Y=+1$, and $1-(-1)(0.2)=1.2$ when $Y=-1$. Its surrogate risk is $0.75(0.8)+0.25(1.2)=0.6+0.3=0.9$.

For score $2$, the replacement loss is $\max(0,1-2)=0$ when $Y=+1$, and $\max(0,1+2)=3$ when $Y=-1$. Its surrogate risk is $0.75(0)+0.25(3)=0.75$.

The target risks tie at $0.25$, but the surrogate risks do not: $0.75<0.9$, so the replacement distinguishes the two scores. That distinction may be useful for a method that works with scores, but it is not evidence that the target error rates differ.

Why the Distinction Matters

A risk is always defined relative to a loss. If a scientific question asks for expected absolute error, a squared-error risk is not simply another calculation of that same quantity. If a classification objective is based on zero-one loss, a replacement that reacts to scores measures more than whether the final decisions are correct. The replacement may be chosen for practical or mathematical reasons, but conclusions about the target objective require a stated comparison condition, a direct calculation, or another argument that connects the two risks.

Keep the comparisons organized:

Hold the prediction fixed to ask how the loss replacement changes its measured risk.
Hold the loss fixed to rank different prediction rules under that criterion.
Do not infer a ranking under the target loss from a ranking under a surrogate unless a justified relationship connects them.

Common Mistake

Wrong reasoning: “The surrogate risk of $f$ is lower than the surrogate risk of $g$, so $f$ must have lower target risk.” That conclusion silently assumes the replacement preserves the target-risk ranking.

Correction: A surrogate-risk comparison directly ranks the rules only under the surrogate loss. To conclude anything about target risk, calculate target risks or establish a valid transfer condition, such as the uniform discrepancy bound proved above. A label such as “surrogate” does not supply that condition.

Key takeaway: Surrogate risk replaces the target loss inside the expectation while keeping the underlying distribution and prediction rule explicit. Comparing both risks on the same rule is well-defined; assuming their minimizers or rankings coincide requires further justification.

Check Your Understanding

Answer each question by distinguishing the loss being used from the prediction rule being evaluated.

In the definition of surrogate risk, which components are held fixed when comparing it with target risk?
Why does a lower surrogate risk for one of two rules not necessarily imply a lower target risk?
State the pointwise condition in the theorem that bounds the difference between target and surrogate risks.
In the absolute-error example, which prediction is preferred by each loss? Verify the two pairs of risks.
In the classification-score example, why can the surrogate risks differ even though the target risks are equal?