Plug-In Multiclass Rules from Probability Estimates

Why Probability Errors Matter for Decisions

A multiclass probability estimate is useful because it can be converted into a decision: predict a class, select a treatment, or choose another action. The best decision depends not only on which class is most probable, but also on the task loss attached to each possible action and outcome. A small error in estimated probabilities therefore has no single fixed effect on decision quality: its effect is controlled by the range of the loss.

This tutorial gives a direct guarantee for a plug-in rule: compute conditional risks using estimated class probabilities, then choose an action with the smallest estimated risk. The bound applies to general finite multiclass loss matrices. It does not require a particular probability-estimation method or a margin condition.

Setup and Definition

Let \(Y\) take values in the finite class set \(\{1,\ldots,K\}\), and let \(\mathcal A\) be a finite set of available actions. The action set need not be identical to the class set. A loss matrix \(L\) assigns a finite loss \(L(a,y)\) to each action \(a\in\mathcal A\) when the true class is \(y\). Smaller loss is preferred.

At a fixed covariate value \(x\), write \(\eta_y(x)=P(Y=y\mid X=x)\) for the true class probabilities. They form a probability vector \(\eta(x)\). Suppose an estimate \(\hat\eta(x)\) is also a probability vector: each entry is nonnegative and its entries sum to one. For any probability vector \(p\), the conditional risk of action \(a\) is the expected loss when the class probabilities are \(p\).

Define the global loss range by

A common additive constant in all entries of the loss matrix does not affect decisions or excess risk; the range \(\Delta\) captures the relevant scale. If \(\Delta=0\), all actions have the same loss for every class, so every decision has zero excess risk.

A Risk Perturbation Bound

The first step is to control how much the risk of a fixed action changes when its probability vector changes. For two probability vectors \(p\) and \(q\), their total variation distance is \(\operatorname{TV}(p,q)=\frac12\sum_y|p_y-q_y|\). For a fixed action, the losses across classes lie within an interval no wider than \(\Delta\).

Proof. Put \(d_y=p_y-q_y\). Since \(p\) and \(q\) both sum to one, \(\sum_y d_y=0\). The total positive mass in \(d\) equals the absolute total negative mass; denote each by \(t\). Then \(t=\frac12\sum_y|d_y|=\operatorname{TV}(p,q)\). Let \(m=\min_y L(a,y)\) and set \(c_y=L(a,y)-m\). We have \(0\leq c_y\leq\Delta\). Subtracting \(m\) does not change the risk difference, because \(\sum_y d_y=0\). The positive and negative parts of \(\sum_y c_y d_y\) each have magnitude between 0 and \(\Delta t\), so their difference has absolute value at most \(\Delta t\). This proves the inequality. \(\square\)

Excess Risk of the Plug-In Rule

Proof. Add and subtract the two risks computed under \(q\):

The middle bracket is nonpositive because \(a_q\) minimizes risk under \(q\). Each of the other brackets is at most its absolute value, which is at most \(\frac{\Delta}{2}\lVert p-q\rVert_1\) by the lemma. Adding these two bounds gives the stated inequality. \(\square\)

The factor of two relative to the one-action perturbation bound is important: the estimated action is chosen by comparing two actions, and the probability error can affect both of their risks. The result requires no unique minimizer; any minimizers selected under the stated tie rule satisfy it.

From Conditional Risk to Population Risk

Let \(g_{\hat\eta}(x)\) be the plug-in action at covariate value \(x\), and let \(g^*(x)\) be a Bayes action. The population risk of a rule \(g\) is \(R(g)=E[L(g(X),Y)]\). Assume the conditional probabilities and estimated probabilities are measurable functions of \(x\), the action set is finite, and the loss values are finite. A fixed tie-breaking rule then makes the decisions measurable. Applying the theorem separately at each \(x\) and taking expectations gives

If \(\hat\eta\) is estimated from a training sample, this statement holds conditional on that sample: treat the fitted probabilities as fixed, apply the pointwise result, and average over a new covariate and outcome. Taking expectation over the training sample then gives the corresponding unconditional bound, provided the expectations exist. In particular, if every coordinate error is at most \(\epsilon(x)\), then \(\lVert\eta(x)-\hat\eta(x)\rVert_1\leq K\epsilon(x)\), yielding the looser bound \(\Delta K E[\epsilon(X)]\).

Interpretation and Common Pitfalls

The result separates two ingredients. The probability estimator determines the size of the error in the class-probability vector, measured here by the sum of absolute coordinate errors. The task determines how costly a given probability error can be through \(\Delta\). A large loss range permits a larger excess-risk bound for the same probability accuracy. Conversely, the bound can be conservative when only a few actions are competitive or when their losses vary little across the classes receiving probability error.

• Using only one risk comparison. The lemma bounds the change in risk for one fixed action by \(\frac{\Delta}{2}\lVert p-q\rVert_1\). Applying that bound once does not bound the regret of a decision selected under \(q\): both the selected action and the true optimal action enter the comparison. The decomposition above uses two perturbation terms, giving \(\Delta\lVert p-q\rVert_1\).
• Assuming the most probable class is always optimal. That is true for zero-one loss, but not for a general loss matrix. The plug-in rule must minimize the estimated conditional task risk.
• Ignoring the scale of the loss. A probability-error bound alone is not a task-risk bound. Multiplying by the loss range converts the probability discrepancy into the correct loss scale.