Weighted Risks for Unequal Observation Costs

Why Weight Observations?

The usual empirical risk gives every observed case the same contribution. That is appropriate when all cases are meant to count equally. In some settings, however, observations have different importance or represent different amounts of the population. A weighted risk lets the analyst specify how strongly each observed case contributes, while retaining the same loss function for each case.

Weights affect the comparison being made: a model that performs well on highly weighted observations can be preferred even if its average loss across the sample is larger. It is therefore important to state both the weights and the normalization convention. Normalization changes the numerical scale of risk, but, when the same positive weights are used to compare all predictors, it does not change which predictors minimize it.

Empirical Weighted Risk

Suppose the observed data are \(z_1,\ldots,z_n\), a predictor or decision rule is \(h\), and the loss on observation \(z_i\) is \(\ell(h,z_i)\). Assign observation \(i\) a fixed weight \(a_i\geq 0\). The weights are not all zero.

The first convention divides by the sample size \(n\); the second divides by the total weight. If all weights equal one, both reduce to the ordinary empirical average. In general, \(\widehat R_{a}^{\,\mathrm{raw}}(h)=(\sum_i a_i/n)\widehat R_{a}^{\,\mathrm{norm}}(h)\). Thus the two conventions have different scales, but their ratio is constant across predictors for a fixed data set and fixed weights.

Population Weighted Risk

For a random observation \(Z\) with distribution \(F\), let \(w(Z)\) be a measurable, nonnegative weight and let \(\ell(h,Z)\) be the loss. The weight may be specified directly as a function of the observation, or may be a recorded nonnegative quantity associated with it. The population definition must distinguish the weighted expectation from the weighted average.

For an unnormalized risk, finiteness of the weighted expectation is needed to obtain a finite numerical value. For a normalized finite risk, both a finite, strictly positive expected weight and a finite weighted loss expectation are required. Nonnegative weights ensure that the contributions do not cancel one another. A weight of zero removes an observation from the weighted objective; it does not change that observation’s loss.

Normalized Weighting as a Change of Distribution

When \(0<E[w(Z)]<\infty\), normalized weighting has a precise population interpretation. It defines a new distribution that gives relatively more probability to observations with larger weights. For a measurable set \(A\), define \(Q(A)=E[w(Z)\mathbf{1}_{\{Z\in A\}}]/E[w(Z)]\).

Proof. For every measurable \(A\), \(Q(A)\geq0\). Also, \(Q(\Omega)=E[w(Z)\mathbf{1}_{\{Z\in\Omega\}}]/E[w(Z)]=E[w(Z)]/E[w(Z)]=1\). If \(A_1,A_2,\ldots\) are pairwise disjoint measurable sets, their indicators sum to the indicator of their union. By monotone convergence for the nonnegative functions \(w(Z)\mathbf{1}_{\{Z\in\cup_j A_j\}}\), it follows that \(Q(\cup_j A_j)=\sum_j Q(A_j)\). Hence \(Q\) is a probability distribution.

For a nonnegative simple function \(g=\sum_{j=1}^m c_j\mathbf{1}_{A_j}\), with disjoint measurable \(A_j\) and \(c_j\geq0\), linearity gives \(E_Q[g]=\sum_j c_jQ(A_j)=E[w(Z)g(Z)]/E[w(Z)]\). For any nonnegative measurable \(g\), there is an increasing sequence of nonnegative simple functions converging pointwise to \(g\). Applying the simple-function identity and then the monotone convergence theorem under both distributions gives the same identity for \(g\). Taking \(g(Z)=\ell(h,Z)\), when the weighted expectation is finite, proves the risk statement. \(\square\)

This interpretation also clarifies what a weight does: normalized weighting changes the distribution against which loss is averaged. It does not, by itself, replace the loss function. Different applications may motivate different weights, so the weighting rule should be reported rather than treated as an inconsequential computational detail.

What Normalization Does—and Does Not—Change

Proof. For every \(h\in\mathcal H\), \(\widehat R_{a}^{\,\mathrm{raw}}(h)=(A/n)\widehat R_{a}^{\,\mathrm{norm}}(h)\). The multiplier \(A/n\) is strictly positive and does not depend on \(h\). Therefore, for any \(h_1,h_2\in\mathcal H\), \(\widehat R_{a}^{\,\mathrm{raw}}(h_1)\leq\widehat R_{a}^{\,\mathrm{raw}}(h_2)\) if and only if \(\widehat R_{a}^{\,\mathrm{norm}}(h_1)\leq\widehat R_{a}^{\,\mathrm{norm}}(h_2)\). The ordering of all predictors, including ties, is the same, so their minimizing sets are identical. \(\square\)

The same reasoning applies to population risk when \(0<E[w(Z)]<\infty\): normalized risk is unnormalized risk multiplied by the same positive constant \(1/E[w(Z)]\) for every predictor. This invariance is about comparing predictors under a fixed weighting scheme. It does not say that changing the weights leaves the comparison unchanged.

Common Mistakes

• Mixing the two denominators. A weighted sum divided by \(n\) is not generally a weighted average. Divide by \(\sum_i a_i\) for the normalized empirical risk.
• Assuming weights only change scale. Multiplying every weight by the same positive constant changes only scale. Changing the relative weights can change which predictor has lower risk, as the first worked example shows.
• Forgetting to check the denominator. Normalization is undefined if all empirical weights are zero; at the population level, it requires \(E[w(Z)]>0\).
• Confusing observation weighting with a changed loss. Weighted risk multiplies the loss contribution of an observation by its weight. It does not, by that operation alone, assign a different loss to each kind of error.