Empirical Risk From Observed Data

From Individual Losses to an Observed-Data Average

A loss function assigns a penalty to a prediction and the outcome that is observed. Once a dataset is in hand, each observation therefore contributes one loss value for a chosen predictor. Empirical risk is the average of those observed loss values. It is a direct summary of performance on the dataset at hand, not an average over outcomes that might have been observed.

This distinction matters in learning: empirical risk gives a concrete quantity that can be calculated from data, while population risk is defined using the distribution of future or generic observations. The two quantities are related, but the observed average is not itself an expectation over the data-generating distribution.

Definition: Let $D_n=((x_1,y_1),\ldots,(x_n,y_n))$ be an observed dataset of $n\geq 1$ feature-response pairs, let $h$ be a prediction function, and let $\ell$ be a loss function. The empirical risk of $h$ on $D_n$ is $$ \widehat R_n(h;D_n)=\frac{1}{n}\sum_{i=1}^{n}\ell\bigl(h(x_i),y_i\bigr). $$ The summand $\ell(h(x_i),y_i)$ is the loss on observation $i$. The empirical risk is their arithmetic mean, with each recorded observation counted once.

For a fixed dataset and a fixed predictor, empirical risk is an ordinary numerical calculation. If the dataset is regarded as random before it is observed, the same formula defines a random quantity: its value depends on which observations are realized. No independence or identical-distribution assumption is needed simply to calculate the average for an observed dataset.

In the earlier material on population risk, the risk of $h$ was defined as the expected loss under the distribution of an observation, when that expectation is defined. Empirical risk instead uses the observed pairs and their average loss. A numerical value for empirical risk does not, by itself, establish what the expected loss will be on new observations.

Three Common Calculations

The calculation has the same structure for different losses: evaluate the loss for each observed pair, add the values, and divide by the number of observations. What changes is how each prediction error is converted into a penalty.

Worked Example: Squared-Error Empirical Risk

Suppose the observed responses are $y_1=2$, $y_2=0$, and $y_3=5$, and a predictor gives the corresponding predictions $h(x_1)=1$, $h(x_2)=2$, and $h(x_3)=3$. Under squared-error loss, $\ell(a,y)=(a-y)^2$, calculate the empirical risk.

For observation 1, the prediction error is $1-2=-1$, so the loss is $(-1)^2=1$. For observation 2, the error is $2-0=2$, so the loss is $2^2=4$. For observation 3, the error is $3-5=-2$, so the loss is $(-2)^2=4$.

The total observed loss is $1+4+4=9$, and there are three observations. Therefore,

\widehat R_3(h;D_3)=\frac{1+4+4}{3}=3.

The value 3 is the average squared error on these three pairs. It is not the square root of the average squared error; that would be a different summary.

Worked Example: Absolute-Error Empirical Risk

Use the same responses and predictions as in the previous calculation, but now take absolute-error loss, $\ell(a,y)=|a-y|$.

The absolute errors are $|1-2|=1$, $|2-0|=2$, and $|3-5|=2$. Thus the observed losses sum to $1+2+2=5$. Dividing by the three recorded pairs gives

\widehat R_3(h;D_3)=\frac{1+2+2}{3}=\frac{5}{3}.

The same predictions and outcomes give squared-error empirical risk $3$ and absolute-error empirical risk $5/3$. These values differ because the losses penalize errors differently: squaring makes larger errors count more heavily relative to smaller ones.

Worked Example: Classification Empirical Risk

Consider four observations with true labels $y=(A,B,B,A)$. A classifier predicts $(A,A,B,B)$. Under zero-one classification loss, $\ell(a,y)=\mathbf{1}\{a\ne y\}$, the loss is 1 for an incorrect label and 0 for a correct label.

The first prediction is correct, so its loss is 0. The second prediction is incorrect, so its loss is 1. The third is correct, giving loss 0, and the fourth is incorrect, giving loss 1. The total loss is $0+1+0+1=2$. Hence

\widehat R_4(h;D_4)=\frac{2}{4}=\frac{1}{2}.

For zero-one loss, empirical risk is the fraction of observed labels predicted incorrectly. Here, the classifier makes errors on two of the four observations, so its empirical risk is 50 percent.

Empirical Risk as an Average Over the Observed Dataset

An empirical average can also be written as an average with respect to a distribution that places equal mass on each observed pair. This representation makes explicit that repeated observations retain their multiplicity: if the same pair appears several times, it contributes several times to the sum.

Result: Define the empirical distribution of the observed pairs by $\widehat P_n=\frac{1}{n}\sum_{i=1}^n\delta_{(x_i,y_i)}$, where $\delta_{(x_i,y_i)}$ puts unit mass at the pair $(x_i,y_i)$. For any loss values defined at the observed pairs, $$ \widehat R_n(h;D_n)=\int \ell\bigl(h(x),y\bigr)\,d\widehat P_n(x,y). $$

Proof. By the definition of integration with respect to a finite discrete distribution, integrating a function against $\delta_{(x_i,y_i)}$ evaluates that function at $(x_i,y_i)$. Linearity of a finite sum therefore gives

\int \ell\bigl(h(x),y\bigr)\,d\widehat P_n(x,y) =\frac{1}{n}\sum_{i=1}^n \int \ell\bigl(h(x),y\bigr)\,d\delta_{(x_i,y_i)}(x,y) =\frac{1}{n}\sum_{i=1}^n\ell\bigl(h(x_i),y_i\bigr) =\widehat R_n(h;D_n).

This identity is just a different expression for the same finite average. It does not replace the observed dataset with the population distribution: $\widehat P_n$ puts mass only on the pairs that were recorded. $\square$

Checking the Average by Partitioning the Data

A useful arithmetic check is to divide the observations into disjoint groups, calculate each group’s average loss, and then combine those averages using the group sizes as weights. Simply averaging group averages is generally wrong when the groups have different sizes.

Result: Suppose the index set $\{1,\ldots,n\}$ is partitioned into nonempty disjoint groups $I_1,\ldots,I_m$. Write $n_j=|I_j|$, and let $\widehat R_{I_j}(h)=\frac{1}{n_j}\sum_{i\in I_j}\ell(h(x_i),y_i)$ be the average loss in group $j$. Then $$ \widehat R_n(h;D_n)=\sum_{j=1}^{m}\frac{n_j}{n}\widehat R_{I_j}(h). $$

Proof. Substitute the definition of each group average into the right-hand side. Since the groups partition all observation indices, every individual loss appears exactly once:

\sum_{j=1}^{m}\frac{n_j}{n}\widehat R_{I_j}(h) =\sum_{j=1}^{m}\frac{n_j}{n} \left(\frac{1}{n_j}\sum_{i\in I_j}\ell(h(x_i),y_i)\right) =\frac{1}{n}\sum_{j=1}^{m}\sum_{i\in I_j}\ell(h(x_i),y_i) =\frac{1}{n}\sum_{i=1}^{n}\ell(h(x_i),y_i).

The final expression is the definition of $\widehat R_n(h;D_n)$, proving the identity. $\square$

Worked Example: Combining Unequal-Sized Groups

Suppose six observed losses are $0,2,1,3,0,2$. The first group contains the first two observations, with average loss $(0+2)/2=1$. The second group contains the remaining four, with average loss $(1+3+0+2)/4=6/4=1.5$.

The overall average weights these group averages by their proportions of the full dataset:

\widehat R_6=\frac{2}{6}(1)+\frac{4}{6}(1.5) =\frac{1}{3}+1=\frac{4}{3}.

Checking directly, the six losses sum to $0+2+1+3+0+2=8$, so the overall average is $8/6=4/3$, as required. An unweighted average of the two group averages would be $(1+1.5)/2=1.25$, which is incorrect because the groups contain different numbers of observations.

Common Mistakes and Interpretation

Using a sum instead of an average. The total loss is $\sum_{i=1}^n\ell(h(x_i),y_i)$; empirical risk divides that total by $n$. Confusing them changes the scale and makes the result depend directly on dataset size.
Confusing empirical risk with population risk. Empirical risk averages the losses on the observed pairs. Population risk averages according to the distribution of an observation. The observed average alone does not make the two quantities equal.
Dropping repeated observations. If a pair occurs more than once in the dataset, each occurrence contributes a loss. The empirical distribution and the empirical-risk sum both preserve these repetitions.
Averaging group averages without weights. This is only valid when the groups have equal sizes. In general, weight each group’s average by its proportion of the total number of observations.
Changing the loss midway through a calculation. Squared error, absolute error, and zero-one loss give different penalties. State the loss first, then apply it consistently to every observation.

Key takeaway: To calculate empirical risk, evaluate the specified loss on every observed pair, add the resulting values, and divide by the number of pairs. The answer describes average loss on that dataset under that predictor and loss function.

Check Your Understanding

Work from the definition and show the individual losses in each calculation.

For responses $1,4$ and predictions $2,2$, calculate empirical risk under squared-error loss.
Using the same responses and predictions, calculate empirical risk under absolute-error loss.
A classifier makes 3 mistakes among 12 observed cases. What is its empirical risk under zero-one loss?
Explain why repeating each observation in a dataset twice leaves its empirical risk unchanged.
Two groups have sizes 3 and 7, with average losses 2 and 5. Calculate the overall empirical risk using the group averages.