F1 Loss and Set-Level Bayes Decisions

Why F1 Requires a Set-Level Decision

In multilabel prediction, the response is a set of labels: several labels may be correct at once, and the prediction is also a set. The F1 score rewards overlap between these two sets while penalizing labels that are missed or added. Because its denominator depends on both set sizes, the expected F1 of a prediction generally cannot be computed from each label’s marginal probability alone.

The Bayes decision is the prediction set with the smallest conditional expected loss, given the available information. For F1 loss, finding it requires considering both which labels to predict and how many. On a finite label set, this problem has an exact solution: for each possible nonzero prediction size, rank labels by size-dependent expected contributions, then compare the best predictions across sizes.

Definition: Let the finite label set be $\mathcal{Y}=\{1,\ldots,m\}$. Write $Y\subseteq\mathcal{Y}$ for the random true label set and $S\subseteq\mathcal{Y}$ for a predicted set. Define $$ \operatorname{F1}(S,Y)= \begin{cases} 1, & S=\varnothing,\ Y=\varnothing,\\ \dfrac{2|S\cap Y|}{|S|+|Y|}, & |S|+|Y|>0. \end{cases} $$ The F1 loss is $\ell_{\mathrm{F1}}(S,Y)=1-\operatorname{F1}(S,Y)$. The convention that two empty sets receive score 1 makes the score defined for every pair of sets.

Suppose a feature vector $X$ is observed. Assume $X$ takes values in a standard Borel space, so a conditional distribution of the finite-valued $Y$ given $X=x$ can be used. At a fixed $x$, abbreviate $p_A=P(Y=A\mid X=x)$ for each $A\subseteq\mathcal{Y}$. A Bayes decision at $x$ minimizes the conditional risk $E[\ell_{\mathrm{F1}}(S,Y)\mid X=x]$ over all prediction sets $S$. Since the loss is one minus the score, this is equivalent to maximizing conditional expected F1.

Expected F1 at a Fixed Prediction Size

For a nonempty prediction $S$, let $k=|S|$. The denominator in F1 is then positive for every possible true set, including the empty set. Expanding the overlap count as a sum of indicators gives

E[\operatorname{F1}(S,Y)\mid X=x] =2\sum_{i\in S} E\left[\frac{\mathbf{1}\{i\in Y\}}{k+|Y|}\,\middle|\,X=x\right].

Define the size-$k$ weight of label $i$ by

a_i(k;x)=E\left[\frac{\mathbf{1}\{i\in Y\}}{k+|Y|}\,\middle|\,X=x\right] =\sum_{\substack{A\subseteq\mathcal{Y}\\i\in A}}\frac{p_A}{k+|A|}.

These weights are not generally just the marginal probabilities $P(i\in Y\mid X=x)$: the contribution from a true set $A$ is downweighted according to its cardinality. They can also change their ordering as $k$ changes. The expected F1 for a nonempty $S$ is $2\sum_{i\in S}a_i(k;x)$, so at a fixed size the best set consists of the $k$ labels with the largest size-$k$ weights.

Theorem: For each $k\in\{1,\ldots,m\}$, order the labels by decreasing $a_i(k;x)$, breaking ties by any fixed rule, and let $S_k(x)$ contain the first $k$. This set maximizes conditional expected F1 among all predictions of size $k$. A Bayes decision is a set with the largest score among $S_1(x),\ldots,S_m(x),\varnothing$, where the empty prediction has conditional expected F1 $p_\varnothing=P(Y=\varnothing\mid X=x)$.

Proof. Fix $k>0$. For any prediction $S$ with $|S|=k$, the expansion above gives the score $2\sum_{i\in S}a_i(k;x)$. If $S$ contains a label $j$ with a smaller weight than some label $i\notin S$, replacing $j$ by $i$ changes the sum by $a_i(k;x)-a_j(k;x)\geq 0$. Repeated replacements produce a set of the $k$ largest weights without decreasing the score. Thus $S_k(x)$ maximizes the score among size-$k$ predictions. For the empty prediction, the score is 1 when $Y=\varnothing$ and 0 otherwise, so its conditional expectation is $p_\varnothing$. Every prediction is either empty or has one of the sizes $1,\ldots,m$. Comparing these finitely many candidates therefore gives a maximizer over all prediction sets, equivalently a minimizer of conditional F1 risk. Since the label set is finite and the conditional probabilities are measurable in $x$, fixed tie-breaking makes this selection measurable. $\square$

Worked Example: Comparing Prediction Sizes

Suppose there are three labels and, at the observed $x$, the conditional distribution is $P(Y=\{1\}\mid x)=0.5$, $P(Y=\{2\}\mid x)=0.3$, and $P(Y=\{3\}\mid x)=0.2$. All other true sets have probability zero. Calculate the best expected F1 at each prediction size.

For size $k=1$, predicting a singleton receives F1 score 1 when it matches the true singleton and 0 otherwise. The expected scores for $\{1\},\{2\},\{3\}$ are therefore $0.5,0.3,0.2$; the best is $\{1\}$, with score $0.5$.

For size $k=2$, a predicted pair receives score $2/(2+1)=2/3$ when it contains the true singleton, and 0 otherwise. The pair $\{1,2\}$ contains labels 1 and 2, so its expected score is $(0.5+0.3)(2/3)=8/15$. The other pairs score $(0.5+0.2)(2/3)=7/15$ and $(0.3+0.2)(2/3)=1/3$. Thus $\{1,2\}$ is best at size 2.

For size $k=3$, the predicted set contains every possible true singleton, but the score is $2/(3+1)=1/2$ for each one. Its expected score is $1/2$. The empty prediction scores 0 because the true set is never empty. Comparing $1/2$, $8/15$, $1/2$, and 0 shows that $\{1,2\}$ is Bayes optimal, with score $8/15$ and risk $1-8/15=7/15$.

Marginal Probabilities Do Not Determine the Decision

The weights $a_i(k;x)$ depend on the probability of label $i$ appearing in true sets of different sizes. Consequently, two conditional distributions can have identical label marginals but require different Bayes decisions. This gives a direct reason that separate decisions based only on each marginal cannot be universally optimal for F1.

Worked Example: Identical Marginals, Different Optimal Sets

Consider two labels, with both marginal probabilities equal to 0.4 in each of two distributions.

Distribution A: $P(Y=\{1\}\mid x)=0.4$, $P(Y=\{2\}\mid x)=0.4$, and $P(Y=\varnothing\mid x)=0.2$. Predicting both labels gives score $2/3$ for either singleton truth and 0 for the empty truth. Its expected score is $0.8(2/3)=8/15$. Predicting $\varnothing$ scores $0.2$; predicting either singleton scores $0.4$. Thus predicting both is optimal.

Distribution B: $P(Y=\{1,2\}\mid x)=0.4$ and $P(Y=\varnothing\mid x)=0.6$. Each label again has marginal probability 0.4. Predicting both scores 1 for the nonempty truth and 0 for the empty truth, giving expected score $0.4$. Predicting $\varnothing$ gives expected score $0.6$. Predicting either singleton gives score $2/3$ when $Y=\{1,2\}$ and 0 otherwise, for expected score $0.4(2/3)=4/15$. Thus $\varnothing$ is optimal.

The marginal probabilities are the same in both cases, but the optimal predictions differ. Any rule that uses only those two marginals must return the same prediction for both distributions, and therefore cannot be Bayes optimal for both. The difference comes from how the labels co-occur and how often the true set is empty.

Result: There is no decision rule based only on the vector of label marginals that is Bayes optimal for every conditional distribution of the true label set under F1 loss.

Justification. The two distributions in the example have the same marginal vector $(0.4,0.4)$. The first has the unique optimal prediction $\{1,2\}$, since $8/15$ is strictly larger than $0.4$, $0.2$, and $4/15$. The second has the unique optimal prediction $\varnothing$, since $0.6$ is strictly larger than $0.4$ and $4/15$. A rule receiving only the shared marginal vector must make the same choice in both cases, so it fails to be optimal for at least one. $\square$

Worked Example: The Empty-Set Candidate Can Win

Let $P(Y=\varnothing\mid x)=0.3$, $P(Y=\{1,2\}\mid x)=0.4$, and $P(Y=\{2,3\}\mid x)=0.3$. Compute the best expected score for each prediction size.

The empty prediction has expected score $0.3$. For a singleton prediction, a true set of size 2 that contains the predicted label gives score $2/(1+2)=2/3$. The singleton scores for labels 1, 2, and 3 are therefore $0.4(2/3)=4/15$, $0.7(2/3)=7/15$, and $0.3(2/3)=1/5$. The best singleton is $\{2\}$, scoring $7/15$.

For size 2, predicting $\{1,2\}$ scores 1 against $\{1,2\}$ and $1/2$ against $\{2,3\}$, so its expectation is $0.4+0.3/2=0.55$. Predicting $\{2,3\}$ gives $0.3+0.4/2=0.50$, while predicting $\{1,3\}$ gives $0.4/2+0.3/2=0.35$.

Predicting all three labels gives score $4/5$ against either true set of size 2, and hence expected score $0.7(4/5)=0.56$. Comparing the best scores at each size, $0.3$, $7/15$, $0.55$, and $0.56$, shows that predicting all three labels is Bayes optimal. In particular, the best set of a fixed size need not be the overall best set.

Common Pitfall and Practical Interpretation

A common mistake is to insert marginal probabilities into the F1 formula as if the expected score were the F1 score of expected counts. In general, $E[2|S\cap Y|/(|S|+|Y|)\mid x]$ is not the ratio obtained by replacing the random counts in that expression with their expectations. The denominator varies with the true set, and the empty-set convention adds a separate contribution for an empty prediction.

Another tempting shortcut is to apply the same threshold independently to every marginal probability. Such a rule ignores the distribution of true-set sizes and label co-occurrence; the matching-marginals example shows that no marginal-only rule can succeed universally. A threshold can be useful as a chosen heuristic or under additional assumptions, but it does not replace the set-level Bayes comparison derived here.

For a finite label set, the decision has a clear interpretation: evaluate the best candidate of each nonempty size using the corresponding weights $a_i(k;x)$, include the empty prediction with score $P(Y=\varnothing\mid x)$, and select the candidate with the largest expected F1. The resulting prediction is tailored to the evaluation loss rather than assembled from separate label decisions.

Check Your Understanding

Use the definitions and conditional-risk rule to answer the following.

Why can the weight $a_i(k;x)$ differ from the marginal probability $P(i\in Y\mid X=x)$?
For a fixed nonzero prediction size $k$, what property must a set maximizing expected F1 have?
If $P(Y=\varnothing\mid X=x)=0.45$, what is the conditional expected F1 of predicting the empty set?
In the matching-marginals example, why must any rule using only the marginal vector return the same prediction under both distributions?
Why is comparing the best prediction at every size necessary, rather than selecting the best labels at one chosen size?

F1 Loss and Set-Level Bayes Decisions

What You'll Learn