Validation of Nondecomposable Performance Criteria

Why These Criteria Need Special Validation

Many validation criteria are averages of individual losses: calculate a loss for each observation and average the results. AUC and F-measure do not have that form. AUC compares scores across pairs of observations with different labels, while F-measure is a ratio formed from several counts. Their validation estimates therefore need to respect the structure of the criterion, rather than treating it as an ordinary per-observation average.

Throughout, suppose a fitted classifier produces a real-valued score $S=s(X)$, with larger scores indicating stronger evidence for the positive class $Y=1$. A threshold $t$ gives the binary prediction $\hat Y_t=I(S\geq t)$. For validation to estimate the performance of this fitted rule, the validation observations must be independent of the data used to fit it. The targets below are conditional on the fitted scoring rule and threshold; they describe how that rule performs on new observations from the target population.

Population and Empirical AUC

Definition: Let $S^+$ have the score distribution conditional on $Y=1$, and let $S^-$ independently have the score distribution conditional on $Y=0$. The population area under the ROC curve is $$ \operatorname{AUC}=P(S^+>S^-)+\frac{1}{2}P(S^+=S^-), $$ provided both classes have positive probability. The half-credit convention makes tied positive-negative scores count as half a correctly ordered pair.

For validation observations $(Y_i,S_i)$, let $n_1$ and $n_0$ be the numbers of positives and negatives. When both are nonzero, the empirical AUC is

\widehat{\operatorname{AUC}} = \frac{1}{n_1n_0} \sum_{i:Y_i=1}\sum_{j:Y_j=0} \left\{I(S_i>S_j)+\frac{1}{2}I(S_i=S_j)\right\}.

Each positive-negative pair contributes one, one-half, or zero according to its ordering. This is a pairwise statistic, not an average of $n$ independent losses: many pair terms share the same validation observation.

Worked Example: Computing Validation AUC With a Tie

Suppose a validation set has positive scores $0.9, 0.6$ and negative scores $0.7, 0.6, 0.2$. There are $2\times3=6$ positive-negative pairs. For the positive score $0.9$, all three comparisons are wins, giving $3$. For $0.6$, the comparison with $0.7$ is a loss, the comparison with $0.6$ is a tie worth $1/2$, and the comparison with $0.2$ is a win. Its contribution is $1.5$.

The total pair score is $3+1.5=4.5$, so $$ \widehat{\operatorname{AUC}}=\frac{4.5}{6}=0.75. $$ The tie is included explicitly; counting it as either a full win or a full loss would not use the stated AUC convention.

Theorem: Suppose validation observations are IID from a population with $0<P(Y=1)<1$, and the score rule is fixed independently of them. Define empirical AUC as above when both classes appear, and assign it any fixed value, such as zero, otherwise. Then $\widehat{\operatorname{AUC}}\xrightarrow{p}\operatorname{AUC}$.

Proof. Conditional on the validation class counts $n_1,n_0$, the positive scores are IID draws from the positive-class score distribution and the negative scores are IID draws from the negative-class score distribution, with the two groups independent. Write $h(a,b)=I(a>b)+\tfrac12 I(a=b)$, so $0\leq h\leq1$. Conditional on positive counts, the empirical AUC is the average of $h(S_i^+,S_j^-)$ over all $n_1n_0$ pairs. Its conditional expectation is the population AUC.

To bound its conditional variance, expand the variance of the pair average as a sum of covariances. Two pair terms are independent if they share neither a positive nor a negative score. There are $n_1n_0$ diagonal pairs of terms; at most $n_1n_0(n_0-1)$ ordered pairs share their positive score, and at most $n_1(n_1-1)n_0$ share their negative score. Each covariance has absolute value at most $1$, since both terms lie in $[0,1]$. Thus, for positive counts, $$ \operatorname{Var}(\widehat{\operatorname{AUC}}\mid n_1,n_0) \leq \frac{1}{n_1n_0}+\frac{1}{n_1}+\frac{1}{n_0}. $$ Because $0<P(Y=1)<1$, the law of large numbers gives $n_1/n\xrightarrow{p}P(Y=1)$ and $n_0/n\xrightarrow{p}P(Y=0)$. Both counts therefore diverge in probability, and the bound tends to zero in probability. Conditional Chebyshev bounds then imply convergence to the population AUC on the event that both classes occur. Finally, the probability that either class is absent is $P(Y=1)^n+P(Y=0)^n$, which tends to zero. The arbitrary extension on that event cannot affect convergence in probability. $\square$

F-Measure: A Ratio of Population Quantities

Fix a threshold $t$. Define the population true-positive, false-positive, and false-negative probabilities by $a=P(Y=1,S\geq t)$, $b=P(Y=0,S\geq t)$, and $c=P(Y=1,S<t)$. The population F-measure, or F1, is the harmonic mean of precision and recall when those component ratios are defined. Its equivalent joint-probability form is more useful for validation.

Definition: If $d=2a+b+c>0$, the population F1 is $$ F_1=\frac{2a}{2a+b+c}. $$ For IID validation observations, let $TP,FP,FN$ be the corresponding counts. If $D=2TP+FP+FN>0$, define $$ \widehat F_1=\frac{2TP}{2TP+FP+FN}. $$ If $D=0$, define $\widehat F_1$ by an explicit convention, for example $\widehat F_1=0$.

The denominator has a useful interpretation: $2a+b+c=P(S\geq t)+P(Y=1)$, and the sample denominator is the predicted-positive count plus the actual-positive count. Thus $d>0$ whenever the target population has positive prevalence $P(Y=1)>0$, even if the classifier never predicts positive.

Worked Example: Calculating F1 From a Confusion Table

A validation sample has 12 true positives, 8 false positives, and 10 false negatives. Then $TP=12$, $FP=8$, and $FN=10$, giving $$ \widehat F_1=\frac{2(12)}{2(12)+8+10}=\frac{24}{42}=\frac{4}{7}\approx0.571. $$ For comparison, precision is $12/(12+8)=0.6$, and recall is $12/(12+10)=6/11\approx0.545$. Their harmonic mean is $$ \frac{2(0.6)(6/11)}{0.6+6/11}=\frac{4}{7}, $$ which checks the count-based calculation.

Theorem: Suppose validation observations are IID, $P(Y=1)>0$, and the fitted threshold rule is fixed independently of them. Define $\widehat F_1=0$ when its denominator is zero. Then $\widehat F_1\xrightarrow{p}F_1$.

Proof. Set $A_n=TP/n$, $B_n=FP/n$, and $C_n=FN/n$. Each is an average of IID indicator variables, so the law of large numbers gives $A_n\xrightarrow{p}a$, $B_n\xrightarrow{p}b$, and $C_n\xrightarrow{p}c$. Consequently, $2A_n+B_n+C_n\xrightarrow{p}d=2a+b+c$. Since $d\geq P(Y=1)>0$, the limiting denominator is strictly positive. The continuous mapping theorem applied to the ratio gives $$ \frac{2A_n}{2A_n+B_n+C_n}\xrightarrow{p}\frac{2a}{2a+b+c}=F_1 $$ whenever the sample denominator is positive.

It remains to handle the zero-denominator event. The sample denominator is zero exactly when there are no actual positives and no predicted positives. In particular, this event is contained in the event that all $n$ outcomes are negative, whose probability is $(1-P(Y=1))^n\to0$. The chosen value zero on the zero-denominator event therefore changes the statistic only on an event whose probability tends to zero. Hence the extended statistic also converges in probability to $F_1$. $\square$

Worked Example: Why the Zero-Denominator Convention Matters

Suppose $P(Y=1)=0.1$, and the rule predicts positive exactly when $Y=1$. Then $a=0.1$, $b=0$, and $c=0$, so the population F1 is $2(0.1)/(2(0.1))=1$. In a validation sample of size $n$, the event of observing only negatives has probability $0.9^n>0$ for every finite $n$. On that event, $TP=FP=FN=0$, so the raw ratio is $0/0$, not a defined statistic.

Assigning, for example, $\widehat F_1=0$ on this event makes the estimate defined for every sample. Since $0.9^n\to0$, the convention does not alter its probability limit. This is why consistency must be stated for an extended statistic, rather than applying the ratio formula on an event where it is undefined.

Prevalence, Stratification, and What the Estimate Means

AUC compares the conditional score distributions in the positive and negative classes. Changing the prevalence while leaving those two conditional distributions unchanged does not change population AUC. F1 is different: its population probabilities combine the classifier's behavior in each class with the target prevalence. As prevalence changes, the balance of true positives, false positives, and false negatives changes, and so can F1.

Worked Example: Adjusting F1 for Stratified Validation

Suppose a target population has prevalence $\pi=0.10$. A fixed rule has sensitivity $P(S\geq t\mid Y=1)=0.80$ and false-positive rate $P(S\geq t\mid Y=0)=0.10$. In the target population, $$ a=0.10(0.80)=0.08,\qquad b=0.90(0.10)=0.09,\qquad c=0.10(0.20)=0.02. $$ Therefore, $$ F_1=\frac{2(0.08)}{2(0.08)+0.09+0.02} =\frac{0.16}{0.27}\approx0.593. $$

Now suppose validation deliberately samples equal numbers from the two classes. The class-conditional rates can still be estimated from that design, but the unweighted validation F1 corresponds to prevalence $0.50$, not $0.10$. At prevalence $0.50$, the probabilities are $a=0.40$, $b=0.05$, and $c=0.10$, giving $F_1=0.80/(0.80+0.05+0.10)\approx0.842$. To target prevalence $0.10$, combine estimated class-conditional rates with the target $\pi$ as in the first calculation. The two values differ because F1 is prevalence-dependent.

Practical Validation Choices and Common Pitfalls

For a single independent holdout set, compute AUC from all positive-negative validation pairs and compute F1 from the combined confusion counts at the specified threshold. Report the threshold and the target population or sampling design alongside F1; an F1 value without those details can be difficult to interpret. AUC and F1 answer different questions: AUC summarizes ranking across thresholds, while F1 summarizes a particular thresholded decision.

In cross-validation, the aggregation rule also matters. Averaging fold-specific F1 values gives equal weight to folds, while pooling the fold confusion counts and forming one ratio gives a count-weighted combined quantity. These need not agree because a ratio of pooled counts is not generally the average of fold ratios. Fold-specific AUC averages compare positive-negative pairs within folds; pooling out-of-fold scores can additionally compare scores produced by different fitted models. If those models have different score scales, cross-fold pair comparisons may not represent the intended performance target.

Pitfall: treating AUC terms as independent. Pair contributions share observations, so the usual independent-loss variance calculation does not apply directly. The pairwise structure must be respected.
Pitfall: leaving a zero-denominator statistic undefined. Even with positive population prevalence, an all-negative validation sample has positive probability at finite $n$. Specify a convention and show that the exceptional event vanishes asymptotically.
Pitfall: using a balanced validation sample's raw F1 as population F1. Unless its class proportions match the target population, the unweighted ratio targets a different prevalence. Reconstruct the joint probabilities using the target prevalence.
Pitfall: choosing a threshold on the same data and reporting its unadjusted F1 as if the threshold were fixed. The reported value then includes the benefit of selection on those observations. Performance assessment after data-driven tuning requires validation that accounts for the selection procedure.

Key takeaway: Validate a nondecomposable criterion in its own form: AUC uses positive-negative pairs, and F1 uses a ratio of population or sample counts. State the sampling target, handle undefined finite-sample cases explicitly, and ensure the validation design matches the performance claim.

Check Your Understanding

Answer these questions using the definitions and validation logic above.

Why does the empirical AUC compare every positive score with every negative score rather than average independent per-observation losses?
What value does the half-credit convention assign to a tied positive-negative score pair?
Why is $P(Y=1)>0$ enough to make the population F1 denominator positive, even if the classifier never predicts positive?
For a stratified validation sample with equal class counts, what additional information is needed to estimate F1 for a population whose prevalence is 0.10?
Explain why pooled out-of-fold AUC can compare predictions on a different scale from the within-fold comparisons used by fold-specific AUC.