Why Review Capacity Changes the Evaluation Question
A model may assign each case a score that ranks its likelihood of requiring follow-up. If staff can investigate only a fixed number of cases, the operational question is not simply, “How well does the model classify cases?” It is also, “Among the cases we can actually review, how many are useful to review?” This calls for evaluation at the available capacity.
A fixed-capacity policy selects the highest-priority \(k\) cases from a batch of \(n\). A fixed-cutoff policy selects every case whose score meets a specified threshold, so the number selected can vary from batch to batch. These policies answer different operational questions and need not select the same cases.
Top-\(k\) precision measures the proportion of reviewed cases that were positive. Top-\(k\) recall measures the proportion of all positive cases that the review capacity captured. When comparing rankings for the same operational capacity, precision answers how productive the review slots were; recall answers how much of the positive population those slots found. Neither measure alone describes every operational objective.
Top-\(k\) Measures and Prevalence
The positive fraction in the evaluation set is its prevalence, \(P/n\). A related measure is lift at capacity: top-\(k\) precision divided by prevalence. It compares the positive rate in the selected cases with the positive rate in the full evaluation set. Lift above one indicates that the selected cases are more concentrated in positives than the full set.
These measures are connected by a simple identity. Write \(r=k/n\) for the fraction of cases that can be reviewed and \(\pi=P/n\) for prevalence. Since \(TP_k=(\text{Recall@}k)P\), substituting into precision gives
Thus lift can also be written as \(\text{Recall@}k/r\). This derivation explains why precision, recall, and lift are not independent summaries when they use the same cases, outcome definition, and capacity. It also shows why a high recall at a very large review fraction does not necessarily mean strong prioritization: selecting a large share of the population makes it easier to capture positives.
Worked Example: Measuring Performance at a Five-Case Capacity
A hypothetical review team can investigate five of 20 claims. A model assigns descending scores to the cases. The observed outcomes, in score rank order, are: \(1,0,1,1,0,0,1,0,0,1,0,0,0,1,0,0,0,0,0,1\). Evaluate the top-five policy. Then compare it with a fixed score cutoff of 0.76, which selects the six highest-scored cases.
Method. Count positive outcomes among the selected cases, then use the total number of positives in the full validation set as the recall denominator.
Step 1: Count all positives. The positive outcomes occur at ranks 1, 3, 4, 7, 10, 14, and 20. Therefore \(P=7\), \(n=20\), and prevalence is \(7/20=0.35\).
Step 2: Evaluate top five. The first five outcomes are \(1,0,1,1,0\), so \(TP_5=3\). There are \(5-3=2\) false positives among the reviewed cases. Precision is \(3/5=0.60\), and recall is \(3/7=0.4286\) to four decimal places. Lift is \(0.60/0.35=1.7143\) to four decimal places.
Step 3: Evaluate the fixed cutoff. At cutoff 0.76, six cases are selected. Their outcomes are \(1,0,1,1,0,0\), giving three true positives and three false positives. Precision is \(3/6=0.50\), while recall remains \(3/7=0.4286\). The cutoff policy reviews one more case but finds no additional positive in this batch.
Check. Both policies find three positives, so their recall is identical. The fixed-cutoff policy uses six slots rather than five, so its precision is lower: \(0.50<0.60\). The common mistake is to describe the cutoff result as top-five performance; a fixed threshold selects all cases meeting the threshold, not necessarily exactly the available capacity.
Fixed Capacity Is Not a Fixed Score Threshold
Under top-\(k\) selection, the number selected is determined by capacity. The score boundary depends on the scores in that batch: when there is no tie at the boundary, the score of the \(k\)th-ranked case separates the selected cases from the rest. Under a fixed-cutoff policy, the threshold is specified in advance and the number selected depends on how many cases meet it. A threshold can therefore exceed capacity in a high-risk batch or leave review slots unused in a low-risk batch.
If a fixed threshold selects more cases than staff can review, an additional rule is needed to decide which cases fit within capacity. If it selects fewer cases, the organization must decide whether to leave capacity unused or review cases below the threshold. Those are operational policy choices, not details that a metric can resolve automatically.
Ties require particular care for top-\(k\) selection. If cases tie at the capacity boundary, score alone does not identify exactly which \(k\) cases should be selected. An explicit rule—such as a secondary priority variable, a documented random selection procedure, or review of all tied cases if capacity permits—is needed. The realized top-\(k\) metrics can depend on that rule.
Worked Example: A Tie at the Capacity Boundary
A hypothetical clinic can review three of eight referrals. The scores and outcomes in descending score order are: A, 0.95, outcome 1; B, 0.90, outcome 0; C, 0.80, outcome 1; D, 0.80, outcome 0; E, 0.70, outcome 1; F, 0.60, outcome 0; G, 0.50, outcome 0; H, 0.40, outcome 1. Compare two possible ways to fill the third review slot.
Method. The first two cases are unambiguously selected. Because C and D tie at the boundary, specify which one receives the final slot before calculating performance.
Step 1: Count positives in the full set. A, C, E, and H are positive, so \(P=4\).
Step 2: Apply a secondary rule that selects C. The reviewed cases are A, B, and C, with outcomes \(1,0,1\). Thus \(TP_3=2\), precision is \(2/3=0.6667\), and recall is \(2/4=0.5000\), both rounded to four decimal places.
Step 3: Apply a different secondary rule that selects D. The reviewed cases are A, B, and D, with outcomes \(1,0,0\). Thus \(TP_3=1\), precision is \(1/3=0.3333\), and recall is \(1/4=0.2500\), to four decimal places.
Check. Both selections include exactly three cases, but the measured performance differs because the tied cases have different outcomes. The common mistake is to claim that the third-ranked score by itself separates selected from unselected cases. It does not when the boundary score is tied; the tie-handling rule must be reported.
Comparing Models at the Same Capacity
When the operational constraint is a fixed number of reviews, compare candidate rankings at the same \(k\) on the same held-out cases. The outcome definition and observation period must also be consistent. Otherwise, a difference in reported precision or recall may reflect a different evaluation population or target rather than a better ranking.
It is useful to report both the count \(TP_k\) and the rates. The count makes the operational yield concrete, while precision and recall put that yield in context. If capacity changes, report performance at the relevant capacities rather than assuming that a model’s performance at one \(k\) carries over to another. Top-\(k\) precision is not guaranteed to decrease smoothly as \(k\) increases: adding a case can raise or lower the cumulative positive fraction.
Worked Example: Comparing Two Rankings for Eight Review Slots
A hypothetical benefits team has 40 held-out cases, of which 10 had the target outcome. Two models rank these same cases. Model A’s first eight outcomes are \(1,1,0,1,0,1,0,1\); Model B’s first eight outcomes are \(1,0,0,1,0,1,0,0\). Compare them when the team can review eight cases.
Method. Use the same outcome denominator, the same eight-case capacity, and the same held-out population for both rankings.
Step 1: Establish the baseline. There are \(P=10\) positives among \(n=40\) cases, so prevalence is \(10/40=0.25\). The review fraction is \(8/40=0.20\).
Step 2: Evaluate Model A. Its first eight outcomes contain five positives. Therefore \(TP_8=5\), precision is \(5/8=0.625\), recall is \(5/10=0.50\), and lift is \(0.625/0.25=2.50\).
Step 3: Evaluate Model B. Its first eight outcomes contain three positives. Therefore \(TP_8=3\), precision is \(3/8=0.375\), recall is \(3/10=0.30\), and lift is \(0.375/0.25=1.50\).
Step 4: Interpret the difference. At this capacity, Model A finds two more positives in the same eight review slots. Its precision is higher by \(0.625-0.375=0.250\), or 25 percentage points, and its recall is higher by \(0.50-0.30=0.20\).
Check. The lift values are consistent with recall divided by the review fraction: Model A has \(0.50/0.20=2.50\), and Model B has \(0.30/0.20=1.50\). The common mistake is to compare models using different numbers of selected cases; that would not answer which ranking makes better use of the stated eight-slot capacity.
Choosing and Reporting the Criterion
The relevant measure follows from the operational question. If the number of cases is fixed and wasted review effort is the primary concern, top-\(k\) precision is directly interpretable. If the objective is to capture as many positives as possible with that capacity, report top-\(k\) recall. Lift provides context for how concentrated the selected cases are relative to the full population. In practice, reporting the selected count, true-positive count, precision, recall, prevalence, and the tie rule makes the result easier to interpret.
A fixed score cutoff is appropriate when the organization has a meaningful threshold that should remain stable across batches, such as a threshold tied to a defined action policy. It should be evaluated by reporting how many cases are selected as well as the resulting confusion counts and rates. A fixed cutoff does not promise a fixed workload; top-\(k\) selection does not promise a fixed score threshold.
Evaluate these criteria on data not used to choose the ranking or tune its evaluation policy. If \(k\), a cutoff, or a tie rule is selected after examining evaluation outcomes, the reported performance can be optimistic. State the capacity and policy in advance where possible, and describe the validation population so that prevalence and review yield can be understood in context. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, use the stated outcome definition and capacity. Round probabilities and rates to four decimal places where needed.
- A validation set has 50 cases and 10 positives. The top five scores include three positives. Find precision@5, recall@5, prevalence, and lift@5. (Answer: precision@5 \(=0.6000\), recall@5 \(=0.3000\), prevalence \(=0.2000\), lift@5 \(=3.0000\).)
- A population of 200 cases contains 10 positives. A team reviews the top 10 cases and finds four positives. Calculate precision@10, recall@10, and lift@10. (Answer: precision@10 \(=0.4000\), recall@10 \(=0.4000\), lift@10 \(=8.0000\).)
- At a fixed score cutoff, a validation set produces \(TP=12\), \(FP=8\), \(FN=18\), and \(TN=162\). How many cases are selected, and what are precision and recall? (Answer: 20 selected; precision \(=0.6000\); recall \(=0.4000\).)
- A set has 10 cases and four positives. The first three ranked outcomes are \(1,0,1\). Two cases tie for the fourth position, one positive and one negative; capacity is four. What are precision and recall if the tie rule selects the positive case, and if it selects the negative case? (Answer: Selecting the positive gives precision \(=0.7500\) and recall \(=0.7500\); selecting the negative gives precision \(=0.5000\) and recall \(=0.5000\).)
- There are 100 cases, including 20 positives. A ranking captures six positives in its top 10 and nine positives in its top 20. Calculate precision and recall at both capacities, and state whether precision must increase when capacity increases. (Answer: At \(k=10\), precision \(=0.6000\) and recall \(=0.3000\); at \(k=20\), precision \(=0.4500\) and recall \(=0.4500\). Precision need not increase; here it decreases.)