From Ranked Claims to a Review Rule
A claim score can help prioritize investigation, but an investigation team has limited time. The operational question is not merely which model ranks claims well; it is which claims should be sent for review given the available capacity? A review threshold turns scores into a yes-or-no decision, and a confusion matrix shows what that decision would have captured in data with known outcomes.
Suppose a model assigns each claim a score that increases with the model’s estimated concern about that claim. The score might be a probability, but it need not be calibrated as one: a score that only ranks claims can still be used to set a cutoff. For a threshold \(t\), mark a claim for review when its score is at least \(t\). The choice of \(t\) determines how many claims are reviewed and the resulting balance between finding positive cases and avoiding unnecessary investigations.
Two measures describe different aspects of the review list. Sensitivity is the share of all positive claims that the rule flags. Precision is the share of flagged claims that are positive. A team may care about both, but they answer different questions: sensitivity measures how many positive cases are found, while precision measures how concentrated the review list is in positive cases.
Sensitivity is undefined if the evaluation data contain no positive claims. Precision is undefined if the rule flags no claims. In ordinary evaluation, a zero denominator should be reported as undefined, not silently replaced by zero. Capacity is commonly expressed as a maximum number of claims or as a fraction of the eligible portfolio; convert a percentage capacity to a claim count before comparing it with candidate thresholds.
Why Capacity Changes the Threshold Choice
Lowering a threshold can only add claims to the review list when the rule is “score at least \(t\).” The list therefore usually grows as the threshold falls. The added claims may include additional positives, which can raise sensitivity, but they can also include negatives, which can lower precision. The direction of the precision change is not guaranteed: it depends on the outcomes among the claims added.
If capacity is a maximum, a basic decision rule is to choose the threshold that captures as many positive claims as possible without exceeding the review limit. If a decision maker also requires a minimum precision or sensitivity, first exclude candidate rules that violate those constraints. The final choice depends on the stated operational objective; capacity alone does not specify how much a missed positive should matter relative to an unnecessary review.
The result follows directly from the definition. Let \(P_+\) be the total number of positive claims in the fixed evaluation set. For every candidate rule, \(\mathrm{TP}+\mathrm{FN}=P_+\), so sensitivity is \(\mathrm{TP}/P_+\). Since \(P_+\) is constant and positive, a larger TP count gives a larger sensitivity. The capacity condition \(\mathrm{TP}+\mathrm{FP}\leq C\) determines which candidates are feasible. This reasoning does not say that the feasible candidate with the most true positives also has the highest precision: its review list may contain more false positives.
Compare Candidate Thresholds with Confusion Matrices
The following process keeps the operational decision separate from the metric calculations. Candidate thresholds should be evaluated on data for which outcomes are available and which are appropriate for estimating performance. The examples use hypothetical claims and data; they are not company or industry results.
Define what counts as positive, and confirm that larger scores mean greater priority for review. State whether capacity is a maximum or must be filled exactly.
For each threshold, count flagged positives and flagged negatives. Count the unflagged positives and negatives as well.
Calculate the total number flagged. Exclude any candidate whose review count exceeds a maximum capacity.
Calculate sensitivity and precision, then apply the stated objective or constraints. Describe the choice in terms of claims reviewed and observed outcomes.
Worked Example: Use a Maximum Capacity to Choose a Candidate Threshold
A hypothetical validation set contains 40 claims, of which 8 are later confirmed positive. The review team can investigate at most 10 claims. Three candidate thresholds produce the following counts: threshold A flags 5 claims, including 4 positives; threshold B flags 9, including 6 positives; threshold C flags 14, including 7 positives. Choose among the candidates to capture as many positives as possible without exceeding capacity, and calculate sensitivity and precision for the eligible choices.
Method. For each candidate, obtain FP by subtracting TP from the number flagged. Obtain FN by subtracting TP from the 8 total positives. There are \(40-8=32\) negatives, so TN equals 32 minus FP.
For threshold A, \(\mathrm{TP}=4\), \(\mathrm{FP}=5-4=1\), \(\mathrm{FN}=8-4=4\), and \(\mathrm{TN}=32-1=31\). Its review count is \(4+1=5\), sensitivity is \(4/8=0.5000\), and precision is \(4/5=0.8000\).
For threshold B, \(\mathrm{TP}=6\), \(\mathrm{FP}=9-6=3\), \(\mathrm{FN}=8-6=2\), and \(\mathrm{TN}=32-3=29\). Its review count is \(6+3=9\), sensitivity is \(6/8=0.7500\), and precision is \(6/9=0.6667\), rounded to four decimal places.
For threshold C, \(\mathrm{TP}=7\), \(\mathrm{FP}=14-7=7\), \(\mathrm{FN}=8-7=1\), and \(\mathrm{TN}=32-7=25\). Its review count is \(7+7=14\), which exceeds capacity. Its sensitivity would be \(7/8=0.8750\) and its precision \(7/14=0.5000\), but those metrics do not make it an eligible rule under the stated capacity.
Thresholds A and B are feasible. B captures 6 observed positives compared with A’s 4, while using 9 of the 10 available investigation slots. Choose threshold B for the stated objective. It leaves one slot unused because a maximum capacity is not a requirement to fill every slot. This avoids the mistake of selecting C just because it has the highest sensitivity while ignoring the capacity constraint.
Worked Example: Add a Minimum Precision Requirement
A hypothetical portfolio has 100 claims, including 10 positives. The team can review at most 8 claims and requires precision of at least 0.75. Candidate A flags 6 claims, including 4 positives. Candidate B flags 8, including 6 positives. Candidate C flags 11, including 8 positives. Select the eligible candidate with the greatest sensitivity.
For A, \(\mathrm{TP}=4\), \(\mathrm{FP}=6-4=2\), \(\mathrm{FN}=10-4=6\), and \(\mathrm{TN}=90-2=88\). Its review count is \(4+2=6\), its sensitivity is \(4/10=0.4000\), and its precision is \(4/6=0.6667\). It meets capacity but fails the precision requirement.
For B, \(\mathrm{TP}=6\), \(\mathrm{FP}=8-6=2\), \(\mathrm{FN}=10-6=4\), and \(\mathrm{TN}=90-2=88\). Its review count is \(6+2=8\), its sensitivity is \(6/10=0.6000\), and its precision is \(6/8=0.7500\). It meets both requirements.
For C, \(\mathrm{TP}=8\), \(\mathrm{FP}=11-8=3\), \(\mathrm{FN}=10-8=2\), and \(\mathrm{TN}=90-3=87\). Its sensitivity is \(8/10=0.8000\), but its review count of 11 exceeds the capacity of 8. Its precision is \(8/11=0.7273\), which also falls below the requirement.
Only B satisfies both constraints, so select B. Its sensitivity is 0.6000 and its precision is 0.7500. The check is that its confusion-matrix counts sum to \(6+2+4+88=100\), the portfolio size. The common mistake is to choose C based on sensitivity alone; it violates both operational conditions.
Thresholds, Exact Slots, and Ties
A threshold rule and an instruction to review exactly \(C\) claims are not always equivalent. If the scores are distinct, reviewing the \(C\) highest-scoring claims can be implemented by placing a cutoff between the \(C\)th and \((C+1)\)th scores. If several claims share the cutoff score, the rule “score at least \(t\)” may flag more than \(C\). The data do not provide a unique ordering among claims with the same score.
Worked Example: Respect a Capacity at a Tied Score
A hypothetical review unit has capacity for exactly 4 claims. Among 10 claims, two higher-scoring claims consist of one positive and one negative. Three more claims share the next score; their observed outcomes are two positives and one negative. The remaining five claims are below that score, and three of them are positive. A simple score threshold that includes the tied group would flag 5 claims, exceeding capacity. If the team randomly selects 2 of the 3 tied claims to fill the remaining slots, calculate expected TP, FP, sensitivity, and precision.
The total number of positives is \(1+2+3=6\). The two higher-scoring claims contribute 1 TP and 1 FP. Among the three tied claims, the probability that a randomly selected claim is positive is \(2/3\). Selecting 2 claims therefore gives an expected \(2(2/3)=4/3\) positive claims and an expected \(2-4/3=2/3\) negative claims.
Adding the higher-scoring claims gives expected \(\mathrm{TP}=1+4/3=7/3\) and expected \(\mathrm{FP}=1+2/3=5/3\). Since 4 claims are reviewed, expected precision is \(E[\mathrm{TP}]/4=(7/3)/4=7/12=0.5833\). Expected sensitivity is \(E[\mathrm{TP}]/6=(7/3)/6=7/18=0.3889\), rounded to four decimal places.
These are expected metrics over the random tie-breaking procedure, not the result of a particular draw. The rule remains within capacity because it selects exactly 2 of the 3 tied claims. A deterministic secondary ranking rule could also fill the slots, but its performance should be evaluated using that same rule. The common mistake is to include the entire tied group and report a five-claim list as if it respected capacity.
Interpret the Choice Carefully
A confusion matrix is conditional on a particular evaluation set and a particular rule for ties. It supports a transparent comparison, but it does not guarantee that the next set of claims will have the same positive rate or score distribution. In particular, precision depends on the prevalence of positives in the evaluated group. If the positive rate changes, precision may change even when sensitivity and specificity remain similar.
Before operational use, make the review target explicit, state whether capacity is a maximum or an exact number, and document how ties are resolved. Report the count flagged alongside sensitivity and precision; the metrics alone can conceal whether a rule is practical. If a requirement is given, such as “at least 70% sensitivity” or “precision of at least 0.80,” check it directly rather than assuming a threshold with a favorable value for another metric will satisfy it.
For example, if a rule flags no claims, it uses no capacity but has undefined precision, and its sensitivity is zero when the evaluation set contains positives. If a rule flags every claim, its sensitivity is 1 when positives are present, but its precision equals the positive rate in the set. Neither extreme is automatically useful. A defensible choice comes from the stated objective and constraints, not from maximizing a single metric without context.
Check Your Understanding
For each hypothetical case, use the stated outcome and capacity. Round probabilities to four decimal places where needed.
- A set contains 50 claims, including 10 positives. A candidate threshold flags 8 claims, of which 6 are positive. Find FP, FN, TN, sensitivity, and precision. (Answer: FP = 2, FN = 4, TN = 38, sensitivity = 0.6000, precision = 0.7500.)
- A validation set has 10 positive claims. Three candidate rules flag, respectively, 5 claims with 4 positives, 7 claims with 5 positives, and 9 claims with 7 positives. Capacity is at most 7 claims, and the objective is to maximize sensitivity. Which rule is selected, and what are its sensitivity and precision? (Answer: Select the rule flagging 7 claims; sensitivity = 0.5000 and precision = 0.7143.)
- A set contains 80 claims, including 15 positives. A candidate flags 12 claims, including 9 positives. Find all four confusion-matrix counts and both metrics. (Answer: TP = 9, FP = 3, FN = 6, TN = 62, sensitivity = 0.6000, precision = 0.7500.)
- A review list has capacity for 4 claims. Two higher-scoring claims contain one positive and one negative. Three claims tied at the next score contain one positive and two negatives; select 2 of these 3 uniformly at random. There are 6 positives in the full set. Find expected TP and the expected sensitivity and precision. (Answer: Expected TP = 1 + 2(1/3) = 5/3; expected sensitivity = 5/18 = 0.2778; expected precision = (5/3)/4 = 5/12 = 0.4167.)
- A set contains 20 positives. Candidate A flags 8 claims, including 7 positives; candidate B flags 10, including 8; candidate C flags 12, including 9. Capacity is at most 12, and minimum precision is 0.80. Which candidate has the greatest sensitivity among those meeting both conditions? (Answer: Candidate B; its precision is 8/10 = 0.8000 and sensitivity is 8/20 = 0.4000. Candidate A has precision 7/8 = 0.8750 and sensitivity 0.3500; C has precision 9/12 = 0.7500 and fails the precision requirement.)