Capacity-Constrained Threshold Selection

Why Capacity Changes the Threshold Decision

A decision rule can be limited by more than the relative costs of false positives and false negatives. A hypothetical screening program may be able to offer a follow-up assessment to only 40 people, even if its model identifies 75 people whose predicted risks exceed a cost-based threshold. In that setting, the decision maker must account for a fixed operational capacity as well as the model’s ranking of cases.

The central distinction is that cost-based thresholding asks which cases are worthwhile to select, while capacity-constrained selection asks which cases should be selected when only a specified number can be chosen. The two rules can agree, but they answer different questions and need not produce the same set of decisions.

Definition: Let $Y=1$ denote that the target event occurs and $Y=0$ that it does not. For case $j$, let $p_j=P(Y_j=1\mid\text{available information})$. A capacity-constrained rule selects a specified number $K$ of cases, or no more than $K$, from the eligible population. If larger scores indicate larger event probabilities, selecting the $K$ largest scores is called top-$K$ selection.

In this tutorial, selecting a case means classifying it as positive or assigning it an intervention. A selected case with $Y=0$ is a false positive; a case not selected with $Y=1$ is a false negative. We assume that the incremental cost of a false positive is $C_{FP}>0$, the incremental cost of a false negative is $C_{FN}>0$, and correct decisions have zero incremental cost. These costs are the same for every case unless stated otherwise.

Expected Cost for One Case

If a case has event probability $p$, selecting it produces expected cost $(1-p)C_{FP}$: a false positive occurs with probability $1-p$. Not selecting it produces expected cost $pC_{FN}$: a false negative occurs with probability $p$. These are conditional expected costs based on the case’s probability, not certain costs for an individual whose outcome is still unknown.

L_{\mathrm{select}}=(1-p)C_{FP},\qquad L_{\mathrm{not\ select}}=pC_{FN}

The expected cost change from selecting rather than not selecting is selected cost minus unselected cost. A negative change means selection reduces expected cost. This sign convention matters: the reduction in cost has the opposite sign.

\Delta L =L_{\mathrm{select}}-L_{\mathrm{not\ select}} =(1-p)C_{FP}-pC_{FN} =C_{FP}-(C_{FP}+C_{FN})p

With no capacity restriction and equal costs across cases, selection is preferred when $\Delta L<0$, or equivalently when the probability is above the cost-based threshold. At equality the two actions have the same expected cost.

p>p_{\mathrm{cost}} \quad\text{where}\quad p_{\mathrm{cost}}=\frac{C_{FP}}{C_{FP}+C_{FN}}

This threshold is the probability at which selecting and not selecting have equal expected cost. It is not automatically the cutoff for selecting exactly $K$ cases. A capacity cutoff is determined by how many cases can be accommodated and by their relative scores; the cost threshold is determined by the error costs.

Why the Highest Probabilities Are Selected

Suppose a fixed number $K$ must be selected and the error costs are constant across cases. For a case with probability $p$, selecting rather than not selecting changes expected cost by $C_{FP}-(C_{FP}+C_{FN})p$. This expression decreases as $p$ increases. Therefore, if one selected case has a lower probability than an unselected case, swapping their decisions cannot increase expected cost.

p_a>p_b \quad\Longrightarrow\quad \Delta L_a-\Delta L_b =-(C_{FP}+C_{FN})(p_a-p_b)<0

Thus selecting case $a$ instead of case $b$ has the more favorable expected cost change. Repeating such swaps leads to the $K$ largest probabilities. The same conclusion follows by considering expected cost reduction: the reduction from selecting rather than not selecting is $(C_{FP}+C_{FN})p-C_{FP}$, which increases with $p$. Do not confuse this cost reduction with the cost change, which decreases with $p$.

Result: If each case has a calibrated event probability, the number to select is fixed, and $C_{FP}$ and $C_{FN}$ are identical across cases, selecting the $K$ largest probabilities minimizes total expected classification cost among all selections of size $K$. If the score only ranks cases, the same top-$K$ selection is justified when the score orders cases in the same way as their event probabilities; the score itself need not be a probability.

This result assumes exactly $K$ cases must be selected. If the rule instead permits selecting up to $K$, the decision maker should not fill unused capacity with cases for which selection increases expected cost. Under the same equal-cost assumptions, select cases above the cost threshold, subject to the capacity limit; cases at the threshold are cost-indifferent. The operational rule is then to select the worthwhile cases with the largest probabilities, up to $K$.

Worked Examples

Worked Example: Selecting Exactly Three Cases

A hypothetical program can offer exactly three follow-up assessments to eight applicants. Their calibrated event probabilities are $0.10, 0.25, 0.35, 0.50, 0.65, 0.70, 0.80,$ and $0.90$. The incremental false-positive cost is $C_{FP}=\$100$, and the false-negative cost is $C_{FN}=\$300$. Determine the optimal selection and its total expected cost. Then compare it with selecting three lower-probability cases.

Method. Because exactly three cases must be selected and both error costs are constant across applicants, rank the probabilities and select the three largest. For a selected case with probability $p$, calculate $(1-p)100$; for an unselected case, calculate $300p$.

The three largest probabilities are $0.90, 0.80,$ and $0.70$. Their expected costs are $100(0.10)=\$10$, $100(0.20)=\$20$, and $100(0.30)=\$30$, respectively. The five unselected probabilities are $0.65,0.50,0.35,0.25,$ and $0.10$; their expected costs are $300(0.65)=\$195$, $300(0.50)=\$150$, $300(0.35)=\$105$, $300(0.25)=\$75$, and $300(0.10)=\$30$.

Adding the selected cases’ expected costs gives $\$10+\$20+\$30=\$60$. Adding the unselected cases’ expected costs gives $\$195+\$150+\$105+\$75+\$30=\$555$. The total expected cost is therefore $\$60+\$555=\$615$.

For comparison, suppose the program instead selects probabilities $0.65,0.50,$ and $0.35$. Their selected-case costs are $100(0.35)=\$35$, $100(0.50)=\$50$, and $100(0.65)=\$65$, totaling $\$150$. The unselected probabilities $0.90,0.80,0.70,0.25,$ and $0.10$ have costs $\$270,\$240,\$210,\$75,$ and $\$30$, totaling $\$825$. This alternative total is $\$150+\$825=\$975$, which exceeds $\$615$.

Check. The selected set contains exactly three cases, and replacing lower-probability selected cases with higher-probability unselected cases lowers expected cost. The common mistake avoided is ranking cases in the wrong direction or comparing only selected-case costs while ignoring the costs of unselected cases.

Worked Example: A Binding Capacity Versus the Cost Threshold

A hypothetical outreach team has five eligible members with calibrated event probabilities $0.15,0.30,0.45,0.60,$ and $0.75$. It can contact at most two members. The error costs are $C_{FP}=\$100$ and $C_{FN}=\$300$. Find the cost threshold, apply the capacity rule, and calculate the expected cost under both the capacity-limited rule and the unrestricted cost-based rule.

Method. First calculate the cost threshold. Then identify whether it would select more cases than the two-person capacity permits. For each policy, sum the appropriate expected cost for every case.

The cost threshold is $100/(100+300)=0.25$. Without a capacity limit, probabilities above $0.25$ are selected: $0.30,0.45,0.60,$ and $0.75$. This is four cases, so the capacity binds. Under the capacity rule, the two largest probabilities, $0.75$ and $0.60$, are selected.

For the capacity-limited rule, the selected cases have expected costs $100(1-0.75)=\$25$ and $100(1-0.60)=\$40$. The unselected probabilities $0.45,0.30,$ and $0.15$ have expected costs $300(0.45)=\$135$, $300(0.30)=\$90$, and $300(0.15)=\$45$. The total is $\$25+\$40+\$135+\$90+\$45=\$335$.

For the unrestricted cost-based rule, probabilities $0.30,0.45,0.60,$ and $0.75$ are selected. Their expected costs are $100(0.70)=\$70$, $100(0.55)=\$55$, $100(0.40)=\$40$, and $100(0.25)=\$25$. The $0.15$ case is not selected and has expected cost $300(0.15)=\$45$. The unrestricted total is $\$70+\$55+\$40+\$25+\$45=\$235$.

The capacity-limited expected cost is $\$335-\$235=\$100$ higher. That difference arises because the two cases at $0.30$ and $0.45$ cannot be contacted; their expected costs increase by $\$90-\$70=\$20$ and $\$135-\$55=\$80$, respectively, when left unselected.

Check. The capacity-limited policy selects exactly two cases and has the larger expected cost, as it is restricted relative to the unrestricted cost-minimizing policy. The common mistake avoided is treating the cost threshold $0.25$ as if it automatically satisfied the two-case capacity.

Worked Example: A Tie at the Capacity Boundary

A hypothetical review unit can select exactly two of six cases. Their calibrated probabilities are $0.80,0.65,0.65,0.40,0.20,$ and $0.10$. Let $C_{FP}=\$80$ and $C_{FN}=\$320$. Identify the capacity-based selection, discuss the tie, and compare its expected cost with the unrestricted cost-based rule.

Method. Rank the probabilities for the fixed-capacity rule, and calculate the cost threshold separately. If two cases tie at the selection boundary, either tied case has the same expected cost under the stated assumptions.

The capacity rule selects the $0.80$ case and one of the two cases with probability $0.65$. Either tied case can be chosen: both have the same expected cost and are interchangeable under equal costs. For the selected cases, expected costs are $80(1-0.80)=\$16$ and $80(1-0.65)=\$28$.

The unselected probabilities are $0.65,0.40,0.20,$ and $0.10$. Their expected costs are $320(0.65)=\$208$, $320(0.40)=\$128$, $320(0.20)=\$64$, and $320(0.10)=\$32$. Thus the fixed-capacity total expected cost is $\$16+\$28+\$208+\$128+\$64+\$32=\$476$.

The cost threshold is $80/(80+320)=0.20$. With no capacity restriction, the cases above $0.20$ are selected: $0.80,0.65,0.65,$ and $0.40$. Their expected costs total $16+28+28+48=\$120$. The $0.20$ case is indifferent because selecting and not selecting it both cost $80(0.80)=\$64$ and $320(0.20)=\$64$. The $0.10$ case is not selected and costs $320(0.10)=\$32$. Total expected cost is $\$120+\$64+\$32=\$216$.

Check. The capacity rule selects two cases, while the unrestricted rule selects four cases with probabilities above the cost threshold. The common mistake avoided is assuming that a threshold rule can always select an exact number of cases: tied scores may require a tie-breaking rule, and a probability threshold can select more cases than capacity allows.

Choosing and Reporting a Capacity Rule

When predicted probabilities are available and the number of available interventions is known, top-$K$ selection is transparent: sort eligible cases from highest to lowest probability and choose the first $K$. If the model outputs a score that is not a probability, the score may still be used for this ranking if higher scores reliably indicate greater risk. However, a cost threshold stated in probability units cannot be applied directly to an uncalibrated score. The score must first be mapped to probabilities or otherwise given a valid cost interpretation.

In practice, a threshold is often reported alongside a capacity rule. For example, a policy might select the top 200 cases and report the score of the 200th-ranked case as the operational cutoff. That cutoff can change when the eligible population or available capacity changes, even when the model and its cost assumptions do not. Conversely, a cost-based probability threshold can remain the same while the number selected changes with the population’s probability distribution.

If multiple cases have the same score at the capacity boundary, a simple threshold may select too many or too few cases. The decision maker must define how to handle the tie, such as using a prespecified secondary criterion or selecting randomly among tied cases. If probabilities and costs are identical for those tied cases, any choice among them has the same expected cost. If their costs or other relevant decision consequences differ, the tie need not be economically neutral.

The top-$K$ result also depends on equal error costs across cases. If case-specific costs vary, ranking by probability alone may not minimize expected cost. For case $j$, the expected cost change from selecting rather than not selecting is $C_{FP,j}-(C_{FP,j}+C_{FN,j})p_j$. With a fixed number to select, cases should instead be compared using this case-specific change, selecting the most favorable changes. Capacity may also be a maximum rather than a requirement to fill every available slot; in that setting, an available slot should not be used when doing so increases expected cost.

Key takeaway: A fixed capacity creates a ranking problem: under equal case-level costs, select the cases with the largest probabilities. A cost threshold creates an individual decision rule: select when the probability exceeds $C_{FP}/(C_{FP}+C_{FN})$. Check separately whether the cost-based selections fit the available capacity, and specify how ties and unused capacity are handled.

Check Your Understanding

For each problem, distinguish the capacity rule from the cost rule where relevant. Assume equal costs across cases unless the question states otherwise.

A program must select exactly two of four cases with probabilities $0.12,0.38,0.55,$ and $0.81$. Which probabilities are selected? (Answer: $0.81$ and $0.55$.)
False-positive and false-negative costs are $\$90$ and $\$210$. What is the cost threshold, and what is the decision at a probability exactly equal to that threshold? (Answer: $0.3000$; the two actions have equal expected cost.)
A unit must select exactly three cases with probabilities $0.20,0.50,0.50,0.70,$ and $0.90$. Which cases are selected, accounting for the tie? (Answer: $0.90$, $0.70$, and either one of the two $0.50$ cases.)
False-positive cost is $\$150$ and false-negative cost is $\$450$. For probabilities $0.10,0.25,0.40,$ and $0.65$, which cases are selected by the unrestricted cost rule, and how should the $0.25$ case be treated? (Answer: Select $0.40$ and $0.65$; the $0.25$ case is cost-indifferent.)
A program may select at most one of three cases with probabilities $0.20,0.40,$ and $0.60$. False-positive and false-negative costs are $\$200$ and $\$300$. Which case should be selected, and what is the expected cost change from selecting the $0.40$ case rather than not selecting it? (Answer: Select the $0.60$ case; the $0.40$ case has expected cost change $\$200-500(0.40)=\$0$.)

This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

What You'll Learn