Why Operational Constraints Change Metric Selection
A model can have strong predictive performance and still be unusable. A screening team may be able to review only a fixed number of cases each day. A real-time system may need a prediction before a request times out. A regulated process may require a result that can be explained using a small set of factors. In each situation, model success depends on both predictive quality and whether the model can operate as intended.
The practical implication is important: do not rank models by a performance metric before checking whether they meet mandatory operating limits. First identify which candidates are usable. Then compare the usable candidates on the performance objective that matters for the decision. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A constraint should be stated in a form that can be checked. “Fast enough” is not a testable requirement; “the 95th-percentile response time must be no more than 100 milliseconds” is. “Explainable” also needs an operational definition, such as requiring a specified set of inputs, a limited number of factors, or documented reasons for individual decisions. Such proxies do not capture every aspect of interpretability, but they make the chosen requirement assessable.
It is useful to distinguish a hard constraint from a preference. A hard constraint rules a candidate out if it is violated. A preference helps rank candidates that remain eligible. For example, a limit of 100 manual reviews per day can be hard, while a preference for fewer reviews can help distinguish two systems that both meet that limit.
A Constrained Selection Rule
Suppose a set of candidate models is being evaluated. Let \(M_j\) be the performance measure for candidate \(j\), and let the required operational conditions be \(C_1,\ldots,C_K\). The feasible set contains only candidates that meet every required condition. If larger values of \(M_j\) are better, select the feasible candidate with the largest value; if smaller values are better, select the feasible candidate with the smallest value.
For a measure that should be minimized, replace the maximum with a minimum. The reasoning is a direct consequence of the requirement: a candidate outside \(\mathcal{F}\) cannot be deployed under the stated rules, regardless of its score. The comparison therefore has to be made over \(\mathcal{F}\), not over all candidates. If \(\mathcal{F}\) is empty, no candidate meets the requirements; the appropriate conclusion is to revisit the model options or the requirements, rather than to quietly ignore a failed constraint.
Specify what action a prediction supports, the required capacity or service level, and which conditions are mandatory.
For example, choose a measure of detected cases when the aim is to find positives, or an error measure when the aim is to predict a numeric quantity.
Apply every limit consistently to each candidate using the same evaluation data and definitions.
State the selected performance measure, the constraints checked, and any relevant limitations.
Worked Examples
Worked Example: Fixed Review Capacity
A hypothetical claims team has 1,000 claims to screen and can manually investigate at most 120. Of the claims, 80 are later confirmed to have a serious issue. Two candidate ranking systems each send their top 120 claims for investigation. System A finds 48 serious cases and includes 72 claims without the issue. System B finds 44 serious cases and includes 76 claims without the issue. If the goal is to find as many serious cases as possible without exceeding capacity, which system is preferred?
Method. Treat the review limit as a hard constraint, then compare the number of confirmed serious cases captured within the allowed 120 reviews. Both systems meet capacity exactly, so both are feasible.
For System A, the number of cases reviewed is \(48+72=120\). Its sensitivity, the fraction of all 80 serious cases found, is \(48/80=0.6000\). Its precision, the fraction of reviewed claims that are serious, is \(48/120=0.4000\).
For System B, the number reviewed is \(44+76=120\). Its sensitivity is \(44/80=0.5500\), and its precision is \(44/120=0.3667\) to four decimal places. Because the operational goal is to find as many serious cases as possible at the fixed capacity, System A is preferred: it finds 48 rather than 44.
Check. Both systems send exactly 120 claims for review, and System A captures four more of the 80 serious cases. The comparison respects the capacity limit.
Common mistake avoided. It would be wrong to select a system using a threshold that sends more than 120 claims to the team just because that threshold has higher sensitivity. The capacity is a hard limit; a performance gain outside the feasible operating range cannot be used as the selection basis.
Worked Example: Joint False-Positive and Response-Time Limits
A hypothetical alert service evaluates 400 cases: 40 have the target event and 360 do not. The operating rules require a false-positive rate no greater than 0.0500 and a 95th-percentile response time no greater than 100 milliseconds. Candidate A produces 30 true positives and 18 false positives, with a 95th-percentile response time of 80 milliseconds. Candidate B produces 32 true positives and 22 false positives, with response time 60 milliseconds. Candidate C produces 28 true positives and 9 false positives, with response time 130 milliseconds. Choose the feasible candidate with the greatest sensitivity.
Method. Calculate the false-positive rate using the 360 cases without the event, check both limits, and only then compare sensitivity among candidates that pass.
For A, the false-positive rate is \(18/360=0.0500\), and its response time is \(80\) milliseconds. It satisfies both limits. Its sensitivity is \(30/40=0.7500\).
For B, the false-positive rate is \(22/360=0.0611\) to four decimal places. Although its response time of \(60\) milliseconds meets the time limit and its sensitivity is \(32/40=0.8000\), it violates the false-positive limit.
For C, the false-positive rate is \(9/360=0.0250\), but its response time of \(130\) milliseconds exceeds the maximum. Its sensitivity is \(28/40=0.7000\), but it is infeasible.
Candidate A is the only candidate meeting both constraints, so A is selected. Its sensitivity does not exceed the unconstrained sensitivity of B, but B cannot be used under the stated error-rate rule.
Check. The denominators are the 360 non-event cases for false-positive rate and the 40 event cases for sensitivity. Candidate A meets both limits; B fails the error-rate limit, and C fails the response-time limit.
Common mistake avoided. Dividing B’s 22 false positives by all 400 cases gives 0.0550, but that is not the false-positive rate. The false-positive rate is conditional on the cases without the event, so its denominator is 360.
Worked Example: Interpretability and Numeric Prediction Error
A hypothetical claims operation predicts the dollar amount of a repair. Its rules require mean absolute error (MAE) no greater than $350, 95th-percentile response time no greater than 1.0 second, and no more than six input factors in the deployed prediction. The three candidates have the following absolute errors on the same five validation claims:
- Simple model: $100, $200, $300, $400, and $600; response time 0.3 seconds; five input factors.
- Complex model: $80, $120, $200, $250, and $350; response time 1.2 seconds; 24 input factors.
- Compact model: $150, $250, $300, $350, and $450; response time 0.7 seconds; six input factors.
Which candidate is feasible, and which feasible candidate has the lowest MAE?
Method. Compute MAE as the sum of absolute errors divided by the number of claims. Apply all three limits to each candidate, then minimize MAE among candidates that pass.
For the simple model, the error total is \(100+200+300+400+600=1{,}600\) dollars. Its MAE is \(1{,}600/5=\$320\). Its MAE is within the $350 limit, its 0.3-second response time is within the 1.0-second limit, and its five inputs are within the six-factor limit. It is feasible.
For the complex model, the total error is \(80+120+200+250+350=1{,}000\) dollars, giving MAE \(1{,}000/5=\$200\). It meets the error limit, but 1.2 seconds exceeds the response-time limit and 24 inputs exceed the six-factor limit. It is infeasible.
For the compact model, the total error is \(150+250+300+350+450=1{,}500\) dollars, giving MAE \(1{,}500/5=\$300\). Its 0.7-second response time and six input factors meet their limits, so it is feasible. Comparing the feasible MAEs, \(\$300<\$320\), so the compact model is selected.
Check. The complex model has the lowest MAE overall, but it violates two mandatory conditions. Among the two feasible models, the compact model has the lower MAE.
Common mistake avoided. Selecting the complex model solely because its MAE is lowest ignores the operating limits. Also, the six-factor rule is a specified operational proxy for interpretability in this scenario; it should not be presented as a universal definition of interpretability.
Applying Constraints Carefully
Operational limits need to be defined on a scale and population that match actual use. A review-capacity limit should specify the time period and whether capacity refers to cases sent for review or cases ultimately completed. A response-time limit should specify the statistic being limited: for example, an average can conceal a long tail, whereas a high percentile describes slower requests. Error-rate limits should specify the relevant denominator and the evaluation population.
Constraints can also conflict. A stricter false-positive limit may reduce the number of true cases detected; a smaller model may be easier to review but predict less accurately. Do not conceal such trade-offs in a single score unless the decision rules support that combination. If requirements are expressed as hard limits, show which candidates pass and which fail. If several candidates pass, compare them using the previously stated performance objective.
| Operational concern | Example of a checkable constraint | Question to verify |
|---|---|---|
| Manual review capacity | No more than 120 cases sent per day | Does the evaluated threshold stay within the daily limit? |
| Response time | 95th-percentile response time at most 100 milliseconds | Is the same latency statistic used for every candidate? |
| Allowable error rate | False-positive rate at most 0.0500 | Is the denominator the correct population for that rate? |
| Interpretability | No more than six specified input factors | Does the operational rule match what users need to understand? |
A measured constraint is not automatically a permanent guarantee. Capacity, response time, and error rates may change when the volume or mix of cases changes. The evaluation should therefore use data and definitions relevant to the intended operating conditions, and the reported conclusion should make clear which conditions were checked. If no candidate passes, report that result directly rather than choosing an infeasible candidate as though all requirements were met.
Check Your Understanding
For each situation, apply the operational limits before comparing performance. Probabilities are reported to four decimal places.
- A review team can handle 50 of 500 cases, of which 25 are positive. At that capacity, system P finds 15 positives and system Q finds 12. Calculate each system’s sensitivity and precision, then identify the one that finds more positives within capacity. (Answer: P has sensitivity \(15/25=0.6000\) and precision \(15/50=0.3000\); Q has sensitivity \(12/25=0.4800\) and precision \(12/50=0.2400\); P finds more positives.)
- Among 200 cases, 20 are positive and 180 are negative. A system must have false-positive rate at most 0.0500 and 95th-percentile response time at most 100 milliseconds. System X has 15 true positives, 9 false positives, and 90-millisecond response time. System Y has 17 true positives, 12 false positives, and 70-millisecond response time. Which system is feasible? (Answer: X is feasible: \(9/180=0.0500\) and 90 milliseconds meets the limit. Y is infeasible because \(12/180=0.0667\) exceeds the false-positive limit.)
- A numeric model has absolute errors of $100, $200, $300, and $400 on four validation cases. The allowable MAE is $300. What is its MAE, and does it meet the error constraint? (Answer: \((100+200+300+400)/4=\$250\); it meets the constraint.)
- Two feasible classifiers are being compared on 100 positive cases under a fixed review limit. Model R detects 62 positives, while model S detects 58. If the stated objective is to maximize sensitivity, which is preferred, and what are their sensitivities? (Answer: R is preferred; its sensitivity is \(62/100=0.6200\), compared with \(58/100=0.5800\) for S.)
- A hypothetical numeric-prediction service requires MAE at most $200, response time at most 0.5 seconds, and no more than four input factors. Model L has MAE $180, response time 0.4 seconds, and four factors. Model M has MAE $150, response time 0.6 seconds, and three factors. Which model is selected if MAE is minimized among feasible models? (Answer: L; it meets all three constraints. M has lower MAE but is infeasible because its response time exceeds 0.5 seconds.)