From Stakeholder Language to Testable Requirements
A stakeholder may say, “We need to catch almost all serious cases,” “The team cannot review too many alerts,” or “Predictions should usually be close.” These statements communicate important goals, but they do not yet define how model performance will be judged. Words such as “almost all,” “too many,” and “close” can mean different things to different people.
Eliciting success requirements means turning those statements into clear, measurable conditions before evaluating candidate models. The task is not to choose a metric in isolation. It is to confirm what outcome matters, to whom, over what period, and what level of performance is acceptable. This process is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A requirement should let another analyst answer, “Does this result meet the agreed condition?” without guessing what the stakeholder meant. For example, “high recall” is not yet testable. “At least 90% of confirmed urgent cases in the evaluation period must be identified” is testable: it identifies the outcome, population, period, metric, and threshold.
A Structured Elicitation Process
Stakeholder discussions are more productive when broad requests are unpacked systematically. Separate the underlying concern from the proposed metric, and ask follow-up questions where the wording leaves room for multiple interpretations.
Ask what action the prediction will support and who may benefit or be harmed if it is wrong. “Reduce missed urgent cases” and “reduce unnecessary referrals” point to different consequences.
State exactly what counts as a positive event, who is included, and how the outcome is established. A documented event within 30 days is not interchangeable with an event recorded at any later time.
Specify its denominator and unit. A limit of “10% false alarms” might mean 10% of people without the event, 10% of all people screened, or 10% of alerts. Those are different requirements.
Replace “low” or “usually” with an agreed value, and state whether the requirement applies per week, per quarter, or across a defined evaluation set.
Identify which conditions are mandatory and which are preferences. If requirements conflict, the stakeholders responsible for the decision must set a priority or revise a condition; analysts should not silently invent a compromise.
A useful requirement often has the form: “For [population] over [period], [measure with stated denominator] must be at least/at most [threshold], because [decision or consequence].” For numeric predictions, clarify both what “close” means and whether errors in one direction are especially harmful. For classification, distinguish the proportion of actual cases detected from the proportion of flagged cases that are genuine.
Worked Example: Urgent Cases and Weekly Review Capacity
Worked Example: Urgent Cases and Weekly Review Capacity
A hypothetical service team reviews model alerts for urgent cases. The stakeholder says, “We need to catch at least nine out of ten urgent cases, but the team can review no more than 30 alerts each week.” On one evaluation week, there are 20 confirmed urgent cases. Candidate A flags 28 people, including 18 urgent cases. Candidate B flags 36 people, including 19 urgent cases. Translate the request into requirements and assess both candidates.
Method. Convert “nine out of ten urgent cases” into a minimum sensitivity requirement. Sensitivity is the number of actual positive cases identified divided by the number of actual positive cases. Convert the review limit into a maximum alert-count requirement. These are separate conditions: satisfying one does not imply satisfying the other.
There are 20 actual urgent cases. Candidate A identifies 18, so its sensitivity is \(18/20=0.90\), or 90%. It flags 28 people, which is no more than the weekly limit of 30. Candidate A meets both stated requirements.
Candidate B identifies 19 of the 20 urgent cases, so its sensitivity is \(19/20=0.95\), or 95%. However, it flags 36 people, which exceeds the capacity limit by \(36-30=6\) alerts. Candidate B meets the detection requirement but fails the workload requirement.
Conclusion. On these results, Candidate A meets both requirements; Candidate B does not. The conclusion does not claim that Candidate A is better under every consideration. It states only how each candidate performs against the requirements elicited here.
Common mistake avoided. It would be wrong to say that Candidate B is successful just because its sensitivity is higher. The stakeholder also stated a hard capacity limit. The two conditions must be checked separately.
This example also shows why the evaluation period matters. A single week with 20 urgent cases may provide only a limited basis for judging future performance. Stakeholders and analysts should agree on the evaluation population and period, and distinguish an observed result from a guarantee about future weeks.
Worked Example: Making “Close Enough” Measurable
Worked Example: Making “Close Enough” Measurable
A hypothetical repair service wants cost predictions to be “close most of the time,” while avoiding very large underpredictions. Stakeholders agree to assess 20 held-out repairs. They define “close” as an absolute prediction error of at most $200 for at least 90% of repairs. They define a very large underprediction as an actual cost more than $500 above the prediction, and allow no more than 5% of repairs to have such an error. Candidate A is within $200 on 18 repairs and has one very large underprediction. Candidate B is within $200 on 17 repairs and has no very large underprediction. Assess the requirements.
Method. Translate “most of the time” into the agreed proportion within the dollar tolerance, and calculate the permitted count. Separately calculate the maximum permitted count of severe underpredictions. An absolute error requirement alone does not describe which direction an error takes, so the underprediction condition needs its own definition.
At least 90% of 20 repairs must be within $200. The required count is \(0.90\times20=18\). No more than 5% may have a very large underprediction. The maximum count is \(0.05\times20=1\).
Candidate A is within $200 on 18 of 20 repairs, so its proportion is \(18/20=0.90\), or 90%. It has one severe underprediction, giving \(1/20=0.05\), or 5%. It meets both conditions, including each condition at its threshold.
Candidate B is within $200 on 17 of 20 repairs, giving \(17/20=0.85\), or 85%, below the required 90%. It has no severe underpredictions, giving \(0/20=0\%\), so it meets the severe-error condition. Because both conditions are required, Candidate B does not meet the full set.
Conclusion. Candidate A meets both agreed requirements on this evaluation set; Candidate B meets only the severe-underprediction requirement. The word “at most” means an error of exactly $200 counts as within tolerance, and a rate of exactly 5% is permitted.
Common mistake avoided. It would be misleading to summarize the request as “minimize average error.” The stakeholders specified a proportion within a tolerance and a separate limit on large errors in one direction. Neither condition can be inferred from an average alone.
Worked Example: Reconciling Competing Requirements
Worked Example: Reconciling Competing Requirements
A hypothetical fraud-review team screens 1,000 transactions. In the evaluation set, 10 transactions are confirmed fraud and 990 are not. The safety stakeholder requires at least 90% of fraud cases to be flagged. The review manager requires no more than 40 total investigations. A customer representative requires the false-positive rate among non-fraud transactions to be at most 2%. Candidate A flags 34 transactions: 9 fraud and 25 non-fraud. Candidate B flags 23 transactions: 8 fraud and 15 non-fraud. Assess the requirements and identify the unresolved decision.
Method. Calculate each requirement using its stated denominator. The 90% detection condition uses the 10 actual fraud cases. The false-positive rate uses the 990 actual non-fraud transactions. Investigation workload uses all flagged transactions. These denominators cannot be substituted for one another.
For Candidate A, sensitivity is \(9/10=0.90\), or 90%, so it meets the safety threshold. Its total investigation count is 34, which is within the limit of 40. Its false-positive rate is \(25/990\approx0.0253\), or approximately 2.53%, which exceeds the 2% limit.
For Candidate B, sensitivity is \(8/10=0.80\), or 80%, so it fails the safety threshold. Its investigation count is 23, within the limit of 40. Its false-positive rate is \(15/990\approx0.0152\), or approximately 1.52%, so it meets the false-positive-rate condition.
Conclusion. Neither candidate meets all three requirements. Candidate A satisfies safety and workload but fails the false-positive-rate limit. Candidate B satisfies workload and false-positive rate but fails the safety threshold. The data do not establish which candidate is acceptable: the stakeholders must decide whether a requirement can be revised, whether a different operating approach is needed, or whether neither candidate is acceptable.
Common mistake avoided. Do not resolve the disagreement by quietly relaxing the safety threshold or by treating a lower investigation count as more important. A trade-off among stakeholder objectives requires an explicit decision by the people accountable for those objectives.
Why Denominators and Priorities Matter
The definitions used in the examples make clear why a metric must be tied to an outcome and denominator. For a classification problem, the actual positive cases consist of identified positives and missed positives. Therefore, if \(TP\) is the number of true positives and \(FN\) the number of false negatives, the number of actual positive cases is \(TP+FN\), and sensitivity is:
This formula follows directly from the question sensitivity answers: among all actual positive cases, what fraction did the model identify? The denominator must therefore include both identified cases and missed cases. By contrast, the total number of alerts is \(TP+FP\), because every flagged transaction is either a true positive or a false positive. That count is relevant to review workload, not to sensitivity.
A stakeholder’s number may also be a target, a limit, or an aspiration. Ask whether “no more than 40 investigations” is a firm operational constraint or a preferred goal, and whether “at least 90% detection” is mandatory or negotiable. If every condition is mandatory, all must be checked. If some are preferences, the stakeholder needs to state how those preferences should be prioritized. Do not combine unlike goals into a single score without an agreed method and rationale.
The following questions help make requirements precise:
- Outcome: What event or quantity is being predicted, and how is it confirmed?
- Population: Which people, transactions, claims, or other units are included?
- Time: What prediction horizon and evaluation period apply?
- Denominator: Is the rate calculated among actual cases, non-cases, all screened units, or alerts?
- Threshold: What value is acceptable, and does equality count as meeting it?
- Priority: Which requirements are hard constraints, and which represent preferences?
- Accountability: Who is authorized to resolve a conflict or approve a change?
Write down agreed definitions and any unresolved questions. This record reduces the risk that different teams interpret the same result differently, or that a threshold is changed after results are seen. It should also distinguish what was observed on the evaluation data from what stakeholders expect in future use.
Check Your Understanding
For each scenario, translate the stated goal into a measurable conclusion. Treat all counts as observed on the stated evaluation set.
- A screening team evaluates 240 records, including 24 confirmed cases. The requirement is to identify at least 87.5% of confirmed cases, with no more than 35 records flagged. A candidate identifies 21 confirmed cases and flags 35 records. Does it meet both requirements? (Answer: Yes; sensitivity is \(21/24=87.5\%\), and 35 flags meets the maximum of 35.)
- A cost model is assessed on 40 repairs. The stakeholder requires at least 85% of predictions to be within $150 of actual cost. What is the minimum number of repairs that must meet that tolerance? (Answer: 34 repairs, since \(0.85\times40=34\).)
- A service screens 5,000 applications, of which 100 are confirmed positive. A candidate identifies 92 positives, has a 3% false-positive rate among the 4,900 negatives, and flags no more than 230 applications under the workload requirement. Does it meet all three conditions? (Answer: No; it identifies 92/100 = 92% and has \(0.03\times4{,}900=147\) false positives, for 239 total flags, exceeding the limit by 9.)
- Two business segments have 80 and 20 observations, with mean absolute errors of $10 and $50, respectively. Calculate the overall mean absolute error across all 100 observations and the unweighted average of the two segment errors. What does the difference illustrate? (Answer: Overall mean absolute error is \((80\times10+20\times50)/100=\$18\); the unweighted segment average is \((10+50)/2=\$30\). The summaries weight segments differently and answer different questions.)
- In a set of 200 cases, 10 are positive. A model has 8 true positives, 2 false negatives, 180 true negatives, and 10 false positives. A stakeholder says only, “Accuracy must be at least 95%.” Calculate accuracy and sensitivity, then explain what remains unclear. (Answer: Accuracy is \((8+180)/200=94\%\), and sensitivity is \(8/10=80\%\); it fails the stated accuracy threshold, but the statement alone does not establish an acceptable sensitivity or the consequences of missed positives.)