Writing Limitations That Change the Decision

Why Some Limitations Matter More Than Others

A model report can list many caveats and still fail to help a decision-maker. The useful question is not simply whether a limitation exists. It is whether the limitation could change the recommendation, the people or situations to which it applies, or the conditions needed for safe use. A decision-relevant limitation has a clear connection to an action.

For example, “the model was developed using historical data” is too general on its own. If the historical data came from a different operating setting, explain what could change: perhaps the predicted risk is too high or too low for the intended users. Then connect that possibility to the decision. Would the recommended intervention still be worthwhile? Should use be restricted to a narrower group until the difference is checked?

This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus. It focuses on selecting and explaining material limitations, not on producing an exhaustive inventory of every possible source of error.

Definition: A decision-relevant limitation is a constraint or uncertainty in the data, model, evaluation, or intended use that could change a recommendation, its expected consequences, or the conditions under which the recommendation is safe to implement. A limitation is material when a plausible effect of that limitation could make the decision-maker choose differently.

A Practical Test for Materiality

Start from the proposed decision, not from a generic list of model weaknesses. State the action being considered and the evidence supporting it. Then ask what a limitation could change: the estimated probability, the estimated benefit, the number of people affected, or whether the model is being used in the setting for which it was assessed.

Name the decision.
Be specific about the action, such as contacting a customer, inspecting equipment, or expanding a service. “Use the model” is usually too vague.

Identify the vulnerable evidence.
State which estimate or assumption the action relies on. For example, the recommendation may depend on predicted risk being above a threshold or estimated benefit exceeding cost.

Show the consequence.
Use a calculation, range, or clear comparison to show whether a plausible change could reverse the decision or restrict where it applies.

State the appropriate response.
Explain whether the decision should be delayed, narrowed, treated as provisional, or supported by further evidence. Do not imply that a caveat automatically makes a model unusable.

A short sensitivity calculation is often more informative than a long caveat. It shows the decision boundary: the point at which the preferred action changes. If plausible values on either side of a limitation lead to different actions, the limitation deserves prominent attention. If the decision is unchanged over a reasonable range, the caveat may still be recorded, but it need not dominate the recommendation.

Worked Examples

Worked Example: A Risk Estimate Near an Intervention Threshold

A hypothetical equipment service is considering an inspection call for a machine predicted to have a 30-day failure probability of 0.18. An inspection call costs $300. If the machine fails without an inspection, the service estimates a loss of $2,000. The model was evaluated mainly on older machines, while the proposed rollout includes a newer machine type. A preliminary review suggests the newer type’s failure probability might be as low as 0.12. Should the report recommend an inspection for every machine with a predicted probability of 0.18?

Method. Compare the expected cost of inspection with the expected failure loss if no inspection is made. This comparison makes the decision threshold explicit.

Step 1: Calculate the threshold. If the failure probability is $p$, the expected loss without inspection is $2{,}000p$ dollars. Inspection is preferred when that amount exceeds $300:

2{,}000p > 300 \quad\Longleftrightarrow\quad p > \frac{300}{2{,}000}=0.15.

Step 2: Apply the model estimate. At $p=0.18$, the expected loss without inspection is $2{,}000(0.18)=\$360$. Inspection costs $300, so its expected cost advantage is $360-300=\$60$. On this estimate, inspection is preferred.

Step 3: Test the limitation’s effect. At the preliminary lower probability $p=0.12$, the expected loss without inspection is $2{,}000(0.12)=\$240$. That is $60 less than the $300 inspection cost, so inspection is no longer preferred.

Conclusion. The difference between older and newer machines is material because a plausible probability of 0.12 falls below the 0.15 decision threshold. A suitable recommendation would not treat 0.18 as sufficient evidence for an unconditional rollout to the newer machines. The report could recommend inspection for the assessed population while making use on the newer type provisional pending evidence that its risk estimates support the same decision.

Common mistake and correction. The incorrect reasoning is to report only that the model estimate exceeds the threshold, $0.18>0.15$, and recommend inspection for all machines. That calculation ignores the stated limitation. The correction is to test the plausible alternative estimate: $0.12<0.15$. Because that changes the preferred action, the population difference must qualify the recommendation.

Worked Example: A Limitation That Could Reverse Expected Value

A hypothetical service team plans to contact 250 customers selected by a model. A historical analysis estimates that contact produces an incremental successful response for 0.14 of contacted customers. Each incremental success is worth $180, and each contact costs $22. The estimate comes from a non-randomized comparison, so the report notes that the incremental response rate could be only 0.08. Is this caveat material to the proposed campaign?

Method. Calculate expected net value per contact at both rates, then multiply by the planned number of contacts. The word “incremental” matters: the calculation concerns additional responses attributable to contact, not all responses among contacted customers.

Step 1: Calculate the break-even rate. Let $r$ be the incremental response rate. Expected value per contact before contact cost is $180r$. Contact is expected to have positive net value when $180r>22$:

r > \frac{22}{180}=0.1222\ldots

Thus, the break-even rate is approximately 0.1222, or 12.22%.

Step 2: Calculate value at the reported estimate. At $r=0.14$, expected value per contact is $180(0.14)=\$25.20$. After the $22 cost, expected net value is $25.20-22=\$3.20$ per contact. For 250 contacts, expected net value is $250(3.20)=\$800$.

Step 3: Calculate value at the plausible lower rate. At $r=0.08$, expected value per contact is $180(0.08)=\$14.40$. After cost, expected net value is $14.40-22=-\$7.60$ per contact. For 250 contacts, expected net value is $250(-7.60)=-\$1{,}900$.

Conclusion. The limitation is material. The reported estimate implies an $800 expected net benefit, while the plausible lower rate implies an expected net loss of $1,900. The report should explain that the recommendation depends on the incremental response estimate and could propose a limited test before committing to all 250 contacts.

Common mistake and correction. A common error is to multiply the 14% response rate by $180, subtract $22, and present the resulting $3.20 per contact as a dependable benefit. This treats an estimate from a non-randomized comparison as certain and ignores the plausible 8% rate. The correction is to show both calculations and make clear that the decision changes below the 12.22% break-even rate.

Worked Example: Incomplete Outcomes and an Unsettled Performance Decision

A hypothetical organization is considering wider use of a binary classifier. Its internal policy is to proceed only if at least 80% of cases are correctly classified in an evaluation of 100 cases. Outcomes are finalized for 80 cases, of which 68 were classified correctly. Outcomes for the remaining 20 cases are still pending. Should the report state that the policy requirement has been met?

Method. Keep the denominator aligned with the decision. The 68 correct classifications are confirmed, but the eventual correct count among the pending cases is not yet known.

Step 1: Calculate performance among cases with finalized outcomes. The observed proportion is $68/80=0.85$, or 85%. This describes the 80 finalized cases; it is not yet the confirmed proportion for all 100 evaluation cases.

Step 2: Find the possible full-sample range. If none of the 20 pending cases is correctly classified, the full-sample proportion is $68/100=0.68$, or 68%. If all 20 are correctly classified, it is $(68+20)/100=88/100=0.88$, or 88%. Therefore, the final proportion could range from 68% to 88%.

Step 3: Compare the range with the policy requirement. The 80% requirement lies inside that range. Some possible outcomes would fall below the requirement and others would exceed it. The available evidence therefore does not establish that the policy requirement has been met.

Conclusion. The pending outcomes are a material limitation for this decision. A defensible report can state the 85% result among finalized cases, but should not present it as a confirmed pass for the full evaluation. The decision could be deferred until the remaining outcomes are finalized, or any provisional use should be explicitly restricted.

Common mistake and correction. The incorrect calculation is $68/80=85\%$, followed by the claim that the 100-case evaluation meets the 80% requirement. The arithmetic is correct, but the conclusion uses a subset as if all outcomes were known. The correction is to show the full-sample bounds, 68% to 88%, and note that they straddle the requirement.

Writing a Limitation That Supports a Decision

A useful limitation statement connects four elements: the evidence that is constrained, the possible direction or range of its effect, the decision that depends on it, and the response. When the effect cannot be quantified credibly, state the uncertainty without inventing a numerical range. Still explain why it matters and what decision it qualifies.

Key takeaway: A limitation should be prominent when a plausible effect could change the action, its expected value, or the safe scope of use. State the limitation in context, show its decision consequence where possible, and qualify the recommendation accordingly.

Avoid two opposite errors. First, do not bury an important limitation among minor points. Second, do not imply that every limitation invalidates the model. A caveat may matter for one decision and not another: a small change in a low-stakes ranking may not alter the action, while the same-sized change near a financial break-even point may reverse it.

A concise report can distinguish material from secondary caveats without dismissing either. For instance, lead with the limitation that could reverse the recommendation, explain its consequence, and then summarize other constraints that do not currently alter the decision. This gives the reader a reasoned priority rather than an undifferentiated list.

Prefer consequence over label. “The estimate may be lower in the new setting, which could move expected risk below the inspection threshold” is more useful than “there may be distribution shift.”
Separate evidence from judgment. Report what was observed, what remains uncertain, and which part of the recommendation follows from judgment rather than established evidence.
Match the response to the limitation. A limitation may call for a narrower recommendation or further evidence, rather than an automatic rejection of the model.

This approach makes limitations part of the decision logic. It allows a reader to see not only what the analysis cannot establish, but also whether that uncertainty changes what should be done now.

Check Your Understanding

For each problem, identify whether the stated limitation could change the decision and give the decision-relevant calculation.

A hypothetical repair team considers a $120 inspection when an uninspected failure would cost $800. A model estimates failure probability as 0.17, but for the proposed equipment group it could be 0.12. Calculate the break-even probability and say whether the limitation could reverse the action. (Answer: The break-even probability is $120/800=0.15$. Since 0.17 is above 0.15 but 0.12 is below it, the limitation could reverse the action.)
A hypothetical outreach campaign costs $15 per contact and earns $100 per incremental success. The estimated incremental success rate is 0.18, but a plausible lower value is 0.12. For 100 contacts, calculate net value at both rates. (Answer: At 0.18, net value is $100(100(0.18)-15)=\$300$. At 0.12, it is $100(100(0.12)-15)=-\$300$, so the limitation is material.)
An evaluation requires at least 75% correct classifications across 80 cases. Outcomes are final for 60 cases, with 48 correct; the other 20 are pending. Find the possible full-sample range and determine whether the requirement is established. (Answer: The minimum is $48/80=60\%$, and the maximum is $(48+20)/80=85\%$. The range crosses 75%, so the requirement is not yet established.)
A hypothetical inspection policy costs $40 per item, and failure without inspection costs $500. A model estimate is 0.10, while a plausible population-specific estimate is 0.07. Calculate the expected cost of each action at both probabilities and assess materiality. (Answer: The threshold is $40/500=0.08$. At 0.10, no inspection costs $0.10(500)=\$50$, so inspection is cheaper by $10. At 0.07, no inspection costs $35, so it is cheaper than inspection by $5. The limitation could reverse the decision.)
A program has a fixed cost of $1,000 and expects 200 eligible cases. Each successful outcome is worth $30. The estimated success rate is 0.20, but a plausible lower rate is 0.15. Calculate expected net value under both rates and state whether the caveat matters. (Answer: At 0.20, expected value is $200(0.20)(30)-1{,}000=\$200$. At 0.15, it is $200(0.15)(30)-1{,}000=-\$100$. The limitation is material because it changes expected net value from positive to negative.)

What You'll Learn