Why Model Purpose Must Come Before Model Detail
A capital model is not fit for use merely because it produces a plausible number. Its output must answer a defined question, at a level of detail and speed suitable for the decision-maker, with enough precision to support the decision. A model designed to support annual solvency assessment may not be suitable for daily underwriting authority, even if both users are interested in risk.
The starting point is therefore not a calculation method but the decision the model is meant to inform. A purpose such as “measure risk” is too broad to guide design or validation. A useful statement identifies the user, the decision, the relevant output, and the conditions under which that output will be used. These ideas are aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
The same model output may be material for one decision and immaterial for another. A small change in a portfolio-wide capital estimate might not affect a high-level planning range, but could change whether a particular account is referred for review. Materiality is therefore not a universal dollar amount. It depends on the model’s use, the decision rule, and the consequences of being wrong.
Translate Decisions into Requirements
Model requirements should be stated in terms that can be assessed. For each intended use, specify what the model must produce, who will use it, when it is needed, what level of detail is necessary, and what degree of uncertainty or error is tolerable. Requirements may differ across underwriting, pricing, planning, and solvency work.
| Intended use | Decision supported | Possible model requirement |
|---|---|---|
| Underwriting | Whether to quote, decline, or refer an account | Timely account-level output with uncertainty small enough to support the referral rule |
| Pricing | Whether a proposed price adequately reflects the selected risk basis | Outputs aligned with the pricing decision and available before the quote is finalized |
| Capital planning | How much capital to plan for and where management attention is needed | Consistent portfolio or segment results available by the planning deadline |
| Solvency assessment | Whether capital is adequate under the specified assessment basis | Results on the required basis and horizon, with sufficient support for governance and review |
These are examples of requirement types, not universal specifications. For example, “the model must be accurate” is not operational: a reviewer cannot determine what accuracy means without a tolerance, comparison basis, and intended decision. A more useful requirement might state that an estimate must be available within two business days, be reported at region level, and have validation evidence supporting an uncertainty range no wider than a specified amount.
State what the user will decide and the rule, threshold, or judgment the output informs.
Identify the measure, reporting level, user, and any interpretation guidance needed for the decision.
State the required precision, granularity, and delivery time. These are requirements for the use, not abstract measures of model quality.
Determine whether a plausible error, change, or missing detail could alter the decision or its consequences.
Compare the model’s demonstrated performance with the stated requirements. If it falls short, constrain the use, add review, or improve the model before relying on it.
Materiality: Quantitative Screens and Decision Consequences
A quantitative materiality screen can use both an absolute and a relative threshold. Let \(R\) be a positive reference value, such as a previously approved capital estimate, and let \(\Delta\) be the change being assessed. A policy may define a change as quantitatively material if it exceeds either a dollar threshold \(A\) or a relative threshold \(r\):
The thresholds, reference value, and strictness of the comparisons must be set in advance for the specific use. A firm might use this screen to decide which changes require escalation, but it should not treat a screen as a complete definition of significance. A change below both thresholds can still be qualitatively material if it affects a control, a regulatory conclusion, a key assumption, or a decision near a boundary. Conversely, a large numerical change may have limited decision consequence if it does not affect the use in question, though it may still require investigation.
Why Decision Boundaries Matter
A model estimate is often uncertain. If the intended decision turns on a threshold, the size of that uncertainty relative to the distance from the threshold matters. Suppose the estimate is \(z\), the decision cutoff is \(c\), and a defensible bound on estimation error is \(e\), so that the value relevant to the decision lies in \([z-e,z+e]\). For a decision made when the value exceeds \(c\), the decision is robustly positive only when the whole interval is above the cutoff:
It is robustly negative when the whole interval is at or below the cutoff, which is equivalent to \(z+e \le c\). Otherwise, the interval straddles the cutoff and the decision is sensitive to the uncertainty. This result follows directly from the interval bound: if every possible value is above \(c\), the condition is always satisfied; if every possible value is at or below \(c\), it is never satisfied. If values on both sides are possible, the estimate alone does not establish which decision is justified.
Worked Example: Testing an Underwriting Referral Rule
A hypothetical insurer uses an underwriting model to support referral decisions. An account is referred when its indicated annual quote exceeds $92,000. For one account, the model indicates $90,000, with an assessed uncertainty bound of $4,000. Decide whether the model output supports a definitive decision under this rule. Assume the quote and uncertainty values are supplied model outputs; no interest rate, mortality assumption, or probability distribution is used.
Method. Compare the full uncertainty interval with the referral cutoff rather than comparing only the point estimate.
The lower end is \(90{,}000-4{,}000=86{,}000\), and the upper end is \(90{,}000+4{,}000=94{,}000\). Thus, the interval is \([$86{,}000,$94{,}000]\).
The referral cutoff of $92,000 lies within the interval: \(86{,}000 \le 92{,}000 \le 94{,}000\). The model’s point estimate is below the cutoff, but the uncertainty interval crosses it. The estimate therefore does not support a robust automated decision either to refer or not to refer. A suitable requirement might be that cases with intervals crossing the authority threshold receive additional review.
Check. The interval contains values below and above $92,000, so it is not wholly on either side of the decision boundary. The conclusion is consistent with the stated rule. The common mistake avoided is treating the point estimate alone as decisive.
Worked Applications of Materiality and Fit for Use
Worked Example: Applying an Absolute and Relative Materiality Screen
A hypothetical company has an approved capital reference of $240.00 million. Its change policy screens a change as quantitatively material if it is greater than $5.00 million in absolute value or greater than 2.00% of the reference amount. A recalculation gives $244.85 million. Apply the screen. Assume these are supplied capital estimates; no interest rate, mortality assumption, or probability distribution is used.
Method. Calculate the absolute change and divide it by the stated reference amount, not by the recalculated amount.
The absolute change is \(244.85-240.00=4.85\) million dollars. It does not exceed the $5.00 million threshold.
The relative change is \(4.85/240.00=0.0202083\ldots\), or \(2.02083\ldots\%\). Rounded to four decimal places, this is 2.0208%. It exceeds the 2.00% threshold. Since the policy uses “or,” exceeding either threshold is sufficient: the change is quantitatively material.
Check. The absolute test fails to trigger, but the relative test triggers, so the combined rule identifies the change. The common mistake avoided is using $244.85 million, the new value, as the denominator; the policy specifies the approved reference amount.
Worked Example: Choosing a Model Version for Capital Planning
A hypothetical insurer needs a quarterly capital-planning estimate. Its requirements are: delivery within two business days, region-level results, and an uncertainty bound no wider than $2.00 million. Three candidate versions have the following validated performance: Version A takes 0.5 business days, reports only a portfolio total, and has an uncertainty bound of $4.00 million; Version B takes 1.5 business days, reports by region, and has a bound of $1.50 million; Version C takes 4 business days, reports by region, and has a bound of $0.50 million. Which version meets all stated requirements? Assume these performance figures are supplied validation results; no interest rate, mortality assumption, or probability distribution is used.
Method. Test each version against every requirement; strong performance on one criterion does not compensate automatically for failure on another.
Version A meets the deadline because \(0.5 \le 2\) business days, but it fails the region-level reporting requirement and its $4.00 million bound exceeds $2.00 million. It does not meet all requirements.
Version B meets the deadline because \(1.5 \le 2\) business days. It reports by region and its uncertainty bound meets the tolerance because \(1.50 \le 2.00\) million. It meets all three requirements.
Version C reports by region and meets the precision tolerance because \(0.50 \le 2.00\) million. However, its four-business-day processing time exceeds the two-day deadline. It does not meet all requirements.
Check. Version B is the only version that passes all three tests. The common mistake avoided is selecting Version C solely because it has the smallest uncertainty, despite missing the planning deadline.
Common Traps and Practical Implications
- Using one materiality threshold for every purpose. A threshold appropriate for high-level planning may be too coarse for a decision made close to an account-level authority limit.
- Confusing precision with fitness for use. A precise result delivered after the decision deadline may be unusable. A timely portfolio total may also be inadequate if the decision requires results by region.
- Assuming quantitative immateriality settles the question. A small change can still matter if it affects a decision boundary, a key control, or the interpretation of a result.
- Making a point estimate do more than it can support. When uncertainty spans the decision cutoff, the output may justify escalation or further review rather than a definitive automated action.
- Failing to state the reference basis. Relative changes depend on the denominator. Define whether the comparison is to an approved estimate, a prior-period result, or another specified reference.
Purpose and intended use should be revisited when the decision, user, timing, or tolerance changes. A model need not serve every organizational need, but any restriction should be explicit: for example, an output may be suitable for planning ranges but not for account-level decisions. This keeps model limitations visible and helps users avoid treating a number as more precise, timely, or detailed than the evidence supports.
Check Your Understanding
For each problem, apply the stated requirements and decision rules exactly.
- A hypothetical capital estimate is $180 million. A change policy flags a change only when it is greater than $3 million in absolute value or greater than 1.5% of the $180 million reference. A recalculation is $182.7 million. Is it quantitatively material under this strict “greater than” rule? (Answer: No; the change is $2.7 million and exactly 1.5% of the reference, so neither threshold is exceeded.)
- A hypothetical model estimates a decision measure of 74, with an error bound of $5 in the same units. A decision is positive only when the true measure exceeds 70. Is the result robustly positive, robustly negative, or uncertain relative to the cutoff? (Answer: Uncertain; the interval is [69, 79] and crosses 70.)
- A hypothetical planning use requires region-level outputs within one business day and an uncertainty bound no wider than $1.00 million. Option A takes 0.5 days, has a $0.80 million bound, and reports only a portfolio total. Option B takes 1.2 days, has a $0.50 million bound, and reports by region. Option C takes 0.8 days, has a $1.40 million bound, and reports by region. Does any option meet every requirement? (Answer: No; A fails granularity, B fails timeliness, and C fails the precision tolerance.)
- A hypothetical approved capital reference is $320 million. The materiality screen is a change greater than $8 million or greater than 2.00% of the reference. A new estimate is $327.5 million. Apply both tests. (Answer: The change is $7.5 million, or \(7.5/320=2.34375\%\); the absolute test does not trigger, but the relative test does, so it is quantitatively material.)
- An underwriting model estimates a measure of 106 with an error bound of 3. Referral is required only when the measure exceeds 110. Is the decision robust? (Answer: Yes, it is robustly not referred under the stated rule; the interval is [103, 109], wholly at or below 110.)