Why Findings Need Structured Follow-Through
A review is useful only if its conclusions lead to controlled action. A validation review might identify an incorrect input mapping, an unexplained performance exception, or a weakness in supporting evidence. If the issue is recorded only as a note in a report, it may have no accountable owner, no deadline, and no reliable way to establish whether it was fixed. Findings and remediation tracking connect the review conclusion to a decision about model use.
This tutorial focuses on individual findings: how to describe and classify them, assign owners and due dates, determine what evidence supports closure, and record residual risk or escalation. The specific labels and deadlines below are an illustrative internal framework, not universal regulatory thresholds. An organization should use its applicable governance policy and regulatory obligations. These practices are aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
A finding is not complete merely because a reviewer has named a problem. A useful record makes it possible for someone who was not in the review meeting to understand what is wrong, why it matters, what action is required, who is accountable, and how the action will be verified.
Classify Severity With Context
Severity should reflect the consequence of the condition if it is not corrected, the likelihood that the condition affects the relevant process, and the reach of that effect. Relevant consequences can include a materially incorrect rate indication, an unreliable validation conclusion, a compliance concern, or a decision made without adequate support. Consider existing controls, but give them credit only when there is evidence that they operate effectively.
For illustration, an organization could assign impact and likelihood ratings from 1 to 4. Impact ranges from negligible (1) to severe (4); likelihood ranges from remote (1) to frequent or already observed (4). Define the categories in advance with examples appropriate to the organization. The following score is a prioritization aid, not a measured probability or a substitute for judgment:
Multiplication gives a higher index when either dimension rises and allows reviewers to compare combinations consistently. For example, impact 3 with likelihood 2 yields 6, while impact 3 with likelihood 4 yields 12. However, the product does not establish that an event has a particular probability or expected dollar cost. Two findings with the same product can have very different implications. Therefore, use documented category definitions and an override rule: a credible threat of unlawful use, materially unsupported production rates, or another urgent control failure may require immediate escalation even if a numerical score is low.
| Illustrative category | Score guide | Typical response |
|---|---|---|
| Critical | 12–16, or an override condition | Escalate promptly; consider restricting or suspending affected use while the decision-maker assesses the issue. |
| High | 8–11 | Set a short deadline, identify interim controls, and require documented verification before affected decisions proceed where warranted. |
| Moderate | 4–7 | Assign a responsible owner and a defined corrective action; monitor progress to the agreed deadline. |
| Low | 1–3 | Document the rationale and address or formally disposition the issue through a proportionate process. |
These score bands are illustrative and should not be applied mechanically. A reviewer should record why the impact and likelihood ratings fit the facts, whether scope is limited or broad, and what controls reduce exposure. If the evidence is incomplete, state that uncertainty rather than assuming the most favorable case. Severity can be revised when new evidence changes the assessment, but the record should preserve the earlier rating, the reason for the change, and who approved it.
Build a Trackable Finding
A concise record usually includes the finding identifier; date raised; reviewer; affected model, process, or data; criterion; condition and evidence; cause if known; impact and likelihood rationale; severity; interim control; corrective action; accountable owner; due date; status; and closure or acceptance decision. Separate the accountable owner, who is responsible for delivery, from contributors who perform specific tasks. A committee may approve a decision, but an individual or role still needs to be answerable for completing the action.
Actions should address the cause where it is known, not only the symptom. “Review the mapping” is difficult to verify. “Correct the territory crosswalk, rerun the affected extraction, and reconcile record counts and modeled outputs to the approved source” is observable. If the cause is not yet known, make diagnosis an explicit action with an owner and deadline, then update the corrective plan when the investigation concludes.
Describe what the evidence shows and what requirement, approved procedure, or documented expectation it fails to meet.
Use the defined rating guide, document the evidence and uncertainty, and apply any required escalation override.
Specify the deliverable, accountable role, due date, and any temporary restriction or control needed before completion.
Review evidence against the action’s acceptance criteria; record closure, approved residual-risk acceptance, or escalation, including the decision-maker and rationale.
Deadlines, Evidence, and Closure
A due date should follow the severity, the time until affected decisions are made, and the effort needed to produce reliable evidence. A deadline is an internal control, not proof that an issue is safe to leave unaddressed until that date. When an unresolved finding could affect a pending filing or rate release, the owner should identify an interim restriction or escalate the decision rather than relying on a distant target date. If a deadline changes, retain the original date, the reason for the change, the approving role, and the revised date.
Closure requires more than a statement that work was performed. Evidence should be appropriate to the action: a controlled code change and test results, a reconciled data extract, a rerun validation report, or an approved procedure and evidence that it operated. A reviewer independent of the person who performed the correction should verify material findings where practical. The reviewer records whether the evidence meets the stated acceptance criteria and whether the original condition is resolved.
If the organization decides not to eliminate a known exposure, record accepted residual risk rather than marking the corrective action complete. The acceptance record should state what remains, the reason for accepting it, the scope and duration, the approving authority, any monitoring or compensating control, and a review date or trigger. Acceptance does not erase the finding or make the underlying condition disappear. Escalation is appropriate when the owner cannot meet the deadline, the severity increases, evidence contradicts the original assessment, or the required acceptance authority is unavailable.
Worked Example: Territory Mapping Finding Before a Rate Release
A hypothetical insurer is reviewing a proposed property rate update. A reviewer finds that a territory crosswalk sends some records with an obsolete code to a default territory. Testing identifies 240 affected records among 8,000 records. The defaulting is already observed, and the affected segment could have a material rate indication difference. Under the organization’s illustrative scale, impact 3 means material effect on a segment, and likelihood 3 means a recurring condition is likely while the crosswalk remains in use. Classify the finding and set a defensible response.
Step 1: Record the condition and evidence. The condition is not merely “territory data may be wrong.” It is that 240 of 8,000 tested records, or \(240/8{,}000=0.03=3\%\), use an obsolete code that maps to the default territory. The crosswalk version and test output are retained as evidence. The criterion is that source territory codes must map according to the approved crosswalk or be explicitly rejected for investigation.
Step 2: Calculate and interpret the score. Impact is 3 and likelihood is 3, so the priority score is \(3\times3=9\). Under the illustrative bands, 9 is High. This rating is supported by an observed defect and potentially material effects on a segment; it is not a claim that there is a 9-unit probability or a quantified loss.
Step 3: Assign action and interim control. The pricing model owner is accountable. The data steward is assigned to correct the crosswalk and produce a controlled replacement extract. Because the rate release is pending, the proposed rate update is held for affected records until correction and review; simply proceeding with the default would leave the identified issue uncontrolled. The target is 10 business days, with escalation to the pricing governance approver if that date cannot be met.
Step 4: Define closure evidence. Closure requires a versioned crosswalk, a test showing all 240 obsolete codes were either validly remapped or explicitly rejected, a reconciliation of the affected and total record counts, and a rerun of the impacted rate indication. An independent reviewer checks these items. The finding closes only if the evidence satisfies those criteria and the decision to release the rate update is recorded.
Check. The result is High severity, score 9, with a named accountable owner, a 10-business-day target, a hold on affected use, and specified verification evidence. A common mistake would be to call this Low because only 3% of records are affected; the correction is to assess the potential consequence and actual recurrence as well as the share of records.
Worked Example: Validation Denominator Error
In a hypothetical review of a proposed commercial auto model, a validation report states that 96 policies failed a performance tolerance among 1,200 policies. The reviewer discovers that 400 policies were ineligible for that test and should have been excluded from its denominator. All 96 failures are among the 800 eligible policies. The report’s threshold is 10% failures. Assume impact 3 for a misleading approval conclusion and likelihood 3 because the report was used in the review. Determine the corrected result and finding response.
Step 1: Recompute the reported rate. The report used \(96/1{,}200=0.08\), or 8.00%. Since 8.00% is below the 10% threshold, the report appears to pass.
Step 2: Apply the stated eligibility condition. The correct eligible denominator is \(1{,}200-400=800\). The corrected failure rate is \(96/800=0.12\), or 12.00%. That exceeds the 10% threshold by \(12\%-10\%=2\) percentage points. The report’s denominator error therefore changes the stated conclusion from pass to fail.
Step 3: Classify and respond. The score is \(3\times3=9\), High under the illustrative framework. The validation lead is accountable for correcting the report and rerunning the affected test; the model owner must not treat the original validation result as evidence of a pass. The deadline is five business days, or earlier if a decision is scheduled sooner. Escalate immediately if the model approval is imminent.
Step 4: Verify closure. Evidence includes the corrected eligibility rule, the reproducible calculation \(96/800=12.00\%\), a revised validation report, and the documented decision about the failed tolerance. A reviewer checks that the numerator and denominator follow the test definition. The finding can be closed after that verification, but closure does not itself mean the model passed; the failure requires its own documented disposition.
Check. The correct rate is 12.00%, not 8.00%, so the test fails its 10% threshold. The common mistake is to divide by all records despite the explicit eligibility condition.
Worked Example: Remediation With an Accepted Residual Risk
A hypothetical rating-data process flags a missing optional building-description field. A review finds 18 missing values in a sample of 6,000 records. The field is not used in the rating calculation, and a documented default allows records to process without changing the indication. The missingness nevertheless makes one supporting report less complete. Under the local guide, impact is 1 (negligible rating effect) and likelihood is 2 (occasional). Choose a disposition and state what the record needs.
Step 1: Compute the score and scope. The missing share is \(18/6{,}000=0.003=0.30\%\). The score is \(1\times2=2\), Low. The rationale must distinguish the reported completeness issue from rating impact: the field is not used in the calculation, and the documented default preserves processing.
Step 2: Assign a proportionate action. The data steward is accountable for adding a completeness check and documenting the fallback. The target is 20 business days. The check should identify missing values without silently changing the approved default or treating the optional field as mandatory.
Step 3: Specify evidence and residual risk. Evidence for remediation is the versioned check, test output confirming it detects the 18 missing values in the sample, and a processing test showing that the approved fallback still operates. If the business elects not to require complete descriptions, the remaining possibility of incomplete reporting is separately recorded as accepted residual risk. The record names the authorized approver, explains that rating calculations do not use the field, identifies the monitoring control, and sets a review trigger if the field becomes a rating input.
Step 4: Avoid false closure. The check may be closed as remediated after independent verification, while the accepted residual risk remains documented and subject to its review trigger. Alternatively, if the check is not implemented, the finding must remain open or be explicitly dispositioned through authorized risk acceptance; it must not be marked “fixed” merely because the score is Low.
Check. The score is 2, the missing share is 0.30%, and the appropriate record separates completed control remediation from the accepted possibility of incomplete reporting. The common mistake is to treat a low score as permission to omit an owner, evidence, or an authorized decision.
Common Traps in Findings Management
- Scoring without explaining the basis. A number alone does not show why an issue matters. Record scope, evidence, consequence, and likelihood rationale.
- Using a finding label without a criterion. State the applicable requirement or expectation; distinguish a confirmed failure from a suggestion for improvement.
- Assigning a committee but no owner. Name an accountable role for delivery and the authority responsible for approval or acceptance.
- Closing on a promise. “The team will fix it” is not closure evidence. Verify the actual change and its effect against stated acceptance criteria.
- Confusing risk acceptance with remediation. Accepted residual risk is an explicit decision about what remains, not evidence that the condition has been corrected.
- Letting a deadline substitute for interim control. If affected use may occur before correction, decide whether to hold, restrict, monitor, or escalate that use.
A sound record preserves the chain from observed evidence to severity, action, verification, and disposition. That chain allows later reviewers to understand what was known and what decision was made, even if personnel or model versions change. Consistent documentation also makes review findings actionable without suggesting that every issue has the same urgency or requires the same remedy.
Check Your Understanding
For each situation, determine the requested classification or disposition using the definitions and illustrative score bands in this tutorial.
- A reviewer documents a recurring input defect with impact 2 and likelihood 3. Calculate the score and category. What minimum tracking elements should accompany the classification? (Answer: Score 6, Moderate; record the evidence and criterion, impact/likelihood rationale, action, accountable owner, due date, and verification evidence.)
- A validation test reports 30 exceptions among 500 records, but 100 records were explicitly out of scope and all 30 exceptions are in the remaining records. The threshold is 8%. Calculate the correct rate and state whether it passes. (Answer: \(30/(500-100)=30/400=7.50\%\); it passes the 8% threshold.)
- A finding has impact 4 and likelihood 2, but evidence indicates a credible risk of releasing materially unsupported rates tomorrow. What does the score indicate, and what additional step is required? (Answer: Score 8, High by the illustrative bands; promptly escalate and apply the required override assessment, including a decision on restricting or holding the affected release.)
- A Moderate finding’s owner requests 15 more business days after the original due date because testing is incomplete. What should the tracking record preserve, and what should happen if affected use will occur before the new date? (Answer: Preserve the original and revised dates, reason, approver, and status; establish an interim control or escalate the use decision rather than relying on the extension.)
- A Low finding’s corrective action is verified, but the organization will continue to tolerate a remaining reporting limitation. What separate documentation is needed? (Answer: Close only the verified corrective action; document the residual condition, rationale, authorized approver, scope or duration, monitoring, and review date or trigger.)