Why Plan Validation Before Testing?
An economic-capital model can combine many risk modules, assumptions, dependencies, and business decisions. Independent validation is the structured assessment of whether the model is fit for its stated purposes, supported by suitable evidence, and subject to appropriate limitations. A validation plan determines what will be assessed, why it matters, who will assess it, and what evidence will be needed before detailed testing begins.
A plan is not a promise to test every model feature equally. The scope should reflect the model's intended uses, materiality, uncertainty, prior diagnostic results, and the consequences of error. Earlier material on sensitivity analysis, benchmarking, and backtesting provides potential signals for directing attention; none of those signals, by itself, replaces an independent validation conclusion. This planning discipline is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Set Scope and Rank Validation Priorities
Start by stating the model's intended purpose and the decisions it informs. A model used to set enterprise capital targets may warrant a different scope from one used only for internal scenario exploration. Identify the material risks and model components in that context, then record known concerns such as sparse data, expert judgment, model uncertainty, significant sensitivities, or prior unexplained differences. These are reasons to prioritize questions, not findings in themselves.
A practical planning tool is a risk register. For each validation question, record the potential consequence of an undetected weakness, its likelihood or plausibility, existing controls, the proposed evidence, and the effort required to assess it. A simple ordinal score can help sort work, provided its limitations are stated. For example, if likelihood and impact are each rated from 1 to 5, a planning score \(s_j=L_j I_j\) can be used to rank question \(j\). The score is a prioritization heuristic; ordinal ratings do not measure risk precisely, and the score must not be represented as a probability or a capital estimate.
Worked Example: Allocating Review Effort by Priority
A hypothetical insurer has 84 independent validation hours available for initial planning. The team identifies three questions: whether enterprise aggregation is adequately supported, whether a parameter based on sparse observations is appropriately governed, and whether a low-materiality report output is clearly described. The team uses a 1-to-5 likelihood rating and a 1-to-5 impact rating, and allocates hours in proportion to the resulting scores. This is a planning convention, not an estimate of expected loss.
Step 1: Score each question. The aggregation question receives ratings 4 and 5, so \(s_1=4\times5=20\). The sparse-parameter question receives ratings 4 and 4, so \(s_2=4\times4=16\). The report-output question receives ratings 2 and 3, so \(s_3=2\times3=6\).
Step 2: Add the scores. The total is \(20+16+6=42\).
Step 3: Allocate the hours. The proportional shares are \(84(20/42)=40\) hours for aggregation, \(84(16/42)=32\) hours for the sparse parameter, and \(84(6/42)=12\) hours for the report output.
Check. The allocated effort totals \(40+32+12=84\) hours, exactly the available budget. The method makes the prioritization visible, but the team should still set minimum coverage for lower-ranked areas and document why the ratings are reasonable.
Method and mistake avoided. Normalize each score by the total score, then multiply by the available effort. A common mistake is to allocate hours directly in proportion to impact alone: that would ignore the distinct likelihood assessment and would not follow the stated scoring rule.
The plan should explain exclusions as well as inclusions. An area may be excluded from a particular validation cycle because it is immaterial to the stated use, recently assessed with adequate evidence, or outside the agreed scope. Record the reason, residual risk, and any compensating review rather than implying that an untested area has been validated.
Connect Questions to Evidence, Owners, and Criteria
Each planned test should be specific enough that another reviewer can understand what conclusion is being sought. A useful test specification names the validation question, the relevant model purpose or risk, the evidence to examine, the test owner, any necessary support, the pass or escalation criterion, and the resulting workpaper or report evidence. The test owner is accountable for conducting and documenting the assessment; model developers may supply explanations or records, but should not determine the independent conclusion about their own work.
Evidence should be appropriate to the question. It may include approved documentation, development records, independent analyses, comparison results, management decisions, or records of limitations and controls. A test plan should not treat the existence of a document as proof that the underlying model is sound. It should specify what the reviewer will assess in that evidence and what would count as an unresolved concern.
Worked Example: Turning a Validation Concern into a Test
A hypothetical economic-capital model uses expert judgment to set a parameter for a risk with limited observations. The validation plan must define a test that can be independently completed, without treating a later detailed parameter review as already done.
Step 1: State the question. The question is whether the selected parameter has a documented basis, reflects the stated model purpose, and includes a transparent account of uncertainty and limitations.
Step 2: Specify evidence. Request the approved parameter rationale, the records of relevant expert input, the materials considered by the decision-makers, and the documentation describing uncertainty and limitations. The plan asks the reviewer to compare these items for consistency; it does not assume that the presence of all four proves adequacy.
Step 3: Assign ownership. An independent validator owns the assessment and its workpaper. The model team supplies the requested records and answers factual questions. A separate validation lead reviews the workpaper and resolves any disagreement about the conclusion.
Step 4: Set a criterion. The test is satisfactory only if the rationale and decision trail are traceable, the stated purpose is addressed, and material uncertainty or limitations are disclosed. Missing support or an unresolved inconsistency is recorded for assessment rather than silently treated as a pass.
Check. The plan identifies a question, evidence, an accountable reviewer, supporting participants, and a decision criterion. It therefore directs the validation work without confusing document collection with a validation conclusion.
Method and mistake avoided. Work backward from the conclusion the reviewer must reach, then name evidence capable of supporting or challenging it. A common mistake is to assign a test to the model developer and count the developer's self-assessment as independent validation; the correction is to keep the independent judgment with the validator.
A clear pass criterion improves consistency but should not force a binary answer when evidence is mixed. The plan can allow outcomes such as pass, pass with limitations, unresolved, or fail, with escalation rules for uncertainty. Criteria should be set before the result is known where practicable; otherwise the reviewer risks changing the standard to fit the observed outcome.
Rate Findings and Specify Remediation
A validation finding describes a weakness, its supporting evidence, its potential consequence, and the action needed. Severity should reflect the potential effect on model fitness for use, including the materiality of affected decisions and the possibility that existing controls mitigate the concern. The number of affected tests, or the ease of fixing a problem, does not by itself determine severity.
| Severity | Planning interpretation | Typical response expectation |
|---|---|---|
| Critical | A credible weakness could make a material use unsafe or invalidate a key capital conclusion. | Escalate promptly; restrict or suspend affected use until an adequate response is approved. |
| High | A material weakness could significantly distort a decision or the reliability of an important result. | Assign an accountable remediation owner and deadline; require independent retesting before closure. |
| Moderate | A bounded weakness affects reliability or documentation but does not, on current evidence, invalidate all material uses. | Set a proportionate action plan, monitor completion, and verify the correction as specified. |
| Low | A limited issue has little apparent effect on the stated use but should still be corrected or documented. | Record a responsible owner and target date; close with proportionate evidence. |
Severity labels are not universal numerical thresholds. A plan should state its own criteria and escalation route, and allow the validator to explain why a particular issue meets them. Remediation expectations should identify the action owner, the target date, interim restrictions or controls where needed, the evidence required to demonstrate correction, and who performs independent retesting. Closure means that the agreed evidence has been assessed and the issue's residual effect has been addressed or explicitly accepted by the appropriate authority; it does not merely mean that a change was made.
Worked Example: Classifying a Material Finding
A hypothetical internal model has an approved comparison tolerance of 2% for a specified benchmark. In a planned validation test, the model reports capital of $1.16 billion while the benchmark is $1.20 billion. The planning rubric classifies an unresolved difference exceeding the tolerance as high severity if it affects a material use. These values are illustrative, not industry data.
Step 1: Calculate the relative difference. The absolute difference is \(|1.20-1.16|=\$0.04\) billion, or $40 million. Using the benchmark as the denominator, the relative difference is \(0.04/1.20=0.03333\), or 3.333%.
Step 2: Compare with the pre-set criterion. The observed 3.333% difference exceeds the 2% tolerance by 1.333 percentage points. The test therefore does not meet its stated comparison criterion.
Step 3: Rate and respond. Because the hypothetical rubric says that an unresolved breach affecting a material use is high severity, the finding is rated high, subject to the validator confirming the scope and relevance of the benchmark. The plan requires a named model-team remediation owner, a target date, a documented explanation and response, and independent retesting before closure. Until the difference is resolved or appropriately addressed, the finding must be escalated and the affected use assessed for restriction.
Check. The comparison uses the benchmark denominator specified by the example, and the result exceeds the stated tolerance. The severity follows the defined rubric rather than an invented universal percentage rule.
Method and mistake avoided. Compute the difference, apply the stated denominator, compare with the pre-established criterion, and then use the severity rubric. A common mistake is to call the finding low because only one test failed; the correction is to assess the potential consequence for the material use, not count failed tests.
Approval Criteria and the Validation Record
Approval is a decision about fitness for specified uses, not a statement that a model is perfect or risk-free. The plan should identify who may approve the validation conclusion, what evidence must be complete, how open findings are treated, and what limitations or restrictions must accompany any approval. The approver should receive a concise view of scope, work completed, exceptions, unresolved issues, remediation commitments, and the validator's overall conclusion.
A practical approval decision can require that:
- the agreed material scope has been covered or exclusions have explicit justification and acceptance;
- planned tests have documented evidence, conclusions, and review sign-off;
- critical issues are resolved or the affected use is restricted, and other open issues have accepted owners, deadlines, and controls;
- material limitations and conditions of use are communicated to decision-makers; and
- the approver has authority to accept the residual risk or refer the matter to a higher decision-maker.
The validation report and supporting workpapers should make the path from question to evidence to conclusion traceable. Record changes to scope or test criteria and explain their rationale. A plan that is revised as new evidence emerges remains useful if changes are controlled, visible, and approved by the appropriate validation lead.
Check Your Understanding
For each hypothetical situation, decide what a sound validation plan should specify. Each item requires more than identifying a label or calculation.
- A team rates a validation question likelihood 3 and impact 4, and another likelihood 2 and impact 5. Using the stated planning rule \(s=LI\), calculate both scores, rank the questions, and explain one limitation of this ranking. (Answer: Scores 12 and 10; prioritize the first under this rule, while noting that ordinal scores are only a heuristic.)
- A validator is assigned a test of a material risk parameter. Name two types of evidence the plan could request, identify who should own the independent conclusion, and state one condition that would prevent an automatic pass. (Answer: For example, request the approved rationale and decision records; the independent validator owns the conclusion; missing support or an unresolved inconsistency prevents an automatic pass.)
- A validation plan has 60 hours and scores of 15, 10, and 5. Under proportional allocation, calculate the hours assigned to each question and check the total. (Answer: 30, 20, and 10 hours; total 60 hours.)
- A material-use test exceeds its pre-set tolerance, but the model team says the discrepancy is easy to fix. What should determine severity, and what should be required before closure? (Answer: Severity should reflect potential effect on the material use under the documented rubric, not ease of correction; require an accountable owner, target date, correction evidence, and independent retesting.)
- A validation cycle is complete except for one high-severity issue affecting a key capital decision. State two items the approval decision should address before allowing that use to continue. (Answer: It should address whether the issue is resolved or the affected use restricted, and whether any interim control, residual limitation, or escalation is acceptable to an authorized decision-maker.)