Governance Review of a Vendor Model Update

Why Review a Vendor Model Update?

A new catastrophe-model version can change indicated loss even when the insured portfolio has not changed. The difference may reflect revised hazard science, vulnerability functions, financial terms, data processing, or a changed run configuration. A reviewer therefore needs to establish what changed, which outputs are affected, and whether the evidence supports the intended use before treating the updated results as reliable.

This review is not simply a comparison of two output files. It links the vendor’s version and assumptions to the organization’s input data, checks reproducibility and validation evidence, measures the impact on the relevant portfolio, and confirms that required approvals and records are complete. A favorable impact result alone does not demonstrate that an update is sound; an apparently reasonable total can conceal a material data or segment-level issue.

Definition: A vendor model update review is a documented assessment of a specific model version and its configuration, input lineage, assumptions, validation evidence, and impact on the intended use. The review concludes whether the outputs are acceptable for that use, acceptable subject to explicit conditions, or not acceptable pending remediation. Acceptance is specific to the reviewed version, data, configuration, and use; it is not a claim that the model is universally accurate.

The review should begin with a defined comparison. Record the old and new model versions, the intended use, the portfolio and valuation date, the exposure and claims data versions, and the run settings. When the goal is to isolate the model-version effect, hold the portfolio, financial terms, and other relevant settings constant. If those items also changed, identify them separately rather than attributing the whole difference to the model update.

Trace the Update from Inputs to Outputs

Lineage means the ability to trace an output back through the data and processing that produced it. For a catastrophe-model review, that typically includes the source exposure and claims extracts, geocoding or other transformations, model version, hazard and vulnerability modules, financial terms, demand-surge or secondary uncertainty settings where applicable, and run identifiers. The reviewer should be able to tell which files and settings were used, not merely that a vendor report says the run was completed.

A useful first test is reconciliation: do the submitted records and exposures agree with the controlled source extracts, and are exceptions visible? Counts alone can be misleading. A small number of ungeocoded properties might account for a large share of insured value or concentrate in a high-hazard area. Completeness should therefore be considered both by record count and by measures relevant to the modeled result, such as insured value and location.

The reviewer should also distinguish the model’s assumptions from the organization’s assumptions. A vendor may change a hazard module or vulnerability function, while the organization may separately choose a financial perspective or event-set setting. Document both. Where a changed assumption is material to the intended use, the review should explain its direction and potential effect; a brief version label is not an adequate explanation.

Worked Example: An Exposure-Lineage Exception

A hypothetical property portfolio is submitted for an update comparison. The controlled exposure file contains 12,000 locations with total insured value of $600 million. The new model run contains 11,760 locations matched to the expected geocoding output; the remaining 240 were assigned a fallback location treatment. Those 240 locations represent $108 million of insured value. The review procedure, set before examining the result, requires at least 99.5% of locations to have the approved geocoding treatment and requires material fallback exposure to be resolved or explicitly assessed.

Method. Reconcile record coverage and insured-value coverage, then compare each with the review conditions.

First, the count of locations receiving the approved treatment is 11,760. Its coverage is $11{,}760/12{,}000=0.98$, or 98.00%. The number falling back is $12{,}000-11{,}760=240$, which is 2.00% of locations.

Second, fallback insured-value share is $$108\text{ million}/$600\text{ million}=0.18$, or 18.00%. The remaining exposure value is $$600-$108=$492$ million.

The count coverage of 98.00% is below the required 99.5%. In addition, the fallback records represent 18.00% of insured value, so the exception is not minor merely because it involves 2.00% of locations. The run should not be accepted on this evidence. The reviewer should request correction or a controlled assessment of those records, rerun or otherwise reconcile the affected outputs, and retain the exception and resolution evidence.

Check. The approved-treatment count plus fallback count is $11{,}760+240=12{,}000$, and the corresponding insured values are $492 million plus $108 million, or $600 million. Common mistake avoided: checking only the 98% record count and overlooking that the affected records carry 18% of insured value.

Assess Validation Evidence and Impact

Validation evidence should be relevant to the model version, configuration, and intended use under review. It may include comparisons with suitable external benchmarks, tests of event or loss components, diagnostics for known limitations, or checks that the updated implementation reproduces expected results. A reviewer should ask how the test was designed, what population and period it covers, what was excluded, and whether the criteria were defined before seeing the results. Passing a test is meaningful only in relation to a stated criterion and an understood limitation.

Catastrophe losses are infrequent and volatile. A short period with few observed events may provide weak evidence about tail behavior, even when average losses appear close. Validation should therefore not be summarized as “the model matched experience” without specifying the measure, benchmark, period, and limitations. A model’s performance on one aggregate statistic does not establish that every peril, region, or layer is adequately represented.

For impact analysis, compare like with like. If the exposure set and run settings are held constant, segment-level expected annual loss (EAL) changes can be attributed to the model version for that comparison. Let $E_{0,j}$ and $E_{1,j}$ be the old and updated EAL for segment $j$. Total EAL is the sum across segments, so:

E_0=\sum_j E_{0,j}, \qquad E_1=\sum_j E_{1,j}, \qquad E_1-E_0=\sum_j(E_{1,j}-E_{0,j}).

This identity follows by subtracting the two sums term by term: $\sum_j E_{1,j}-\sum_j E_{0,j}=\sum_j(E_{1,j}-E_{0,j})$. It provides a reconciliation check and a way to show which segments explain the total change. It does not, by itself, determine whether the change is reasonable or whether a rate should change.

Worked Example: Reconciling a Change in Modeled Loss

A hypothetical insurer compares two versions of a catastrophe model using the same portfolio, valuation date, policy terms, and run configuration. The old EAL is $4.00 million for Coastal and $4.40 million for Inland. The updated EAL is $4.80 million for Coastal and $4.44 million for Inland. The review team has not set a universal pass threshold; it requires the dollar and percentage changes to be reconciled and the drivers to be investigated before deciding whether the update is acceptable.

Method. Calculate each segment’s dollar change and percentage change, sum the segment changes, and compare the result with the portfolio total change.

Coastal change is $$4.80-$4.00=$0.80$ million. Relative to the old Coastal EAL, this is $$0.80/$4.00=0.20$, or 20.00%.

Inland change is $$4.44-$4.40=$0.04$ million. Relative to the old Inland EAL, this is $$0.04/$4.40=0.0090909$, or approximately 0.91%.

Old total EAL is $$4.00+$4.40=$8.40$ million. Updated total EAL is $$4.80+$4.44=$9.24$ million. The total increase is $$9.24-$8.40=$0.84$ million, or $$0.84/$8.40=0.10$, which is 10.00%.

The segment changes reconcile: $$0.80+$0.04=$0.84$ million. Coastal contributes $$0.80/$0.84=95.24\%$ of the net increase, while Inland contributes 4.76%. The concentration of the change in Coastal is a reason to examine the changed coastal hazard or vulnerability assumptions and relevant validation evidence. It is not, by itself, proof that the new result is wrong or right.

Check. Segment totals give $8.40 million before and $9.24 million after, consistent with the calculated 10.00% total increase. Common mistake avoided: averaging the two segment percentage changes, which would give an unweighted figure that does not equal the portfolio change. The aggregate percentage must use the old total as its denominator.

Worked Example: Applying Predefined Validation Criteria

A hypothetical review compares updated-model EAL with an independently prepared benchmark for 20 region-peril cells. Before reviewing results, the team set these criteria: at least 18 of 20 cell-level relative differences must be within 15% in absolute value; no cell may exceed 25% in absolute value; and aggregate updated EAL must be within 5% of aggregate benchmark EAL. Define each cell’s relative difference as $(\text{updated EAL}-\text{benchmark EAL})/\text{benchmark EAL}$. The results are 18 cells within 15%, one cell at +21%, and one cell at -18%. Aggregate benchmark EAL is $100 million and aggregate updated EAL is $103 million.

Method. Evaluate each criterion separately, using the stated denominator and absolute-value conditions where required.

The first condition requires at least 18 cells within 15%; exactly 18 meet it, so the condition passes. The two other cells have absolute differences of 21% and 18%. Both are below the 25% maximum, so the second condition passes.

For the aggregate condition, the difference is $$103-$100=$3$ million. Relative to the benchmark, the difference is $$3/$100=0.03$, or 3.00%. Since 3.00% is within 5%, the third condition passes. All three stated numerical criteria are met.

This result supports proceeding to the broader review, but it does not establish that all catastrophe risk is well represented. The reviewer must still examine whether the benchmark is suitable, whether the cells cover the intended use, and whether known limitations remain. The decision is therefore “validation criteria met,” not “model proven accurate.”

Check. The aggregate ratio of updated to benchmark EAL is $103/100=1.03$, consistent with a 3.00% excess. Common mistake avoided: treating the +21% cell as a failure of the 15% condition even though the predefined rule allows up to two cells outside 15%, subject to the separate 25% cap.

Make the Decision and Preserve the Evidence

A review decision should follow the evidence rather than a desired outcome. The reviewer can conclude that the update is acceptable for the stated use, acceptable only after named conditions are met, or not acceptable until material issues are resolved. Conditional acceptance should identify the condition, owner, due date or review point, and the consequence if it remains unresolved. An open issue should not disappear into a general statement that the model was reviewed.

Before acceptance, confirm that the evidence package is internally consistent: the version in the approval record matches the version used for validation and impact analysis; material exceptions have dispositions; the run can be identified and, where required, reproduced; and the appropriate reviewers and approvers have acted under the organization’s approval controls. A reviewer should also distinguish review findings from approval: performing calculations or raising issues does not itself authorize use.

The record should preserve enough information for a later reviewer to understand and reproduce the decision. At minimum, document the intended use, old and new versions, data and configuration identifiers, assumptions examined, validation criteria and results, portfolio and segment impacts, limitations, findings and remediation status, decision rationale, and approval evidence. Record whether the comparison isolated the version change or included other changes. Where uncertainty remains, describe it specifically rather than replacing it with a vague caveat.

Identify the reviewed change.
Record the model versions, intended use, data dates, configuration, and comparison design.

Reconcile lineage and assumptions.
Trace inputs and transformations, identify exceptions, and document assumptions that changed or remained fixed.

Evaluate evidence and impact.
Test validation results against predefined criteria, reconcile total and segment impacts, and assess limitations.

Resolve findings and obtain approval.
Track material issues to disposition and confirm the appropriate approval for the specific use and version.

Retain the decision record.
Preserve the evidence, rationale, conditions, and approval so the conclusion can be understood later.

Common Review Traps

Equating a small total change with a sound update. An aggregate can conceal large or offsetting changes by peril, region, or exposure segment. Reconcile the components and investigate material concentrations.
Confounding version effects with data effects. If exposure, financial terms, or run settings changed at the same time, a simple old-versus-new comparison does not isolate the model update.
Using criteria chosen after seeing results. Post hoc thresholds make it difficult to interpret a pass. State the test and decision rule before reviewing the result, or label a later analysis exploratory.
Treating validation as proof. A few passing aggregate checks cannot eliminate uncertainty about rare events, tail outcomes, or poorly represented segments.
Accepting without version-specific approval. Approval for one version, configuration, or use does not automatically cover another.
Recording conclusions but not evidence. “Reviewed and acceptable” is insufficient if the underlying data, calculations, assumptions, and rationale cannot be identified.

Key takeaway: A defensible vendor-update decision connects a specific version and configuration to traced inputs, explicit assumptions, relevant validation, reconciled impact, resolved findings, and documented approval. The decision must be bounded to the use and evidence actually reviewed.

A disciplined review helps ratemaking professionals distinguish a supportable model change from an unexplained output shift. It also makes uncertainty visible without claiming more certainty than the evidence can provide, a practice aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Check Your Understanding

For each hypothetical situation, identify the appropriate review calculation or governance conclusion.

A submitted file has 8,000 locations, of which 7,920 use the approved geocoding method. The 80 fallback locations represent $24 million of a $240 million total insured value. The count threshold is 99.0%, and fallback exposure must be investigated if it exceeds 5% of insured value. Calculate both percentages and state whether the lineage conditions pass. (Answer: Count coverage is 99.00%, meeting the threshold; fallback insured value is 10.00%, exceeding 5%, so the lineage review is not clear to pass.)
With portfolio and settings held constant, old segment EALs are $3.0 million and $5.0 million; updated EALs are $3.6 million and $5.2 million. Calculate the total percentage change and identify the segment contributing most of the dollar increase. (Answer: Old total $8.0 million, updated total $8.8 million, increase $0.8 million or 10.00%; the first segment contributes $0.6 million, most of the increase.)
A validation rule requires at least 9 of 10 cells within 10% and no cell above 20% in absolute difference. Results are 9 cells within 10%, and the remaining cell is -18%. The aggregate benchmark is $50 million and updated EAL is $53 million, with an aggregate tolerance of 7%. Do the stated criteria pass? (Answer: Yes; 9 cells meet the first rule, 18% is below the 20% cap, and the aggregate difference is $3/$50=6.00%, within 7%.)
A comparison shows a 12% increase in total EAL, but the new run also uses a revised exposure file and a different financial-term setting. Can the full change be attributed to the model version? State the next review step. (Answer: No; the comparison is confounded. Hold the exposure and financial-term setting constant in a controlled comparison, or separately quantify their effects before attributing the residual change.)
All numerical impact and validation checks pass, but the approval record refers to the prior model version and one material fallback-data finding has no disposition. Is the update ready for unconditional acceptance? (Answer: No; resolve or formally disposition the material finding and obtain approval tied to the reviewed version, configuration, and intended use.)

What You'll Learn