Why Vendor Due Diligence Matters
A vendor model can influence indicated rates even when an insurer did not build it and cannot inspect every detail of its implementation. Before relying on such a model, the insurer needs evidence about what data and methods it uses, how the vendor tests and updates it, and whether the insurer can use and review it under appropriate security and contractual controls. A polished report or a strong performance claim is not, by itself, sufficient evidence.
Vendor due diligence is an evidence-based assessment of whether a specific product, model version, and proposed use are acceptably supported. It is not a substitute for the insurer's own review of the model's suitability for its ratemaking purpose. Nor does it require assuming that all vendor information must be public: proprietary details may be reviewed under confidentiality, controlled access, or an independent review arrangement.
Define the Intended Use Before Reviewing Evidence
The first question is not simply whether the vendor is reputable. Specify the decision the model will support, the population and coverage involved, the role of its output in the rating process, and the consequences if the output is wrong or unavailable. A model used as one input to a controlled rating analysis may require different access and continuity arrangements from a model used to determine a consequential rating classification directly.
Use the intended use to set the review boundary. Identify the vendor product, model version, data snapshot or reference period, implementation or interface, and any insurer-specific configuration. Record material exclusions, such as a territory or class for which evidence is not available. Earlier material on data quality, model cards, independent review, and validation evidence provides useful review practices; vendor diligence connects those practices to information and rights held by an external party.
Write down the rating decision, covered population, model version, and how the output enters the process.
Ask for source descriptions, methodology and validation materials, release records, security evidence, and relevant contract terms.
Check that evidence identifies the same product, version, period, population, and use as the proposed deployment.
Record limitations, owners, due dates, and any conditions that must be met before reliance.
Six Evidence Areas
| Area | Evidence to seek | Questions for the reviewer |
|---|---|---|
| Data provenance | Source descriptions, collection periods, permitted-use statements, transformations, and lineage records | Can the vendor identify where the inputs came from, how they were prepared, and whether they represent the intended population and period? |
| Methodology documentation | Model purpose, inputs, output definitions, major processing steps, limitations, and reproducibility information | Can a competent reviewer understand how the output is produced and what the model is not designed to do? |
| Validation evidence | Test design, data scope, metrics, results, limitations, and independent review where available | Do the tests address the intended use, and are results tied to the version under review? |
| Update practices | Release history, change descriptions, version identifiers, notification terms, and rollback or continuity procedures | Can the insurer identify what changed and assess the change before relying on an updated version? |
| Security controls | Access management, data handling descriptions, incident processes, and relevant independent assurance | Are controls proportionate to the sensitivity of information and the service's access to it? |
| Contractual rights | Use scope, review and audit arrangements, notification duties, data handling obligations, and exit provisions | Can the insurer obtain enough information and cooperation to meet its governance responsibilities? |
Evidence should be specific enough to test. For example, a vendor statement that inputs are “industry data” does not establish source, period, coverage, or transformation. A validation summary that reports a favorable metric without describing the test population or metric definition does not establish relevance. Record what was reviewed, its date, the version it covers, and any limitation on access.
How to Evaluate the Evidence
Data provenance. Seek a lineage account from source through transformations to model inputs. Check whether the source population, time period, geographic scope, and exposure or claim definitions are compatible with the intended use. Determine whether the vendor has rights to use the data and whether those rights permit the proposed service. Do not infer that data are representative simply because they are large or come from multiple contributors.
Methodology documentation. The documentation should explain the model's purpose, required inputs, output meaning and units, processing stages at a reviewable level, known limitations, and dependencies. Proprietary restrictions do not necessarily make a model unusable, but they can limit what the insurer can verify. Where detailed disclosure is unavailable, consider whether controlled review by an independent party or other contractual evidence can address the specific gap.
Validation evidence. Check the validation design before interpreting reported results. Ask what population and period were tested, whether the test data were separate from data used to develop or tune the model, how the metrics are defined, and what important segments or failure modes were not assessed. Evidence is strongest when the test is relevant to the intended use, reproducible enough to review, and clearly tied to the version being considered. A validation claim is not proof of future performance.
Update practices. Determine how versions are labeled, how the vendor records changes, how much notice is provided, and whether the insurer can defer adoption while reviewing a material change. Ask what happens if a release introduces a defect or the service becomes unavailable. A generic promise of regular updates is not a substitute for version-specific change information.
Security and contract. Review controls that apply to the actual information exchanged and service arrangement. The contract should make relevant responsibilities and rights usable in practice: for example, what data may be processed, who may access it, how incidents are communicated, what information the vendor must provide, and what happens on termination. Deidentification and controlled data access can reduce exposure, but do not by themselves settle questions of permitted use, security, or model review.
A Structured Decision Framework
A simple scoring framework can help organize evidence, but it is only a prioritization aid. It cannot establish that a model is accurate, make incompatible evidence relevant, or override a mandatory control. Define the scale and thresholds before scoring, and apply any minimum requirements as gates rather than allowing a high score elsewhere to compensate for them.
The weighted average follows from expressing each area as a contribution to a common 0-to-4 scale. Multiplying its rating by its relative importance gives that area's contribution; adding contributions gives the overall summary. Because the weights sum to 1, if every area has the same rating \(r\), then \(S=r\sum_jw_j=r\). This explains the arithmetic, not the choice of weights: those require a documented rationale based on intended use and risk.
Worked Example: Combining Evidence Ratings Without Hiding a Gate
A hypothetical insurer is assessing a vendor model used as one input to a commercial property rating review. The team has set weights of 0.20 for provenance, 0.20 for methodology, 0.20 for validation, 0.15 for updates, 0.15 for security, and 0.10 for contract. Its internal illustrative rule is that a weighted score of at least 3.0 is needed for conditional acceptance, but any unresolved critical security requirement blocks use.
Method: weighted evidence summary plus a separate hard-stop test. The documented ratings, each on the 0-to-4 scale, are 3, 3, 2, 3, 1, and 3 in the order listed. The weighted contributions are:
The contributions are 0.60, 0.60, 0.40, 0.45, 0.15, and 0.30. Their sum is \(S=2.50\). Thus the score is below the stated 3.0 threshold. In addition, the security rating of 1 reflects an unresolved critical requirement, so the hard-stop condition independently prevents use.
The decision is do not rely on the model yet. The insurer can request evidence or remediation addressing the security gap and the weak validation support, then reassess. The common mistake is to average the ratings and treat the score as permission to proceed. Here that would be wrong twice: the score does not meet the internal threshold, and a hard stop cannot be offset by stronger ratings in other areas.
Worked Review Examples
Worked Example: Checking Whether Validation Evidence Matches the Proposed Use
A hypothetical vendor offers a model that estimates expected annual claim cost for a regional equipment policy rating review. Its report describes a holdout test of 8,000 records from an earlier product version. The proposed use covers 12,000 policies in a different region. The report supplies a mean absolute error of $420 per policy but does not define whether the error is measured on annual cost, per claim, or at another unit. The vendor also provides a version identifier, but the insurer cannot confirm that the tested version is the version being offered.
Method: scope-and-measure alignment review. First, compare the test and proposed populations: the test is 8,000 records from an earlier version and another region, while the proposed application is a 12,000-policy population in a different region. This is a material scope difference requiring explanation or additional evidence; the record count alone does not resolve it.
Second, compare the metric definition with the intended output. A value of $420 is not interpretable for this decision until the vendor specifies the unit, the prediction horizon, the comparison target, and how records were weighted or aggregated. Third, compare version identifiers. Since the tested and offered versions are not confirmed to match, the result cannot yet be attributed to the offered version.
Conclusion. The report is evidence that a test was performed, but it is insufficient to establish validation for this intended use. Request the metric definition, population details, and version linkage; then assess whether the test is sufficiently relevant or whether further testing is needed. The common mistake is to regard a numerical error metric and a large test sample as sufficient validation. Neither fixes unclear units, population mismatch, or version uncertainty.
Worked Example: Assessing Update Notice Against Review Time
A hypothetical insurer's internal procedure requires 30 calendar days to review a material vendor model change before adoption. The vendor contract provides 20 days' advance notice. For the next release, the vendor sent notice 24 days before its proposed effective date. Assume the insurer's procedure and the contract are measured in calendar days, and no exception has been approved.
Method: compare the required review interval with the available interval. The shortfall is \(30-24=6\) days. The contract's general notice term is also \(30-20=10\) days shorter than the insurer's requirement. The release notice therefore does not provide the required review period, even though it exceeds the contract's 20-day minimum by \(24-20=4\) days.
The insurer should not treat the vendor's contractual compliance as equivalent to satisfying its own review procedure. It should seek a delayed effective date, an approved exception, or another documented arrangement that supplies the necessary review time before adoption. The common mistake is to compare 24 days only with the 20-day contract term and conclude that the change is ready for use. That calculation answers whether the notice meets the contract's stated minimum, not whether the insurer has completed its required review.
Worked Example: Identifying a Contractual Evidence Gap
A hypothetical vendor will process deidentified policy records for a rating support service. The proposed agreement permits service delivery but is silent on model documentation access, notice of material version changes, and return or deletion of the insurer's records at termination. The vendor offers an annual summary report but no other review mechanism.
Method: map each governance need to a usable right. For documentation access, the annual report does not establish that the insurer can review methods or limitations for the relevant version. For version changes, no notice obligation is stated, so the insurer may not learn of a material change in time to assess it. For termination, the agreement does not specify how records are returned or deleted. Each is an identified gap; deidentification does not fill any of these contractual gaps.
Before relying on the service, the insurer should seek terms that specify suitable documentation or controlled review access, advance notice and version information for relevant changes, and an agreed process for data return or deletion consistent with applicable obligations. If a requested right cannot be obtained, document the resulting restriction and decide whether an alternative control genuinely addresses it. The common mistake is to treat the annual report or deidentified inputs as a complete substitute for review, change-notice, and exit rights.
Document the Decision and Its Limits
A due diligence record should make the conclusion reproducible by another reviewer. At minimum, preserve the intended use, product and version identifiers, evidence reviewed, scope limitations, ratings and their rationale if a score is used, unresolved findings, decision owner, conditions for use, and date or trigger for reassessment. Distinguish a missing document from an adverse finding: “not provided” does not prove a control is absent, but it does mean that the control has not been evidenced to the reviewer.
Use conditions should be precise and monitorable. For example, “use only for the reviewed product and version” is more actionable than “use cautiously.” A condition such as “obtain and review version-specific validation documentation before adopting a material update” identifies both an action and a trigger. Escalate a gap when its consequence is significant or when the insurer cannot obtain information needed to perform its responsibilities.
Vendor due diligence is an ongoing relationship, not a one-time questionnaire. Revisit the assessment when the intended use changes, the vendor changes a material model component or data source, a security or service incident occurs, or contract terms change. The precise reassessment trigger should reflect the insurer's governance process; a new release should not be assumed equivalent merely because it comes from the same vendor.
Common Exam Traps and Practical Takeaways
- Confusing a vendor claim with evidence. Ask for the test design, scope, version, and metric definition behind a performance statement.
- Scoring away a critical gap. Keep mandatory requirements as explicit gates rather than allowing a weighted average to conceal them.
- Ignoring version linkage. A test of one version does not automatically support another version.
- Equating contract compliance with internal readiness. A notice period can meet a contract and still be too short for the insurer's required review.
- Treating proprietary status as either automatic rejection or automatic exemption. Consider controlled review and other evidence, but document what cannot be verified.
- Assuming deidentification resolves every concern. It does not establish data provenance, model validity, security adequacy, or contractual access rights.
Check Your Understanding
For each hypothetical case, identify the due diligence issue and the defensible next step.
- A vendor reports a favorable test on one region but provides no information about the proposed region. What evidence gap matters most? (Answer: The validation population may not match the intended use; request region-specific support or document and assess the limitation before relying on the result.)
- A contract guarantees 15 days' notice, while the insurer's documented review process requires 25 calendar days. A release is announced 18 days before adoption. How many additional days are needed to meet the procedure? (Answer: 7 days, since \(25-18=7\).)
- A due diligence team assigns ratings 4, 3, and 2 to three areas with respective weights 0.50, 0.30, and 0.20. Calculate the weighted score. (Answer: \(0.50(4)+0.30(3)+0.20(2)=2.00+0.90+0.40=3.30\).)
- A vendor supplies a test metric but does not identify its unit or tested version. Is the metric sufficient to support the offered version? State the key reason. (Answer: No; the metric cannot be interpreted without its unit and cannot be linked to the offered version without version identification.)
- A contract permits processing deidentified records but says nothing about deletion at termination or notice of material model changes. Name both unresolved controls. (Answer: The agreement lacks a defined data return or deletion process and a material-change notification obligation.)