Challenging Vendor Assumptions and Uncertainty

Why Vendor Assumptions Need a Challenge

A vendor model can provide useful structure, data, and analytical capacity, but its output is conditional on choices: which events are represented, how often they occur, how losses respond to those events, and what exposures and coverage terms enter the calculation. A modeled result is therefore not a fact independent of assumptions. It is an estimate produced under a specified set of assumptions and data.

A sound challenge does not mean rejecting a vendor result whenever it differs from internal experience. Internal data may be sparse, affected by changing exposure or coverage, or silent about events that have not occurred. Nor does a close match prove that the model is reliable: different assumptions can offset one another. The task is to make comparisons meaningful, identify which assumptions drive material differences, and state what the evidence supports.

Definition: A vendor-assumption challenge is a documented evaluation of the assumptions, inputs, and modeled results supplied by a vendor against relevant internal experience, alternative scenarios, and external benchmarks. A like-for-like comparison places the sources on a sufficiently consistent basis—including exposure, coverage, loss measure, time period, and dollar level—to interpret their differences. A limitation on reliance is a specific condition, such as sparse data or a narrow modeled scope, that restricts how a result should be used or interpreted.

Build a Comparable Evidence Set

Before comparing numbers, identify exactly what each number measures. For example, annual expected loss is not interchangeable with a single-year realized loss, a stress scenario, or a premium indication that includes expenses and risk loads. Record the measure, population, coverage terms, period, and dollar basis for each source. If those cannot be aligned, describe the mismatch and treat the comparison as qualitative or partial.

A useful challenge considers three kinds of evidence. Internal experience can test whether modeled results are broadly consistent with observed losses after relevant adjustments. Alternative scenarios reveal how sensitive the output is to assumptions that are uncertain or disputed. External benchmarks provide context, but may differ in geography, coverage, data quality, model purpose, or exposure mix. None is automatically a definitive answer.

Evidence source	Useful challenge	Important limitation to document
Internal experience	Does modeled loss appear plausible relative to normalized observed experience?	Limited observation period, changing exposure, loss development, or incomplete coverage alignment
Alternative scenario	How much does the result change when a key assumption changes?	Scenario values may not have assigned probabilities and are not necessarily confidence bounds
External benchmark	Is the result in a plausible range relative to a relevant independent reference?	Benchmark scope, methods, and population may not match the modeled portfolio

Key principle: A difference is a signal to investigate, not by itself proof that one source is right. Explain the basis of comparison, the likely causes of the difference, and the uncertainty that remains.

Expected Loss and Assumption Sensitivity

For a simple event model, let $N$ be the number of loss events in a year, with $E[N]=\lambda$. Suppose each event has loss $X_j$, with expected severity $E[X_j]=m$, and the annual aggregate loss is $L=\sum_{j=1}^{N}X_j$. If event severities have the same mean and are independent of the event count, expected annual loss is $\lambda m$. This compact relationship helps isolate the effect of frequency and severity assumptions. More complex vendor models may require richer calculations, but the same discipline applies: identify which inputs generate the output.

E[L]=E\!\left[E[L\mid N]\right] =E[Nm] =\lambda m

The justification is conditional expectation. Given $N=n$, the aggregate loss has expected value $nm$, because it is the sum of $n$ event losses, each with mean $m$. Averaging $nm$ over the distribution of $N$ gives $E[N]m=\lambda m$. This result requires the stated common-mean and independence conditions. If severity depends on event count, or the model includes event-specific terms that change with conditions, the simple product may not represent the vendor calculation.

When frequency changes by a factor $a$ and severity by a factor $b$, the product model changes expected loss by a factor $ab$, not generally by the sum of the two percentage changes. This interaction matters when testing combined scenarios.

Worked Example: Comparing a Vendor Estimate With Internal Experience

A hypothetical insurer is reviewing vendor-modeled annual catastrophe loss for a portfolio of 3,600 policies. The vendor estimates an event frequency of 0.06 events per policy-year and mean loss of $24,000 per event. Internal records contain $25,920,000 of catastrophe loss over 21,600 policy-years. Assume, solely for this comparison, that the internal losses have been adjusted to the same coverage, exposure mix, and dollar level as the vendor estimate. Compare the results and state an appropriate limitation.

Method. Calculate each source’s annual expected loss on a per-policy-year basis, then scale the vendor estimate to the portfolio. The internal total is observed experience; it is not assumed to be a complete estimate of future catastrophe risk.

Vendor estimate per policy-year. Under the simple frequency-severity model:

0.06 \times \$24{,}000=\$1{,}440

For 3,600 policies, the vendor portfolio estimate is $3{,}600\times \$1{,}440=\$5{,}184{,}000$ per year.

Internal experience per policy-year. The observed total divided by the exposure is:

\frac{\$25{,}920{,}000}{21{,}600} =\$1{,}200

That is $1,200 per policy-year. On 3,600 policy-years, this rate corresponds to $3{,}600\times \$1{,}200=\$4{,}320{,}000$.

Difference. The vendor estimate exceeds the internal rate by $\$1{,}440-\$1{,}200=\$240$ per policy-year. Relative to internal experience, the difference is $\$240/\$1{,}200=0.20$, or 20%. The portfolio difference is $\$5{,}184{,}000-\$4{,}320{,}000=\$864{,}000$, also 20% of the internal-based amount.

Interpretation and limitation. The comparison identifies a 20% difference on the stated common basis; it does not establish that the vendor has overstated loss. The internal period may not include rare severe events, and its experience may be statistically unstable. The analyst should investigate the event assumptions and exposure alignment before drawing a conclusion.

Common mistake. Dividing the $240 difference by the vendor rate gives 16.7% after rounding, but that answers a different question: the shortfall of internal experience relative to the vendor estimate. For the stated question—how much higher the vendor estimate is than internal experience—the denominator is $1,200, so the answer is 20%.

Test Assumptions With Scenarios

A scenario is a deliberate change to one or more assumptions, with the other inputs held fixed unless the scenario explicitly changes them too. It helps identify sensitivity and supports a discussion of uncertainty. A scenario is not a probability statement unless probabilities have been assigned and justified. In particular, a low and high scenario should not be called a confidence interval merely because it brackets a central estimate.

Worked Example: Separating Frequency and Severity Effects

A hypothetical vendor model estimates annual loss using frequency of 0.08 events per year and mean severity of $30,000,000 per event. The analyst tests a higher-frequency scenario that increases frequency by 25%, a higher-severity scenario that increases severity by 10%, and a combined scenario applying both changes. Calculate all four expected losses and explain what the combined change means.

Method. Apply $E[L]=\lambda m$ to the baseline and each scenario, changing only the stated assumption in each single-factor test.

Baseline. $0.08\times \$30{,}000{,}000=\$2{,}400{,}000$.

Higher frequency only. The revised frequency is $0.08\times1.25=0.10$. Expected loss is $0.10\times \$30{,}000{,}000=\$3{,}000{,}000$, a 25% increase from baseline.

Higher severity only. The revised severity is $\$30{,}000{,}000\times1.10=\$33{,}000{,}000$. Expected loss is $0.08\times \$33{,}000{,}000=\$2{,}640{,}000$, a 10% increase.

Combined scenario. Applying both changes gives $0.10\times \$33{,}000{,}000=\$3{,}300{,}000$. Relative to baseline, the increase is $\$900{,}000/\$2{,}400{,}000=37.5\%$. Equivalently, the combined multiplier is $1.25\times1.10=1.375$.

Interpretation and limitation. The combined scenario is 37.5% above baseline, not 35%, because the severity increase also applies to the increased frequency. The result quantifies sensitivity under the specified assumptions; it does not say that the combined scenario is 37.5% likely, or that the actual loss must fall within any range formed by these scenarios.

Common mistake. Adding 25% and 10% to report a 35% combined increase ignores the interaction. Multiplying the factors gives $1.25\times1.10=1.375$, so the correct increase is 37.5%.

Use External Benchmarks With Care

An external benchmark can be a useful reasonableness check when it is genuinely relevant. Before interpreting it, ask whether it measures the same kind of loss, uses a comparable exposure base, reflects similar coverage and geography, and represents a compatible time and dollar basis. If not, quantify only what can be compared and disclose the mismatch. Do not average a vendor estimate, internal experience, and an external benchmark simply because all three are numerical estimates of a related quantity.

Worked Example: Reconciling Three Sources on an Exposure Basis

A hypothetical portfolio has 2,500 exposure units for the coming year. The vendor estimates total annual loss of $510,000. Internal adjusted losses are $3,600,000 over 20,000 exposure-years. An external reference indicates $196 per exposure-year. Assume all three sources refer to the same loss measure and dollar level, but the external reference has a broader population than this portfolio. Compare the amounts and explain how to use the benchmark.

Method. Convert each source to a rate per exposure-year, then apply the internal and external rates to the target exposure. This makes the arithmetic comparable without asserting that the broader benchmark is portfolio-specific.

Vendor rate. $\$510{,}000/2{,}500=\$204$ per exposure-year.

Internal rate and projected amount. The internal rate is $\$3{,}600{,}000/20{,}000=\$180$ per exposure-year. Applied to 2,500 exposure units, it gives $2{,}500\times \$180=\$450{,}000$.

External benchmark amount. Applied to the same exposure count, the reference gives $2{,}500\times \$196=\$490{,}000$.

Compare. The vendor amount is $\$510{,}000-\$450{,}000=\$60{,}000$, or $\$60{,}000/\$450{,}000=13.3\%$ above the internal-based amount. It is $\$20{,}000/\$490{,}000=4.1\%$ above the external-based amount. The external estimate is $40,000 above the internal-based amount.

Interpretation and limitation. The benchmark provides context: the vendor rate of $204 is relatively close to the broader reference rate of $196, while both exceed the internal rate of $180. The difference could reflect portfolio mix, internal experience variability, or model assumptions. Because the external population is broader, its proximity to the vendor estimate does not independently validate the vendor result. The comparison supports further inquiry, not automatic adoption or averaging.

Common mistake. Comparing the vendor’s $510,000 total directly with the internal $3,600,000 total would be misleading because the internal figure covers 20,000 exposure-years, not 2,500. Converting to a common exposure basis first yields the relevant internal projection of $450,000.

Document Uncertainty and Limits on Reliance

A useful conclusion distinguishes evidence, interpretation, and unresolved questions. A concise record should identify the vendor result and its version or source, the assumptions and data challenged, the comparison basis, calculations performed, findings, and limitations. If the result is sensitive to an assumption that cannot be independently verified, say so directly. If the evidence is thin, describe the reason—such as a short observation period or a different benchmark population—rather than using a generic statement that “uncertainty exists.”

Define the quantity.
State whether the comparison concerns annual expected loss, an observed loss total, or another measure, and specify the exposure and coverage basis.

Align sources where possible.
Normalize period, exposure, scope, and dollar level. Record any mismatch that cannot be resolved rather than implying a perfect comparison.

Test important assumptions.
Change a clearly identified input, calculate the resulting output, and separate single-factor sensitivity from combined scenarios.

Interpret the evidence proportionately.
Explain whether the sources agree, differ, or are not directly comparable. Avoid treating a benchmark as proof or a scenario as a probability range.

State appropriate reliance.
Record what the result can support and which uncertainties or data limitations prevent stronger conclusions.

The distinction between uncertainty and error is important. A sensitivity result shows how output responds to selected assumptions; it does not establish that the assumptions are wrong. A discrepancy may reveal a data or scope mismatch, a difference in model structure, or ordinary variability in observed experience. The analyst’s conclusion should identify plausible explanations and any evidence needed to distinguish them, rather than claiming more certainty than the comparison provides.

Key takeaway: Challenge vendor results by comparing defined quantities on a consistent basis, testing the assumptions that drive them, and documenting what remains uncertain. A well-supported limitation is more informative than an unsupported declaration that a model is either reliable or unreliable.

Check Your Understanding

For each problem, calculate the requested result and keep the stated comparison basis in view.

A hypothetical model uses frequency of 0.04 events per policy-year and mean severity of $50,000. It is applied to 2,000 policy-years. Calculate the expected annual loss. (Answer: $4,000,000.)
Internal adjusted losses are $1,260,000 over 9,000 exposure-years. A vendor estimates $161 per exposure-year on the same basis. Calculate the internal rate and the vendor’s percentage excess relative to internal experience. (Answer: $140 per exposure-year; 15.0%.)
A vendor baseline has frequency 0.12 and severity $18,000,000. A scenario raises frequency by 10% and reduces severity by 5%. Calculate baseline and scenario expected losses and the percentage change. (Answer: baseline $2,160,000; scenario $2,257,200; increase 4.5%.)
A portfolio has 2,500 exposure units. Internal adjusted losses are $3,600,000 over 20,000 exposure-years, a vendor estimate is $510,000, and a broader external benchmark is $196 per exposure-year. Calculate the internal-based and benchmark-based amounts for the target portfolio and the vendor excess over the internal-based amount. (Answer: $450,000; $490,000; $60,000, or 13.3%. The broader benchmark is context, not portfolio-specific validation.)
A baseline modeled annual loss is $1,500,000. A scenario decreases frequency by 20% and increases severity by 15%. Calculate the scenario loss and state whether the percentage change alone identifies its probability. (Answer: $1,380,000, an 8.0% decrease; no, the scenario does not specify a probability.)

What You'll Learn