Why Scenario Estimates and Loss Data Can Disagree
Operational risk estimates often draw on two different kinds of evidence. Loss data record events that have occurred and been captured by an organization’s reporting process. Scenario estimates describe plausible events, often including severe events that have not occurred in the observed period. A difference between the two estimates is a reason to investigate, not by itself proof that either is wrong.
The central diagnostic question is whether both estimates refer to the same quantity. A reported event count above a threshold cannot be compared directly with a scenario frequency for all events. A historical loss total under an old operating footprint is not automatically a current annual estimate. Nor is an estimate for a severe, multi-unit incident necessarily comparable with the average of routine losses.
The loss basis matters too. “Gross” may mean before insurance or other recoveries; “net” may mean after specified recoveries. Estimates can also differ because one includes allocated expenses, multiple losses from a single incident, or losses outside the data collection system. These choices must be made explicit rather than inferred from a label.
A Structured Reconciliation
A practical diagnosis separates definition differences from estimation differences. The sequence below is useful when an observed estimate and an expert scenario appear inconsistent.
Specify what counts as one event, the covered process or business activity, the loss components included, and whether amounts are gross or net.
Identify the relevant exposure unit, such as transactions, system-years, or operating units, and put both estimates on a common time horizon and current operating basis.
Determine whether losses below a threshold, events reported through another channel, or events outside a collection period are missing from the data.
Clarify the scenario’s frequency, severity, controls, dependencies, and interpretation. Check whether it represents an ordinary event, a tail event, or a stress condition.
Ask whether the data or model can represent the scenario at all, given the sample size, event definitions, tail extrapolation, and operating changes.
This process does not require the estimates to become equal. It determines whether the comparison is meaningful, identifies the assumptions driving any remaining difference, and records where judgment is needed.
Reporting Thresholds and Event Definitions
Suppose losses are modeled as individual events with frequency rate \(\lambda\) per exposure period. Let \(X\) be the loss amount for an event, and let \(c\) be a reporting threshold. Assume, for this calculation, that every event with \(X \ge c\) is captured and that the loss amount distribution is the same for captured and uncaptured events. The expected reported event rate is then the underlying rate multiplied by the probability an event crosses the threshold.
This relation follows by conditioning on whether an event is captured. In a small interval with expected event count \(\lambda\), each event has probability \(P(X\ge c)\) of being reported. The expected number reported is therefore the expected number of events times that probability. Equivalently, if the event count is Poisson and capture decisions are independent across events, thresholding produces a Poisson reported count with rate \(\lambda P(X\ge c)\). The formula does not correct for missed reports above the threshold, changing thresholds, or a threshold that depends on event type; those require additional information.
Worked Example: Reconciling a Thresholded Event Count
A hypothetical scenario estimate is 0.20 events per year for losses meeting a specified event definition. Internal reporting captures only losses of at least $30,000. Experts estimate that 40% of events under that definition reach the threshold. The data contain 8 reported events over 100 comparable exposure-years. Determine whether the reported rate is inconsistent with the scenario estimate after allowing for the threshold.
Method. Convert the scenario rate to a rate of threshold-crossing events, then compare it with the observed reported rate. Assume a stable definition and exposure basis, complete capture above $30,000, and the stated 0.40 capture probability.
The scenario-based rate of reported events is
The observed rate is
The two rates agree on this basis. This agreement is a reconciliation of point estimates, not proof that the scenario is correct: sampling variation and uncertainty in the 0.40 capture probability remain.
Common mistake avoided. Comparing 0.20 directly with 0.08 treats the thresholded data as if they included every event. The correction is to apply the capture probability to the scenario rate before comparing.
The event boundary can be just as consequential as the threshold. For example, a scenario may count one system outage affecting many customers as one event, while a database records each customer claim separately. Comparing the raw counts would mix different units. The analyst should establish a consistent event rule, then identify whether records need to be grouped or whether the scenario needs to be restated on the data’s unit.
Exposure Changes and Control Assumptions
Historical counts describe the exposure that existed during the observation period. If the number of operating units, transactions, or system-years changes, a raw count comparison can mislead. A rate per exposure unit is useful only when the exposure unit is defined consistently and the rate is reasonably transferable. Changes in controls or operating processes may require a separate adjustment; they should not be hidden inside an unexplained scaling factor.
Worked Example: Adjusting Frequency for Current Exposure
A hypothetical service has 6 operational events over 300 unit-years. It now operates 500 comparable units. A scenario assessment assumes that a recent control degradation increases event frequency by 40% relative to the historical rate. Estimate the annual event count under the scenario assumptions.
Method. Calculate the historical rate per unit-year, project it to current exposure, and then apply the stated control multiplier. Assume the historical events and current units use the same event definition and that the underlying rate is transferable before the control adjustment.
The historical rate is
At 500 units, the count implied by the unchanged rate is
A 40% increase corresponds to a multiplier of \(1+0.40=1.40\). The adjusted estimate is
The result is 14 events per year under the stated assumptions. It is not a claim that the control change has been measured precisely; it is the consequence of the scenario’s specified multiplier.
Common mistake avoided. Dividing 6 events by 10 calendar years gives 0.6 events per year, but that rate is for the historical operation, not for 500 current units. Multiplying 0.6 by 500 would produce 300 events per year and would mix a corporate-year rate with a unit count. The unit-year rate is the compatible quantity.
The control adjustment also needs a clear direction and scope. If a scenario already describes the current degraded-control environment, applying an additional 1.40 multiplier would count the degradation twice. Conversely, applying a historical rate without adjustment may understate frequency if a material control deterioration is not represented in the historical data.
Severe Scenarios and Sparse Data
A severe scenario may describe a loss that is absent from a short historical record. Absence is informative only in relation to the scenario’s assumed frequency and the length and scope of the observation period. It does not establish that the event is impossible. A data model fitted to routine events may also fail to represent a common-cause incident or a large-scale system failure, even if its average fit to routine losses is acceptable.
Worked Example: Interpreting a Severe Scenario With No Historical Event
A hypothetical organization has observed six routine operational losses over 10 years, with total losses of $360,000. Each was below $100,000. Experts separately estimate a 5% annual probability of a distinct, severe incident causing a $400,000 loss. Assess whether zero such severe incidents in the 10-year record contradicts the scenario, and calculate the scenario’s annual expected loss contribution.
Method. Keep the routine-loss estimate separate from the severe-event scenario. Assume the severe event has the stated 5% probability independently in each year, its loss is $400,000 if it occurs and zero otherwise, and the observed routine events do not include this severe-event category.
The routine losses have an observed average annual amount of
The severe scenario’s annual expected loss contribution is
Under the stated assumption that the severe category is distinct from the routine losses, the combined expected annual amount is
The probability of observing no severe incident in 10 independent years is
Thus, zero severe incidents over the period is not surprising under the scenario: the probability is 0.5987. The data’s routine-loss average and the severe-scenario contribution describe different parts of the loss picture, so comparing $36,000 directly with $20,000 as competing estimates would be misleading.
Common mistake avoided. Treating “no observed severe event” as evidence that its annual probability is zero ignores the finite observation period. Also, adding the two estimates is justified here only because the problem defines the severe category as distinct from the observed routine category; overlapping categories could double count.
A scenario may also exceed the data because experts assume different controls, dependencies, or event aggregation than those embedded in the historical model. The appropriate response is to make those differences explicit and test their consequences. It is not appropriate to force a scenario into a data distribution that cannot represent its event mechanism simply to obtain a closer numerical match.
Interpreting the Remaining Difference
After definitions and bases are aligned, a residual disagreement may still remain. It can reflect sampling variability, uncertainty in expert judgment, changing operations, or limitations in the data and model. Record which explanation is supported and which remains uncertain. Where a parameter or assumption is decisive, show how the estimate changes when it is varied rather than presenting a single point estimate as definitive.
| Diagnostic finding | What to check |
|---|---|
| Scenario frequency exceeds reported frequency | Event definition, exposure units, reporting threshold, and capture completeness |
| Scenario severity exceeds observed losses | Whether the scenario is a tail event, whether losses are gross or net, and whether the data cover comparable events |
| Historical rate differs from the current scenario | Changes in exposure, controls, processes, or the operating environment |
| Two estimates agree after adjustment | Whether the adjustment is independently supported; agreement alone does not validate the inputs |
A useful conclusion identifies the common comparison basis, the adjustment made, the assumptions that remain, and the resulting limitation. “The estimates differ” is not yet a diagnosis; “the scenario includes events below the internal reporting threshold, while the observed rate counts only threshold-crossing events” is a testable explanation.
Check Your Understanding
For each problem, identify the appropriate comparison and calculate the requested quantity.
- A hypothetical scenario estimates 0.30 events per exposure-year. A reporting threshold captures 25% of these events. Over 120 comparable exposure-years, 9 events are reported. Calculate the scenario-implied reported rate and the observed rate. Are their point estimates aligned? (Answer: Scenario-implied rate = 0.075 per exposure-year; observed rate = 9/120 = 0.075 per exposure-year; aligned.)
- A hypothetical operation records 12 events over 400 unit-years. Current exposure is 600 units, and a scenario assumes a 25% frequency increase over the historical rate. Estimate the current annual event count. (Answer: Historical rate = 12/400 = 0.03 per unit-year; unchanged-rate count = 0.03(600) = 18; adjusted count = 18(1.25) = 22.5 events per year.)
- A reporting system captures half of qualifying events. It records 3 events over 5 exposure-years. Assuming complete capture of qualifying events and a stable capture probability, estimate the underlying event rate. (Answer: Reported rate = 3/5 = 0.6 per exposure-year; underlying rate = 0.6/0.5 = 1.2 events per exposure-year.)
- A hypothetical severe scenario has an 8% independent annual probability and a $250,000 loss if it occurs. Calculate its annual expected loss and the probability of no such event in five years. (Answer: Annual expected loss = 0.08($250,000) = $20,000; probability of no event = 0.92^5 = 0.6591.)
- A data-based estimate of routine losses is $50,000 per year. A distinct scenario category has a 4% annual probability of a $250,000 loss. Assuming the categories do not overlap, calculate the combined expected annual loss. State why the same addition would need review if overlap were possible. (Answer: Scenario contribution = 0.04($250,000) = $10,000; combined estimate = $50,000 + $10,000 = $60,000 per year; possible overlap could double count losses.)