Why Scenario Elicitation Requires Structure
Internal loss records can be sparse precisely where operational risk matters most: severe events may be rare, unfamiliar, or altered by changes in systems and controls. Expert elicitation provides a disciplined way to describe such events and record informed judgments about them. It does not turn judgment into observed data. Its value depends on making the event and the basis for each judgment clear enough to challenge, compare, and update.
A scenario should not be a vague label such as “cyber event” or “processing failure.” It should describe a plausible chain from cause to loss, specify what is included and excluded, and distinguish how often the event might occur from how large the resulting loss might be. The earlier material on operational loss event units and model boundaries helps frame the unit being described; elicitation adds a structured account of what experts believe about a severe event within that boundary.
A useful narrative identifies the affected process, the initiating conditions, the failure or event, the resulting consequences, and the period and organizational scope being considered. It also states exclusions. For example, a payment-system outage might include unrecovered customer remediation and transaction-processing costs, while excluding losses assigned to a separate scenario for malicious credential use. Without explicit boundaries, experts may answer different questions while appearing to disagree about the same one.
Separate the Event, Frequency, and Severity Judgments
Frequency judgments concern how often a defined event occurs over a stated exposure period. They may be expressed as a judgment about the probability of at least one occurrence, or as judgments across event-count categories. Severity judgments concern the loss conditional on the event occurring. If experts are asked for both in one question, they may mix “how likely” with “how bad,” making their answers difficult to interpret.
For a binary annual event \(A\), \(P(A)\) is the probability that the event occurs at least once during the year. For a count \(N\), a frequency elicitation may instead distinguish \(N=0\), \(N=1\), and \(N\geq 2\). A severity judgment must specify its conditioning event—for example, loss conditional on a defined event occurring—and the loss components in scope. These descriptions do not by themselves prescribe a particular probability or loss model.
When experts give ranges or quantile judgments, record what each endpoint means and what assumptions the expert used. A range may represent a plausible interval, a confidence interval in the expert’s own judgment, or simply a low-to-high assessment; these are not interchangeable. Likewise, a “central estimate” is not automatically a mean, median, or percentile unless the elicitation question defines it that way.
Check Event Logic Before Combining Judgments
Probability judgments must be compatible with the event narrative. If event \(B\) is defined as occurring only as a consequence of event \(A\), then \(B\) is a subset of \(A\). Therefore, \(P(B\mid A^c)=0\), and \(P(B)=P(A)P(B\mid A)\). This is not a modeling preference; it follows from the stated event definitions.
The identity follows by partitioning \(B\) into the mutually exclusive cases \(B\cap A\) and \(B\cap A^c\). Their probabilities are \(P(B\mid A)P(A)\) and \(P(B\mid A^c)P(A^c)\), respectively; adding them gives \(P(B)\). When \(B\) can happen only if \(A\) happens, the second term is zero. Elicitation notes should preserve that logical restriction rather than forcing inconsistent marginal and conditional judgments to coexist.
Worked Example: Checking a Conditional Event Judgment
A hypothetical organization defines \(A\) as at least one compromise of a specified payment-vendor credential during a calendar year. It defines \(B\) as at least one fraudulent settlement instruction caused by that specified compromise during the same year. Experts judge \(P(A)=0.0400\) and \(P(B\mid A)=0.2500\). Find the implied annual probability of \(B\), and check the event-table probabilities.
Method. First use the event definition to establish that \(B\) can occur only when \(A\) occurs. Thus \(P(B\mid A^c)=0\). Then apply the law of total probability:
The complementary probability is \(P(A^c)=1-0.0400=0.9600\). The four joint cells are:
Check. The cells sum to \(0.0100+0.0300+0+0.9600=1.0000\). Also, \(P(B)=P(A\cap B)+P(A^c\cap B)=0.0100+0=0.0100\), as required.
Common mistake avoided. A marginal judgment of \(P(B)=0.0200\) would contradict the narrative and the conditional judgment: the definition requires \(P(B)=0.0100\). If experts intend \(B\) to include fraudulent instructions from other causes, they must broaden its definition before eliciting a nonzero \(P(B\mid A^c)\).
A Practical Elicitation Sequence
A structured session helps experts answer the same question and makes uncertainty visible. The following sequence is a useful governance discipline; it is not a formula for replacing professional judgment.
State the business process, organizational boundary, exposure period, and loss components. Identify adjacent events that are out of scope.
Describe initiating conditions, the operational failure or event, the path to consequences, and the assumptions needed for the chain to occur.
Elicit occurrence or count judgments separately from loss conditional on occurrence. State exactly what “low,” “central,” or “high” means if those terms are used.
Ask which assumptions drive disagreement, what evidence or experience informs each judgment, and whether another event shares a cause or control failure.
Preserve individual judgments, rationale, dissent, definitions, and revisions. Check logical constraints and ask experts to explain material differences before summarizing them.
A facilitator should avoid suggesting a preferred answer or allowing the most senior participant to anchor the room before others have considered the question. Independent initial judgments can expose differing interpretations; discussion can then clarify assumptions without erasing the record of disagreement. Documentation should distinguish facts, scenario assumptions, expert judgment, and unresolved uncertainty.
Frequency Judgments Across Event Counts
For a count-based elicitation, probabilities assigned to all count categories must be nonnegative and sum to one. An open-ended category such as “two or more” is useful for distinguishing no event, one event, and repeated events, but it does not specify how many events occur within that category. Consequently, those categories alone do not determine an expected event count.
Worked Example: Interpreting Elicited Frequency Categories
For a hypothetical service-reconciliation failure, experts assign the following probabilities over one year: \(P(N=0)=0.70\), \(P(N=1)=0.20\), and \(P(N\geq2)=0.10\), where \(N\) is the number of occurrences meeting the scenario definition. Find the probability of at least one occurrence and the probability of exactly one occurrence given at least one. State whether the expected count is determined.
Method. The categories are mutually exclusive and exhaustive. First check their total: \(0.70+0.20+0.10=1.00\). The probability of at least one occurrence is the sum of the last two categories:
The conditional probability of exactly one occurrence, given at least one, is:
The expected count is not determined. The probability \(0.10\) is assigned to the entire category \(N\geq2\), but the elicitation does not say how probability is distributed between 2, 3, or larger counts.
Check. The conditional probability of \(N\geq2\), given at least one, is \(0.10/0.30=1/3\); the two conditional categories sum to \(2/3+1/3=1\).
Common mistake avoided. Treating the \(N\geq2\) category as if it meant exactly two would add an unstated assumption. Record the category as open-ended unless experts explicitly provide further count detail.
Document Severity Uncertainty Without Hiding Disagreement
Experts may differ because they picture different event paths, use different loss boundaries, or rely on different evidence. A single blended number can conceal those distinctions. A useful record preserves each judgment and its rationale, then summarizes only in a way that matches the question asked. If several experts provide central estimates, their median is one possible descriptive summary; it does not establish that the experts share a common probability distribution or that the median is a mathematically defined severity parameter.
For every severity judgment, document the conditioning event, currency and price basis, included consequences, and assumptions about recovery or mitigation. Explain whether the stated amount is a central view, a selected percentile, or a plausible range. If experts use different interpretations, clarify the scenario rather than averaging incompatible answers.
Worked Example: Summarizing Expert Central Judgments
Three experts independently assess the loss, in millions of dollars, conditional on a hypothetical data-reconciliation event occurring under the same stated scope. Each gives a low, central, and high judgment. Their triplets are: Expert 1: \(2, 6, 15\); Expert 2: \(3, 7, 18\); Expert 3: \(4, 9, 24\). Find the median of the three central judgments and check whether the low, central, and high summaries remain ordered when each column is summarized by its median.
Method. Keep each expert’s three judgments in the same order and summarize each column separately. The central judgments are \(6,7,9\) million dollars; ordered, they are \(6,7,9\), so their median is \(7\) million dollars. The low judgments are \(2,3,4\), with median \(3\) million dollars. The high judgments are \(15,18,24\), with median \(18\) million dollars.
The column-wise summaries are therefore \(3,7,18\) million dollars. They preserve the required ordering:
Check. Each column has three values, so its median is the middle value after ordering. The median of the central judgments is \(7\), not \(6\); \(6\) is the smallest central judgment. The units remain millions of dollars throughout.
Common mistake avoided. Do not select the smallest central estimate as the group median. Also, label these results as descriptive medians of expert judgments—not as a fitted loss distribution or a claim that the low, central, and high figures have particular percentile meanings.
Dependence Belongs in the Scenario Narrative
Two operational consequences may be linked by a common cause, a shared system, or a control failure. Asking experts only for each event’s marginal probability can miss that connection. Elicitation should ask whether each event can occur without the proposed cause and, when relevant, how likely combinations are conditional on that cause. Independence is an assumption to be examined, not a default conclusion from having separate event names.
Worked Example: Eliciting Joint Consequences of a Common Cause
For a hypothetical annual scenario, let \(A\) be a shared routing-service outage, \(B\) a payment-processing disruption caused by that outage, and \(C\) a claims-file delay caused by that outage. Experts judge \(P(A)=0.0300\), \(P(B\mid A)=0.6000\), \(P(C\mid A)=0.4000\), and \(P(B\cap C\mid A)=0.3000\). By definition, neither \(B\) nor \(C\) occurs in this scenario unless \(A\) occurs. Find both marginal probabilities, the probability of both consequences, and the probability of at least one consequence. Check whether the joint conditional judgment is feasible.
Method. Since \(B\) and \(C\) are defined as consequences of \(A\), multiply their conditional probabilities by \(P(A)\):
The probability of both consequences is:
Within \(A\), the probability of at least one consequence is \(0.6000+0.4000-0.3000=0.7000\), using the addition rule for two events. Because neither consequence occurs in the stated scenario when \(A\) does not occur:
For feasibility, the joint conditional probability must satisfy \(\max(0,0.6000+0.4000-1)\leq P(B\cap C\mid A)\leq\min(0.6000,0.4000)\). Thus \(0\leq0.3000\leq0.4000\), so the judgment is feasible.
Check. The union also equals \(P(B)+P(C)-P(B\cap C)=0.0180+0.0120-0.0090=0.0210\). If \(B\) and \(C\) were conditionally independent given \(A\), their joint conditional probability would be \((0.6000)(0.4000)=0.2400\), not \(0.3000\). The elicited joint judgment therefore should not be replaced by conditional independence without a reasoned change in assumptions.
Common mistake avoided. Multiplying the two marginal probabilities \(0.0180\) and \(0.0120\) would impose unconditional independence, which conflicts with the shared-cause narrative and the given conditional joint judgment.
Governance, Challenge, and Common Traps
Scenario judgments are most useful when another reviewer can reconstruct the question that was asked and understand why an answer was given. The record should include the narrative version, event definitions, scope, elicitation date, participants’ relevant roles, individual judgments, discussion points, and any final summary. It should also identify evidence considered and distinguish it from assumptions or expert opinion. A later change in systems, controls, or business processes may invalidate a previous judgment even if the event label is unchanged.
- Vague event boundaries. Different experts may include different causes or loss components. Define the event and exclusions before asking for judgments.
- Mixing frequency with severity. A rare-event judgment and a conditional loss estimate answer different questions. Elicit and document them separately.
- Ignoring conditional logic. If \(B\) can only follow \(A\), a positive judgment for \(P(B\mid A^c)\) is inconsistent unless the definition of \(B\) changes.
- Assuming independence. Shared causes can make consequences dependent. Ask directly about common causes and joint occurrence.
- Overstating precision. A numerical judgment is not automatically precise evidence. Preserve ranges, rationale, and dissent rather than implying certainty from a decimal place.
- Turning categories into unsupported detail. A judgment for “two or more” events does not say how probability divides among higher counts.
Check Your Understanding
For each problem, state any event-definition issue before calculating.
- In a hypothetical annual scenario, \(A\) is a specified database-access failure with \(P(A)=0.0500\). Event \(B\) is a fraudulent record change caused only by that failure, and \(P(B\mid A)=0.2000\). Find \(P(B)\) and \(P(B\mid A^c)\). (Answer: \(P(B)=0.0100\); \(P(B\mid A^c)=0\).)
- Experts assign \(P(N=0)=0.60\), \(P(N=1)=0.25\), and \(P(N\geq2)=0.15\) for a defined event over one year. Find \(P(N\geq1)\) and \(P(N=1\mid N\geq1)\). (Answer: \(0.40\); \(0.6250\).)
- Three experts give central severity judgments of 5, 8, and 14 million dollars for the same hypothetical event and scope. What is the median central judgment, and what does it not establish by itself? (Answer: 8 million dollars; it does not establish a fitted severity distribution or a percentile interpretation.)
- Let \(A\) be a hypothetical common-cause outage with \(P(A)=0.0200\). Events \(B\) and \(C\) can occur in this scenario only under \(A\); experts judge \(P(B\mid A)=0.50\), \(P(C\mid A)=0.30\), and \(P(B\cap C\mid A)=0.20\). Find \(P(B\cup C)\), and check feasibility of the joint judgment. (Answer: \(0.0120\); feasible because \(0\leq0.20\leq0.30\).)
- Experts assign \(P(A)=0.0400\), \(P(B\mid A)=0.25\), and \(P(B\mid A^c)=0.0100\), where \(B\) is not defined as requiring \(A\). Find \(P(B)\). (Answer: \(0.0400(0.25)+0.9600(0.0100)=0.0196\).)