Why Limited Evidence Needs Explicit Treatment
Risk assessments often rely on incomplete incident records, short periods of observation, changing processes, or expert judgment. A rating may still help decision-makers compare concerns and select actions, but it is only useful if readers can see what supports it and what remains unknown. A category such as “Occasional” is not a measured frequency unless the underlying evidence and method justify that interpretation.
The practical task is not to eliminate uncertainty by assigning more decimal places. It is to describe the evidence, assumptions, and gaps clearly enough that another assessor can understand the rating, challenge it, and decide whether more information is needed. This approach is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Keep the risk rating distinct from confidence in that rating. “Possible, with low confidence” communicates two different things: the assigned likelihood category and the weakness of its support. Combining them into a single score can conceal whether a risk is judged less likely or simply less well understood. A risk register should preserve both pieces of information.
Describe the Evidence Before Interpreting It
Before assigning a category, identify what information is being used and how it relates to the risk being assessed. An incident count is meaningful only if the event definition, observation period, and relevant exposure are clear. An expert estimate should identify whose judgment it represents, what scenario was considered, and which assumptions were used. A record of “no reported incidents” is not necessarily evidence that no incidents occurred.
When event counts and exposure are suitable, a simple observed rate can help describe the evidence:
The expression follows directly from the meaning of a rate: it divides the number of events by the amount of time or other exposure during which those events could occur. For example, events per year require exposure measured in years. This calculation describes observed experience; it does not, by itself, establish a stable future frequency. Interpreting it as representative requires assumptions, including that the exposure is relevant and that event capture and operating conditions are sufficiently comparable over the period.
Even when the rate can be calculated, avoid treating it as exact. A small number of observations can be highly sensitive to one additional event. Changes in reporting practice, process design, or exposure can further limit comparison with future conditions. State what the calculation does and does not support.
Worked Example: A Short Incident History for a Service Provider
A hypothetical insurer assesses interruptions at a claims-processing vendor. Its log shows two qualifying interruptions during five full years of operation. The assessment team’s established likelihood categories are Rare for fewer than 0.1 observed events per year, Occasional for at least 0.1 but fewer than 0.5, and Frequent for at least 0.5. Assume the log uses a consistent event definition and captures all qualifying interruptions during those five years. No probability distribution is assumed.
Method. Calculate the observed rate, compare it with the stated category boundaries, and then document the limitations of using a short history to support a forward-looking judgment.
Step 1: Set out the numerator and exposure. The numerator is 2 recorded interruptions. The exposure is 5 years.
Step 2: Calculate the observed rate.
Step 3: Compare with the category boundaries. Since \(0.1 \leq 0.4 < 0.5\), the observed rate falls in the Occasional category under the stated scale.
Step 4: State the interpretation and its limit. The team may record Occasional as a provisional likelihood rating supported by this observed experience. It should also say that the rate is based on only two events over five years and assumes the historical operating conditions remain relevant. The calculation does not establish that future interruptions will occur at exactly 0.4 per year.
Common mistake avoided. Reporting “0.4 interruptions per year” as if it were a precise forecast overstates what the records show. The category is a judgment informed by a short historical rate, not a guarantee about future frequency.
Make Assumptions and Gaps Visible
An assessment record should make its reasoning reviewable. A concise entry can identify the risk scenario, evidence source and period, assumptions used, evidence gaps, rating, confidence, and review trigger. Include only assumptions that affect the assessment; vague statements such as “based on experience” do not let a reader evaluate the reasoning.
State what event is being assessed, the relevant process or exposure, and what counts as an occurrence. Different event definitions can produce different evidence and ratings.
Name the source, period, and scope. Distinguish recorded events from confirmed events, expert estimates, and information that has not been verified.
For example, state whether the assessment assumes complete reporting, stable exposure, or continued use of a particular control. Note where those conditions are unknown.
Use the defined qualitative categories, but record whether support is strong, moderate, or limited under the organization’s convention. Do not create a numerical confidence score unless it has a defined, justified interpretation.
Name a useful next evidence source, an action to close the gap, or a condition that should trigger reassessment.
Impact assessments require the same discipline. A stated impact category should identify the dimensions considered, such as financial, operational, customer, or legal consequences, and the scenario assumptions behind them. If reasonable assumptions produce different categories, record the range or the relevant scenarios rather than concealing the variation in a single unsupported point estimate.
Worked Example: A Risk With No Recorded Events
A hypothetical insurer considers a cyber incident that could prevent policy renewals from being processed. Its internal incident log covers two years and contains no qualifying events. The team has not confirmed whether the log captures all near misses or interruptions. For this example, the established likelihood scale labels an event Rare if its observed rate is below 0.1 per year. A scenario review suggests financial impact could exceed $1,000,000 if recovery takes several days; the amount is an assumption for assessment, not a measured loss. No probability distribution is assumed.
Method. Separate the observed record from the forward-looking judgment, state what the record cannot establish, and rate impact from the defined scenario.
Step 1: Summarize the record. There are 0 recorded qualifying events over 2 years. The crude observed rate is \(0/2=0\) recorded events per year.
Step 2: Identify why that rate does not settle likelihood. The log’s completeness is unverified. Therefore, the zero count may reflect missing events as well as no events. It is not adequate support for concluding that the true event rate is zero or that the risk is Rare.
Step 3: Record the provisional likelihood conclusion. The team should mark likelihood as not rated pending evidence, or assign a provisional category only if its assessment process requires one and the supporting judgment is stated. Here, it records “not rated—limited evidence; log completeness unverified” rather than asserting a category.
Step 4: Assess impact under the stated scenario. The assumed recovery scenario could exceed $1,000,000. If the organization’s existing impact scale defines that threshold as Severe, the provisional impact rating is Severe. The record should name the assumed recovery period and financial scope; it should not describe the assumed amount as an observed loss.
Step 5: Identify a next step. The team should reconcile the incident log with available service and near-miss records, then review the likelihood assessment. The finding is that evidence is inadequate, not that the event cannot occur.
Common mistake avoided. Assigning Rare solely because \(0/2=0\) confuses the recorded rate with a well-supported estimate of future likelihood. The correction is to expose the data-quality gap and leave the likelihood rating provisional or unrated.
Handle Disagreement Without Hiding It
Different sources may describe different event scopes. One system may record every interruption; another may record only incidents that generated a claim. Before averaging or choosing between their counts, determine whether they refer to the same event definition, time period, and exposure. A numerical compromise does not resolve a mismatch in what was counted.
If the sources can be reconciled, show how. If they cannot, describe the competing interpretations and how each affects the rating. Expert judgment may help interpret records, but it should not be presented as independent data. Record the judgment’s basis and any assumptions about future conditions.
Worked Example: Reconciling Two Sources With Different Event Definitions
A hypothetical payment operation assesses service interruptions over six years. The operations log records three interruptions of any duration. A claims log records one interruption that caused a claim; the two sources confirm that this event is included in the operations log. Review establishes that two of the three operations-log events meet the team’s defined threshold for a material interruption. The team’s likelihood scale classifies rates of at least 0.1 but below 0.5 material interruptions per year as Occasional, and rates of at least 0.5 as Frequent. Assume six years is the relevant exposure and no other qualifying events were found.
Method. Reconcile event definitions first, then calculate the rate for the defined scenario. Do not treat claims as a complete count of all interruptions.
Step 1: Identify the mismatch. The operations log counts interruptions of any duration; the claims log counts only interruptions that generated a claim. The counts of 3 and 1 are therefore not estimates of the same event set.
Step 2: Apply the assessment definition. The defined scenario is a material interruption. Of the three operations-log events, two meet that definition. The claims log confirms one of those events but does not add another.
Step 3: Calculate the relevant observed rate.
Step 4: Assign and qualify the category. Since \(0.1 \leq 0.333 < 0.5\), the observed rate is Occasional under the stated scale. Record that the category is based on two qualifying events, that the claims log is incomplete for interruptions without claims, and that the operations log is the primary source for this scenario.
Step 5: State the remaining limitation. The category summarizes the available history; six years of records do not by themselves establish that future operating conditions will be unchanged.
Common mistake avoided. Averaging the counts 3 and 1 would give 2 events, but that arithmetic has no defensible interpretation because the sources use different inclusion rules and overlap. Reconcile the event definition before calculating.
Write Ratings That Can Be Reviewed
A clear assessment does not need to be long, but it should let a reviewer distinguish fact, assumption, and judgment. For instance: “Likelihood: Occasional, provisional; two material interruptions in six years in the operations log. Claims records are not a complete event source. Assumes the current process and exposure remain broadly comparable. Review after log reconciliation or a material process change.” This wording is more useful than a bare category because it identifies the basis and a path to reconsideration.
Avoid false precision in both numbers and labels. A qualitative score may be helpful for prioritization, but arithmetic on category labels does not automatically create a meaningful measure. In particular, adding or averaging ordinal ratings can imply equal distances between categories when those distances have not been established. Use the organization’s defined matrix as intended, and preserve the narrative needed to interpret it.
Check Your Understanding
For each hypothetical assessment, focus on what the evidence supports and what should be documented.
- A service team records 3 qualifying outages in 10 years. Assume a category scale defines Occasional as at least 0.1 but below 0.5 events per year. Calculate the observed rate, give the category, and state one limitation to note. (Answer: \(3/10=0.3\) events per year; Occasional; note that the rate describes the recorded history and may not represent future conditions.)
- A risk log shows no vendor failures in one year, but the team does not know whether staff reported brief failures consistently. Is “zero likelihood” supported? State the appropriate evidence qualification. (Answer: No; the record supports zero reported failures, not zero likelihood. Mark the assessment as limited or provisional because reporting completeness is unknown.)
- Two sources report 4 and 2 events over the same five-year period. The second source records only events that caused customer refunds, and those 2 events are included among the 4. If the assessed scenario is any qualifying service interruption, which count should be used, and what should be disclosed? (Answer: Use 4 for the broader interruption definition, subject to confirming it matches the scenario; disclose that the refund source is a narrower, overlapping subset and is not a complete count.)
- An expert estimates that an event could cause a financial impact between $200,000 and $800,000, depending on how long recovery takes. The impact scale classifies $250,000 to below $1,000,000 as Material. What should the assessment say about the impact category? (Answer: The plausible range spans Minor and Material under that scale; document the recovery-time assumption and range rather than presenting a single precise impact as certain.)
- A team calculates 1 recorded event over 4 years and reports 0.25 events per year. Its scale places rates from 0.1 to below 0.5 in Occasional. Give the category and one reason not to interpret 0.25 as a precise forecast. (Answer: Occasional; the rate summarizes limited historical evidence and may not predict the future if reporting, exposure, or operating conditions change.)