Why Calibrate Qualitative Scores?
A likelihood-impact matrix can give risk assessments a consistent format, but it cannot ensure that different assessors interpret its categories in the same way. One assessor may call an event “likely” because it has occurred several times in a year; another may reserve that category for events expected to occur at least once every year. If both scores are entered without examining the difference, the risk register may suggest a level of precision that the assessment process did not achieve.
Calibration is the process of checking whether assessors apply shared scoring definitions consistently, identifying the reasons for differences, and clarifying the definitions with reference to examples. It is not a process for pressuring assessors to agree. A well-calibrated process makes the basis for each score more comparable while allowing a genuine difference in judgment to remain visible.
The earlier material on likelihood-impact matrices establishes the role of categories in a qualitative assessment. Calibration focuses on how people use those categories. The comparison is most informative when assessors first score the same clearly described scenarios independently, using the same assessment scope, time horizon, and information. Discussion comes afterward, so one person’s initial interpretation does not automatically anchor the others.
Comparing Scores Fairly
For two assessors scoring the same set of scenarios, exact agreement is the number of scenarios with matching scores divided by the number of scenarios scored. This is easy to explain, but it treats every mismatch alike: a one-category difference and a three-category difference each count as one disagreement. The score difference and its reason should therefore be reviewed as well as the percentage.
When three or more assessors score a scenario, agreement can also be summarized by counting the matching pairs of assessors. If \(n_c\) assessors assign category \(c\), the number of pairs agreeing in that category is \(\binom{n_c}{2}\). The total number of assessor pairs is \(\binom{n}{2}\), where \(n\) is the number of assessors. Pairwise agreement is the number of agreeing pairs divided by the total number of pairs.
The formula counts unordered pairs: each distinct pair is counted once, and an assessor is never paired with themself. For example, when two assessors choose the same category, the count is \(\binom{2}{2}=2\times1/2=1\) agreeing pair. Writing \(1\times1/2\) would be an arithmetic error, even though the intended final count is one.
A Calibration Process
A useful process separates measurement from discussion. The team should decide what is being assessed before comparing scores: for example, the same risk statement, the same time horizon, and the same impact dimension. Otherwise a score difference may reflect different questions rather than inconsistent use of the same scale.
Specify the risk or scenario, assessment horizon, impact dimension, and whether the score describes the same risk state for every assessor.
Each assessor selects a category and briefly records the evidence or interpretation supporting it before seeing the other scores.
Identify exact agreements and disagreements, then ask which fact, boundary, or phrase in the category definition led to each different score.
Agree how the existing definitions apply to representative scenarios. Record the clarifying wording so it can be applied beyond the examples discussed.
Have assessors score the examples again independently. Record remaining differences and their reasons rather than treating consensus as a required outcome.
A useful shared example states enough facts to apply the category rule and makes clear which facts are in scope. An example with missing information may test how assessors handle uncertainty rather than whether they understand the category. That can be a valid discussion, but it is a different question from whether the categories are being interpreted consistently.
Worked Example: Counting Agreement Among Four Assessors
Four assessors independently score one hypothetical risk using a four-category impact scale. Their ratings are 2, 2, 2, and 3. Calculate pairwise agreement and state what the result does and does not show.
Method. Count all unordered pairs of assessors, then count the pairs that selected the same category. There are four assessors, so the total number of pairs is
Three assessors selected category 2. The number of pairs within those three assessors is
The single assessor who selected category 3 creates no agreeing pair in that category:
Thus there are 3 agreeing pairs out of 6, and pairwise agreement is \(3/6=0.5\), or 50%. This result summarizes how often a pair matched for this score. It does not establish whether category 2 or category 3 is correct.
Common mistake avoided. Do not divide the number of assessors by two without first counting distinct pairs within each category. For two assessors in the same category, the correct calculation is \(\binom{2}{2}=2\times1/2=1\), not \(1\times1/2\). The latter is not the pair count.
Resolve Differences by Testing the Definitions
When two scores differ, the first step is not to average them. Instead, compare the assessors’ interpretations with the written scoring rule. A difference may arise because a boundary is vague, a scenario was understood differently, or an assessor used facts outside the agreed scope. These have different remedies: clarify the category boundary, make the scenario more precise, or restate the assessment basis.
Suppose an impact scale defines category 2 as a service interruption of up to four hours and category 3 as an interruption of more than four hours through two business days. A hypothetical scenario describes a one-day interruption to a customer service. If one assessor scores the interruption to the service as a whole while another considers only the core system, they have not applied the same scenario definition. A shared example can make the intended unit of assessment explicit. It should not be used to hide a difference in judgment that persists after the scope is clear.
Worked Example: Diagnosing Disagreement on Five Scenarios
Two assessors use the same impact scale to rate five hypothetical service scenarios. Their scores are:
The written scale defines category 1 as no customer service interruption, category 2 as an interruption of up to four hours, and category 3 as an interruption of more than four hours through two business days. The scenarios describe the customer service outcome, not the duration of an internal system fault.
Step 1: Calculate exact agreement. The scores match on scenarios 1, 3, and 4. There are three matches among five scenarios:
Step 2: Locate the mismatches. On scenario 2, the ratings are 2 and 3. The hypothetical service is unavailable to customers for one day, so the written definition places it in category 3. Assessor A may have focused on a core system that recovered sooner; that is not the stated outcome being scored. On scenario 5, the ratings are 1 and 2. The scenario describes a three-hour customer service interruption, which falls in category 2 rather than category 1.
Step 3: Clarify and test. The assessors should confirm that the customer-facing service interruption is the impact dimension and that the stated duration is the relevant duration. They should then score the scenarios independently again. The example tests whether both assessors apply the same rule; it does not justify changing a score merely to raise the agreement percentage.
Common mistake avoided. Calling the two differences “close enough” because each is only one category apart would conceal that both are resolved by the stated boundary and scope. Exact agreement remains 60% before clarification; a separate, documented rescore would be needed to report any later agreement.
Shared Examples and Consistency Checks
Shared examples are most useful when they are varied enough to test meaningful distinctions: one example near a category boundary, one comfortably within a category, and one that tests a commonly misunderstood phrase. The team should state the facts that determine the score and explain which definition applies. Examples should illustrate the rule, not replace it. Otherwise assessors may memorize a list of precedent scores and struggle with new scenarios.
After definitions are clarified, independent rescoring gives a simple consistency check. A higher exact-agreement rate on the same examples suggests that assessors now apply the shared rules more consistently. It does not show that the rules themselves are well chosen or that the resulting scores accurately represent the risk. Agreement and validity are distinct: assessors can consistently apply a poor definition.
Worked Example: Checking Agreement After Clarification
Two assessors initially score six shared likelihood examples. Their category ratings are:
The assessors clarify that the scale refers to the probability of the defined event during the next 12 months, not to how often a related event has occurred over an unspecified period. They then rescore independently. Assessor A’s scores remain \(1,1,2,2,3,3\), while Assessor B’s scores become \(1,1,2,3,3,3\).
Step 1: Count initial matches. Before clarification, the scores match on examples 1, 3, and 6. Therefore, initial exact agreement is
Step 2: Count matches after clarification. After rescoring, the scores match on examples 1, 2, 3, 5, and 6. They differ on example 4, so the rescore agreement is
Step 3: Interpret the change. Exact agreement increased by \(83.33\%-50\%=33.33\) percentage points on these six examples. The result is consistent with improved alignment on the time horizon, but it is not proof that the likelihood categories are accurate. The remaining disagreement should be examined against the event definition and category boundary.
Common mistake avoided. Do not report the change as a 33.33% increase in agreement: that confuses percentage points with relative percent change. The absolute change is 33.33 percentage points. The relative increase is \((5/6-3/6)/(3/6)=2/3\), or 66.67%, if that comparison is specifically required.
Common Traps and Practical Use
- Discussing before independent scoring. Early discussion can anchor assessors to the first proposed score. Collect initial ratings separately, then compare.
- Changing category meanings to make scores match. Clarify ambiguous wording consistently; do not redefine a category for only one scenario.
- Averaging ordinal categories. A mean of categories 1 and 3 is not automatically a defensible category 2. Categories are labels with order, not necessarily equal numerical intervals.
- Treating agreement as accuracy. Consistent use of a category does not validate the category boundary or the underlying risk judgment.
- Ignoring the comparison basis. Different horizons, event definitions, or impact dimensions can produce different scores without demonstrating inconsistent scoring.
- Forcing unanimity. If a difference remains after scope and definitions are clear, document the rationale and retain the differing views for appropriate review.
Check Your Understanding
For each problem, use the stated assessment basis and show the key comparison or calculation before checking the answer.
- Four assessors score a hypothetical risk as 1, 1, 2, and 2. How many total assessor pairs are there, how many pairs agree, and what is the pairwise agreement? (Answer: 6 total pairs; 2 agreeing pairs; \(2/6=33.33\%\).)
- Two assessors score four shared scenarios as \(1,2,2,3\) and \(1,3,2,2\). Calculate exact agreement and state how many scenarios differ by one category. (Answer: 2 of 4 match, so 50% exact agreement; the other 2 differ by one category.)
- A hypothetical scale defines category 1 as annual likelihood below 2% and category 2 as at least 2% but below 8%. Which category applies to an event assessed at exactly 2% annual likelihood, and what wording resolves the boundary? (Answer: Category 2; “at least 2%” includes the boundary.)
- Three assessors score four shared examples. Their ratings are, respectively: example 1, \(1,1,2\); example 2, \(2,2,2\); example 3, \(3,3,4\); example 4, \(1,2,3\). Calculate pairwise agreement across all assessor pairs and examples. (Answer: \(5\) agreeing pairs out of \(12\) total pairs, or \(41.67\%\).)
- Two assessors initially agree on 4 of 8 examples. After clarifying the time horizon and independently rescoring, they agree on 6 of 8. State the change in exact agreement in percentage points and one conclusion that the comparison does not support. (Answer: From 50% to 75%, an increase of 25 percentage points; it does not prove that the scores are accurate.)