Why RCSA Ratings Can Conflict
A risk and control self-assessment (RCSA) translates judgments about operational risk into ratings that help an organization discuss and monitor its exposures. Two teams can assign different ratings to what appears to be the same risk. That difference may reflect a real difference in exposure or controls, but it can also arise because the teams used different scales, evidence, assumptions, control assessments, or boundaries. A rating comparison is useful only after those possibilities have been checked.
The task here is diagnosis, not choosing which risk should receive the most attention. In particular, a numerical rating is not automatically a measurement on a common scale. A rating of 3 from one team cannot safely be compared with a 3 from another until their rating rules and assessment scopes are understood.
An RCSA commonly distinguishes inherent risk, assessed before considering controls, from residual risk, assessed after considering controls. Teams may also rate likelihood and impact separately, then combine those dimensions under a defined matrix or other rule. For a sound comparison, establish which rating is being discussed, what each dimension means, and how the final label is assigned.
A Structured Diagnostic
Compare ratings in a fixed sequence. Start with the label and scope of the risk, then inspect the rating method, evidence, assumptions, and control assessment. This sequence helps separate a real change in risk from a difference in how it was described or scored.
Check that both assessments refer to the same process, event type, business unit, time period, and risk state (inherent or residual). Similar wording does not guarantee identical scope.
Read the definitions for each level, including likelihood and impact thresholds, any time horizon, and the rule that converts dimensions into an overall rating. Do not assume identical numbers mean identical risk.
Identify the observation period, data sources, exposure denominator, scenario assumptions, and treatment of rare but severe outcomes. Determine whether one team is using a different or incomplete evidence base.
Check that controls are within scope, designed for the stated risk, implemented, and supported by appropriate operating-effectiveness evidence. A control's existence alone does not establish that it reduces residual risk.
State whether the difference is explained by scope, method, evidence, or control performance; whether a like-for-like reassessment is needed; and what remains unresolved. Preserve the rationale rather than forcing agreement.
Why Like-for-Like Comparison Matters
A rating is the output of a process: evidence and assumptions are interpreted under definitions and rules for a specified risk and scope. If those inputs differ, a difference in output does not isolate a difference in underlying risk. This is why matching the meaning of a rating is a necessary condition for interpreting its value.
For example, suppose one team calls a likelihood level “high” when an event is expected at least once per year, while another uses “high” for an event expected at least once every five years. The same label does not represent the same frequency range. Similarly, if one impact definition considers gross financial loss and the other considers net loss after recoveries, the impact labels do not refer to the same quantity.
The same logic applies to numerical scales. RCSA levels are generally ordered categories, not measurements with equal intervals. If 1 means low, 2 means moderate, and 3 means high, it does not follow that the step from 1 to 2 equals the step from 2 to 3. Consequently, averaging incompatible ratings does not resolve the underlying mismatch.
Worked Examples
Worked Example: Identifying a Scale-Definition Mismatch
Two hypothetical insurance operations assess the residual likelihood of a payment-processing error. Both report a rating of 3. Team North defines level 3 as “likely: at least one occurrence per year.” Team South uses a five-level scale and defines level 3 as “possible: one occurrence in two to five years”; its level 4 is “likely: at least one occurrence per year.” What can be concluded, and what comparison is appropriate?
Method. Compare the definitions attached to the ratings before interpreting their numerical values.
Step 1: Identify the reported values. North reports 3 and South reports 3. That is equality of the numbers, not yet evidence of equal likelihood.
Step 2: Translate each number using its own scale. North’s 3 means at least one occurrence per year. South’s 3 means one occurrence in two to five years. Those definitions represent different likelihood ranges.
Step 3: Match the definitions. North’s “at least one occurrence per year” corresponds to South’s level 4, based on the definitions given. The like-for-like comparison is therefore North 3 against South 4, not North 3 against South 3.
Conclusion. The initial apparent agreement is a scale mismatch. The available information does not establish which operation has greater actual exposure; it establishes that the reported numbers are not directly comparable. The mistake avoided is treating identical rating numbers as identical risk levels.
Worked Example: Separating Evidence From Assumptions
Two hypothetical claims teams assess the residual likelihood of a duplicate reimbursement. Team A reports level 2, using 4 duplicates among 2,000 transactions in the past year. Team B reports level 4, using 3 duplicates among 600 transactions in the past six months and assuming the next six months will have the same transaction volume and event rate. Both use the same likelihood bands: level 2 means fewer than 5 events per 1,000 transactions; level 4 means at least 5 per 1,000. Diagnose the difference, assuming the counts are complete and the bands apply to the observed rate per transaction.
Method. Check the denominator and stated rating rule, then compare the evidence on a common per-transaction basis. Do not compare raw event counts from different exposure volumes.
Step 1: Calculate Team A’s observed rate. \(4 \div 2{,}000 = 0.002\), or \(0.002 \times 1{,}000 = 2\) events per 1,000 transactions. This is below 5 per 1,000, consistent with level 2.
Step 2: Calculate Team B’s observed rate. \(3 \div 600 = 0.005\), or \(0.005 \times 1{,}000 = 5\) events per 1,000 transactions. This meets the stated threshold for level 4.
Step 3: Examine the time assumption. Team B observes six months of transactions and explicitly assumes the same event rate for the next six months. Its rating band is based on events per 1,000 transactions, not an annual event count, so that assumption does not change the calculated observed rate per transaction. It would matter if the rating definition were annual frequency rather than a transaction-based rate.
Conclusion. Given the stated common bands and complete counts, the different ratings are consistent with the observed rates: 2 per 1,000 for A and 5 per 1,000 for B. The teams should still verify that their transaction populations and counting rules are comparable. The mistake avoided is ranking the teams by raw counts or silently applying an annual-frequency assumption to a per-transaction rating.
Worked Example: Diagnosing a Control and Scope Difference
Two hypothetical units rate residual risk from unauthorized supplier-bank-detail changes on the same four-level scale. Unit East assesses only its manual-change queue and rates residual risk 2 because a supervisor reviews each change. A sample of 200 completed manual changes has 2 exceptions. Unit West assesses both manual and automated changes and rates residual risk 4 because its automated channel has no independent review. In a sample of 300 automated changes, 12 lack the required supporting approval. The shared scale defines level 2 as controlled with limited exceptions and level 4 as a material control weakness. Diagnose whether the ratings conflict.
Method. Separate scope from control evidence, calculate the observed exception rates, and avoid treating a control in one channel as evidence about another.
Step 1: Confirm the scopes. East includes manual changes only. West includes both manual and automated changes. The assessments therefore do not cover the same population.
Step 2: Calculate the manual exception rate. East observes \(2 \div 200 = 0.01\), or 1%. Its evidence is consistent with a supervisor review and a limited number of exceptions, although the sample alone does not establish the review’s quality or whether exceptions were resolved.
Step 3: Calculate the automated exception rate. West observes \(12 \div 300 = 0.04\), or 4%, without independent review. This evidence points to a different control condition in the automated channel.
Step 4: Interpret the ratings in context. East’s rating describes the manual queue; West’s rating includes an automated channel with a stated control weakness. The ratings may both be reasonable for their respective scopes, but they are not like-for-like assessments of the same population.
Conclusion. To compare the overall risk, the units need a common scope and a documented assessment of both channels, including control evidence for each. The mistake avoided is assuming East’s supervisor review covers automated changes, or averaging ratings that refer to different populations.
Common Mistakes and Documentation
When ratings disagree, the aim is to identify the reason, not to make the disagreement disappear. A defensible record describes the comparison and its limits. It should identify the risk statement and boundaries, the scale version, the evidence period and sources, key assumptions, the relevant controls and their test results, and the reason for any final rating or reassessment.
- Comparing labels without definitions. Two “medium” ratings may use different likelihood or impact thresholds.
- Using raw counts without exposure. Four events in a large transaction population may indicate a lower rate than three events in a much smaller population.
- Treating a control as effective because it exists. Design, implementation, and operating evidence are distinct considerations.
- Ignoring assessment boundaries. A rating for one channel or process segment cannot automatically represent a broader operation.
- Averaging ordinal ratings to create agreement. An average can conceal differences in definitions, evidence, or scope without resolving them.
If both ratings use the same definitions and scope but rely on different evidence, document which evidence is more complete, relevant, and reliable; do not discard a source solely because it conflicts with another. If a control assessment is disputed, identify the control objective, test period, population, exceptions, and remediation status. If the cause remains uncertain, record the uncertainty and specify what further evidence or reassessment is needed.
Check Your Understanding
For each problem, identify the comparison issue and state the appropriate next step.
- Two teams report impact level 3. One defines level 3 as gross loss from $100,000 to $250,000; the other defines it as net loss from $100,000 to $250,000 after recoveries. What must be checked before comparing the ratings? (Answer: Confirm whether both are rating gross or net loss and use a common impact definition; identical numbers do not establish comparable impact.)
- Team A records 6 processing errors in 3,000 transactions; Team B records 4 in 1,000. Both use the same bands: level 2 is below 5 errors per 1,000, and level 3 is at least 5 per 1,000. Calculate each rate and identify the expected levels. (Answer: A: \(6/3{,}000 \times 1{,}000=2\) per 1,000, level 2; B: \(4/1{,}000 \times 1{,}000=4\) per 1,000, also level 2.)
- A unit lowers residual risk because a new approval control was introduced last month. It has no test results yet. What does the rating evidence support, and what should be documented? (Answer: The control’s introduction supports that it exists or has been implemented, not that it operates effectively. Document the limited evidence and obtain appropriate operating-effectiveness evidence before relying on that claim.)
- One assessment covers a call center’s manual refunds, while another covers all refunds, including an automated channel with separate access controls. The second rating is higher. Is this necessarily a rating inconsistency? (Answer: No. The scopes differ. Compare the same populations and assess the automated-channel controls separately before drawing a like-for-like conclusion.)
- Two teams use identical scales and scopes. One rates likelihood 2 based on a complete year of records; the other rates likelihood 4 based on a short period that includes a one-time system outage. What should the reviewer establish before reconciling the ratings? (Answer: Check the evidence periods, completeness, event definitions, and whether the outage is representative or exceptional under the rating assumptions; then document the evidence-supported rating or the remaining uncertainty.)