Why RCSA Facilitation Needs Consistent Evidence
A risk and control self-assessment (RCSA) is only useful when its ratings can be traced to a defined risk scenario, evidence, and a reasoned judgment. Process owners understand how work is actually performed, but their experience alone may not capture rare failures, workarounds, or control exceptions. Conversely, a spreadsheet of incidents or test results can be misleading if it is detached from the process and exposure it describes.
Facilitation provides the discipline that brings these sources together. The facilitator is not there to defend a preferred rating or to replace the process owner's accountability. The role is to establish a common basis for discussion, ask for support for material claims, and make the reasoning transparent. The approach below is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Plan the Assessment Before the Meeting
A workshop should not begin with a blank matrix and a request to “pick a score.” First define the process boundary, the period under review, the participants, and the evidence to be assembled. This prevents two people from rating different versions of the same process. For example, one participant may include a manual exception queue while another considers only the automated workflow.
The facilitator should ask the process owner to describe the principal steps, handoffs, systems, and points where an error could cause harm. Then state each risk in cause-event-effect form: a cause or weakness may lead to an event, which may produce a consequence. The purpose is not to build a complete risk taxonomy in the session; it is to keep the discussion anchored to a specific, assessable scenario.
Before the meeting, request evidence for a defined period and population. Useful items include a process map, current procedures, control descriptions, control-owner records, test results, incident and near-miss logs, exception reports, and relevant exposure counts. Record the source, date, scope, and limitations of each item. A control description shows what is intended; it does not by itself establish that the control operated as described.
Name the process, included activities, relevant systems, assessment period, and any excluded activities. Ask the owner to confirm the boundary before assigning ratings.
Collect incident and near-miss records, operating volumes, control documentation, and available test results. Note missing periods, incomplete populations, and changes in the process.
Include the process owner and people who perform or monitor the work. Where appropriate, include control or risk specialists to clarify evidence, not to take ownership away from the business.
Discuss the scenario, controls, and evidence in a consistent order. Capture the rating, rationale, unresolved uncertainty, and any evidence follow-up needed.
Use Evidence Without Overstating What It Shows
Different evidence answers different questions. Incident records show recorded outcomes, but may omit unreported events or events outside the collection period. Near misses can reveal exposure even when no loss occurred. Operating data provide denominators that make counts interpretable. Control documentation describes design and assigned responsibilities; testing and operating records provide evidence about whether a control worked in practice.
No single source should automatically settle a rating. If the owner reports no failures but an exception log shows repeated overrides, establish whether the log covers the same process and period, what counts as an exception, and whether the entries represent actual control failures. If sources disagree, document the discrepancy and its implications rather than silently selecting the more convenient number.
A useful evidence statement is specific: “The exception report covers 8,000 payments from January through December and records four duplicate-payment attempts; three were blocked and one was paid.” This is more useful than “duplicate payments are rare.” It also distinguishes exposure and attempted events from losses actually incurred. Where a data set is incomplete, say so; do not interpret missing records as zero events.
Structured Challenge Without Anchoring
Anchoring occurs when an early estimate or a confident first speaker sets the reference point for the rest of the discussion. Optimism can arise when participants assume a control will always work, or treat a clean period as proof that the exposure is negligible. A structured sequence reduces these effects by collecting views before debating ratings and by asking the same core questions across scenarios.
For example, ask participants to state their initial rating independently before the group discussion. Then ask for the scenario and evidence behind each view. Discuss the evidence before revisiting the rating. Avoid leading questions such as “This is clearly low, isn't it?” Prefer neutral prompts: “What evidence would support that rating?” and “What would have to be true for the control to prevent the event?”
- Ask what happened during the assessment period, including near misses, overrides, and manual workarounds.
- Ask whether the evidence covers the whole population, all relevant systems, and the period being assessed.
- Ask how the control could fail, and whether the observed test actually examined that failure mode.
- Ask what assumptions are required for a favorable rating and whether those assumptions are documented.
- Record a reasoned range or uncertainty where the evidence does not support a single confident conclusion.
Structured challenge is not an instruction to raise ratings whenever evidence is imperfect. It is a way to make the evidential basis visible and to distinguish a supported conclusion from an untested assumption. Where evidence is missing, the facilitator should record the limitation and identify a proportionate follow-up rather than inventing precision.
Worked Examples
Worked Example: Duplicate Payments in a Defined Population
A hypothetical insurer's payment team assesses duplicate payments for the last 12 months. The agreed population is 8,000 claim payments. The exception report records four duplicate-payment attempts: three were blocked by the duplicate-detection control, and one was paid, causing a gross loss of $18,000. Assume the RCSA uses this illustrative likelihood scale: 1 means fewer than 1 event per 10,000 transactions; 2 means 1 to fewer than 5; 3 means 5 to fewer than 20; 4 means 20 to fewer than 50; and 5 means 50 or more. Impact is 1 for less than $5,000, 2 for $5,000 to less than $25,000, 3 for $25,000 to less than $100,000, 4 for $100,000 to less than $500,000, and 5 for $500,000 or more. For this exercise, the score is likelihood multiplied by impact; scores 1–4 are low, 5–9 moderate, 10–15 high, and 16–25 very high. These are workshop assumptions, not a universal RCSA scale.
Method. First confirm that the four records belong to the stated population and period. For inherent likelihood, use the four observed attempts as events that could have caused a duplicate payment if the control were absent. The exposure-adjusted rate is:
Five events per 10,000 transactions falls in likelihood band 3. The maximum stated gross impact is $18,000, which falls in impact band 2. The illustrative inherent score is \(3\times2=6\), or moderate.
For residual likelihood, count the one attempt that resulted in payment: \(1/8{,}000\times10{,}000=1.25\) events per 10,000, which is likelihood band 2. The same $18,000 impact gives band 2, so the residual score is \(2\times2=4\), or low. The control blocked three of four recorded attempts, a descriptive proportion of \(3/4=75\%\). That proportion supports a discussion of control performance, but it is not by itself proof that future effectiveness will be 75%.
Check. The residual event count is one, not zero, and the event rate is 1.25 per 10,000, which is within band 2. The score is \(2\times2=4\). A common mistake is to use only the one paid loss for both inherent and residual likelihood. That would ignore the three attempts that could have caused loss without the control; the correction is to distinguish the four potential events from the one event that escaped the control.
Worked Example: Challenging Optimism About a Manual Review
A hypothetical policy-administration team says its manual review control “catches almost everything.” The agreed RCSA population is 1,200 endorsements over six months. The control log records 48 reviews, while an independent sample of 50 completed files finds three with no evidence of the required review. The incident register records two processing errors in the same period, one of which required a $32,000 correction. Assume the likelihood and impact bands from the prior example, with event rates expressed per 10,000 endorsements. For the residual frequency calculation, use the two recorded processing errors; do not extrapolate the three sample exceptions to the full population.
Method. Separate the sources before asking for a rating. The log documents 48 reviews, but does not establish that all 1,200 endorsements were reviewed. The sample identifies three files lacking review evidence, but it is a sample and does not directly establish three additional loss events. The incident register records two errors, providing the stated event count for the illustrative residual likelihood calculation.
Calculate the residual event rate: \(2/1{,}200\times10{,}000=16.67\) errors per 10,000 endorsements. This lies in likelihood band 3 (5 to fewer than 20). The $32,000 correction lies in impact band 3 ($25,000 to less than $100,000). The illustrative residual score is \(3\times3=9\), or moderate.
Next, challenge the control claim with neutral questions: “What explains the difference between 1,200 endorsements and 48 logged reviews?” “Does the sample cover both systems and all six months?” “Were the three files actually reviewed but not documented, or was the review absent?” These questions identify what must be clarified without assuming either that the control worked or that every undocumented review failed.
Check. The rate calculation uses the two recorded errors and the full stated exposure: \(16.67\) per 10,000, band 3. The impact is band 3, giving score 9. A common mistake is to add the three sample exceptions to the two incidents and score five events. The sample findings are not established additional losses, so that calculation mixes unlike evidence and overstates what is known. Keep the sample result as a control-evidence concern and investigate its meaning.
Worked Example: Preparing a Consistent Cross-Team Discussion
Two hypothetical teams process separate batches of the same type of benefit adjustment. Team North handles 4,000 adjustments in a year and records two errors requiring correction; the largest correction is $8,000. Team South handles 2,000 adjustments and records two errors; the largest correction is $40,000. Both teams use the illustrative likelihood and impact scale defined in the first example. Each team initially proposes a low residual rating, citing a procedure that requires a second-person check.
Method. Normalize the evidence to the same exposure unit before comparing event frequency. North's rate is \(2/4{,}000\times10{,}000=5\) errors per 10,000, giving likelihood band 3. South's rate is \(2/2{,}000\times10{,}000=10\) per 10,000, also band 3. Their equal raw counts do not imply equal rates because the exposures differ.
North's largest correction, $8,000, is impact band 2, so its illustrative score is \(3\times2=6\), moderate. South's $40,000 correction is impact band 3, so its score is \(3\times3=9\), also moderate. The same category does not mean the evidence is identical: South has a higher rate and a higher observed maximum impact within the stated data.
The facilitator should then check whether both event registers use the same definition of “error,” whether the one-year periods align, and whether second-person checks are evidenced by completed records or only described in procedure. If the evidence confirms that both teams have the same control requirement, assess operation using comparable records and state any data limitations. If it does not, record the difference rather than forcing an artificial comparison.
Check. The normalized rates are 5 and 10 per 10,000, both likelihood band 3; the impact bands are 2 and 3, producing scores 6 and 9. A common mistake is to compare the two counts of two errors and conclude that the teams have equal frequency. The correction is to divide each count by its own exposure before applying the common scale.
Make the Assessment Reproducible
A useful RCSA record allows a reviewer to understand not only the selected rating but how the group arrived at it. For each material scenario, retain the scope and period, risk statement, evidence sources and limitations, control description, rating rationale, challenge raised, and any unresolved question. Record whether a figure is observed, estimated, or judgmental. If the group changes an initial view, capture the evidence or reasoning that prompted the change.
A concise record is better than a long transcript. For example: “Four duplicate attempts in 8,000 payments; three blocked and one paid; control log and incident register reconciled; residual likelihood band 2 based on one paid event; impact band 2 based on $18,000 gross loss.” This records a traceable conclusion without implying that the score is more precise than the data.
Ratings are decision aids, not measurements with unlimited precision. Multiplying ordinal bands is valid only when the organization's stated methodology calls for it, and the resulting product should not be interpreted as a probability or expected loss. The key facilitation standard is consistent application of defined criteria, evidence-based challenge, and a clear account of uncertainty.
Check Your Understanding
Use the illustrative likelihood and impact bands stated above where a question requires them. These problems are hypothetical.
- A payment process handles 5,000 transactions and records three potential duplicate events before controls. What is the event rate per 10,000, and which likelihood band applies? (Answer: 6 events per 10,000; band 3.)
- A control procedure requires daily review, but the evidence pack contains review logs for 18 of 20 business days. What is one neutral follow-up question, and why is the procedure alone insufficient to establish operation? (Answer: Ask what explains the two missing log days and whether review occurred; the procedure describes intended design, not evidence that the control operated.)
- A team reports no losses, but its near-miss register contains four blocked events over 2,000 transactions. What rate do the blocked events represent per 10,000, and what should the facilitator clarify before treating them as loss events? (Answer: 20 per 10,000; clarify whether each was a genuine event that could have caused the defined loss and how the register defines and captures near misses.)
- Two processes each record four errors. Process A has 8,000 transactions and Process B has 2,000. Calculate both rates per 10,000 and explain why comparing counts alone is misleading. (Answer: A is 5 per 10,000 and B is 20 per 10,000; equal counts conceal different exposure-adjusted frequencies.)
- A process records one $12,000 loss in 4,000 transactions. Under the stated illustrative scale, calculate likelihood band, impact band, and score, assuming the score is the product of the bands. (Answer: Rate is 2.5 per 10,000, likelihood band 2; $12,000 is impact band 2; score \(2\times2=4\), low.)