Why Control and Mitigation Assessments Go Wrong
A control assessment can appear rigorous and still overstate risk reduction. A written policy may be mistaken for proof of consistent operation; two safeguards may be credited separately for the same effect; or insurance may be treated as if it removes every consequence of a loss. Each error can make reported residual exposure too low and lead to a decision based on unsupported assurance.
The discipline is to distinguish what a control is intended to do from what evidence shows it did, and to trace each material consequence through the full mitigation arrangement. The examples below use hypothetical situations and amounts. Their purpose is to show how to identify the error, calculate its effect, and state what the evidence does and does not support.
A sound assessment also keeps gross exposure, control effects, and retained exposure distinct. Evidence of a control activity is not automatically evidence that losses were prevented, and a transfer arrangement changes who bears specified losses rather than making the underlying event impossible. The earlier material on control dependencies and common-mode failure is relevant when deciding whether safeguards act independently or share a vulnerability.
Policy Existence Is Not Proof of Operation
A policy states what should happen. To assess operation, an assessor needs evidence that the required action occurred for the relevant transactions and period. Depending on the control, evidence might include dated approvals, system logs, reconciliations, exception records, or independent test results. The evidence should be tied to the control objective and the population under review.
A sample can show what happened in the tested items. It does not, without an appropriate sampling and inference basis, prove that the same result holds for every item. Nor does a completed checklist alone establish that the review was sufficiently accurate or that exceptions were resolved. The assessor should identify the population, the sample, the test performed, the exceptions, and the limits of the conclusion.
Worked Example: A Review Policy and a Sample Test
A hypothetical claims administrator has a written procedure requiring every manual reserve adjustment to receive supervisor approval before posting. During a quarter, there were 80 adjustments. An assessor selects 15 adjustments at random and finds dated approval evidence for 14. What can the assessor conclude, and what would be an error?
Method. Separate the control requirement from the test evidence, then calculate the sample’s observed evidence and exception rates. The procedure’s requirement applies to all adjustments in the population; the sample test concerns only the 15 selected adjustments.
The number of sampled adjustments with approval evidence is 14, so the observed sample evidence rate is \(14/15=0.9333\), or 93.33%. One sampled adjustment lacks approval evidence, giving an observed sample exception rate of \(1/15=0.0667\), or 6.67%.
Those calculations describe the sample. They do not establish that 93.33% of the population complied, and they do not establish that approval evidence exists for every adjustment in the population. The exception should be investigated: for example, determine whether approval occurred but was not retained, whether the adjustment bypassed the control, or whether the control’s evidence requirements were unclear. The assessor should also consider the implications for the assessment and whether further testing is warranted.
Common mistake and correction. The wrong conclusion is, “The policy requires approval, so all 80 adjustments were approved.” The policy specifies the required behavior; it is not evidence that the behavior occurred. A more defensible conclusion is, “Approval evidence was found for 14 of the 15 sampled adjustments; one sampled item was an exception, and this test alone does not establish operation for all adjustments in the population.”
Do Not Count the Same Safeguard Effect Twice
A safeguard may be described in several ways, or two safeguards may address the same failure pathway. Crediting each with its full standalone reduction can exaggerate the combined effect. Before combining estimated effects, establish what each measure changes, whether the effects overlap, and whether independence or another justified joint model is supported by evidence.
For a simple frequency-severity model, let \(\lambda\) be the expected number of losses per year and let \(m\) be the mean loss amount per event. If the expected annual loss is the sum of the loss amounts over events, then \(E[L]=\lambda m\). This follows by conditioning on the event count \(N\): if each event has mean severity \(m\), then \(E[L\mid N]=Nm\), so \(E[L]=E[Nm]=mE[N]=m\lambda\). This formula assumes a finite expected event count and severity and that the stated mean severity applies to the events being counted. It does not, by itself, justify multiplying control-effect percentages.
Worked Example: Two Controls With an Overlapping Effect
A hypothetical payment platform expects 0.20 unauthorized-payment events per year, with a fixed average loss of $400,000 per event. A transaction alert and a manual callback are both intended to prevent payments caused by the same compromised contact channel. Each control was initially assigned a 40% frequency reduction. A joint test of the controls, however, supports only a 40% combined reduction because the safeguards largely address the same pathway. Calculate the supported expected annual loss and compare it with an independence-based calculation.
Method. Calculate baseline expected annual loss, apply the supported combined frequency reduction once, and then show the result of the unsupported independent multiplication.
Baseline expected annual loss is \(0.20\times \$400{,}000=\$80{,}000\). A 40% combined reduction leaves 60% of the baseline event frequency: \(0.20\times 0.60=0.12\) expected events per year. The supported expected annual loss is therefore \(0.12\times \$400{,}000=\$48{,}000\).
The incorrect calculation treats the two 40% reductions as independent sequential effects. It gives a remaining frequency of \(0.20\times 0.60\times 0.60=0.072\) events per year, and an expected annual loss of \(0.072\times \$400{,}000=\$28{,}800\). That is $19,200 below the amount supported by the joint test: \(\$48{,}000-\$28{,}800=\$19{,}200\).
Common mistake and correction. The error is to multiply the two remaining-frequency factors merely because there are two controls. That multiplication requires a defensible model for the joint effects; the facts here instead give a measured combined reduction of 40%. Use the joint evidence, so the estimated expected annual loss is $48,000, not $28,800. This is an expected-loss estimate under the stated assumptions, not a guarantee that the actual loss in a particular year will equal that amount.
This issue is not solved by assuming that safeguards are always dependent, either. If separate evidence supports distinct effects and a suitable joint model, a combined calculation may be appropriate. The assessment should make the basis explicit: a standalone estimate is not automatically a combined estimate, and a shared failure mode can undermine both controls at once.
Transferred Risk Still Has a Residual
A contract or insurance policy may transfer a defined portion of a loss, subject to its wording and the ability of the counterparty to perform. The assessment should account for deductibles, limits, exclusions, timing, disputes, and counterparty failure where relevant. It should not treat a nominal transfer as equivalent to eliminating the event, the underlying gross loss, or every related consequence.
For a simple model with a single possible loss, let the loss occur with probability \(p\). If the company’s net cost conditional on the event is \(C\), and there is no such cost when the event does not occur, expected annual cost excluding fixed premiums is \(pC\). This follows from the two possible outcomes: \(E[\text{cost}]=pC+(1-p)0=pC\). If a fixed premium is paid regardless of whether a loss occurs, add that premium to expected annual outflow. This calculation assumes the stated probability, payment terms, and counterparty behavior apply; it does not capture uncertainty beyond those assumptions.
Worked Example: Insurance Limits and Counterparty Default
A hypothetical insurer faces a 10% annual probability of one covered event causing a $1,000,000 loss. Its policy has a $100,000 deductible and a maximum payment of $600,000 per claim. Assume the insurer pays nothing if it defaults, default occurs with 5% probability conditional on a claim, and default is independent of the loss amount. The annual premium is $70,000 and is payable whether or not a claim occurs. Find the expected annual outflow including premium.
Method. First calculate the promised claim payment under the contract, then the expected amount actually collected allowing for default. Subtract the expected collection from the gross event loss to find the company’s expected conditional cost, and finally include event probability and premium.
For a $1,000,000 loss, the amount above the deductible is \(\$1{,}000{,}000-\$100{,}000=\$900{,}000\). The payment is capped at $600,000, so the promised payment is \(\min(\$900{,}000,\$600{,}000)=\$600{,}000\). If the insurer performs, the company retains \(\$1{,}000{,}000-\$600{,}000=\$400{,}000\) of the event loss.
The probability of insurer performance conditional on a claim is \(1-0.05=0.95\). The expected collection conditional on a claim is \(0.95\times \$600{,}000+0.05\times \$0=\$570{,}000\). Thus the company’s expected conditional event cost, including the deductible, loss above the limit, and nonpayment in default, is \(\$1{,}000{,}000-\$570{,}000=\$430{,}000\).
The expected annual event cost is \(0.10\times \$430{,}000=\$43{,}000\). Adding the fixed annual premium gives expected annual outflow of \(\$43{,}000+\$70{,}000=\$113{,}000\).
Check. If the insurer performs, the company bears $400,000 on an event; if it defaults, the company bears $1,000,000. The conditional expectation is \(0.95\times \$400{,}000+0.05\times \$1{,}000{,}000=\$430{,}000\), matching the calculation above. The expected annual outflow including premium is therefore $113,000.
Common mistake and correction. A wrong calculation might treat the full $600,000 limit as certain recovery and report expected annual outflow of \(\$70{,}000+0.10\times \$400{,}000=\$110{,}000\). It ignores the stated default risk. Under the assumptions, expected collection is $570,000 per claim, not $600,000, so the corrected expected annual outflow is $113,000. The contract reduces the company’s expected event cost, but does not remove the deductible, the amount above the limit, or counterparty risk.
A Practical Diagnostic Sequence
When reviewing a control or mitigation assessment, use a sequence that makes unsupported assumptions visible. Keep the conclusion proportionate to the evidence, and state unresolved limitations instead of converting them into assumed protection.
State the event or loss being addressed and the action the control is supposed to perform. A control cannot be assessed meaningfully without a clear objective.
Use policies and procedures to understand intended design. Use relevant, dated evidence to assess whether the control operated over the period and population in scope.
Record what was tested, what passed, what failed, and what the test cannot establish. Do not turn sample results into certainty about untested items.
Determine whether safeguards address distinct pathways, share dependencies, or have a supported combined-effect estimate. Do not multiply standalone reductions without a justified joint model.
Apply contractual terms and relevant counterparty assumptions. Keep uncovered amounts, payment uncertainty, and other retained consequences visible in the assessment.
| Assessment claim | Why it may be wrong | More defensible treatment |
|---|---|---|
| “The policy exists, so the control operated.” | Policy evidence describes the requirement, not necessarily the transactions performed. | Assess operating evidence for the period and state the scope and limits of testing. |
| “Two controls each reduce losses by 40%, so together they reduce losses by 64%.” | The effects may overlap or depend on the same failure pathway. | Use a supported combined-effect model or report the uncertainty. |
| “Insurance transfers the loss.” | Deductibles, limits, exclusions, nonpayment, and other conditions can leave exposure with the insured. | Calculate the covered payment and identify the residual under stated assumptions. |
Common Assessment Traps
- Confusing a requirement with evidence. “Must be approved” describes the control objective; it does not prove that a particular item was approved.
- Overstating what a sample shows. A sample test supports a conclusion about tested items. Any conclusion beyond them needs a suitable basis and must reflect uncertainty.
- Adding or multiplying control effects without checking overlap. A second safeguard may not provide a second full reduction if it addresses the same pathway.
- Reporting transferred exposure as zero. The contractual payment is limited by terms and may be affected by counterparty performance; identify what remains with the organization.
- Mixing expected cost with certainty. An expected annual loss is an average under a model, not a prediction that the exact amount will occur in a given year.
Check Your Understanding
For each scenario, identify the defensible conclusion or calculate the requested amount. Treat the stated assumptions as hypothetical and use only the evidence provided.
- A procedure requires supervisor review of every adjustment. An assessor samples 18 adjustments and finds evidence of review for 16. Calculate the sample evidence rate and sample exception rate. What does this test establish about the entire population? (Answer: The sample evidence rate is \(16/18=0.8889\), or 88.89%; the sample exception rate is \(2/18=0.1111\), or 11.11%. This alone does not establish review for all adjustments in the population.)
- A hypothetical process has an expected frequency of 0.10 loss events per year and a mean severity of $900,000. Two safeguards address the same failure pathway. Joint testing supports a combined frequency reduction of 30%. Calculate the supported expected annual loss, and compare it with incorrectly applying two independent 30% reductions. (Answer: Supported frequency is \(0.10\times0.70=0.07\), so expected annual loss is \(0.07\times\$900{,}000=\$63{,}000\). The unsupported multiplication gives \(0.10\times0.70^2=0.049\) and \(\$44{,}100\).)
- A hypothetical event has a 20% annual probability and causes a $500,000 loss. A contract has a $50,000 deductible and a $300,000 maximum payment. Conditional on a claim, the counterparty has a 10% default probability with zero recovery in default; assume default is independent of the event amount. Excluding any premium, calculate expected annual retained loss. (Answer: The promised payment is \(\min(\$500{,}000-\$50{,}000,\$300{,}000)=\$300{,}000\). Expected collection conditional on a claim is \(0.90\times\$300{,}000=\$270{,}000\), leaving expected conditional cost of \(\$500{,}000-\$270{,}000=\$230{,}000\). Expected annual retained loss is \(0.20\times\$230{,}000=\$46{,}000\).)
- A payment procedure requires approval before release. A test finds approval evidence for every item in a random sample of 25 payments. State one conclusion supported by the test and one conclusion it does not establish. (Answer: The test found approval evidence for all 25 sampled payments. It does not by itself establish approval for every payment in the population or establish the quality of each review.)
- A hypothetical control model starts with an expected 0.50 loss events per year. Two safeguards have distinct failure pathways, and testing supports independent sequential frequency reductions of 20% and 30%. Mean severity is $100,000. Calculate the modeled expected annual loss. (Answer: Remaining frequency is \(0.50\times0.80\times0.70=0.28\) events per year; modeled expected annual loss is \(0.28\times\$100{,}000=\$28{,}000\).)