What an Operating-Effectiveness Test Establishes
A control can be well designed but not performed consistently. Testing operating effectiveness asks whether the control operated as intended, for the relevant population and period, and whether reliable evidence supports that conclusion. For example, a payment approval control may specify an appropriate reviewer and approval timing; a test must still determine whether the sampled payments actually received that approval before release.
A test conclusion is bounded by its scope and evidence. Examining a sample does not prove that every item in the population complied. Instead, it provides evidence about the sampled items and, when the sample and assumptions support it, information about the broader population. This distinction is important when reporting results aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Plan the Test Before Examining Items
A useful test starts with a precise, observable attribute. “Payments are reviewed appropriately” is too vague to test consistently. “Each payment above the stated threshold has a dated approval from an authorized reviewer before release” identifies what evidence to seek and what conditions constitute a deviation. The test should also specify the relevant population, period, unit of selection, evidence sources, and how exceptions or unverified items will be treated.
The population must match the control’s actual scope. If a control applies to all payment releases during a quarter, a list of approved payments alone may omit payments that bypassed approval. Reconcile the population to a suitable complete source, check that the period and unit are correct, and investigate apparent duplicates or omissions before selecting items. For a frequent control, the sampling unit might be one payment or one daily reconciliation; for a monthly control, it may be one monthly performance instance.
Selection should be appropriate to the question. A random sample can support statements about the population under stated sampling assumptions. A targeted selection of unusually large or otherwise higher-risk items can be useful for investigating specific concerns, but it does not automatically represent the population. Do not describe a judgmental selection as though it were a random sample.
State the control action, the required timing or authorization, and the dates the test covers.
Identify all eligible control instances and the item that will be selected and evaluated.
Use a selection method suited to the objective, and obtain records capable of showing what happened and when.
For each item, record whether the attribute is met, a deviation is confirmed, or the evidence is insufficient. Compare results with any acceptance criteria established before testing.
Evidence, Exceptions, and Sample Fractions
Evidence must be relevant to the attribute and reliable enough to support the conclusion. A dated approval record linked to the specific transaction may support whether approval occurred before release. An employee’s recollection may help explain a record, but inquiry alone is generally weaker evidence of a particular control performance than a contemporaneous record. Consider whether records are complete, attributable to the stated reviewer, and protected from alteration. Where possible, corroborate evidence against an independent source, such as a transaction log or system timestamp.
For a sample of \(n\) examined items, let \(e\) denote the number classified as exceptions. The fraction of examined items that meet the control attribute is \((n-e)/n\). The fraction that are exceptions is \(e/n\). These are descriptive fractions for the sample; they are not automatically the true population rates.
These expressions follow by counting the two possible classifications for an item with sufficient evidence: it either meets the attribute or is an exception. If the \(n\) examined items are classified this way, the number meeting the attribute is \(n-e\). Dividing each count by the total \(n\) gives the corresponding sample fraction. Items with insufficient evidence should be tracked separately rather than silently counted among the items that meet the attribute.
A test plan may specify an acceptance rule, such as a maximum number of deviations in a defined sample. That rule must be set before results are known and interpreted within its assumptions. There is no universal sample size or exception cutoff that proves a control is effective in every circumstance. The control’s frequency, the population, the significance of the attribute, the reliability of available evidence, and the consequence of missed deviations all matter.
Worked Example: Testing Approval Before Payment Release
A hypothetical insurer’s payment control requires an authorized manager to approve each claim payment above a stated threshold before release. The control population contains 240 eligible payments during the quarter. A tester selects 30 payments at random from a reconciled population list and finds two with no evidence of approval before release. The test plan, established in advance, permits at most one deviation in this sample. Calculate the sample fractions and apply that plan.
Method. Classify each selected payment against the stated attribute: documented approval by an authorized manager before release. The two items lacking required evidence of prior approval are confirmed deviations for purposes of this hypothetical test.
Step 1: Count items meeting the attribute. There are \(n=30\) examined payments and \(e=2\) exceptions, so \(30-2=28\) sampled payments meet the attribute.
Step 2: Calculate both sample fractions. The fraction meeting the attribute is \(28/30=0.9333\), or 93.33% to two decimal places. The exception fraction is \(2/30=0.0667\), or 6.67% to two decimal places. The two fractions sum to 1.0000.
Step 3: Apply the pre-established plan. The plan permits at most one exception, but the sample has two. Therefore, the sample does not meet the specified acceptance rule. This is a conclusion about the test plan’s criterion, not proof that exactly 6.67% of all quarterly payments deviated.
Check and common mistake. The counts reconcile: 28 meeting the attribute plus 2 exceptions equals 30 examined items. A common error is to label \(e/n=2/30\) as the compliant fraction. It is the exception fraction; the compliant fraction is \((n-e)/n=28/30\).
How to Handle Incomplete or Unreliable Documentation
A missing record does not, by itself, establish every fact about what occurred. The control may have been performed but not documented, or it may not have been performed. The tester should seek relevant corroboration, such as an independently maintained approval log, while avoiding an unsupported assumption that the control worked. If the evidence remains insufficient, classify the item as unverified and explain the limitation.
The control’s requirements affect the interpretation. If retaining an approval record is itself an explicit part of the control, failure to retain it may be a confirmed deviation from that requirement. If the test concerns whether approval occurred and the record is simply unavailable, the underlying approval may be unverified rather than conclusively absent. In either case, do not treat an unverified item as demonstrated compliance.
A clear workpaper records the population source and period, selection method, items selected, attribute tested, evidence inspected, results, follow-up performed, and rationale for each classification. This makes it possible for another reviewer to understand how the conclusion was reached and to distinguish a confirmed deviation from a documentation limitation.
Worked Example: Separating a Confirmed Deviation From Missing Evidence
A hypothetical insurer tests a control requiring a second reviewer to approve each of 120 policy endorsements before the endorsement is issued. The tester randomly selects 20 endorsements. Seventeen have reliable, dated evidence of approval before issue. One has reliable evidence that the second approval occurred only after issue. For two others, the approval records are unavailable, and the tester finds no reliable corroborating evidence. The pre-established test plan requires every selected item to have sufficient evidence and permits no confirmed deviations. Classify the results and evaluate the plan.
Method. Separate items with evidence of compliance, confirmed deviations, and insufficient evidence; do not combine the last two categories as though they establish the same fact.
Step 1: Tally the classifications. Seventeen endorsements meet the attribute. One is a confirmed deviation because approval was after issue. Two are unverified because the available evidence cannot establish whether timely approval occurred. The counts reconcile: \(17+1+2=20\).
Step 2: Calculate relevant fractions. The verified-compliant fraction is \(17/20=0.8500\), or 85.00%. The confirmed-deviation fraction is \(1/20=0.0500\), or 5.00%. The unverified fraction is \(2/20=0.1000\), or 10.00%. If the tester reports the combined fraction not demonstrated to meet the attribute, it is \((1+2)/20=3/20=0.1500\), or 15.00%; this combined figure must not be described as a confirmed deviation rate.
Step 3: Apply the stated plan. The plan requires adequate evidence for every selected item and zero confirmed deviations. The test does not satisfy the plan: there is one confirmed deviation, and two items lack sufficient evidence. The result does not establish that the two unverified approvals were absent; it establishes that this test could not verify them.
Check and common mistake. The three fractions, 0.8500, 0.0500, and 0.1000, sum to 1.0000. Counting all three items as confirmed control failures overstates what the evidence shows. Counting the two undocumented items as compliant makes the opposite error.
What a Sample Can—and Cannot—Show
A sample result is evidence, not certainty about unexamined items. Under a simple model in which each tested item is an independent draw with the same probability \(p\) of being a deviation, the number of exceptions in a sample of size \(n\) has a binomial distribution. Under that model, the probability of observing zero exceptions is \((1-p)^n\). This calculation depends on the independent, equal-probability model; it is not automatically an exact description of a finite population sampled without replacement.
Worked Example: Interpreting a Sample With No Exceptions
A hypothetical insurer performs a daily reconciliation control. A tester randomly selects 12 days from a stable process and finds no deviations. To illustrate the limits of a zero-exception sample, assume each selected day independently has a 10% probability of a deviation, with the same probability on every day. Under that model, calculate the probability of seeing zero deviations even when the deviation probability is 10%.
Method. Use the independent Bernoulli model specified in the problem. For one day, the probability of no deviation is \(1-0.10=0.90\). Independence means the probability of no deviations on all 12 selected days is the product of the 12 individual probabilities.
Calculation.
Rounded to four decimal places, the probability is 0.2824, or 28.24% to two decimal places. Thus, even under the stated 10% deviation model, a sample of 12 can produce no observed exceptions. Zero exceptions in a sample are not proof that the population has no deviations.
Check and common mistake. The probability is \(0.90^{12}\), not \(0.10^{12}\): the calculation concerns twelve days with no deviation. This result illustrates uncertainty under the stated model; it should not be used as an exact finite-population probability unless the sampling assumptions justify that model.
Common Exam Traps and Reporting Discipline
- Reversing the fractions. If \(e\) is the number of exceptions, \(e/n\) is the exception fraction, while \((n-e)/n\) is the fraction meeting the attribute.
- Calling every missing record a confirmed failure. First determine whether the control requirement itself includes documentation, and whether other reliable evidence resolves what happened.
- Calling missing evidence compliance. An item that cannot be verified has not been demonstrated to meet the attribute.
- Generalizing a targeted sample. A sample selected to investigate specific high-risk items may be useful, but its proportions do not automatically estimate the whole population.
- Treating a sample cutoff as a universal rule. Apply only the acceptance criteria specified for the particular test and keep the conclusion within the scope of those criteria.
A precise conclusion states what was tested, what evidence supported the classifications, how many sampled items met the attribute, and how many were exceptions or unverified. It also states the selection method and any limits on inference. This discipline prevents both unwarranted assurance from a clean sample and unwarranted claims from incomplete records.
Check Your Understanding
For each problem, distinguish what the sample shows from what it cannot establish about the population.
- A hypothetical insurer selects 18 claim files to test a required pre-payment review. Three are confirmed exceptions and the remaining files have reliable evidence of timely review. Calculate the sample fraction meeting the attribute and the exception fraction. (Answer: \(15/18=0.8333\) meet the attribute; \(3/18=0.1667\) are exceptions.)
- A sample of 40 transactions contains two confirmed deviations and one item whose records are unavailable. What are the confirmed-deviation fraction and the fraction not demonstrated to meet the attribute if the unverified item is included? (Answer: Confirmed deviations \(2/40=0.0500\); not demonstrated \(3/40=0.0750\).)
- A test plan set before testing permits at most one deviation in a random sample of 25. The tester finds two confirmed deviations and no unverified items. Does the sample meet the stated plan? (Answer: No; two deviations exceed the maximum of one.)
- Under an independent Bernoulli model, each tested item has a 20% probability of deviation. What is the probability of observing zero deviations in a sample of six? (Answer: \(0.80^6=0.2621\), to four decimal places.)
- A tester randomly selects 16 items and finds one confirmed exception; all other items have reliable evidence of compliance. Calculate the fraction meeting the attribute and state whether the result proves that the full population has the same rate. (Answer: \(15/16=0.9375\) meet the attribute and \(1/16=0.0625\) are exceptions; the sample does not prove that the population has the same rate.)