What Operating-Effectiveness Testing Establishes
A control may be designed appropriately but still fail to reduce risk if it is skipped, performed late, or carried out by someone without the required authority. Testing operating effectiveness asks a different question from evaluating control design: did the control operate as required, for the relevant instances, throughout the period being assessed? That answer helps risk managers judge whether reliance on the control is justified.
A conclusion applies only to a stated control, a defined period, and the instances examined. It is not proof that no loss occurred or that every future control execution will succeed. Testing should therefore begin with clear criteria and finish with a conclusion whose scope matches the evidence.
Design and operation are related but separate. A control can be well designed and operate poorly; it can also be performed exactly as specified even though its design does not address the risk adequately. Testing operation does not repair a design weakness. It assesses whether the control that was specified actually functioned in practice.
Set the Test Criteria Before Reviewing Evidence
A useful test translates the control description into observable criteria. For example, “review unusual payments” is too vague by itself. The tester needs to know which payments are covered, what counts as unusual, who must review them, what the reviewer must do, and whether review must occur before payment. Those details determine whether a particular instance conforms.
Before examining results, state the period and the relevant population: the full set of control instances that should have occurred under the control’s stated frequency and scope. A monthly control expected to operate in each month has 12 expected instances in a full year. A control applied to every transaction has a population defined by the transactions meeting its stated criteria during the period. A missing or incomplete population can make even carefully reviewed evidence misleading.
Specify the required action, responsible role or system, timing, frequency, and any required approval or documentation.
Identify which instances should have been controlled and how the list of instances is determined. Reconcile the list to a reliable source where practical.
Determine whether the evidence supports that the specified action occurred, at the required time, and by an authorized person or system.
Record conforming instances, exceptions, and unresolved evidence gaps. Explain how each finding affects reliance on the control.
Evidence must be relevant to the requirement being tested. A dated approval record may support who approved an item and when; it does not by itself establish that the approver checked the required information. Conversely, a completed checklist without a reliable date may not establish that the review occurred before release. A test should distinguish what evidence demonstrates from what it merely suggests.
An exception is an instance in which a stated requirement was not met. Examples include an omitted review, late approval when approval must precede action, an unauthorized performer, or missing evidence when execution cannot otherwise be established. A missing record does not always prove that the control was not performed, but it means operation has not been demonstrated by that record. If reliable alternative evidence is unavailable, the conclusion should not treat the instance as confirmed successful.
Worked Examples
Worked Example: Monthly Approval Before Payment
A hypothetical insurer requires the finance manager to review and approve a monthly schedule of unusually large claim payments before those payments are released. The review covers January through December. The tester verifies the schedule for each month and matches the dated approval to the payment-release date. Eleven approvals occurred before release; in one month, payment was released on the 18th and approval was recorded on the 20th. Calculate the observed exception rate and assess the result.
Method. Define one expected control instance per month, so the population contains \(12\) instances. The requirement is approval before release, not simply an approval at some point in the month.
The number of instances meeting the timing requirement is \(11\). The number of exceptions is \(12-11=1\). Thus:
The calculation is rounded to one decimal place. The late approval is an exception because the sequence required by the control was not followed. The evidence supports 11 timely approvals and one failure for the period. Whether that finding is acceptable for reliance depends on pre-established criteria and the significance of the missed approval; the rate alone does not settle that judgment.
Common mistake avoided: Counting all 12 approvals as successful because each month eventually had an approval. That ignores the explicit “before payment” requirement. The correction is to compare the approval date with the release date for every instance.
Worked Example: Daily Reconciliation With Incomplete Execution Evidence
A hypothetical claims operation is required to reconcile a payment file each business day. The control period contains \(60\) business days. System records show that the reconciliation was completed on time on \(58\) days, completed after the required deadline on one day, and have no reliable execution record for one day. Assume the control requirement is completion by the end of each business day. Determine the demonstrated timely-completion rate and classify the results.
Method. Use the 60 expected daily executions as the denominator. Only the 58 documented, timely completions meet the stated requirement. The late execution is a timing exception; the day without reliable evidence is unverified and cannot be counted as demonstrated success.
The complementary share is \(2/60=0.0333\ldots\), or \(3.3\%\), consisting of one late instance and one instance whose operation was not demonstrated. The percentages sum to \(100.0\%\) after rounding. The evidence supports 58 timely executions, not 59: an absent record cannot be silently treated as proof that the reconciliation occurred.
If the control requires timely completion every business day, the observed record does not show full compliance with that requirement. The conclusion should identify the two different findings rather than describing both simply as “missed reconciliations.” Further evidence could resolve the undocumented day, but until then that instance remains unverified.
Common mistake avoided: Reporting \(59/60=98.3\%\) by counting the undocumented day as successful, or treating late completion as timely. The correction is to classify each instance against the actual deadline and count only supported, conforming execution as successful.
Worked Example: Different Exceptions Have Different Significance
A hypothetical review examines \(30\) payment approvals against a control requiring an authorized manager to approve each payment before release and to retain a specified supporting attachment. Twenty-seven instances satisfy both requirements. Two have an approval before release but no attachment, and one was approved before release by an employee who lacked the required authority. Assume no reliable alternative evidence establishes the missing attachments. Calculate the observed exception rate and describe the findings.
Method. Treat an instance as conforming only if it meets every stated requirement. The two missing attachments and the unauthorized approval each fail at least one requirement, so the exception count is \(2+1=3\). The other \(27\) instances conform.
The rate is rounded to one decimal place. The findings should be described by type: two instances lack evidence required by the control, while one was approved by an unauthorized person. Those are not interchangeable. The missing attachments prevent the tester from establishing that a required part of the control was completed; the unauthorized approval directly violates the authority requirement.
The observed rate summarizes this review set. If the set was selected to focus on particular risk indicators rather than to represent the entire population, the \(10.0\%\) must not be presented as an estimate of the population-wide exception rate. The conclusion can still report the observed exceptions and their nature.
Common mistake avoided: Calling the control effective because \(90\%\) of reviewed instances passed, without considering the type of failure or how the review set was chosen. The correction is to report the rate alongside the exception details and the limits of the evidence.
Interpret Results Without Overstating Them
An observed exception rate is a descriptive summary of the instances examined. If \(n\) instances are examined and \(e\) are exceptions, then the observed rate is \(e/n\). This follows because the \(n\) examined instances divide into \(e\) exceptions and \(n-e\) non-exceptions; the fraction of examined instances that are exceptions is therefore \(e/n\). This calculation does not establish the rate for unexamined instances. Drawing a population conclusion from a sample requires an appropriate sampling approach, which is a separate question from describing the results in hand.
A percentage also does not capture severity by itself. One unauthorized approval may matter more than several minor documentation defects, depending on the control objective and risk. Conversely, a low observed rate does not automatically establish reliable operation, particularly where the control is intended to prevent a severe exposure. Apply criteria set before testing, and state any qualitative judgment explicitly.
A defensible conclusion makes clear:
- which control and control requirements were assessed;
- the period and population covered, and any limitation in that coverage;
- what evidence supported execution and what remained unverified;
- the number and nature of exceptions; and
- whether the evidence supports reliance on the control for the stated purpose.
A control can operate exactly as designed and still be inadequate if its design does not address the risk. Equally, a design that appears sound does not justify assuming consistent operation. Keeping the two assessments separate is essential to a useful control evaluation.
Check Your Understanding
For each problem, classify the evidence against the stated requirement and avoid inferring more than the facts support.
- A hypothetical quarterly reserve review must be completed in each of four quarters before the reserve estimate is submitted. Three reviews were completed beforehand; one was completed the day after submission. How many exceptions occurred, and what is the observed exception rate? (Answer: 1 exception; \(1/4=25.0\%\).)
- A hypothetical automated access report should be reviewed monthly for six months. Five dated records show review by the authorized reviewer; for the sixth month there is no review record and no alternative evidence. How many successful instances are demonstrated, and what is the demonstrated completion rate? (Answer: 5 of 6; \(5/6=83.3\%\) rounded to one decimal place; the sixth instance is unverified, not demonstrated success.)
- A hypothetical transaction control requires approval by a designated officer before execution. In a review of 25 instances, 24 satisfy both conditions and one was approved by the officer after execution. Calculate the observed exception rate and identify the failed condition. (Answer: \(1/25=4.0\%\); the approval was not obtained before execution.)
- A hypothetical review examines 20 high-value transactions selected specifically because they triggered a risk alert. Two lack evidence of the required review. What is the exception rate in the reviewed set, and may it be reported as the rate for all transactions? (Answer: \(2/20=10.0\%\) in the reviewed set; no, the alert-based selection does not establish that the set represents all transactions.)
- A hypothetical daily control has 40 expected executions. Records show 37 timely completions, two late completions, and one execution with no reliable evidence. State the timely-completion rate and the number of exceptions or unverified instances. (Answer: \(37/40=92.5\%\); 3 instances are either exceptions or unverified, comprising 2 late and 1 unverified instance.)