Why Audit a Completed Case Response?
A polished report can still make a decision less reliable if its conclusions exceed its evidence. For example, a model may predict an outcome accurately without identifying what would change that outcome; a reported accuracy may use a different threshold from the one the recommendation depends on; or a test set may have been reused until a preferred model emerged. The final audit is a check that the response’s claims and proposed action follow from the analysis actually performed.
This audit is not a substitute for fitting or validating a model. It checks whether a completed case response accurately describes its evidence, recognizes material limitations, and tells a decision-maker what to do next. The approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A useful audit follows the reasoning chain rather than simply proofreading sentences. For each major conclusion, ask what evidence supports it, whether the evidence was evaluated in an appropriate way, and whether the proposed action follows from the result. The following sequence keeps those checks distinct.
Separate a statement about prediction from a statement about cause, impact, or a recommended intervention.
Confirm that the data used to select or tune the model were not described as an independent final test.
Match the stated probability threshold to the confusion matrix and any threshold-dependent metrics.
Qualify feature-importance claims and state limitations that affect how the findings can be used.
Specify the action, the conditions for taking it, and what should be monitored or evaluated next.
Prediction Is Not a Causal Conclusion
A predictive model estimates outcomes from observed information. A feature that helps distinguish higher-risk from lower-risk cases may be useful for targeting review, but that association alone does not show that changing the feature will change the outcome. A causal conclusion requires appropriate evidence and assumptions for the causal question; predictive performance by itself does not supply them.
For example, a reminder may be sent more often to customers already considered at higher risk. If reminded customers subsequently have more missed payments, the pattern does not establish that reminders caused the missed payments. The selection process may help explain the association. A report should describe what the model predicts and avoid describing an intervention effect unless the study design and analysis support that effect.
Worked Example: Auditing a Causal Claim About Reminders
A hypothetical service team fits a model to predict missed payments. In an observational portfolio, 36 of 200 customers who received a reminder missed a payment; 40 of 500 customers who did not receive a reminder missed a payment. The model has a reported test AUC of 0.78. A draft says, “Reminders increase missed payments, so we should stop sending them.” Audit the numerical comparison and the conclusion.
Method. Calculate the observed outcome rates, then check whether the comparison identifies a causal effect. The missed-payment rate among reminded customers is \(36/200=0.18\), or 18%. Among customers not reminded, it is \(40/500=0.08\), or 8%. The observed difference is \(0.18-0.08=0.10\), or 10 percentage points. The observed rate ratio is \(0.18/0.08=2.25\).
Those calculations describe an association in the observed groups. They do not show what would have happened to the same customers under the opposite reminder decision. If reminders were preferentially sent to customers already considered more likely to miss payments, the groups differ before the reminder. The AUC describes the model’s ranking performance; it does not correct that group comparison or establish a reminder effect.
Audit correction. Replace the causal sentence with a statement such as: “Customers who received reminders had a higher observed missed-payment rate in this portfolio. Because reminder assignment may reflect pre-existing risk, this comparison does not establish that reminders increase missed payments. The model’s AUC of 0.78 measures predictive discrimination, not the causal effect of reminders.” If the team wants to estimate the effect of a reminder policy, it needs an appropriate causal design and analysis.
Common mistake avoided. Treating the difference between two observed rates as the effect of an intervention confuses association with causation. The arithmetic is valid; the causal interpretation is not justified by the information provided.
Protect the Role of the Test Set
An evaluation set can provide an independent assessment only when it has not been used to make the modelling choices being assessed. If several candidate models or configurations are compared on the same test set and the best result is selected, that set has influenced selection. The selected score can then reflect both model quality and favourable random variation in that particular evaluation sample.
The reason is visible in a simple case. Suppose several candidates have the same underlying performance, and each observed score is the true performance plus a mean-zero evaluation error. The selected score is the largest of those observed values. Since the largest error is at least the average error, its expected value cannot be below the average error, which has expectation zero:
This argument illustrates selection optimism when candidates have equal true performance; it does not calculate the size of the bias for a particular case. When candidate models differ in true performance, the selected observed score also reflects those differences. The audit question is practical: was the reported test result used to choose, tune, or repeatedly revise the model? If so, do not present it as though it were an untouched final evaluation. Use a new, appropriately protected test set for a final assessment, or use a validation design that accounts for the model-selection process.
Worked Example: Auditing Repeated Test-Set Selection
A hypothetical review team compares 12 model configurations on one test set. It records AUC values of 0.82, 0.84, and 0.81 for three shortlisted configurations, tries further variations after inspecting the results, and then reports the 0.84 result as “the final independent test AUC.” What can the audit conclude, and what should the team report instead?
Method. Trace whether the test set influenced selection. It did: the team compared configurations, inspected their results, and used those results to guide further variations. The reported 0.84 is the largest of the three displayed values, but those values are not the full selection record because nine other configurations and subsequent variations were also considered.
The audit cannot calculate an unbiased replacement AUC from the values 0.82, 0.84, and 0.81. They do not reveal the true performance, the variability of the scores, or the results of all later variations. It would be incorrect to subtract an invented “selection adjustment” from 0.84.
Audit correction. Report that 0.84 was obtained on data used in model selection and is therefore not an independent final test estimate. To assess the selected process, evaluate it once on a genuinely untouched test set, if one is available. If not, describe the limitation and use an appropriate validation strategy for any new performance estimate. Keep the model-selection record transparent.
Common mistake avoided. Calling a data set “test” does not make its result independent after the team has used it to choose the model. The data’s role is determined by how they were used.
Thresholds Must Match the Reported Metrics
Many classification models produce a predicted probability \(p\) for a positive outcome. To create a class prediction, a decision rule must specify a threshold \(t\): predict positive when \(p\geq t\), and negative otherwise. A confusion matrix and metrics such as accuracy, sensitivity, and precision depend on that threshold. AUC, by contrast, summarizes ranking across thresholds and does not identify one operational cutoff.
An audit should therefore check that the threshold in the recommendation is the threshold used for every reported confusion-matrix result. It should also check whether the threshold was selected using training or validation data, rather than chosen after examining the final test results.
Worked Example: Correcting a Threshold Mismatch
A hypothetical screening model is evaluated on 400 cases: 40 have the positive outcome and 360 do not. At threshold \(t=0.50\), the confusion matrix has 30 true positives, 10 false negatives, 40 false positives, and 320 true negatives. The draft claims that the operational threshold is \(t=0.25\), but reports the \(t=0.50\) metrics as evidence for that policy. Audit the metrics at 0.50 and explain what is missing.
Method. Calculate the threshold-specific metrics from the stated counts. The accuracy is \((30+320)/400=350/400=0.875\). The sensitivity is \(30/(30+10)=30/40=0.75\). The precision is \(30/(30+40)=30/70\approx0.4286\). These metrics are correct for the stated \(t=0.50\) matrix.
The matrix does not give the counts at \(t=0.25\). Lowering the threshold can change which cases are classified positive, but the direction alone does not provide the new counts or metrics. The draft therefore cannot use the displayed accuracy, sensitivity, or precision as though they described the \(t=0.25\) operating rule.
Audit correction. Either report the matrix and its metrics explicitly for \(t=0.50\), or provide predictions at \(t=0.25\) and calculate the corresponding results. Then justify the chosen threshold in the context of the decision, such as the relative costs of missed positives and unnecessary reviews.
Common mistake avoided. Reporting an accuracy from one cutoff beside a recommendation using another cutoff is not a minor labelling issue: it gives the reader performance for a different classification rule.
Interpret Importance Carefully, Then Make the Action Clear
A feature-importance measure describes a model-specific contribution under a particular definition and evaluation procedure. For example, permutation importance measures how much a chosen performance measure changes when a feature’s values are permuted. A large importance value can suggest that the fitted model relies on information associated with that feature. It does not, on its own, establish that the feature causes the outcome, indicate the direction of an effect, or guarantee that the feature will remain important in another population.
Importance can also be difficult to interpret when predictors are correlated: information shared across predictors can make their individual importance values unstable or distribute importance among them. An audit should ask which importance method was used, what it measures, and whether the wording in the report stays within that scope. “The model relied more on feature A than feature B under this calculation” is more defensible than “feature A drives the outcome.”
A useful recommendation is specific enough to guide a decision but conditional enough to respect the evidence. It identifies an action, the population or setting to which the action applies, the reason for taking it, and any evaluation or monitoring needed before broader use. A list of metrics without a decision, or a decision with no stated basis, leaves the reader to supply the missing reasoning.
Worked Example: Auditing Importance, Limitations, and a Recommendation
A hypothetical team evaluates a model to flag applications for manual fraud review. At threshold \(t=0.20\), its held-out assessment contains 60 true positives, 40 false negatives, 100 false positives, and 800 true negatives. At \(t=0.35\), it contains 50 true positives, 50 false negatives, 50 false positives, and 850 true negatives. The team assigns an assumed decision cost of $500 per false negative and $20 per false positive. A permutation analysis shows that permuting income reduces AUC by 0.06, while permuting address-change count reduces it by 0.01. The draft says, “Income causes fraud; deploy the model for all applicants because the $0.20 threshold is cheapest.” Audit the claim and formulate a better recommendation.
Method. First check the threshold comparison using the supplied costs. At \(t=0.20\), the assumed cost is \(40\times \$500+100\times \$20=\$20{,}000+\$2{,}000=\$22{,}000\). At \(t=0.35\), it is \(50\times \$500+50\times \$20=\$25{,}000+\$1{,}000=\$26{,}000\). Under these stated costs and counts, \(t=0.20\) has an assumed cost $4,000 lower for the assessed cases.
The permutation results support only a model-specific comparison under the stated importance procedure: permuting income reduced AUC more than permuting address-change count in this analysis. They do not show that income causes fraud, establish the direction of any relationship, or prove that income will have the same importance elsewhere.
The calculated costs also depend on the assumed cost amounts and the evaluated cases. The draft does not establish that these assumptions apply to all applicants, that all relevant review costs are included, or that performance will persist after deployment. These are material limitations to an immediate all-applicant rollout.
Audit correction. A decision-focused recommendation could be: “Under the stated false-negative and false-positive costs, threshold \(t=0.20\) has the lower estimated cost in this assessment. Consider a limited, monitored pilot using this threshold, with human review of flagged applications. Treat income as an important predictive feature under the reported permutation analysis, not as a cause of fraud. Before expanding the policy, verify that the cost assumptions and performance are appropriate for the intended applicant population and monitor outcomes.” This names the action, the evidence, and conditions for broader use.
Common mistake avoided. The lower calculated cost does not prove that the feature causes the outcome or that the policy is suitable for every population. A comparison under stated assumptions supports a conditional decision, not an unconditional deployment claim.
Final Audit Checklist
Before accepting a completed case response, examine whether each conclusion survives the following checks. A response need not claim certainty; it should make the boundary between evidence and interpretation visible.
- Causal scope: Does the response distinguish observed association and prediction from an intervention effect?
- Evaluation integrity: Was the reported test result used to choose or tune the model? If so, is that limitation acknowledged and the result described accurately?
- Threshold consistency: Do the confusion matrix and threshold-dependent metrics match the threshold in the recommendation?
- Importance scope: Does the report name what the importance measure indicates without turning it into a causal or universal claim?
- Limitations: Are relevant restrictions on data, population, assumptions, or evaluation stated in a way that affects how the result should be used?
- Action clarity: Does the recommendation say what to do, for whom, on what evidence, and what should be checked next?
A concise audit does not need to repeat every modelling detail. It does need to ensure that the decision-maker is not asked to infer an unstated threshold, overlook reuse of evaluation data, or accept a causal claim that the analysis did not establish.
Check Your Understanding
For each case, identify the audit issue and determine the defensible conclusion or calculation.
- A classifier is assessed on 300 cases, with 30 positives and 270 negatives. At the stated threshold, it has 24 true positives, 6 false negatives, 30 false positives, and 240 true negatives. Calculate accuracy, sensitivity, and precision. (Answer: Accuracy \(=264/300=0.8800\); sensitivity \(=24/30=0.8000\); precision \(=24/54\approx0.4444\).)
- A team tests five model configurations on its test set, selects the highest-AUC model, then reports that AUC as an independent final estimate. What is the audit issue, and what would provide a more defensible final assessment? (Answer: The test set influenced model selection, so the reported score is not an independent final assessment; evaluate the selected process on an untouched test set or use a validation design that accounts for selection.)
- A fitted model has higher permutation importance for a customer’s account age than for a service-contact count. A draft claims that account age prevents the outcome. What can the importance result support, and what does it not establish? (Answer: It supports that account age had greater model-specific importance under the stated permutation procedure; it does not establish a causal effect, direction, or universal importance.)
- Under an explicitly assumed cost of $300 per false negative and $15 per false positive, threshold A has 20 false negatives and 40 false positives; threshold B has 30 false negatives and 10 false positives. Which has lower assumed cost, and by how much? (Answer: A costs \(20\times\$300+40\times\$15=\$6{,}600\); B costs \(30\times\$300+10\times\$15=\$9{,}150\); A is lower by $2,550 under these assumptions.)
- A model performs well on a historical population, but a proposed deployment covers a new region and the report gives no information about population comparability. State a defensible recommendation style. (Answer: Recommend only a limited, monitored evaluation in the new region, describe the historical-population evidence as a limitation, and check performance and relevant outcomes before broader deployment.)