Why an Evidence Chain Matters
A model report can contain accurate tables and still fail to justify its recommendation. That happens when a reader cannot trace the proposed action back through the validation results, modeling choices, and data to the original objective. An auditable evidence chain makes those connections explicit: each link states what was done, what evidence resulted, and what that evidence permits the report to conclude.
For example, a report might recommend sending predicted high-risk policies for manual review. To assess that recommendation, a reviewer needs to know what “high risk” means, which population was evaluated, how the model produced the predictions, how validation was performed, and whether the reported results measure performance relevant to manual review. A polished summary cannot substitute for those links. This approach to reporting is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The chain is not simply a list of model facts. For each conclusion, ask three questions: What evidence supports it? Where did that evidence come from? What does it justify—and what does it not justify? A recommendation should not claim more than the linked evidence establishes.
Build the Chain in Traceable Steps
A useful report gives each link enough detail to be checked without making the reader infer important decisions. The records need not all appear in the main narrative; a concise report can point to clearly named tables, code, or data specifications. The essential point is that a reviewer can follow the references and establish which version of each item was used.
Name the outcome to predict, the prediction point, the population of interest, and the intended use. Keep the target and the action distinct: predicting an outcome does not by itself determine what action to take.
Identify the observation unit, inclusion period, target definition, data sources, and any exclusions. State which predictors are available at the prediction point. A later-created field cannot be treated as information available at that earlier point.
Name the model version and the material choices needed to interpret its results, including the predictor set, preprocessing, and any classification threshold. Make it possible to connect the fitted model to the data and code that produced it.
Identify the validation sample and evaluation metric, report the relevant result, and provide counts or calculations that allow a reviewer to check it. Do not present a metric without its population, denominator, or evaluation context.
Explain which evidence supports the proposed action, note material limitations, and avoid extending a result beyond the population, outcome, or use that was evaluated.
The links are logical as well as documentary. If the objective is to identify as many future events as possible for review, a metric that mainly reflects the number of correct non-event classifications may not be enough. If the validation data do not represent the intended population, even correctly calculated metrics may provide weak support for deployment. The chain is useful precisely because it exposes these mismatches.
Check That the Numbers Reconcile
Reconciliation is one practical way to make a report auditable. The metric, the cases it summarizes, and any subgroup results should agree. For regression, the root mean squared error (RMSE) on \(n\) evaluated cases is the square root of the mean of the squared residuals, where each residual is observed outcome minus predicted outcome. This aggregation assumes the segment results use the same outcome scale and the same squared-error definition, and that each case receives equal weight. It does not apply unchanged when segment results use different weights or definitions.
Here, \(n_j\) is the number of evaluated cases in segment \(j\). The result is not the ordinary weighted average of the segment RMSE values. To see why, square each segment RMSE to recover that segment’s mean squared error. Multiply by its case count to recover its sum of squared residuals. Add those sums, divide by the total number of cases, and then take the square root. Each step preserves the definition of the overall RMSE.
Worked Example: Reconciling Segment and Overall Error
A hypothetical insurer evaluates a model’s predicted claim cost on 100 held-out policies. All errors are measured in dollars, every policy has equal weight, and the two disjoint segments use the same target definition. Segment A contains 40 policies and has RMSE of $100; Segment B contains 60 policies and has RMSE of $200. Verify the overall RMSE and identify what a report should say.
Method. Recover each segment’s sum of squared residuals by squaring its RMSE and multiplying by its number of policies.
For Segment A, the mean squared error is \(100^2=10{,}000\) dollars squared. Its sum of squared residuals is \(40\times10{,}000=400{,}000\).
For Segment B, the mean squared error is \(200^2=40{,}000\) dollars squared. Its sum of squared residuals is \(60\times40{,}000=2{,}400{,}000\).
The total sum of squared residuals is \(400{,}000+2{,}400{,}000=2{,}800{,}000\), and the total case count is \(40+60=100\). Therefore, the overall mean squared error is \(2{,}800{,}000/100=28{,}000\) dollars squared, and the overall RMSE is \(\sqrt{28{,}000}=\$167.332\ldots\). Rounded to the nearest cent, it is $167.33.
Check. The overall value lies between the two segment RMSE values, and the larger segment with higher error pulls it closer to $200 than to $100. A report that instead gives \((40\times100+60\times200)/100=\$160\) has averaged the RMSE values rather than combining squared errors. That common mistake does not reproduce the overall RMSE.
Reporting implication. The evidence supports an overall held-out RMSE of $167.33 for these 100 equally weighted policies under the stated target definition. It does not, by itself, establish that the model is accurate enough for a particular business action; that requires an objective and a decision standard against which to interpret the error.
Connect Validation Results to the Proposed Action
Classification results also need a visible path from counts to metrics and from metrics to the use proposed. Consider a binary outcome with a chosen classification threshold. A true positive is an actual positive classified as positive; a false positive is an actual negative classified as positive; a false negative is an actual positive classified as negative; a true negative is an actual negative classified as negative. Sensitivity is the proportion of actual positives classified as positive. Precision is the proportion of positive classifications that are actual positives. These measures answer different questions, so the report should name the one relevant to its stated objective.
Worked Example: Checking a Review Recommendation
A hypothetical health plan evaluates a fixed classification threshold on 100 held-out members. There are 20 actual high-cost cases. The model classifies 25 members as positive; 15 of those are actual high-cost cases. The proposed action is to send positive classifications for manual review. Check the counts, calculate key measures, and state what the evidence supports.
Method. First fill in the four outcome counts. Of the 20 actual positive cases, 15 are classified positive, so there are \(20-15=5\) false negatives. There are 25 positive classifications, of which 15 are true positives, so there are \(25-15=10\) false positives. The remaining actual negatives number \(100-20=80\); after subtracting the 10 false positives, there are \(80-10=70\) true negatives.
The counts sum to \(15+5+10+70=100\), as required. Sensitivity is \(15/(15+5)=15/20=0.75\), or 75%. Precision is \(15/(15+10)=15/25=0.60\), or 60%. Accuracy is \((15+70)/100=85/100=0.85\), or 85%.
Check. The model identifies 15 of the 20 actual high-cost cases and sends 25 members for review. If the stated operational aim is to identify at least 70% of actual high-cost cases for review, the observed sensitivity of 75% clears that specified target on this evaluation sample. The evidence does not establish that a review will reduce costs, nor does it establish performance in a different population.
Common mistake. Reporting “85% successful” from accuracy alone and treating that as proof that the review action is worthwhile skips the objective. Accuracy combines correct positives and correct negatives; it does not show what fraction of actual high-cost cases were captured. Here, sensitivity answers that capture question, while precision describes how many of the referred members were actual high-cost cases. Both should be interpreted in the context of the proposed review process.
Preserve Data and Model Lineage
A reviewer also needs to know that the reported validation result belongs to the stated data and model. A compact audit trail can identify the data extract and date, the target and predictor definitions, the split or validation procedure, the preprocessing and model versions, the threshold if one is used, and the output table from which reported metrics were calculated. If a result is revised, the report should make clear which version is current and what changed.
Timing is particularly important. A predictor must be available at the prediction point specified by the objective. A field recorded after the outcome cannot support a prediction made before that outcome, even if it improves validation results. Similarly, a metric computed on one population should not silently be presented as evidence about another. These are breaks in the chain: they sever the connection between the stated use and the evidence offered for it.
Worked Example: Finding a Broken Validation Link
A hypothetical hospital wants to flag patients for follow-up using information available at discharge. A report evaluates 1,000 held-out patients, including 100 who are readmitted within 30 days. Its initial confusion counts are 90 true positives, 10 false negatives, 70 false positives, and 830 true negatives. The resulting accuracy is \((90+830)/1{,}000=920/1{,}000=92\%\). A review of the predictor specification then finds a “follow-up appointment completed” field that is populated after the 30-day outcome window.
Method. Trace the prediction point in the objective—discharge—to the time each predictor is available. The appointment-completion field is recorded later, so it cannot be used as evidence about what the model could know at discharge. Its presence breaks the link between the stated use and the evaluated model.
The team removes that field and re-evaluates the model on the same 1,000 held-out patients. The corrected counts are 72 true positives, 28 false negatives, 180 false positives, and 720 true negatives. They sum to \(72+28+180+720=1{,}000\). Corrected accuracy is \((72+720)/1{,}000=792/1{,}000=79.2\%\). Corrected sensitivity is \(72/(72+28)=72/100=72\%\).
Check. The corrected count of actual positive cases is \(72+28=100\), matching the 100 readmissions in the evaluation sample. The corrected negative count is \(180+720=900\), matching the other 900 patients. The calculations therefore reconcile with the stated sample.
Recommendation and limitation. The original 92% accuracy does not support a discharge-time recommendation because its model used information unavailable at discharge. The corrected result is the relevant validation evidence, but whether 72% sensitivity is adequate depends on a pre-specified operational standard and the cost and capacity of follow-up. The report should state that the corrected model was evaluated on these 1,000 held-out patients and should not present the invalid result as a basis for action.
State What the Chain Does Not Establish
An auditable report is not necessarily a report in favor of deployment. Traceability makes it possible to distinguish a supported conclusion from an unsupported one, including when the correct recommendation is to gather more evidence. Before finalizing, verify that the stated population matches the data, the predictors were available at the prediction point, the evaluation procedure matches the claim, and the reported numbers reconcile with their underlying counts or outputs.
Also distinguish a validation finding from a business effect. A model’s ability to predict an outcome does not, on its own, show that acting on predictions will improve outcomes. Nor does a result on one held-out sample guarantee the same performance elsewhere. A report should name these limits where they affect the proposed action rather than relying on a general disclaimer detached from the evidence.
A practical final review is to select each sentence that supports the recommendation and trace it backward: to its metric, to its validation cases, to the model version, to the eligible data, and ultimately to the objective. If any link is missing or inconsistent, qualify the conclusion or repair the analysis before relying on it.
Check Your Understanding
For each situation, identify the audit check or calculation needed and compare your conclusion with the answer.
- A held-out regression sample has 100 cases in Segment A with RMSE 10 and 300 cases in Segment B with RMSE 20. The outcome scale and equal case weighting are the same in both segments. What is the overall RMSE, rounded to two decimals? (Answer: \(\sqrt{325}=18.03\).)
- On 200 held-out cases, a classifier has 24 true positives, 6 false negatives, 18 false positives, and 152 true negatives. Calculate sensitivity, precision, and accuracy. (Answer: Sensitivity \(=24/30=80.00\%\); precision \(=24/42=57.14\%\); accuracy \(=176/200=88.00\%\).)
- A report predicts an event using a field recorded only after the event, although the proposed prediction is made before the event. What link is broken, and what should happen before the result supports the proposed use? (Answer: Predictor availability does not match the prediction point; remove or appropriately redefine the field and re-evaluate the model using information available at prediction time.)
- A report states that 1,000 cases were evaluated and gives 30 true positives and 70 false positives. What is precision, and can sensitivity be calculated from these figures alone? (Answer: Precision \(=30/(30+70)=30\%\); no, because the number of actual positive cases, including false negatives, is not given.)
- A model report gives a validation metric but does not identify the validation population or the model version. Why is its recommendation not fully auditable? (Answer: A reviewer cannot establish which cases and model produced the metric or whether they match the stated objective; the report must identify the population and model version and connect them to the result.)