Why Consistency and Traceability Matter
A decision-maker may encounter the same result in a paragraph, a table, and a chart. Those three presentations should agree, but agreement alone is not enough: a reviewer should also be able to identify the output and calculation that support each claim. Editing for consistency and traceability is the process of checking both.
This is not a substitute for assessing whether the data, model, or evaluation design are appropriate. It is a disciplined final check that the report accurately describes the analysis that was performed. A perfectly consistent report can still describe a poorly designed analysis; conversely, a sound analysis can be misrepresented by a copied number, an unclear denominator, or an outdated chart.
A useful review treats each material claim as a link in a chain: report statement, table or figure, calculation, and source output. The chain should be specific enough that another reviewer can reproduce the reported value without guessing which data, model version, or denominator was intended.
Record exactly what the prose, table, or figure says, including its population, unit, and time period.
Identify the output file, table, model version, or calculation used to support the claim. A filename alone may be insufficient if it does not identify the run or evaluation sample.
Where practical, recalculate the reported statistic from its underlying counts or values. Check that the source and the report use the same definitions.
Compare narrative, tables, figures, captions, and recommendations. Correct inconsistent presentations or explain a genuine difference in scope.
Keep a concise note of the corrected value or wording, the supporting source, and the final version checked.
What to Reconcile
The most important checks concern more than whether two numbers look alike. Two different values can both be correct if they refer to different populations or definitions, while two identical values can be misleading if one is attached to the wrong label. Review the meaning around each number as carefully as the number itself.
| Item to check | Questions for the editor |
|---|---|
| Value and calculation | Does the reported value follow from the supporting output? Are rounding and displayed precision consistent? |
| Population and denominator | Which records are included? Is the denominator the same in the prose, table, and figure? |
| Metric definition | Is the statistic named correctly, and are its component counts or inputs clear? |
| Units and scale | Are amounts in dollars, thousands of dollars, percentages, or proportions? Do axes and captions use the same scale? |
| Model and evaluation | Do descriptions identify the same model and evaluation sample as the output being cited? |
| Labels and recommendations | Do captions, categories, thresholds, and proposed actions agree with the analysis actually reported? |
For example, an RMSE of 0.42 when the outcome is recorded in thousands of dollars corresponds to $420, not $0.42. A report should make that conversion explicit. Similarly, a metric calculated on a test sample should not be described as if it came from the full population unless that is genuinely the evaluation design.
Recomputing a Reported Statistic
Recalculation is especially useful for statistics derived from counts. Consider a binary outcome and a classification rule. Let \(TP\), \(FP\), \(FN\), and \(TN\) denote the numbers of true positives, false positives, false negatives, and true negatives, respectively. Each evaluated record belongs to exactly one of these four cells.
These definitions follow directly from the questions each metric answers. Accuracy is the number of correct classifications divided by all classifications; the correct cells are \(TP\) and \(TN\). Precision asks what fraction of predicted positives are actual positives, so its denominator is \(TP+FP\). Recall asks what fraction of actual positives were identified, so its denominator is \(TP+FN\). This also explains why these metrics cannot be substituted for one another: they use different denominators.
The formulas require the relevant denominator to be nonzero. If there are no predicted positives, precision is undefined; if there are no actual positives, recall is undefined. An editor should not silently replace an undefined metric with zero or another value. The report should state how that case was handled, if it arises.
Worked Example: Checking a Confusion-Matrix Summary
A hypothetical screening model is evaluated on 200 records. Its output gives \(TP=36\), \(FP=24\), \(FN=12\), and \(TN=128\). The draft narrative reports 82% accuracy and 75% recall; a table reports 80% accuracy and 60% precision. Reconcile the figures. Method: recompute each metric from its defining cells. This avoids the common mistake of treating accuracy, precision, and recall as interchangeable percentages.
Step 1: Check the total. The four cells contain \(36+24+12+128=200\) records, matching the stated evaluation size.
Step 2: Calculate accuracy. Correct classifications are the true positives and true negatives, or \(36+128=164\). Thus accuracy is \(164/200=0.82\), or 82%.
Step 3: Calculate precision. The model predicted positive for \(36+24=60\) records. Of those, 36 were positive, so precision is \(36/60=0.60\), or 60%.
Step 4: Calculate recall. There were \(36+12=48\) actual positives. The model identified 36 of them, so recall is \(36/48=0.75\), or 75%.
Step 5: Resolve the discrepancy. The narrative's 82% accuracy and 75% recall agree with the output. The table's 60% precision also agrees, but its 80% accuracy does not. Replace the table value with 82%, then check any figure or summary that repeats accuracy.
Check: The corrected report gives accuracy 82%, precision 60%, and recall 75%. The values match their separate formulas and the total of 200 records. The trap was not an arithmetic error in the confusion matrix; it was an inconsistent copy of the accuracy value.
Trace the Population, Units, and Metric Labels
A report can contain correct calculations that are attached to the wrong description. For every reported result, identify the population used, the outcome scale, the metric definition, and the model or run that produced it. These details are part of the result, not optional decoration.
Worked Example: Reconciling Test-Sample Size and Error Units
A hypothetical renewal model is developed using 1,200 eligible policies, with 300 held out as a test sample. The recorded outputs for the test sample are MAE = $280 and RMSE = 0.42 when claim amounts are stored in thousands of dollars. The draft says “performance was evaluated on 1,200 test policies,” while its table lists “RMSE = $280” and a chart caption says “average absolute error = $280.” Correct the report. Method: trace each statement to the evaluation output, then check its population, metric, and unit. This avoids assuming that similar-looking error labels mean the same statistic.
Step 1: Verify the test population. The stated design has 1,200 eligible policies in total and 300 held out for testing. Therefore, the result described as test performance is based on 300 policies, not 1,200. The draft sentence should be changed to “performance was evaluated on the 300-policy test sample.”
Step 2: Convert the RMSE scale. The stored outcome is in thousands of dollars, so an RMSE of 0.42 thousand dollars is \(0.42 \times \$1{,}000=\$420\). It is not $280. The table should label RMSE as $420, or retain 0.42 only if it clearly states that the unit is thousands of dollars.
Step 3: Check the caption against the metric. MAE is already reported as $280. The chart caption “average absolute error = $280” is consistent if the chart displays MAE, because MAE is the mean of absolute prediction errors. If the chart is intended to display RMSE instead, its caption and value must be changed to RMSE and $420.
Step 4: Make the revised set of statements agree. The report should identify the 300-policy test sample, report MAE as $280, and report RMSE as $420 (or 0.42 thousand dollars). Its figure title and caption must name the metric actually plotted.
Check: The unit conversion is \(0.42 \times 1{,}000=420\), and the test size is 300 out of 1,200 total policies. MAE and RMSE remain distinct, so the report must not use one metric's label for the other's value. The common trap is to copy the dollar-denominated MAE into an RMSE row simply because both are error measures.
Check Model Descriptions and Recommendations Against Outputs
Traceability applies to claims as well as numerical metrics. If a recommendation refers to a score threshold, a segment, or a model version, the supporting output should establish the same rule. A reader should not have to infer whether a paragraph uses a different threshold or population from the table beside it.
Worked Example: Reconciling a Referral Threshold
A hypothetical service team reviews a model score for 1,000 accounts. The score-output summary shows 30 accounts with scores at least 0.25, 20 accounts with scores from 0.20 up to but not including 0.25, and 950 accounts below 0.20. A report table says “review accounts scoring at least 0.25: 30 accounts,” while the recommendation says “send the 50 accounts scoring at least 0.20 for review.” A figure caption states “referral threshold: 0.25.” Determine what must be edited. Method: reconcile the threshold definition with the counts and the action wording. This avoids treating a threshold and its resulting count as independent facts.
Step 1: Check the total. The three score bands contain \(30+20+950=1{,}000\) accounts, matching the stated population.
Step 2: Calculate the count at the recommendation threshold. At a threshold of at least 0.20, both the 30 accounts at or above 0.25 and the 20 accounts from 0.20 up to 0.25 qualify. The total is \(30+20=50\).
Step 3: Compare the presentations. The table and figure caption both describe a threshold of 0.25 and a count of 30. The recommendation describes a threshold of 0.20 and a count of 50. Each statement is internally consistent with the score bands, but they do not describe the same proposed rule.
Step 4: Resolve the report according to the intended action. If the intended action is to review 50 accounts, change the table's threshold to 0.20 and its count to 50, and change the caption to 0.20. If the intended action is to review 30 accounts, retain the table and caption and revise the recommendation to specify a 0.25 threshold and 30 accounts. The editor must verify the intended threshold with the analysis owner rather than choose one by preference.
Check: The report can be made consistent under either documented choice, but it cannot claim both that the threshold is 0.25 and that the corresponding count is 50. The trap is to edit only the number while leaving the threshold label unchanged.
A Practical Editing Record
A compact audit record makes later revisions safer. For each material statement, note the location in the report, the source output, the population and definition, and the check performed. If the report changes after review, the record helps the editor identify every place where the same result appears.
| Report claim | Supporting source | Reconciliation check | Resolution |
|---|---|---|---|
| Test accuracy is 82% | Confusion matrix for the specified test run | Recompute correct classifications over all test records | Update any table that says 80% |
| RMSE is $420 | Evaluation output; outcome stored in thousands | Convert 0.42 thousand dollars to dollars | Use a clear dollar or thousands label |
| Review 50 accounts at score 0.20 | Score-band count for the stated population | Sum counts at or above 0.20 | Match table, figure, and recommendation to the approved threshold |
Record enough detail to identify the exact output, such as the model run or evaluation sample, rather than writing only “model results.” Use consistent rounding across presentations, but retain sufficient precision in the source to verify the displayed value. If a result has been intentionally updated, ensure that obsolete values do not remain in figure captions, executive summaries, or copied tables.
Common Editing Traps
- Checking digits but not definitions. The same value can refer to different metrics, populations, or units. Verify the label and denominator as well as the number.
- Assuming two samples are interchangeable. A development population and a test sample may have different sizes and purposes. Name the population tied to each result.
- Correcting only the main table. Search for repeated claims in prose, captions, callouts, and recommendations so that a correction is carried through the full report.
- Hiding a genuine difference. If two values refer to different periods or groups, explain the distinction rather than forcing them to match.
- Treating consistency as validation. Agreement across report sections shows faithful editing, not that the model or analysis is appropriate. Keep those judgments separate.
Careful consistency and traceability make a report easier to review, revise, and use. They also help preserve the boundary between what the analysis produced and what the report recommends. This editing discipline is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, reconcile the report claim with the stated supporting output.
- A confusion matrix has \(TP=18\), \(FP=12\), \(FN=6\), and \(TN=64\). A narrative reports 82% accuracy, while a table reports 80%. Which accuracy is supported by the matrix? (Answer: \(82/100=82\%\); the table should be corrected.)
- A model is evaluated on 240 test records from a total of 960 records. A caption says that the reported test metric uses all 960 records. What should the caption say about the evaluation population? (Answer: It uses the 240-record test sample, not all 960 records.)
- An error output is 0.35 in thousands of dollars. A report displays $350 in one place and $0.35 in another, with neither unit specified. What is the dollar value and what must the report clarify? (Answer: $350; state that the output is in thousands of dollars or convert consistently to dollars.)
- There are 14 scores at least 0.30, 9 scores from 0.25 up to but not including 0.30, and 77 scores below 0.25. A recommendation says to review scores at least 0.25, but a table gives 14 accounts. What count and total population should the table report for that rule? (Answer: 23 accounts out of 100.)
- A report describes precision as \(24/(24+16)\), but the output has \(TP=24\), \(FP=16\), and \(FN=8\). A separate paragraph calls \(24/(24+8)\) precision. Which value is precision, and what metric does the second fraction calculate? (Answer: Precision is \(24/40=60\%\); \(24/32=75\%\) is recall.)