Why Classify Review Findings?
An independent review can uncover issues that differ greatly in consequence. A misspelled variable label, an unexplained difference between runs, and a dose-unit error are all findings, but they should not receive the same follow-up priority. Classification makes the reason for a priority explicit: it connects an observed issue to its effect on the model’s results, their interpretation, the ability to reproduce them, or the evidence supporting them.
This tutorial treats classification as a structured judgment, not as a substitute for technical review. A finding should be supported by observed evidence, described in the context of the intended use, and assessed for its plausible reach. The categories and severity levels below are a practical framework; an organization’s approved quality procedures may define different labels or escalation rules.
Four Impact Categories
Assign a primary category based on the most direct consequence of the finding. Add a secondary category if the same issue has a distinct additional effect. For example, an undocumented change in a simulation input may affect both reproducibility and validity if the change also alters the simulated population. Naming the categories separately helps reviewers avoid compressing different problems into a vague label such as “model issue.”
- Validity: Could the finding make the computation or conclusion incorrect for the stated analysis purpose? Examples include an incorrect dose conversion, a coding error in a model component, or a simulation that does not implement the specified model. Validity concerns whether the results are fit to support the intended conclusions.
- Interpretation: Could the results be numerically correct but described, labeled, or interpreted in a misleading way? Examples include an incorrect unit in a table, a parameter described as a percentage when it is a ratio, or a conclusion that overstates what the analysis supports.
- Reproducibility: Could another qualified analyst, using the available inputs and instructions, fail to recreate or verify the analysis? Missing code, input versions, run settings, or random seeds can matter. A missing item is consequential when it obstructs reproduction or verification, not merely because a field is blank.
- Documentation: Is the supporting record incomplete, inconsistent, or insufficiently traceable? This includes absent rationale, unclear version history, or missing records of review. A documentation gap may remain a documentation finding, but its severity increases if it prevents verification or conceals an unresolved validity concern.
These categories describe effects, not the location where an issue was found. A problem in a report is not automatically “documentation”: if the report presents an incorrect dose unit that changes the reader’s understanding of exposure, interpretation is the primary effect. Likewise, a missing seed is not automatically severe; assess whether the simulation can still be reproduced adequately and whether the missing information changes a decision.
Severity and Follow-Up Priority
A useful classification separates consequence from urgency. Severity concerns the plausible impact if the finding is real and left unresolved. Priority also considers whether affected results are currently being used, how broad the affected outputs are, and whether a decision or release is imminent. A high-impact finding in a draft that is not being used may still require urgent containment, while a limited documentation gap may be scheduled for routine correction.
| Severity | General meaning | Typical follow-up priority |
|---|---|---|
| Critical | Could invalidate a central result or lead to a materially harmful decision; scope may be broad or the affected conclusion essential. | Immediate escalation; do not rely on affected results until the issue is resolved or appropriately bounded. |
| High | Could materially alter an important result or its interpretation, but the effect may be localized or otherwise bounded. | Resolve before affected results are accepted for their intended use. |
| Moderate | Limits verification or confidence, or affects a secondary result, without currently indicating a material change to the main conclusion. | Plan resolution before finalization when practicable; document any justified interim use and its limits. |
| Low | Has a small, well-bounded effect on presentation or traceability and does not materially impair the analysis or its verification. | Address through routine correction and retain a clear record of the disposition. |
These are decision aids, not universal regulatory definitions. Apply local quality procedures where they exist. If the impact is uncertain, do not automatically assign low severity. Record what is known, what remains uncertain, and whether the uncertainty itself limits reliance on the result.
A Justification for the Classification Sequence
The sequence matters because severity labels without an impact rationale are difficult to compare. Begin with the affected result or process, not with the apparent size of the discrepancy. Then ask what could happen if it remains unresolved. Only after identifying that consequence should you consider how widely it reaches and whether anyone is relying on it.
Name the output, model component, interpretation, or record involved. A vague finding such as “simulation concern” does not define what may be affected.
Determine whether the concern bears on validity, interpretation, reproducibility, documentation, or more than one of these.
Consider how many outputs, analyses, or intended decisions could be affected, and whether those results are already being used.
Use the plausible consequence to justify severity; use scope, evidence, and timing of use to justify urgency. If the effect is uncertain, state that uncertainty rather than disguising it with a low rating.
This sequence prevents a common logical error: inferring seriousness from appearance alone. A discrepancy of one unit in a printed table might be minor if it is a harmless formatting issue, but a unit mismatch that changes the reported exposure by a factor of 1,000 is not minor merely because it occupies one cell. Conversely, a large difference in an intermediate diagnostic is not automatically critical if the affected output is irrelevant to the intended use and this has been established.
Worked Example: A Dose-Unit Error in a Simulation
In a hypothetical simulation workflow, the specification requires a dose of 2 mg/kg. Review shows that the code supplies 2 micrograms/kg, while the model input expects milligrams/kg. The simulated subjects weigh 70 kg. Determine the impact category, severity, and priority.
Check the size of the mismatch. The specified dose per subject is \(2\ \text{mg/kg} \times 70\ \text{kg} = 140\ \text{mg}\). The supplied dose is \(2\ \text{micrograms/kg} \times 70\ \text{kg} = 140\ \text{micrograms} = 0.140\ \text{mg}\). The supplied amount is therefore \(0.140/140 = 0.001\) of the specified amount: a 1,000-fold difference.
Classify the effect. The primary category is validity: the implemented input does not match the specified simulation and may materially alter its outputs. Interpretation may be a secondary concern if the resulting outputs are being presented as predictions under the specified 2 mg/kg dose.
Assign severity and priority. If the simulation results are central to the analysis, a 1,000-fold dose mismatch is potentially critical. Follow-up is immediate: affected results should not be relied on until the issue has been addressed and the impact on those results is established. The classification is based on the demonstrated input discrepancy and its scale, not on an assumption about the direction or exact size of every downstream change.
Worked Classification in Less Direct Cases
Worked Example: Correct Calculation, Incorrect Reported Unit
A hypothetical model fit produces a clearance estimate of 4.2 L/h. The calculation file records the value and unit correctly, but a results table reports “4.2 mL/h.” The table is used in a review of dose selection. Classify the finding.
Check the conversion. Since \(1\ \text{L}=1000\ \text{mL}\), the estimate equals \(4.2 \times 1000 = 4200\ \text{mL/h}\), not 4.2 mL/h. The table’s stated value and unit together imply a clearance 1,000 times smaller than the calculated value.
Classify the effect. Interpretation is primary because the underlying calculation is correct but the reported result communicates a materially different quantity. The fit’s validity is not shown to be affected by this reporting error. If the table also becomes the input to another analysis, validity could become a secondary category for that downstream work.
Assign severity and priority. Because the table informs dose selection, the error is high severity and should be corrected before the table is used to support that decision. If the table had not been circulated and was caught during internal formatting review, urgency could be lower, but the impact of the stated unit would still need to be documented.
Worked Example: Missing Simulation Run Information
A hypothetical report summarizes 1,000 simulation replicates and gives a median exposure of 18.6 mg·h/L. The scripts and model are archived, but neither the random seed nor the software version is recorded. A rerun gives a median of 18.4 mg·h/L. The decision rule in this workflow is to flag an exposure above 20.0 mg·h/L; both reported medians are below that threshold. Classify the finding using only the information given.
Identify what is missing. Without the seed, the exact random stream may not be recreated. Without the software version, exact computational behavior may be harder to verify. The rerun shows that the summaries differ by \(18.6-18.4=0.2\) mg·h/L, but this difference alone does not establish a model defect or explain why the values differ.
Classify the effect. Reproducibility is primary, with documentation also affected: the run cannot be fully specified from the recorded information. The stated facts do not establish that the simulation is invalid. Since both summaries remain below the stated decision threshold, the available evidence does not show a changed decision under this rule.
Assign severity and priority. On these facts, moderate severity is justified: exact reproduction and verification are impaired, but the reported difference does not cross the stated threshold. Follow-up should be planned before finalizing the simulation record. If the result were being used for an imminent decision close to the threshold, or if exact replication were required for the intended review, urgency would increase. That change in priority would follow from the decision context, not from treating all missing seeds as automatically critical.
Diagnostics, Alternatives, and Limitations
Classification is going wrong when the label is disconnected from a concrete effect. Revisit the assessment if the finding uses only broad language (“major issue”), if severity is based on how difficult a correction seems, or if a documentation gap is marked low without checking whether verification remains possible. A classification should also be reconsidered when new evidence changes the affected scope or shows that an assumed consequence does not occur.
The earlier material on Using Benchmark Runs to Detect Regressions concerns identifying changes between a current run and a reference. Classification answers a different question: given an observed issue or change, what does it affect, how serious could that effect be, and how urgently should it be followed up? A benchmark difference is not itself a severity rating. Its importance depends on what changed, whether the change is expected, and what results or decisions it can influence.
A simple “count the findings” approach is useful for tracking workload but poor for prioritization: one high-impact validity issue can outweigh many low-impact presentation defects. A single severity label is quick to communicate, but can conceal whether the concern is validity, interpretation, reproducibility, or documentation. A category-plus-severity assessment preserves that distinction. A numerical risk score may support consistent triage if an organization has defined and validated its scoring rules; an improvised multiplication of ordinal labels can create false precision. For independent review, a concise written rationale is often more informative than a score without context.
Severity remains a judgment under uncertainty. It does not prove that a result is wrong, quantify the probability of harm, or determine by itself whether an analysis is acceptable. Nor does “low” mean “ignore”: low-priority findings still need a recorded disposition. Conversely, a high-priority finding does not establish its cause. Determining the cause and confirming a resolution require further review beyond classification.
Common Pitfalls
- Equating severity with urgency. A potentially serious issue may be discovered before any affected result is used, while a smaller issue may need quick attention because a decision is imminent. State both judgments and their reasons.
- Rating by appearance or effort. A tiny code change can alter a central result, and a difficult-to-fix formatting defect may have little analytical consequence. Judge effects, not the size of the edit or the work needed.
- Calling every missing record “low.” Missing documentation can prevent independent verification. Assess what cannot be checked as a result.
- Treating uncertainty as evidence of safety. If the impact is unknown, say what is unknown and whether reliance should pause. Do not use a low rating merely because the effect has not yet been measured.
- Assigning multiple categories without distinction. Name the primary effect and explain any secondary effect. A list of categories without that reasoning is not a useful classification.
Check Your Understanding
For each situation, identify the primary impact category, propose a severity and follow-up priority, and briefly explain your reasoning. Solutions follow the questions.
- A report gives an estimated half-life of 12 h, but the underlying analysis and saved output give 12 days. The report is being used to plan a monitoring schedule. What is the primary impact, and why?
- A simulation report omits the seed, but records the software version, inputs, code, and replicate-level outputs. The reported conclusion is not close to a decision threshold. What finding remains, and what priority is reasonable?
- A typo in a section heading does not change any label, result, or cross-reference. How would you classify its severity and priority?
- A model output differs from a benchmark run, but the review has not established whether the difference is expected or whether it affects a decision. Should it be labeled low severity? Explain.
Solutions. (1) Interpretation is primary: 12 h and 12 days differ by a factor of 24, and the monitoring schedule may be based on the incorrect description. Because the report informs a decision, classify it as high severity and resolve it before reliance on the stated half-life. (2) Reproducibility is the primary concern because the exact random stream may not be recreated; documentation is also affected if the run record is incomplete. Moderate severity and planned follow-up are reasonable on the stated facts, with higher urgency if exact replication is required or a decision is imminent. (3) This is a documentation or presentation defect with low severity and routine priority, provided the typo creates no ambiguity or broken reference. (4) No. The effect is unresolved, so low severity is not justified by lack of evidence. Record the uncertainty and assess the affected outputs and decision context before assigning a final severity; use a precautionary priority if material impact cannot yet be ruled out.