Why a Correction Is Not Yet a Resolution
A change to a model file may look plausible and still fail to resolve the review finding. The edited file might not be the one used by the analysis, a dependent output might be stale, or the correction might fix one discrepancy while introducing another. Resolution therefore requires more than inspecting the changed line: it requires checking the corrected artifact and the consequences of using it.
The task here is narrow and evidence-based. Start from the finding and its stated expected behavior, identify the correction and its dependencies, then rerun the affected analyses and compare the results with explicit criteria. The earlier material on tracing discrepancies to their root causes and writing actionable independent review comments provides useful context: a resolution check should return to the original discrepancy and the requested change, not substitute a different question.
This definition makes the decision depend on verifiable evidence rather than on the author’s intention. It also separates three questions that are easy to conflate: Was the requested change made? Does the affected analysis now behave as expected? Did the change cause other results to depart from their expected behavior?
Set the Verification Criteria Before Rerunning
A useful resolution criterion states what will be inspected and what outcome counts as success. It should be specific enough that another reviewer could apply it to the same files and outputs. For example, “fix the dose-time error” is not a complete criterion. “For a dose recorded at 0 h, the amount enters the model at 0 h, and the concentration prediction at 0 h matches the hand-calculated value within the prespecified numerical tolerance” is checkable.
Use the finding’s original evidence to define the comparison. Depending on the discrepancy, success might require an exact match in a configuration value, the disappearance of a diagnostic pattern, agreement with an independent test case, or a specified numerical tolerance. Do not choose a tolerance after seeing the revised result. For floating-point output, establish in advance whether the expected difference is exact reproducibility or an acceptable numerical deviation, and justify the tolerance for the calculation and its units.
Before running anything, record the finding identifier, the files changed, the analysis or test cases affected, and the expected evidence. Check that the correction is present in the intended version of the files. Record file names and version identifiers or checksums where practical. This guards against a common failure: rerunning an old copy and mistakenly treating its output as evidence about the correction.
State the expected behavior and the result that would demonstrate it. Preserve the original discrepancy as the reference point.
List changed files, dependent scripts or model components, required test cases, and outputs that could change. Include shared dependencies when the correction can affect multiple analyses.
Use the corrected files, the required software and settings, and documented inputs. Capture enough run information to establish which artifacts produced the output.
Compare the relevant result with the criterion, then inspect other outputs that the change could reasonably affect. Record pass, fail, or not verified with evidence.
Why the Scope of the Rerun Matters
The rerun should be broad enough to test the correction’s consequences and narrow enough to remain interpretable. A single unit test may show that a corrected function works on one input, but it may not establish that the production analysis calls that function or that downstream outputs are updated. Conversely, rerunning every analysis without identifying dependencies can produce a large volume of results without a clear link to the finding.
A practical scope begins with the changed component and follows its dependencies to outputs. Include the smallest test that directly checks the corrected behavior, the affected analysis that uses the component, and any relevant downstream outputs. If a shared model component or common script changed, consider all analyses that depend on it. The earlier material on independent test cases and benchmark runs is relevant here: targeted tests check intended behavior, while benchmark comparisons can reveal unintended changes. Neither alone necessarily covers the entire resolution question.
The logic is straightforward. If a finding concerns an output \(Y\) calculated from an input file \(F\) and analysis code \(C\), evidence from a run that did not use the corrected \(F\) or \(C\) cannot establish the behavior of the corrected analysis. Likewise, if the correction changes a shared component used by \(Y_1\) and \(Y_2\), checking only \(Y_1\) leaves possible effects on \(Y_2\) unexamined. The verification scope must therefore include the inputs and outputs connected to the correction, not merely the visible edit.
Worked Example: Checking a Corrected Dose-Time Input
In a hypothetical model review, a dose at nominal time 0 h was found to be encoded at 1 h. The correction changes the dose time to 0 h. The review criterion requires the dose record to show 0 h and the initial predicted concentration to agree with an independent calculation within 0.01 mg/L. A separate test record predicts 2.00 mg/L at 0 h under the stated hypothetical setup.
First, inspect the corrected input and confirm that the intended record now has dose time 0 h. Then rerun the affected prediction using that input, not a cached output. Suppose the recorded prediction is 2.004 mg/L. Its absolute difference from the expected value is
Since \(0.004\ \text{mg/L} < 0.01\ \text{mg/L}\), the numerical criterion passes, and the input-time criterion passes. The original finding is supported as resolved if the run record confirms that it used the corrected input and the relevant dependent output was regenerated. The conclusion would not be justified by the matching prediction alone if that output came from a run using the old file.
Finally, inspect outputs that could be affected by shifting the dose time, such as the early concentration predictions and the timing of the dose event. If an unexpected change appears, investigate it before closing the finding; a successful check of the initial prediction does not demonstrate that every consequence is acceptable.
Comparing Results and Interpreting Differences
Use a comparison method suited to the output. For a configuration value or a categorical result, exact comparison is usually appropriate. For numerical computations, define whether results should match exactly or within a justified tolerance. For stochastic simulations, a changed random seed or replicate variation can make exact equality an inappropriate criterion; use the previously specified simulation checks and distinguish expected variability from a change in model behavior. The earlier material on random-seed and replicate variability can guide that distinction.
A numerical tolerance should have a clear scale and unit. If an expected prediction is 2.00 mg/L and the acceptable absolute difference is 0.01 mg/L, calculate the absolute difference and compare it directly with that limit. If the expected value can vary substantially in magnitude, a relative tolerance may be more appropriate, but define its denominator and handling of values near zero in advance. Never describe a result as “close enough” without stating the comparison rule.
Worked Example: Applying a Prespecified Tolerance
Suppose a hypothetical finding concerns a derived exposure output expected to be 120.0 mg·h/L. The prespecified acceptable absolute difference is 1.0 mg·h/L. After the correction and rerun, the output is 120.7 mg·h/L.
The absolute difference is
Because \(0.7\ \text{mg·h/L} \leq 1.0\ \text{mg·h/L}\), this comparison passes. It supports resolution only for this criterion; the reviewer must still verify that the intended corrected files were used and check any other affected outputs. If the observed value were 121.2 mg·h/L, the difference would be 1.2 mg·h/L, exceeding the limit, so the criterion would fail even if the value appeared visually close to 120.0.
For outputs without a single expected number, compare the specific feature named in the finding. For example, if the concern is a discrepancy between individual and population predictions, verify the affected records and the pattern that was originally reported. Do not replace that check with an unrelated summary that could conceal the problem. Diagnostic plots and simulation checks may be part of the evidence, but they should be interpreted against the stated criterion and the appropriate model context.
Check for Regressions and New Discrepancies
A regression is an unintended change in behavior that appears after a correction. A resolution check should therefore examine more than whether the original symptom disappeared. Start with outputs most directly affected by the edit, then check relevant benchmark results, test cases, and downstream outputs. When a benchmark differs, determine whether the difference is expected from the correction or signals an unintended change. Document that reasoning rather than treating every difference as either automatically acceptable or automatically a failure.
The comparison needs an appropriate baseline. A prior output can be useful only if it came from a sufficiently comparable, known state and its inputs and settings are documented. If software, data, random seeds, or execution settings also changed, differences may have more than one possible cause. Where feasible, isolate the correction by holding other conditions fixed. If that is not feasible, state the confounding changes and limit the conclusion accordingly.
Worked Example: Finding a New Discrepancy During Verification
In a hypothetical review, a unit conversion in a prediction script was corrected. The original finding concerned a prediction reported in ng/mL rather than mg/L. The revised output now uses mg/L and matches the expected value of 0.80 mg/L. However, a benchmark record for a different analyte, which the changed conversion function also processes, previously returned 1.25 mg/L and now returns 1250 mg/L.
The original finding’s unit criterion may be satisfied for the first prediction, but the changed benchmark is a new discrepancy requiring investigation. The factor-of-1000 difference is consistent with a possible unit-conversion error, so the reviewer should verify the benchmark’s input units, conversion path, and expected output before deciding whether it is an actual regression. The finding should not be closed as fully resolved while this directly affected result remains unexplained.
If the second output is confirmed to be wrong, record it as a new finding or link it to the existing item according to the review process. If it is expected because the benchmark input uses a different unit and the conversion is correct, record the evidence and why it does not violate the criterion. In either case, the review decision follows the checked behavior, not the appearance of the code change.
Failure Modes, Alternatives, and Limits
Several shortcuts produce weak resolution evidence:
- Inspecting code without running it. A plausible edit does not show that the corrected artifact is used or that its outputs are correct.
- Rerunning without checking inputs. A successful run may still use stale files, cached results, or different settings.
- Checking only the original symptom. This can miss regressions in other outputs dependent on the changed component.
- Changing the acceptance rule after seeing the result. This makes the comparison difficult to reproduce and can bias the decision.
- Using an overly broad rerun without a comparison plan. Many outputs are generated, but the evidence for the finding remains unclear.
- Calling an unexplained difference harmless. A difference is not acceptable merely because the original finding appears fixed.
A code review, a unit test, a full analysis rerun, and a benchmark comparison are complementary approaches rather than interchangeable proofs. Code review can verify that a change appears consistent with the intended correction, but it cannot establish runtime behavior. A unit test gives focused evidence for a defined input, but may not test integration with the full analysis. A full rerun checks the integrated workflow but can make cause-and-effect harder to isolate. A benchmark comparison detects departures from known behavior, but a benchmark may not cover the specific edge case in the finding. Select evidence that directly tests the criterion and use additional checks to cover the correction’s plausible effects.
Verification also has limits. It cannot prove that no untested input will fail, and a tolerance-based match does not establish that a model is scientifically appropriate. If the criterion is ambiguous, the necessary baseline is unavailable, or the run cannot be reproduced, report the limitation and do not overstate the result. The conclusion should identify what was tested, what passed or failed, and what remains unverified.
Make the Resolution Decision Traceable
For each finding, preserve a compact evidence trail: the original criterion, changed artifact identifiers, run inputs and settings, relevant output locations, comparison rule, observed result, and decision. This makes it possible to distinguish a verified correction from an undocumented assertion. Keep the conclusion proportionate: “resolved for the specified test case and rerun” is more precise than “the model is correct.”
Use the finding’s status to reflect the evidence. Mark it resolved only when all required criteria pass and relevant new discrepancies have been addressed. Mark it partially resolved when a subset passes but another required check remains open. Mark it unresolved if a criterion fails or the evidence cannot establish the result. A new regression should be recorded and assessed rather than hidden within a successful result for the original symptom.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A corrected script produces the expected output, but the run log shows that it read a copy of the old input file. Is the finding resolved? Why?
- A criterion requires an output within 0.5 mg/L of 10.0 mg/L. A rerun gives 10.6 mg/L. Calculate the absolute difference and decide whether this criterion passes.
- A fix resolves the reported output discrepancy, but a benchmark that uses the same changed function now fails. What should the reviewer do before closing the finding?
- Why might a unit test and a full analysis rerun both be needed to verify a correction?
- A finding’s expected behavior is described only as “make the results reasonable.” What is missing before a reproducible resolution decision can be made?
Solutions. (1) No. The run did not use the corrected input, so its output is not evidence about the corrected analysis; rerun with the intended file and verify its identity. (2) The absolute difference is \(|10.6 - 10.0| = 0.6\ \text{mg/L}\). Since \(0.6 > 0.5\), the criterion fails. (3) Investigate the benchmark discrepancy, determine whether it is a genuine regression or an expected change using documented inputs and criteria, and record any confirmed new finding. Do not close the original item while a directly affected discrepancy remains unexplained. (4) A unit test checks the corrected behavior on a focused case, while the full rerun checks that the integrated analysis uses the correction and regenerates dependent outputs. Each covers a different failure mode. (5) A checkable expected behavior and acceptance rule are missing. Specify the output or behavior to inspect and the exact or tolerance-based condition that counts as passing before evaluating the result.