From a Concern to an Actionable Comment
A review finding is useful only if its recipient can understand what was checked, reproduce the concern, judge its importance, and tell what response is being requested. “The simulation looks wrong” does not provide enough information to do any of those things. It does not identify which simulation, what was observed, or what would count as an adequate response.
An actionable independent review comment connects a specific artifact to evidence and an analytical consequence, then asks for a verifiable resolution. The comment should report what the reviewer can demonstrate, not speculate about an author’s intent or prescribe a particular code change when several remedies could be appropriate.
A practical comment usually contains four linked elements:
- Artifact: name the file, table, figure, script, configuration, or report section, and give a useful version, commit, line, or run identifier.
- Reproduction: provide enough steps and inputs for another person to obtain the reported result. State the observed result and, where the specification or independent calculation gives a basis, the expected result.
- Consequence: explain which output, interpretation, or decision could be affected. Separate a demonstrated discrepancy from its potential downstream impact.
- Request: ask for a checkable response, such as a corrected output, a documented rationale tied to the specification, or a rerun with a stated comparison criterion.
These elements form a reasoning chain. If the artifact or reproduction is missing, the concern may be impossible to investigate. If the consequence is missing, its relevance is unclear. If the request is not verifiable, the recipient may respond without resolving the underlying uncertainty. Strong comments make each link explicit while staying concise enough to be usable.
Evidence, Expected Behavior, and Consequence
Before writing, separate observation from interpretation. “The run produced 498 impaired subjects out of 1,000” is an observation. “The run used equal probabilities for impaired and non-impaired subjects” is an interpretation that should be supported by the configuration or a reproducible check. “The reported population exposure may not represent the specified covariate distribution” is a consequence, whose scope depends on the study’s purpose.
A concise justification for the structure is that a review claim is only as strong as its weakest link: artifact identification permits location, reproduction permits independent checking, and a stated consequence explains why the issue matters. A proposed resolution then gives the recipient a test of closure. If any link is absent, the reviewer should narrow the claim or gather more evidence rather than fill the gap with an assumption.
The expected behavior should come from an authoritative basis, such as the analysis plan, model specification, validated test case, or documented input. If the basis is unclear, the comment should state the ambiguity and request clarification instead of presenting the reviewer’s preference as a requirement. Likewise, distinguish a confirmed error from a possible risk: “The output differs from the specified target” is stronger and more precise than “This might be wrong” when the discrepancy is directly reproduced; “may affect” is appropriate when the downstream consequence has not been quantified.
Worked Example: Covariate Sampling Does Not Match the Stated Target
Hypothetical review evidence. The simulation specification for a 1,000-subject run states that 30% of subjects should have renal impairment. In simulation/config.R, the covariate is generated using an equal-probability draw between 0 and 1. With the submitted code and seed 741, the reviewer obtains 498 impaired subjects out of 1,000. The configuration therefore appears to use a 50:50 sampling probability, rather than the stated 30:70 target.
The observed proportion is \(498/1000=0.498\), or 49.8%. The target count implied by the specification is \(1000(0.30)=300\). The difference is \(498-300=198\) subjects. This comparison does not, by itself, establish the effect on every reported exposure statistic. It does establish that the reproduced sample composition does not match the stated target.
Actionable comment. “In simulation/config.R at the covariate-generation statement, run the submitted simulation with seed 741 and 1,000 subjects. I obtain 498 impaired subjects (49.8%); the simulation specification, section 4.2, calls for 30% (300 of 1,000). This mismatch means the aggregate simulated outputs may reflect a different renal-impairment composition from the one specified. Please either align the sampling with the specified 30:70 distribution or document why the implemented distribution is intentional, and provide the resulting impaired-subject count for the same run.”
The comment names the artifact, gives reproducible inputs and an observed result, states the reference for expected behavior, and limits the consequence to a potential mismatch in aggregate outputs. It requests evidence that can be checked without dictating a particular implementation.
Make Reproduction Practical
A useful reproduction is specific enough to be repeated, but not padded with irrelevant detail. Include the run command or interface steps when they matter; identify the input data, configuration, seed, software or model version when those affect the result; and quote the relevant output or point to a stable location. If the concern depends on a particular environment, say so. Do not assume that “run the model” means the same thing to every reviewer or analyst.
For a numerical discrepancy, show enough arithmetic to make the comparison transparent. State units and precision as appropriate. For a visual concern, identify the figure, panel, subgroup, or time range and describe the pattern rather than merely attaching an image. For a code concern, cite the relevant location and describe the behavior; a line number alone is fragile if files change, so pair it with a filename and a stable identifier when available.
Worked Example: Time Units Produce a Reproducible Prediction Difference
Hypothetical review evidence. A submitted test case uses an intravenous bolus dose of 100 mg, \(CL=5\) L/h, \(V=50\) L, and a prediction time of 1 hour. For a one-compartment model with these inputs, the expected concentration is calculated as \(C(t)=(100/50)\exp[-(5/50)(1)]\), which is approximately 1.810 mg/L. The submitted output is 0.00496 mg/L. The test log shows that the prediction time was passed as 60 while the model parameters are expressed per hour; evaluating the same expression with \(t=60\) gives \(2\exp(-6)\approx0.00496\) mg/L.
Actionable comment. “In tests/prediction_time.csv, case IV-01, the input time is 1 hour, but the test log reports that the model receives 60 with parameters expressed per hour. Running the submitted prediction gives 0.00496 mg/L; using the stated one-hour time gives \(2\exp(-0.1)=1.810\) mg/L. The discrepancy is consistent with a minutes-versus-hours mismatch and changes this test prediction by a factor of about \(1.810/0.00496=365\). Please clarify the time unit at the model interface and provide the prediction for case IV-01 with time and parameter units consistent with the specification.”
The comment calls the unit mismatch “consistent with” the evidence rather than claiming to know the author’s intent. The requested output tests the concrete issue, and the specified model calculation makes the expected value independently checkable.
Request a Checkable Resolution
A request should define what evidence would address the concern, not merely ask someone to “look into it.” Depending on the finding, the requested response might be a rerun with identified inputs, a revised table whose values can be compared with a stated target, an explanation referencing the governing specification, or a clarification that the submitted artifact is not the intended version. The request should leave room for a justified alternative when the evidence does not uniquely determine the remedy.
Avoid demanding a specific code edit if the same analytical requirement could be met in more than one sound way. A reviewer may identify the behavior and expected property while leaving implementation choices to the modeler. Conversely, when a requirement is explicit, name it: “please make this consistent with section 4.2” is more useful than “please fix the sampling.”
Worked Example: Benchmark Output Exceeds the Stated Tolerance
Hypothetical review evidence. A benchmark fixture specifies a predicted \(C_{\max}\) of 1.25 mg/L, with an acceptance tolerance of 1%. The submitted run reports 1.52 mg/L for the same fixture and parameter set. The relative difference from the reference is \(|1.52-1.25|/1.25=0.216\), or 21.6%, which exceeds the stated 1% tolerance.
Actionable comment. “In benchmark fixture oral_case_03, using parameter file model_v7.json, the submitted run reports \(C_{\max}=1.52\) mg/L against the fixture reference of 1.25 mg/L. The relative difference is 21.6%, above the fixture’s 1% acceptance tolerance. This changes the benchmarked exposure prediction and may affect comparisons based on it. Please provide a run for the same fixture and parameter file that meets the stated tolerance, or document which input or expected value is intended to differ and provide the corresponding benchmark basis.”
The arithmetic supports the tolerance claim directly. The comment does not assert that a particular downstream decision has changed; it identifies the exposure prediction and comparisons that could be affected. Its alternative request allows the possibility that the reference or inputs, rather than the model output, require clarification.
Choosing the Format and Avoiding Failure Modes
Inline comments are effective when the concern is tied to a particular line, table cell, or figure panel and the surrounding context is visible. A structured issue or review log is often better for a finding that spans files, requires several reproduction steps, or needs a durable record of evidence and requested response. An email can draw attention to an urgent concern, but an email alone may be difficult to trace to the exact artifact and version. Whichever format is used, preserve the same core elements and link the comment to the relevant artifact.
| Approach | Useful when | Common limitation |
|---|---|---|
| Inline comment | The issue is local and the artifact location is stable. | May not provide enough room for reproduction steps or cross-file context. |
| Structured review log or issue | The concern needs evidence, references, and a clear request in one place. | Can become hard to navigate if findings are combined or locations are vague. |
| Email or informal message | A brief notification is needed to direct attention. | May not remain connected to the reviewed version or provide a checkable record. |
Common mistakes weaken otherwise valid findings:
- Vague location: “The model file is wrong” gives no entry point. Name the file and the relevant section, statement, run, or output.
- Conclusion without evidence: “This is biased” may be an interpretation, not an observation. State the data or calculation that supports the concern and explain the inference separately.
- Unreproducible evidence: A screenshot without the run conditions may not establish how the result arose. Include the inputs needed to repeat the check.
- Overstated consequence: A difference in one test output does not alone prove a clinical decision is invalid. Describe what the discrepancy could affect, and quantify downstream impact only when it has been assessed.
- Unbounded request: “Please investigate” does not specify what would resolve the concern. Request a result, comparison, or explanation that can be evaluated.
- Premature prescription: Naming a specific code change can distract from the analytical requirement. State the required behavior and allow a different implementation if it demonstrably satisfies that requirement.
A comment should also be proportionate. A minor documentation ambiguity may need a clarification, while a reproducible numerical discrepancy against a specified acceptance criterion warrants a more direct request. Severity classification is a separate judgment; the writing task here is to make the evidence and requested response clear enough that the finding can be assessed.
Check Your Understanding
For each question, identify what makes a comment actionable or how it could be improved. Solutions follow each question.
- A comment says, “The simulation output looks unreasonable.” What information is missing? Solution: It needs the specific output and artifact version, reproduction inputs or steps, the observed result, a basis for expected behavior, the possible analytical consequence, and a checkable request.
- A specification calls for 20% of 500 simulated subjects to have a covariate value of 1. The reproduced run contains 127 such subjects. What are the observed proportion, target count, and difference? Solution: The observed proportion is \(127/500=0.254\), or 25.4%. The target count is \(0.20(500)=100\), so the observed count is 27 above the target. The comment should report those facts and avoid claiming downstream impact that has not been assessed.
- A reviewer sees an unexpected value but cannot tell whether the input file matches the version named in the specification. How should the reviewer word the finding? Solution: State the value and the uncertainty about the input version, identify the files or run examined, and request clarification of the intended input or a reproducible run with the specified version. Do not state that the model is defective until the input discrepancy is resolved.
- Why is “Please change the code to use function X” often less useful than a requirement-focused request? Solution: Function X may not be the only valid implementation. State the required behavior or acceptance criterion and request evidence that it is met; prescribe a specific change only when the requirement itself mandates it.
- A benchmark reference is 2.00 mg/L and the submitted output is 2.10 mg/L. The allowed tolerance is 2%. Does the output meet the tolerance? Solution: The relative difference is \(|2.10-2.00|/2.00=0.05\), or 5%. Because 5% exceeds 2%, it does not meet the stated tolerance. A review comment should name the fixture and inputs and request a result within tolerance or a documented clarification of the reference or inputs.