Why Separate-Team Validation Needs a Plan
A model can be evaluated against relevant external data and still be difficult for another group to run or interpret. A separate team tests an additional question: can people who did not build the model follow the supplied materials, reproduce the specified analysis, and identify where interpretation depends on judgment? This is not simply a second analysis by the model developers. The separation makes assumptions, undocumented steps, and fragile computational choices easier to expose.
The purpose is not to guarantee that two teams will obtain identical outputs under every software environment, nor to treat every difference as evidence against the biology. The purpose is to make the comparison traceable. The validation record should show what was supplied, what the team did, what it obtained, which discrepancies occurred, and which limitations remain.
This activity builds on external validation across laboratories and assays: the biological setting and measurement context still matter. Its distinctive focus is the handoff and independent execution. The independent team must be able to distinguish the intended analysis from incidental choices that were never specified.
Plan the Handoff Before the Run
A usable handoff is a set of controlled, identifiable artifacts rather than a folder of files sent informally. At minimum, it should include the executable model or implementation, the model description, parameter values and their units, input data and data definitions, and a prediction specification. The specification states the intended population or experimental conditions, dosing and simulation schedule, outputs to calculate, time points, transformations, and any summaries to report. It should also identify which outputs are model predictions and which are observed data.
Record version identifiers, dates, file checksums or other integrity identifiers, software and dependency versions, and the expected run command or workflow. A checksum is a compact value computed from a file that can be used to detect whether its contents have changed. For data, record provenance, inclusion rules, missing-value handling, and units. For the model, document state variables, parameter conventions, initialization, event handling, and known numerical settings. The earlier material on dimensional, positivity, and boundary-condition assertions and on convergence diagnostics is useful background for specifying what should be checked, but the independent team should not have to infer the intended configuration.
Agree on roles and communication before execution. The independent team should control its run records and discrepancy log. Developers can answer questions and provide technical support, but each question, response, artifact change, and rerun should be recorded. If the package changes during the exercise, preserve the original version and identify precisely which results came from each version. Otherwise, an apparently successful rerun may no longer correspond to the materials originally handed off.
Reproduction Is Not Predictive Agreement
Two comparisons must remain distinct. First, the team asks whether it can reproduce the specified model calculation from the supplied artifacts. Second, the team examines the resulting prediction in relation to the designated validation observations and the scientific claim. Reproducing the developers’ output does not establish that the prediction is accurate. Conversely, a prediction that appears close to observations does not show that the delivered model and analysis are reproducible.
For numerical outputs, compare like with like: the same endpoint definition, units, time point, cohort or experimental condition, and summary. Exact equality may be appropriate for deterministic calculations if the software and numerical environment are fixed. Otherwise, report an absolute difference and, when meaningful, a relative difference. For scalar outputs \(y_I\) from the independent run and \(y_R\) from the reference run, define
The relative difference is undefined when \(y_R=0\), and can be misleading when the reference value is close to zero. In that case, report the absolute difference with units and, if a scale-based comparison is needed, state the prespecified scale used. A discrepancy measure describes the size of a difference; it does not by itself decide whether that difference is acceptable or scientifically consequential.
Worked Example: Comparing a Reproduced Prediction
Suppose a hypothetical reference run reports a drug-induced reduction in a circulating biomarker of 24.0% at day 14. The independent team runs the supplied model and reports 23.4% for the same endpoint and conditions. Both values are percentages, and the prediction specification defines the reduction relative to the same baseline.
The signed difference is \( \Delta=23.4\%-24.0\%=-0.6 \) percentage points. The relative difference is \( |\Delta_{\mathrm{rel}}|=|23.4-24.0|/24.0\times100\%=2.50\% \).
The record should state both results, the unit, the calculation, and the software and configuration used. The independent result is 0.6 percentage points lower than the reference. That arithmetic alone does not establish whether the difference is consequential: the team should check numerical settings, model version, and calculation definitions, then report any remaining difference for interpretation against criteria set for the intended use.
Classify, Investigate, and Record Discrepancies
A discrepancy log makes an independent exercise interpretable. Each entry should have an identifier, date, artifact and version, observed behavior, expected behavior and its source, evidence such as logs or output files, investigation steps, resolution or unresolved status, and potential effect on the prediction or its interpretation. Do not replace the original observation with a corrected result; preserve both the initial result and any rerun after a documented change.
A practical classification separates several sources that can otherwise be conflated:
- Handoff or specification: a missing file, ambiguous endpoint definition, unclear inclusion rule, or inconsistent units in the supplied materials.
- Execution or implementation: a different event schedule, initialization, model version, code path, or dependency than the one intended.
- Numerical: differences associated with solver settings, tolerances, precision, or stochastic variation, where relevant.
- Data processing: a mismatch in filtering, transformation, missingness handling, or mapping between records and model inputs.
- Scientific or structural: a remaining mismatch between predictions and observations that cannot be explained by a reproducibility issue alone.
These categories are prompts for investigation, not mutually exclusive diagnoses. For example, an ambiguous unit can cause a data-processing error and then a prediction discrepancy. First establish which artifacts and operations each team actually used; then determine whether the difference is computational, interpretive, or scientific. Avoid silently choosing the interpretation that gives the closest fit.
Worked Example: Tracing a Unit Discrepancy
In a hypothetical model, a dose input is specified as 2 mg/kg for a 70 kg subject. The independent team enters the value as 2 mg, interpreting the data column as a total dose. The model therefore receives 2 mg rather than the intended \(2\times70=140\) mg.
The dose ratio is \(2/140=0.01429\), so the entered amount is about 1.43% of the intended amount, or 70-fold smaller. The team observes a much smaller simulated exposure than the reference run. It checks the model code and finds no change in the dose-event implementation; the discrepancy is traced to the data dictionary, which did not state whether the value was normalized by body weight.
The log should retain the original input and output, cite the ambiguous field definition, state the unit interpretation used in each run, and identify the corrected rerun separately. This is evidence of a handoff and input-interpretation problem, not by itself evidence that the model’s exposure mechanism is invalid. The team should still report whether the corrected run reproduces the reference result and whether the ambiguity affected other inputs.
Justifying a Traceable Comparison
A useful comparison requires more than placing two final numbers side by side. It requires a chain from the handed-off materials to each reported value. The logic is straightforward: if inputs, model version, execution settings, and output definition are identified, then a difference can be investigated at those links rather than attributed vaguely to “the model.” If one link is unknown, multiple explanations remain possible.
Record data versions, parameter values, units, and any transformations applied by each team.
Record model and code versions, software environment, event schedule, solver configuration, and run instructions.
Confirm that endpoint, time, population summary, and units correspond to the prediction specification.
Calculate the output difference and investigate each recorded link before assigning a cause or scientific meaning.
This procedure does not prove that the model is correct. It limits unsupported attribution: only after differences in inputs, execution, and output definitions have been examined can a remaining prediction–observation mismatch be discussed as evidence about model performance. Even then, the conclusion is bounded by the data, conditions, and claim being evaluated.
Worked Example: Separating a Run Difference from a Model Limitation
A hypothetical independent team receives a reference prediction of 18.0 ng/mL for a biomarker at 8 hours after dosing. Its first run gives 12.0 ng/mL. The team checks the log and finds that its dosing event was placed at time 1 hour rather than time 0, although the specification states time 0.
The first-run absolute difference is \(12.0-18.0=-6.0\) ng/mL, and its relative magnitude is \(6.0/18.0\times100\%=33.3\%\). This comparison is not yet evidence of a model-performance discrepancy because the run did not implement the specified schedule.
After correcting the event time, the team obtains 17.7 ng/mL. The corrected difference is \(17.7-18.0=-0.3\) ng/mL, with relative magnitude \(0.3/18.0\times100\%=1.67\%\). The report should preserve both runs, identify the event-time error, and describe the corrected result. If the observed validation value were, for example, 11.0 ng/mL, its comparison with the corrected prediction would be a separate predictive assessment; it must not be confused with the original implementation mismatch.
Diagnostics, Limitations, and Reporting
Warning signs include unexplained differences in units or time origins, results that cannot be regenerated from saved inputs, undocumented manual edits, inconsistent predictions across output files, and developer explanations that are not captured in the record. For stochastic models, a single run may differ by chance; the team must document random-number settings and the replication procedure specified for the analysis rather than treating one realization as a deterministic reference. For stiff or event-driven models, solver and event handling can be especially consequential, so record them and inspect output around discontinuities.
Independent validation also has limits. A separate team may share tools, expertise, or assumptions with developers, so organizational separation does not eliminate shared error. A successful reproduction demonstrates that the supplied workflow can generate the specified output in the tested environment; it does not establish biological truth, broad applicability, or validity beyond the tested conditions. A discrepancy log can reveal ambiguity and fragility, but does not by itself resolve structural uncertainty or define the model’s domain of applicability.
The final report should summarize the handoff versions, execution conditions, reproduction status, discrepancies and their dispositions, and limitations. State what was reproduced, what was not, and what evidence supports each statement. Where a discrepancy remains unresolved, report it as unresolved rather than selecting a convenient explanation. Whether evidence meets formal qualification requirements depends on acceptance criteria specified for the decision; this tutorial’s reproducibility record should supply evidence for that decision, not substitute for it.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A team reproduces the developers’ biomarker prediction exactly. Does this establish predictive accuracy? Explain.
- A reference output is 40.0 mg/L and an independent output is 42.0 mg/L. Calculate the signed and relative differences.
- An independent run disagrees with the reference because the team used a different data-filtering rule that was not specified. What should be recorded before drawing a scientific conclusion?
- Name two pieces of evidence that help distinguish an implementation discrepancy from a remaining prediction–observation mismatch.
Solutions. (1) No. It supports reproducibility of the specified calculation, not agreement with validation observations or accuracy beyond the evaluated conditions. (2) The signed difference is \(42.0-40.0=+2.0\) mg/L; the relative difference is \(2.0/40.0\times100\%=5.00\%\). (3) Preserve the original inputs and result, document the filtering rule used and the source of the ambiguity, identify the intended rule if established, and report any corrected rerun separately. The original comparison cannot by itself be interpreted as a scientific model discrepancy. (4) Examples include versioned model and data files, run logs identifying software and settings, a documented event schedule, and an explicit endpoint definition with units. These establish what was run and what quantity was compared.