Why Lifecycle Model Versions Matter
A model supporting a label claim does not stand alone. Its predictions depend on data and data-processing choices, model structure, parameter estimates, assumptions, software, and the settings used to run it. If any of these change after approval, a later reviewer needs to know what changed, why it changed, and whether the resulting claim still follows from the evidence and analysis.
Version control for lifecycle models is therefore more than saving successive copies of a model file. It is a controlled record of the analytical state behind a decision. The aim is to make a prior analysis reproducible, to make a later analysis comparable with it, and to preserve the reasoning that connects results to label language. This supports the earlier material on claim-to-evidence traceability without revisiting how to formulate the claim itself.
A practical release should distinguish the analysis that was approved from the analysis that is being evaluated now. Preserve the approved state rather than overwriting it, then create a new release for each substantively distinct update. Changes can be small, such as correcting a data mapping, or consequential, such as adding postapproval observations or changing the model structure. Both require traceable records; the level of review should reflect their potential effect on the analysis and label claim.
What a Reproducible Release Records
A release record should make clear what was analyzed and how. It can be implemented in a controlled repository, a validated document system, or a combination of systems, provided that access, change history, retention, and retrieval are suitable for the intended use. The record should include, at a minimum:
- Identity and lineage: a unique release identifier, creation date, responsible analysts and reviewers, the preceding release, and a concise reason for the update.
- Data provenance: the source and extraction date, inclusion criteria, data cut, relevant data dictionary, and any permitted transformations. Record the exact analysis dataset or a controlled reference to it, not merely a database name.
- Model and implementation: the model specification, estimation or simulation code, scripts for data preparation, configuration files, and software versions or dependencies needed to execute them.
- Assumptions and run settings: assumptions that affect interpretation, analysis choices, random seeds where relevant, and settings such as estimation controls, simulation size, or convergence criteria.
- Results and decision context: outputs used in the labeling assessment, diagnostics relevant to their reliability, the claim or label section they support, and the rationale for the decision.
A short change log should describe what changed, what remained fixed, and why the change was made. “Model updated” is not informative enough: it does not tell a reviewer whether the data, parameter estimates, code, assumptions, or only the reporting format changed. Where a change is not applicable, state that explicitly rather than leaving an ambiguous blank.
Cryptographic hashes can help check whether a file is likely to be the same as a previously recorded file. A matching cryptographic hash provides strong evidence of byte identity, subject to collision risk; it is not mathematical proof that the byte sequences are identical. Direct byte comparison of the complete files establishes their byte identity. Neither a matching hash nor byte identity shows that the file is scientifically appropriate, correctly executed, or correctly linked to the relevant analysis. Retain controlled copies and provenance as well as integrity checks.
Worked Example: Recording a Release for a Label Claim
Suppose a hypothetical approved release, LC-1.0, supports a dosing statement for adults with a specified level of renal impairment. A postapproval data cut is planned. The team needs to prepare a release that another qualified analyst can identify and reproduce.
The team records LC-1.0 as the parent release and creates LC-1.1 for the new analysis. The record identifies the new data extraction date and analysis population, stores the derived analysis dataset in a controlled location, and documents the transformation code and any changed inclusion rules. It references the model specification, estimation script, configuration, and software environment used to produce the updated parameter estimates.
The run record lists the random seed for simulations, the number of simulated profiles, the convergence criteria, and the principal outputs used to assess the claim. The change log states that the model structure and dosing assumptions were unchanged, while the data cut and parameter estimates changed. The decision record identifies the affected label statement, summarizes the evidence reviewed, and records the rationale and reviewers.
This record does not establish that the revised statement is warranted; that requires scientific assessment. It does allow a reviewer to determine which analysis was used, reproduce its computational steps as far as the recorded environment permits, and distinguish the new evidence from unchanged methods.
Comparing Releases Without Confounding Changes
A release-to-release comparison should identify both the observed difference in results and the changes that could have produced it. If new data, a revised model, and a software change are introduced at the same time, a difference in prediction cannot automatically be attributed to any one of them. Record a comparison plan before interpreting the results: define the target output, relevant population and dosing conditions, and which components will be held fixed for each comparison.
For a scalar prediction, write the output as \(g(D,M,A,C)\), where \(D\) denotes the data and processing, \(M\) the model specification and estimated parameters, \(A\) the assumptions, and \(C\) the computational implementation and settings. This notation emphasizes that an output belongs to a full analytical state, not to the model name alone.
A one-component-at-a-time comparison is useful for diagnosis, but it can miss interactions. For example, a revised model may behave differently specifically because the new data cover a population absent from the earlier data. In such cases, evaluate the relevant combinations or report that the effects cannot be uniquely separated without an attribution convention.
Worked Example: Separating Data and Model Changes
Consider hypothetical predictions of the proportion of patients expected to exceed a predefined exposure threshold. The earlier data are \(D_0\), the updated data are \(D_1\), the earlier model is \(M_0\), and the revised model is \(M_1\). For a controlled comparison, suppose all other assumptions and computational settings are fixed. The four predictions are:
- \(g(D_0,M_0)=0.18\)
- \(g(D_1,M_0)=0.20\)
- \(g(D_0,M_1)=0.19\)
- \(g(D_1,M_1)=0.23\)
The total change from the old release to the new release is \(0.23-0.18=0.05\), or 5 percentage points. Using the old model, the data change alone raises the prediction by \(0.20-0.18=0.02\), or 2 percentage points. Using the old data, the model change alone raises it by \(0.19-0.18=0.01\), or 1 percentage point.
The interaction contrast is the difference in the data effect under the two models:
Thus, the data effect is 4 percentage points under the revised model but 2 percentage points under the old model. The interaction is 2 percentage points. The total change is not just the two “alone” changes: \(0.02+0.01+0.02=0.05\).
Attribution also depends on the path used. Changing data first gives \(0.20-0.18=0.02\) for data, followed by \(0.23-0.20=0.03\) for the model. Changing the model first gives \(0.19-0.18=0.01\) for the model, followed by \(0.23-0.19=0.04\) for data. Both paths sum to 0.05, but allocate the interaction differently. Report the chosen comparison path or interaction; do not present one allocation as the uniquely true cause.
Reproduction Checks and Failure Modes
Reproduction means more than obtaining a number that looks similar. First determine whether the aim is computational reproduction—executing the recorded inputs and code—or analytical reproduction—independently implementing the stated analysis. These checks answer different questions. Repeating the original code may reveal that the archived run is reproducible, while an independent implementation can help detect errors in that code or in the written model description.
Exact equality may be expected for stored source files, fixed input data, or deterministic intermediate outputs. Numerical model results can differ slightly across hardware, compilers, or software versions because of rounding, algorithms, and parallel computation. Specify in advance which outputs must match exactly and which should agree within a scientifically justified tolerance. A tolerance should be appropriate to the output and decision; it should not be chosen after seeing an inconvenient discrepancy.
Worked Example: Investigating a Failed Reproduction
A reviewer reruns a hypothetical exposure simulation from release LC-1.1. The archived summary gives a mean exposure of 240 mg·h/L, while the rerun gives 246 mg·h/L. The protocol for this example specifies that the mean should reproduce within 1 mg·h/L. The difference is \(246-240=6\) mg·h/L, so the rerun fails that criterion.
The reviewer first checks the release identifier, dataset reference, and simulation configuration rather than immediately concluding that the model has changed. The archived configuration specifies 10,000 simulated profiles and a fixed random seed; the rerun used 1,000 profiles and no recorded seed. This is a plausible implementation of the wrong run configuration, not evidence that the archived result is erroneous.
After restoring the recorded profile count and seed, suppose the rerun gives 240.3 mg·h/L. The absolute difference is \(240.3-240=0.3\) mg·h/L, within the stated 1 mg·h/L tolerance. The reviewer records the initial failure, its identified cause, the corrected run settings, and the final comparison. If the discrepancy had remained outside tolerance, the next checks would include data version, transformations, software dependencies, units, and model code. A close numerical result alone would not establish that the correct analysis was reproduced.
Common failure modes include overwriting the approved analysis, preserving only final reports rather than executable inputs, relying on filenames such as “final_v2,” omitting data transformations, and failing to record the software environment or run settings. Another is to compare outputs produced under different populations or dosing conditions as if they were directly comparable. Diagnostics should therefore examine both the provenance record and the output definition.
For lifecycle use, keep a concise, reviewable difference report. It should identify changed components, their intended rationale, any checks used to isolate their effects, differences in key predictions and uncertainty summaries, and implications for the supported label claim. Preserve the earlier release and its decision record even if the newer release supersedes it operationally.
Choosing a Versioning Approach and Its Limits
Several approaches can contribute to control, but none is sufficient alone. A source-code repository supports reviewable code history and branching; it does not necessarily preserve restricted datasets or the full computational environment. A data catalog can document dataset lineage but may not capture model assumptions or decision rationale. A document-management system can preserve approved records and review history, but may not make code execution reproducible. A release manifest that links these controlled resources provides an integrating record.
Likewise, a simple change log is easy to read but can be too informal to reproduce an analysis. A fully containerized environment can improve computational reproducibility but cannot establish that data were correctly derived or assumptions were scientifically justified. Version identifiers can follow local conventions—such as distinguishing major analytical changes from minor corrections—but their meaning must be defined and consistently applied. The identifier is an index to the record, not a substitute for it.
Version control also has limits. It cannot resolve scientific disagreement, guarantee that a model is appropriate for a changed population, or determine whether a label claim is justified. Controlled storage may be constrained by privacy, data-use agreements, or long-term software availability; in such cases, preserve access-controlled references and sufficient metadata to identify the source and transformations. Reproducibility should be planned within those constraints rather than assumed from the presence of an archived report.
Common Mistakes
- “The model version is recorded, so the analysis is reproducible.” A model name does not identify the data cut, code, assumptions, environment, or run settings. Record the complete analytical state or controlled references to it.
- “The hash matches, so the files are certainly identical.” A matching cryptographic hash is strong evidence of byte identity, subject to collision risk. Direct byte comparison of the complete files establishes byte identity; neither check establishes scientific correctness.
- “The prediction changed, so the model caused the change.” Data, assumptions, code, and settings may also have changed. Use controlled comparisons or report the joint change and attribution limits.
- “A rerun is close enough because the conclusion is unchanged.” Reproduction criteria should be specified for relevant outputs and justified in advance. A similar conclusion does not explain a numerical discrepancy.
- “Only major scientific changes need a record.” Even a small data-processing or implementation correction can affect results. Record the change and scale review to its potential impact.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A new release uses the same model structure but a later data cut and a new data-cleaning script. Which elements must be recorded to support a meaningful comparison?
- Two files have the same cryptographic hash. What does this support, and what does it not prove?
- In the worked data-and-model comparison, calculate the total prediction change and interaction in percentage points.
- A rerun differs from the archived output by more than the prespecified tolerance. Name two checks to perform before interpreting the discrepancy scientifically.
- Why can changing data first and changing the model first produce different attributions even when both paths give the same total change?
Solutions. (1) Record the data source and cut, analysis population, cleaning script and transformations, model specification and code, assumptions, environment and run settings, relevant outputs, and the reason for the update. Identify which elements changed and which were held fixed. (2) A matching cryptographic hash provides strong evidence of byte identity, subject to collision risk; it does not prove identity with mathematical certainty, correctness, or appropriate use. Direct byte comparison of the complete files establishes byte identity. (3) The total is \(0.23-0.18=0.05\), or 5 percentage points. The interaction is \((0.23-0.19)-(0.20-0.18)=0.02\), or 2 percentage points. (4) Check that the intended release, dataset and transformations were used, and verify software versions, code, units, and run settings such as simulation size or seed. (5) The data effect depends on which model is used, and the model effect depends on which data are used. Their interaction is allocated differently by the two comparison sequences, although both sequences telescope to the same total change.