Why Run-Tracking Errors Matter
A run comparison is only as reliable as the link between each reported result and the exact analysis that produced it. A table can contain plausible estimates and objective-function values yet still mislead if an output file was overwritten, one candidate used a different data set, or a retry was silently substituted for an earlier run. These are not cosmetic record-keeping problems: they can change which model appears preferable and whether a comparison is valid at all.
The earlier material on run directories, model lineage, candidate versions, run status, retries, common inputs, comparison tables, and discrepancy investigation provides the basic workflow. This tutorial concentrates on failure prevention: checks that expose breaks in that workflow before a numerical result is treated as evidence.
A practical comparison should be treated as a collection of linked records, not as a list of filenames. At minimum, each candidate needs a unique run identifier and references to the exact model and data versions used. The record should also identify relevant settings, software version, start and completion status, warnings or errors, and the location or checksum of retained outputs. A checksum is a compact value calculated from a file's contents; if the contents change, its checksum will ordinarily change as well. It helps detect replacement or modification, but it does not explain what changed or prove that the file is scientifically appropriate.
Establish the Comparison Conditions
Before interpreting a ranking, check that the candidates answer the same comparison question. The central conditions depend on the criterion. A likelihood-based comparison requires compatible likelihood definitions and the same observations under the same data treatment. Information criteria also depend on the parameter count and likelihood used. A likelihood-ratio comparison has additional requirements, including the relevant relationship between models and the validity of its reference distribution. A numerical value alone does not establish that those conditions hold.
For a direct comparison intended to isolate model differences, confirm that both candidates use the same analysis population, records, units, transformations, missing-data handling, and observation-level inclusion rules. Confirm that the objective or likelihood is calculated on that same basis. If some of these differ, the comparison may still answer a deliberately defined question, but it is not a like-for-like model comparison and must be described accordingly.
A useful check is to define the comparison identity as a tuple:
For a comparison that is meant to isolate model specification, these components should agree across candidates, except for the model feature being investigated. This is not a claim that all scientific comparisons must hold every input fixed: sometimes a data update is itself the question. It is a requirement to make the intended contrast explicit, so that differences in results are not mistakenly attributed to the model when they may instead reflect changed inputs or procedures.
Worked Example: Detecting an Overwritten Output
In a hypothetical analysis, run R17 fits candidate A and run R18 fits candidate B. Both outputs were initially written to a shared file named final_results.csv. The comparison table lists an objective value of 812.4 for A and 805.9 for B, but the file currently in that location contains the estimates for B.
Check the run-to-output links. The output file's current contents cannot independently support both rows. Its filename does not establish which run produced it, and the stored B estimates cannot be assigned to R17 merely because the table has a row for A.
Check retained evidence. Suppose the run log for R17 records a completed output checksum of 6f2a, while the current shared file has checksum 91bc. Since these checksums differ, the file is not the same artifact recorded for R17. The arithmetic check is simply: 6f2a is not equal to 91bc. The R17 estimates therefore cannot be verified from the current file.
Conclusion and corrective action. The objective values might still be correct if independently recorded in reliable run logs, but the parameter comparison is not reproducible from the available output. Retrieve R17's original output from an archived, run-specific location or rerun it from its recorded inputs. Preserve each result under a unique run path, then regenerate the comparison table from those verified outputs. Do not infer that B is preferable based on the displayed values until their provenance and the comparison conditions are confirmed.
Keep Retries and Data Changes Visible
A retry is a new execution, not a quiet replacement of the old one. It may use the same model and data, but it can differ in initialization, estimation settings, software environment, or other execution details. Preserve the original run record and create a distinct record for each retry. Link the retry to the run it follows, state why it was made, and document any changed settings. Retain failed and incomplete attempts with their status; they should not be presented as successful estimates, but they may explain how the final result was obtained.
Input changes deserve the same visibility. A data file can keep its name while its contents change, for example after a correction to a record or a revised preprocessing step. Record the data version or checksum and the preprocessing specification, not just a path such as analysis_data.csv. If one candidate was fit to an earlier data version, either rerun it on the common version or label the comparison as involving different inputs and limit the interpretation.
A concise audit trail can record the run ID, parent or retry ID, model version, data version, settings, software version, status, key warnings, output locations, and reason for any change. This makes it possible to answer two separate questions: which result is being reported, and what sequence of decisions led to it?
Worked Example: Checking Whether Candidates Used Common Data
Two hypothetical runs compare alternative absorption models. Run R21 reports 48 subjects and 576 concentration records; run R22 reports 48 subjects and 574 concentration records. The data manifest shows that R21 used data version D3 and R22 used version D4, after two records were excluded during a preprocessing update.
Quantify the discrepancy. The difference in included records is \(576 - 574 = 2\) records. The subject counts agree, but that does not establish that the observation sets agree. The two omitted records could affect likelihood and parameter estimates.
Assess the comparison. If the question is which model fits better to one common analysis data set, the stated comparison condition is not met: the candidates used D3 and D4. A difference in objective values could reflect both the model and the changed records. Matching subject counts alone is insufficient because the observation-level inputs differ.
Correct the workflow. Choose the intended data version, document its inclusion rules, and fit both models to that version. If the exclusion itself is under investigation, treat that as an explicitly separate comparison and avoid attributing the entire result difference to the absorption-model change.
Verify Likelihood Compatibility Before Ranking
Likelihood-based measures can only be compared meaningfully when their definitions and data basis are compatible. In particular, confirm that candidates use the same observations, response scale, likelihood contribution conventions, and treatment of records. Check whether a reported objective is actually the negative twice-log-likelihood or another objective function; similar-looking labels do not guarantee identical quantities. Also verify that parameter counts used in criteria are defined consistently.
For example, when the reported criterion is AIC computed from the maximized log-likelihood \(\ell\) and parameter count \(k\), its definition is \(AIC = -2\ell + 2k\). Lower AIC is preferred among the candidates being compared, under the criterion's stated assumptions and scope. That numerical ordering is not proof that a model is true, that it will perform best in every setting, or that the difference is statistically significant. The earlier material on information criteria and likelihood-ratio comparisons covers their interpretation; here the key safeguard is verifying that the inputs to the calculation are comparable before using the result.
Worked Example: Rejecting an Unsupported AIC Ranking
Suppose two hypothetical candidate records report the following: candidate P has \(\ell=-120\), \(k=4\), and was fit to 300 observations; candidate Q has \(\ell=-117\), \(k=5\), and was fit to 298 observations after two records were excluded. For illustration, calculate the reported AIC values.
Candidate P: \(AIC_P=-2(-120)+2(4)=240+8=248\).
Candidate Q: \(AIC_Q=-2(-117)+2(5)=234+10=244\).
The arithmetic gives \(AIC_Q < AIC_P\), a numerical difference of \(248-244=4\). However, P and Q used different observation sets. The computed values do not support a like-for-like ranking of these models on one common data set. It would be incorrect to report simply that Q is preferred because its AIC is lower. First fit both candidates to the same intended observations using compatible likelihood definitions, then recalculate the criterion. The displayed arithmetic is correct, but the proposed comparison is not supported by the recorded inputs.
Make Ranking Claims Match the Evidence
A comparison table should separate measured results from interpretations. Report the criterion, its units or definition where relevant, the run IDs, and any qualifications on comparability. Distinguish a raw numerical ordering from a supported inference. A model with a smaller minimized objective is not automatically the scientifically best model: the objective must have the intended meaning, the fits must be comparable, and the model must also be considered for plausibility, stability, and adequacy for the analysis purpose.
Do not rank candidates when a required value is missing, a run failed, or the objective definitions are unclear. Mark those entries as unavailable or not comparable and state what must be checked. Do not use a single successful retry to conceal earlier failures, or describe estimates from a failed run as final. If convergence or numerical diagnostics are concerning, report the issue and investigate it rather than treating a smaller objective as a definitive win.
The most useful comparison record includes both a result and its basis. For example: “Among candidates P and Q, fitted to data version D5 with the same likelihood definition, Q had the lower AIC by 3.2 units.” That statement is narrower and more defensible than “Q is the correct model.” If the inputs or assumptions differ, say so directly and do not make a ranking that the comparison cannot support.
Diagnostics, Failure Modes, and Practical Alternatives
Several simple checks catch a large share of tracking errors:
- Confirm that every comparison-table row points to one unique run ID and one retained output set.
- Reconcile model, data, settings, software, and output references against the run record before interpreting differences.
- Compare record counts and, where available, record-level identifiers or checksums; counts alone cannot confirm identical data.
- Check that retries are distinct records and that changed settings or inputs are stated.
- Verify the definition of the objective or likelihood and the parameter count used in any criterion.
- Review status, warnings, and numerical diagnostics before treating a run as a valid candidate result.
A common mistake is to assume that matching filenames or subject counts proves that runs used identical inputs. The correction is to verify versioned content and analysis rules, including the observation set. Another is to overwrite a failed run's output with a retry and keep only the successful file. This hides the execution history; preserve both attempts and clearly identify which output is being interpreted. A third is to interpret the lowest objective as an unconditional model ranking. First establish that the objective is comparable and that its direction and definition are known.
Manual logs and automated manifests are complementary approaches. A manual log can capture scientific intent and explain why a retry was needed, but it is vulnerable to omission or transcription error. Automated capture of file checksums, software versions, and run paths can improve consistency, but it cannot decide whether a likelihood comparison is scientifically appropriate. A robust workflow combines automated capture with human review of the comparison question and its assumptions.
These checks reduce preventable errors; they do not guarantee that a model is adequate or that every computational problem has been detected. A checksum establishes file identity, not data correctness. A complete run record establishes traceability, not scientific validity. Model diagnostics and the substantive interpretation of results remain necessary.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A run table identifies two estimates by the same output filename, but the file was overwritten after the second run. What can and cannot be concluded?
- Two candidates include the same number of subjects, but one has 410 observations and the other 408. Is this enough to establish a common data basis? What should be checked?
- A retry uses a different initial estimate and produces a lower objective. What should be recorded before interpreting the change?
- Two AIC values are computed correctly, but the candidates used different observation sets. What is the appropriate conclusion?
Solutions.
1. The current file cannot verify both runs' estimates because its contents correspond only to the file state that remains. The objective values may be usable only if independently preserved and linked to each run in reliable records. Retrieve the original artifact or rerun the analysis, and retain outputs in run-specific locations.
2. No. Matching subject counts do not prove identical observations. The difference is \(410-408=2\) records; check record-level identifiers, data versions, preprocessing, and inclusion rules. If the candidate fits are intended to isolate model differences, refit them on the same analysis data.
3. Give the retry a distinct run ID and link it to the original. Record the changed initial estimate, the reason for retrying, other settings, model and data versions, software environment, status, warnings, and retained outputs. Then evaluate both the objective and relevant run diagnostics; a lower value alone does not establish a valid or preferable fit.
4. The arithmetic may be correct, but the values do not support a like-for-like AIC ranking on one common data set. State the data mismatch and avoid claiming one candidate is preferred on that basis. Refit both candidates to the intended common observations and recalculate AIC if that is the comparison question.