Why a Comparison Workflow Needs More Than a Ranking
A run comparison is defensible only when the results being compared belong to identifiable, suitable runs. A smaller objective function or information criterion is not enough by itself: the runs must use comparable inputs, their results must be interpretable, and the comparison method must fit the relationship between the candidate models. A documented workflow makes those checks visible before a preferred candidate is named.
The goal is not to declare that one model is universally true. It is to assemble evidence for a decision among specified candidates, while preserving the reasons that some runs were excluded or not selected. Earlier material on immutable run artifacts, lineage, common inputs, fit criteria, likelihood-ratio comparisons, and run-to-run discrepancies supplies the component concepts; here they are brought together into an auditable sequence.
Start With the Comparison Question and the Manifest
Before opening a results table, state the question: for example, whether adding a specified covariate improves a reference model, or which of several plausible structures provides a more useful fit. Identify the candidate runs and their intended relationship. A nested-model question may support a likelihood-ratio comparison; a comparison of non-nested models does not. Information criteria can compare candidates under suitable common-data and likelihood conditions, but do not turn an unsuitable run into valid evidence.
A manifest is the run-level record used to establish identity and provenance. For a comparison, inspect at least the run ID, candidate model and version lineage, data identity (including a checksum or other stable identifier), analysis settings, software and estimation configuration, run status, and links to the immutable output artifacts. Verify that output files correspond to the declared run rather than to a later retry or overwritten directory. Then check whether the candidates used the same observations, response definition, likelihood, and compatible analysis settings. Model differences are expected; unexplained differences in data or analysis conditions are not.
Name the reference candidate, the alternatives, and the intended comparison. Record whether each alternative is nested in the reference or is a non-nested candidate.
Match each run ID to its manifest and artifacts. Confirm common data and analysis conditions where required, and flag differences that prevent a fair comparison.
Check completion status, convergence indications, parameter estimates and precision, covariance or uncertainty diagnostics, and any discrepancy that needs investigation.
Use a likelihood-ratio comparison only when nesting and regularity conditions are credible. Use information criteria for appropriate common-data comparisons, and interpret them as relative measures.
State whether each run is retained as a candidate, not preferred, or excluded from comparison, and cite the evidence behind that decision.
Worked Example: Validating Candidate Run Manifests
Consider five hypothetical runs prepared for a comparison. Runs R-A, R-B, and R-C use the same 240 records, response definition, likelihood, and analysis settings. Their data checksum is the same string, 7F2A; they differ in model version as intended. Run R-D is marked complete but its manifest reports 238 records and checksum 91C4. Run R-E uses the same declared inputs as R-A but has a failed estimation status. The checksum labels and records are invented for this example.
R-A, R-B, and R-C. Their shared checksum, record count, response, and settings support proceeding to the estimation and model-comparison checks. Their different model versions are not a comparability defect; those differences define the candidates. The manifest and immutable output links must still agree on each run's identity.
R-D. Its record count and checksum disagree with the common input set. Therefore it is not eligible for this comparison, even though its status says complete. Its objective function should not be ranked alongside the other candidates as if it had used the same data. The data discrepancy should be resolved or R-D should be documented as excluded.
R-E. Its input identity is compatible, but a failed run does not supply a valid completed estimate for model ranking. Record the failure and any relevant logs; do not treat its missing or partial results as a poor-fitting model. A retry, if performed, needs its own recorded identity and outcome.
The manifest review therefore does not choose a model. It establishes which results may be evaluated further and prevents input or status differences from being mistaken for evidence about model quality.
Use Each Comparison Only Under Its Conditions
For a nested comparison, the likelihood-ratio statistic is the difference in objective function values, where the objective function is \(-2\) times the maximized log-likelihood. Suppose model 0 is nested within model 1, both are fitted to the same observations with the same likelihood definition, and model 1 has \(q\) additional freely estimated parameters. Under the null model, with regularity conditions satisfied (including an interior parameter value and identifiable parameters), the statistic is approximately chi-square distributed with \(q\) degrees of freedom. Boundary parameters, non-identifiability, or other failures of regularity can invalidate that reference distribution.
The equality follows directly from \(\mathrm{OFV}=-2\log(L)\): subtracting the alternative model's objective function from the null model's gives twice the difference in maximized log-likelihoods. Under the stated null and regularity conditions, Wilks' result gives the approximate chi-square reference distribution. A small p-value is evidence against the nested null in this comparison; it does not establish that the larger model is scientifically correct or that its estimates are reliable.
Worked Example: A Nested Likelihood-Ratio Comparison
Suppose hypothetical run R-A is the reference model with 8 estimated parameters and objective function 1000.0. R-B adds two covariate effects, is nested in the same likelihood framework, and has objective function 984.0. Both runs use the same 240 observations. Assume the added effects are identifiable, the null values are interior, and no boundary or other regularity concern is present.
The observed difference is:
The degrees of freedom are the number of added parameters, \(q=10-8=2\). For a chi-square variable with 2 degrees of freedom, the upper-tail probability at 16.0 is \(P(\chi^2_2\ge16.0)=e^{-16.0/2}=e^{-8}\approx0.0003355\). At a prespecified \(\alpha=0.05\), this is evidence against the nested reference model under the stated assumptions. It supports considering R-B, but does not remove the need to assess its precision, plausibility, and other diagnostics.
For candidates that are not nested, a likelihood-ratio test is not justified merely because both models have objective function values. Information criteria offer a different kind of comparison. For maximized likelihood \(L\), \(k\) estimated parameters, and \(n\) observations, the criteria used here are \(\mathrm{AIC}=\mathrm{OFV}+2k\) and \(\mathrm{BIC}=\mathrm{OFV}+k\ln(n)\). Lower values indicate a better relative trade-off between fit and complexity among the candidates being compared. These criteria are not hypothesis tests, and a difference between them is not a p-value. Their comparison requires compatible likelihoods and data; the criteria do not repair differences in input records or estimation validity.
Worked Example: Completing a Results Table for Three Candidates
Return to the hypothetical eligible runs. R-A has \(\mathrm{OFV}=1000.0\), \(k=8\); R-B has \(\mathrm{OFV}=984.0\), \(k=10\); and non-nested R-C has \(\mathrm{OFV}=978.0\), \(k=13\). Each uses the same \(n=240\) observations and compatible likelihood definition. For the table, suppose R-B has acceptable estimation diagnostics. Suppose R-C has a high relative standard error of 85% for one parameter, a near-singular covariance diagnostic, and correlations above 0.98 between two parameter estimates. These are hypothetical diagnostics.
For R-A, \(\mathrm{AIC}=1000.0+2(8)=1016.0\). For R-B, \(\mathrm{AIC}=984.0+2(10)=1004.0\). For R-C, \(\mathrm{AIC}=978.0+2(13)=1004.0\). Thus R-B and R-C tie on AIC in this example, despite R-C's lower objective function.
For BIC, \(\ln(240)\approx5.4806389\). Therefore R-A has \(1000.0+8\ln(240)\approx1000.0+43.8451=1043.8\); R-B has \(984.0+10\ln(240)\approx984.0+54.8064=1038.8\); and R-C has \(978.0+13\ln(240)\approx978.0+71.2483=1049.2\). The penalty for R-A is \(8\ln(240)\approx43.8451\), calculated using the unrounded logarithm before rounding the result.
| Run | Manifest and status | OFV; parameters | AIC; BIC | Evidence and disposition |
|---|---|---|---|---|
| R-A | Valid; complete; common 240 records | 1000.0; 8 | 1016.0; 1043.8 | Retain as reference. R-B improves fit in the nested comparison; R-A is not preferred on these criteria. |
| R-B | Valid; complete; common 240 records | 984.0; 10 | 1004.0; 1038.8 | Retain as leading candidate, subject to scientific review. Nested comparison favors added effects; diagnostics are assumed acceptable. |
| R-C | Valid; complete; common 240 records | 978.0; 13 | 1004.0; 1049.2 | Do not retain as preferred: AIC ties R-B, BIC is higher, and precision/covariance diagnostics indicate weak estimation. |
| R-D | Complete, but 238 records and different checksum | Not compared | Not compared | Exclude from this comparison because inputs do not match. |
| R-E | Failed estimation; common inputs declared | No valid estimate | Not calculated | Exclude from model ranking; record the failed outcome. |
The table gives R-B a leading, not absolute, status. R-C's lower objective function alone is insufficient: its extra parameters yield no AIC advantage over R-B, BIC is less favorable, and its reported diagnostics raise a substantive concern about precision. The evidence supports not selecting R-C as preferred here; it does not prove that the model structure is false. R-D and R-E are excluded for different reasons, which should remain explicit in the record.
Read the Evidence Together and Record the Decision
A completed comparison table should keep provenance, numerical results, diagnostics, and disposition connected. At minimum, include run ID and model lineage; manifest validation and status; data identity and observation count; objective function and parameter count; the relevant comparison statistic or information criteria; key estimation diagnostics; and a concise rationale. When a value is unavailable because a run failed or was not comparable, mark it as unavailable and explain why rather than leaving a blank that could be mistaken for oversight.
Diagnostics are part of the evidence, not decorative additions to a ranking. A run can have a lower objective function and still be a poor candidate if estimates are unstable, precision is inadequate for the intended use, or the covariance estimate signals weak identifiability. Conversely, an isolated diagnostic should be interpreted in context and investigated rather than converted automatically into a universal rejection rule. If a discrepancy between runs is unexpected, resolve its origin before interpreting the difference as a model effect.
The workflow also makes distinctions that protect interpretation. A run may be excluded because it used incompatible inputs or did not produce a valid estimate. A valid run may be not preferred because its evidence is weaker than another candidate. A candidate may be retained for further review without being established as the final scientific model. Record which meaning applies and the evidence supporting it.
Common Pitfalls and Limitations
- Ranking before validating inputs. A completed run on a different data set is not directly comparable just because its output includes an objective function. Confirm data identity and analysis conditions first.
- Using a likelihood-ratio test for non-nested models. The chi-square result requires nesting and regularity conditions. Use an appropriate relative criterion for a non-nested comparison instead, and do not report a likelihood-ratio p-value.
- Treating a smaller OFV as sufficient evidence. Adding parameters generally allows a model to fit at least as well. Consider complexity, estimation diagnostics, and scientific plausibility along with fit.
- Calling the lowest criterion “the true model.” AIC and BIC rank relative fit-complexity trade-offs for the specified candidates and data; they do not prove that a candidate is true or quantify the probability that it is true.
- Omitting excluded runs. Hiding failed or non-comparable runs makes the decision difficult to audit. Preserve their identities, statuses, and reasons for exclusion in the documented record.
The workflow has limits: it can organize evidence but cannot guarantee that candidate models are scientifically adequate, that all relevant alternatives were considered, or that an asymptotic approximation is accurate in a particular analysis. Its value is transparency: a reviewer can trace which runs were considered, what assumptions governed each comparison, and why the recorded decision followed from the available evidence.
Check Your Understanding
Answer each question, then compare your reasoning with the short solutions.
- A run is marked complete but has a different data checksum from the other candidates. Should its OFV be ranked in the same table? Why?
- Two nested models use the same data and likelihood. The larger model adds three parameters and reduces OFV by 7.2. State the likelihood-ratio statistic and degrees of freedom. What conditions are needed for a chi-square reference distribution?
- For the same \(n=240\), calculate the BIC penalty for a candidate with \(k=8\) parameters and report the penalty to four decimal places.
- A non-nested candidate has a lower OFV but worse BIC and a near-singular covariance diagnostic. Give two reasons not to select it solely because its OFV is lower.
- What is the difference between excluding a run and deciding that a valid run is not preferred?
Solutions. (1) No. The differing checksum means the input identity is not the same, so first resolve the difference or document the run as excluded from this comparison. (2) The statistic is \(\Delta\mathrm{OFV}=7.2\) with \(df=3\). A chi-square reference requires nested models, common data and likelihood, and suitable regularity conditions, including identifiable parameters with null values not on problematic boundaries. (3) The penalty is \(8\ln(240)=8(5.4806389\ldots)=43.8451\). (4) The lower OFV does not account for model complexity, and the worse BIC provides less favorable relative evidence after its complexity penalty; the near-singular covariance diagnostic also raises concern about estimation reliability. (5) Exclusion means the run is ineligible for that comparison, such as because inputs differ or estimation failed. Not preferred means the run is valid and comparable but the documented evidence favors another candidate.