Why Common Inputs Matter
Two runs can be placed side by side without being meaningfully comparable. If they use different analysis data, likelihood definitions, or estimation settings, a difference in their outputs may have several possible explanations. A fair comparison starts by specifying the question: are the runs intended to be repeated executions of the same analysis, or is one planned feature being changed while the rest are held constant?
For example, repeating the same model with the same setup can help reveal whether an estimation result is reproducible. Comparing two candidate model versions asks a different question: what changes when the model version changes under otherwise common conditions? Neither question is answered merely by having two run records. The relevant inputs must be controlled or any departures from them made explicit.
“Common” means more than using files with the same name or models with the same label. It means establishing what was actually supplied to the estimation procedure and how it was interpreted. The earlier material on run manifests, execution environments, model version lineage, and immutable run artifacts provides the basis for documenting those details.
Define the Comparison Before Launching Runs
Represent the inputs to a run conceptually as \(R=(D,M,L,E,S)\), where \(D\) is the analysis data and its preparation, \(M\) is the model specification, \(L\) is the likelihood definition, \(E\) is the estimation configuration, and \(S\) is the software and execution environment. The run produces an outcome \(Y(R)\), such as a completed fit and its recorded outputs. This notation is a bookkeeping device, not an assumption that outcomes are a simple sum of input effects.
For a comparison intended to study a change in model \(M\), the contrast of interest is conceptually \(Y(D,M_1,L,E,S)-Y(D,M_0,L,E,S)\). The shared \(D,L,E,S\) make the model change the designated difference. If instead the runs use \(D_1\) and \(D_0\), or different estimation settings, their outcome difference cannot be attributed to the model change alone. A controlled comparison does not guarantee that the model is adequate or that estimation succeeds; it makes the comparison interpretable with respect to the stated question.
Before submitting runs, write down the comparison question in one sentence and identify the intended difference. Then specify which fields must match exactly and which differences will be accepted but disclosed. The choice depends on the purpose. For a strict repeatability check, model, data, likelihood, estimation settings, and software should ordinarily be the same. For a planned algorithm comparison, the algorithm is intentionally different, but the data, model, likelihood, and other relevant settings should be kept common where possible.
Inputs to Hold Constant or Disclose
The following categories are common sources of accidental differences. A comparison record should contain enough detail to verify equivalence, not merely assert it.
| Input category | What to match or record | Why it matters |
|---|---|---|
| Data and preparation | Exact input artifact or checksum; included records; exclusions; transformations; units; categorical coding; missing-data handling; sort or merge steps | Different records or values change the information presented to the model. |
| Model specification | Model version; structural and statistical model; parameterization; covariates; parameter constraints; event and observation definitions | A label such as “candidate A” does not establish that the executable specification is identical. |
| Likelihood | Observation distribution; residual-error structure; handling of censored or below-quantification observations; likelihood conventions and any objective scaling | Runs may optimize different quantities even if their model descriptions appear similar. |
| Estimation configuration | Estimator and algorithm; tolerances; iteration limits; initial estimates; fixed or released parameters; numerical options; random seed where applicable | Different settings can lead to different numerical paths, termination, or outcomes. |
| Software and environment | Program and version; relevant libraries; operating system or platform; compiler or numerical library where relevant; command-line options | Implementation and numerical differences can affect execution and reproducibility. |
Some fields are meaningful only for a particular software or analysis. A seed, for example, should be recorded when stochastic operations use it, but it does not make two runs identical if their data or model differ. Conversely, fields that appear operational can matter: parallel execution, compiler settings, or a changed numerical library may affect a calculation. Record such differences even when their influence is not known in advance.
If a field differs, decide whether that difference is the planned contrast or a confounder for the question. If its relevance cannot be assessed, document it and qualify the comparison rather than silently treating the runs as equivalent. Do not “fix” comparability by changing a run record after execution; preserve the actual artifacts and describe the mismatch.
Worked Example: Checking Whether Two Fits Share the Same Data
Two hypothetical runs are intended to compare model versions. Both manifests refer to a file called “analysis.csv” and report 1,200 rows. Run A records checksum “a81f,” and Run B records checksum “c20d.” Both files contain 1,200 rows, but Run B was created after a preprocessing change that converts dose from milligrams to micrograms.
Step 1: State the intended contrast. The question is whether changing the model version from \(M_0\) to \(M_1\) changes the fit, with data and other inputs held constant.
Step 2: Check the evidence for data identity. Matching filenames and row counts do not establish identical data. The different checksums show that the file contents differ. The documented dose-unit conversion confirms a substantive difference, not just a change in file metadata.
Step 3: Decide whether the comparison is controlled. It is not a common-data comparison as specified: the runs differ in both model version and dose units. The apparent contrast is \(Y(D_A,M_1,\ldots)-Y(D_B,M_0,\ldots)\), not a contrast that changes only the model.
Step 4: Resolve or qualify. If the intended question is the effect of model version, rerun both versions using the same correctly prepared data artifact and document its checksum and units. If the runs cannot be repeated, report that the comparison is confounded by data preparation and do not attribute their differences to model version alone.
Justification: Why One Uncontrolled Difference Weakens Attribution
The logic can be stated without assuming that model outputs respond linearly to their inputs. Let \(Y(D,M,L,E,S)\) denote an outcome recorded for a run. The comparison intended to isolate model version is:
Suppose instead that the data also differ. A useful complete decomposition is to add and subtract the outcome for \(D_1\) with the old model:
The first bracket is a model-version contrast at \(D_1\); the second is a data contrast at \(M_0\). The observed difference contains both. This algebra does not claim that every possible interaction has been separated into universal causal effects. Rather, it demonstrates why a two-run difference with multiple changing inputs does not identify the effect of the designated change by itself. Holding inputs constant removes those specific competing differences from the comparison.
Worked Example: Comparing an Estimation Setting
Two hypothetical runs use the same 480-subject dataset, model version, likelihood, software version, and initial estimates. Run A uses a maximum of 500 iterations; Run B uses a maximum of 1,000. All other recorded settings match. The planned question is whether increasing the iteration limit changes whether the algorithm reaches its stated termination condition.
Step 1: Identify the planned change. The iteration limit is the intended difference, so it should not be “matched away.” The common inputs include the same data artifact, model, likelihood, starting values, estimator, tolerances, and software.
Step 2: Verify the comparison is aligned with the question. Because the iteration limit differs and the listed other inputs match, the runs are suitable for this targeted comparison, provided the manifests confirm those values and no unrecorded setting changed.
Step 3: State what the design can establish. The pair can show whether the recorded termination outcome differs under these two iteration limits for this setup. It does not by itself establish that the longer limit is generally preferable, or that any other difference in outputs is caused only by extra iterations if other execution details were omitted.
Step 4: Preserve the distinction. Record the iteration limit as the planned contrast and retain both run manifests and artifacts. If the estimator, initial estimates, or tolerance also differed, the comparison would not isolate the iteration-limit change and should be repeated or explicitly qualified.
Distinguish Repeatability, Sensitivity, and Cross-Environment Comparisons
A fair comparison is defined by the question, so “hold everything constant” is not a universal rule. Three common designs illustrate the distinction.
- Same-configuration rerun: use the same model, data, likelihood, estimation settings, and software to check whether repeated executions yield consistent recorded behavior. If a random seed or parallel strategy is intentionally varied, identify that as the planned change rather than calling the runs exact repeats.
- One-factor sensitivity comparison: change one named setting, such as an initial estimate or algorithm option, while keeping other relevant inputs fixed. This helps isolate the setting’s role, but the result applies to the specified setup.
- Cross-environment reproduction: keep the analysis inputs as consistent as possible while changing a software version or platform deliberately. The environment is then the planned contrast and must be described precisely; “same software” or “different machine” is not sufficiently specific.
In practice, more than one factor may need to change—for example, a software update may require a compatible library. In that case, name the bundle of changes being compared. Do not imply that the effect of one component has been isolated when several changed together. If the goal is to isolate components, design additional runs that vary them separately, where feasible.
Worked Example: Reviewing a Cross-Environment Rerun
A hypothetical analysis is run twice using the same immutable dataset, model file, likelihood definition, initial estimates, and estimation options. Run A uses software version 1.8 on a Linux system; Run B uses version 2.0 on a Windows system. The comparison is described as a test of whether the new software version reproduces the earlier run.
Step 1: Inspect the intended change. The stated target is the software version, but both software version and operating system changed. The comparison therefore changes an environment bundle, not software version alone.
Step 2: Check what remains common. The immutable data and model artifacts and the matching recorded estimation settings support a useful cross-environment comparison. The manifests should also record relevant libraries, numerical dependencies, and execution options so their status is known.
Step 3: Limit the conclusion to the design. The pair can assess whether this analysis behaves consistently across the two recorded environment bundles. It cannot isolate a version-specific effect from an operating-system or dependency effect unless those other factors are controlled or additional runs separate them.
Step 4: Improve isolation if needed. To target software version more narrowly, run both versions on the same platform with other dependencies held fixed where possible. To study platform effects, run a version on both platforms. Record any unavoidable coupled changes rather than describing them as controlled.
Diagnostics, Failure Modes, and Limits
Before interpreting any run-to-run difference, audit the comparison for hidden changes. Useful warning signs include different data checksums despite matching filenames, unmatched observation counts, unexplained changes in units or coding, model files that share a label but not a version, differing likelihood treatment, or missing estimation options. A comparison can also fail at the record-keeping level: “default settings” is not enough if software versions use different defaults.
A practical review can follow this sequence:
Name the one change the comparison is meant to examine, or say that the runs are intended as same-configuration repeats.
Check immutable artifact identifiers and preparation details, then confirm the model version and likelihood definition.
Match relevant settings for the question; list deliberate changes and any unresolved differences.
Call it controlled for the stated contrast only when the intended change is distinguished from other relevant changes. Otherwise, qualify it or design additional runs.
A comparison can be carefully controlled and still be limited. It concerns the inputs and environment actually tested; it does not establish general behavior under other datasets, models, or platforms. Exact equivalence may also be impossible to verify if artifacts or settings were not preserved. In that case, record the uncertainty rather than inferring identity from similar labels or apparently similar outputs.
The main alternative to changing one factor at a time is to compare a bundle of changes that reflects a realistic workflow update. That can be useful when the practical question concerns the entire bundle, but it provides less information about which component accounts for a difference. More elaborate designs can separate multiple factors, but they require additional runs and a clearly defined comparison plan. The right choice depends on whether the question concerns a specific input or a combined operational change.
Common Mistake
A common mistake is to call two runs “the same analysis” because they used the same model name and input filename. The reasoning is that matching labels imply matching inputs. They do not: a file may have been overwritten, preprocessing may have changed, and defaults may differ across software versions. The correction is to verify the actual artifacts and relevant settings using the run records, identify the intended contrast, and disclose every unresolved or intentional difference. If that evidence is unavailable, the comparison is uncertain rather than confirmed as common-input.
Check Your Understanding
Answer each question, then compare with the solutions.
- Two runs have the same filename and row count for the dataset, but different checksums. What should be concluded about data identity?
- A comparison changes the residual-error model and the initial estimates. Can it isolate the effect of the residual-error change? Explain.
- For a same-configuration rerun, list four categories of inputs that should ordinarily be matched or recorded.
- Two runs use the same data, model, likelihood, and estimation options but different software versions. What can this design compare, and what must be disclosed?
Solutions. (1) Data identity is not established; different checksums indicate different file contents, so the preparation and values must be investigated. (2) No. Both the residual-error model and initial estimates changed, so the comparison cannot attribute a difference to the residual-error change alone; hold initial estimates common or make the two-factor contrast explicit. (3) Any four of data and preparation, model specification, likelihood, estimation configuration, and software or environment; the exact relevant fields should be recorded. (4) It can compare behavior across the two software versions under the shared recorded analysis inputs, but versions and relevant libraries or environment details must be disclosed; other environment differences should be controlled or acknowledged.