Managing Retries And Resubmissions

Why Retries Need Their Own Records

A model execution can fail, stop before producing usable results, or finish with a status that prompts further work. Starting it again may seem simple, but the new execution must not erase the evidence of the first one. In a pharmacometric workflow, a later reviewer may need to establish which specification was used for each attempt, why it was repeated, and whether the modeling inputs or settings changed.

The practical distinction is between repeating an execution under the same declared specification and submitting a new execution with a changed specification. Both need a unique execution record. Neither should be represented by editing the original record until it appears to describe the most recent attempt.

Definition: A retry is a new execution of the same declared run specification as an earlier execution, linked to that execution and given its own identifier and record. A resubmission is a new execution linked to an earlier execution for which one or more declared inputs, model settings, estimation settings, or execution controls have changed. The earlier execution remains preserved in either case.

A run request is the continuing analytical task, such as fitting a specified candidate model to a specified data snapshot. An execution is one actual attempt to carry out that request. A request can therefore have several executions: an initial execution, a retry of that execution, or one or more resubmissions based on an earlier attempt. The request identifier groups the history; the execution identifier distinguishes each attempt.

The earlier material on run identifiers, manifests, execution environments, immutable artifacts, model version lineage, candidate model versions, and run status and outcomes supplies the records that each execution should reference. Here the focus is the relation between executions: what the new record points to, what remains unchanged, and how any differences are described.

Representing the Execution Lineage

Give every execution a unique identifier. In addition, record the stable request identifier, the parent execution identifier (if any), the relationship to that parent, and a concise reason for starting the new execution. The first execution has no parent. Each later retry or resubmission points to the specific execution from which it proceeds—not merely to the latest execution in a directory or to a broad model name.

A useful lineage can branch. For example, after an initial execution, one child might retry the unchanged specification while another child might resubmit with a revised initial estimate. These are separate paths from the same parent. Do not make one child appear to be the parent of the other unless the second execution actually proceeded from the first.

Recordkeeping rule: For each new execution, preserve its own identifier, request identifier, parent execution identifier, relationship type (retry or resubmission), creation time, reason, and references to its manifest and environment record. For a resubmission, also record each changed field, its prior and new values, and the reason for the change. For a retry, record that the declared specification was checked and unchanged.

The declared specification must be clear enough to support that classification. It normally includes the data and model inputs, model version, estimation settings, and computational controls that can affect the execution. Record the environment as well. If an environment difference is known, it must be documented even when the analyst did not intend to change the scientific specification. A changed environment means the executions are not exact repeats; describe the difference and classify the new execution according to the workflow’s declared scope rather than silently calling it an identical retry.

A scheduler-generated job number, start time, or execution identifier will ordinarily differ between attempts. Those changes identify the new execution; they do not by themselves mean the declared run specification changed. By contrast, changing an optimizer setting, an initial estimate, a timeout, a data file, or a model control file is a specification change if that field is within the declared scope. State the scope in the workflow so that the retry/resubmission label is not a matter of intuition.

Preserve the completed record.
Do not replace the earlier manifest, environment information, status, logs, or artifacts with information from the new attempt.

Compare the declared specification.
Check the referenced input and model versions, estimation settings, execution controls, and recorded environment against the proposed new execution.

Classify and link.
If the declared specification is unchanged, record a retry. If one or more declared fields change, record a resubmission and name the parent execution.

Create a new execution record.
Assign a fresh identifier, retain the original request identifier, capture the reason and any changed fields, and store new outputs separately.

Worked Examples

Worked Example: Retrying After an Interrupted Execution

Hypothetical case. Request RQ-042 is to fit candidate model M7 using data snapshot D12. Execution EX-042-01 ends with a recorded interruption before the optimizer completes. The analyst proposes another execution with the same data snapshot, model version, estimation settings, software build, and declared execution controls.

The relationship can be written as:

\text{EX-042-02} \xrightarrow{\text{retry}} \text{EX-042-01}

The proposed child has request ID RQ-042 and parent ID EX-042-01. Suppose the specification check gives 5 matching declared components out of 5 checked components. The match proportion is \(5/5=1.00\), or 100%. That supports classifying the new execution as a retry, provided the recorded environment also matches the declared specification. The new execution still receives its own ID, start time, status, logs, and output location.

The reason should be specific, such as “repeat after scheduler interruption; no model or execution settings changed,” rather than simply “rerun.” EX-042-01 remains recorded as interrupted; EX-042-02 receives its own eventual status. The outcome of the second attempt does not retroactively change what happened in the first.

Worked Example: Resubmitting With a Changed Initial Estimate

Hypothetical case. Execution EX-081-01 used an initial estimate of 0.20 for a positive scale parameter and ended with an unsuccessful fit. The analyst proposes a new execution using an initial estimate of 0.35, leaving the data snapshot, model version, software build, and other estimation settings unchanged.

The changed value is \(0.35-0.20=0.15\) in the parameter’s units. Because an estimation setting changed, this is a resubmission, not an exact retry. A concise change record would identify the field as the initial estimate for that parameter, record the old value 0.20 and new value 0.35, and state the reason for choosing the new value.

\text{EX-081-02} \xrightarrow{\text{resubmission}} \text{EX-081-01}

EX-081-02 keeps the request identifier used by EX-081-01, receives a new execution identifier, and references EX-081-01 as its parent. The manifest for EX-081-01 continues to describe the 0.20 starting value; the new manifest describes 0.35. Preserving both records makes it possible to tell which execution used which starting value without relying on memory or overwritten files.

Worked Example: A Changed Time Limit and a Branching History

Hypothetical case. Execution EX-106-01 is submitted with a maximum wall-clock time of 2 hours and stops when that limit is reached. The analyst decides to allow 4 hours on the next attempt. The changed control is \(4-2=2\) additional hours, so the declared execution specification is not identical.

Record the new execution as a resubmission linked to EX-106-01, and document the wall-clock limit as changing from 2 hours to 4 hours. Do not label it an unchanged retry merely because the model and data are the same. If the workflow treats that control as part of the declared specification, the change determines the classification.

Now suppose a second execution is also launched from EX-106-01, this time with the original 2-hour limit and all other declared fields unchanged. That second child is a retry of EX-106-01. The lineage has two children with the same parent: one resubmission and one retry. Each child needs its own manifest, status, and artifacts; neither should be recorded as a continuation of the other.

Why the Original Attempt Must Remain Intact

Preservation is necessary for a basic logical reason. Let \(R_1\) denote the complete record of the first execution and \(R_2\) the record of a later execution. If the original record is overwritten, the stored record may retain only \(R_2\). The mapping from the pair \((R_1,R_2)\) to the overwritten record is not one-to-one: many different first attempts could lead to the same final stored record. Therefore the original attempt cannot reliably be reconstructed from that record alone.

(R_1,R_2) \longrightarrow R_2 \qquad\text{does not preserve information about }R_1

An append-only history avoids this loss: it stores \(R_1\) and \(R_2\) as distinct records and makes the parent-child relation explicit. Immutability does not mean errors in a record can never be corrected. It means a correction should be made as a traceable amendment or new record, with the correction reason and timing retained, rather than silently replacing the evidence.

Diagnostics, Alternatives, and Limitations

Several checks help identify a lineage that may be misleading or incomplete:

Missing parent reference. A noninitial execution without a parent is difficult to interpret. Confirm whether it is genuinely a new request or a retry/resubmission whose link was omitted.
Retry label but changed specification. Compare the new manifest and environment record with the parent. A changed declared field requires a resubmission label, or an explicit explanation if the workflow defines a different classification rule.
Resubmission with no change description. A new execution with a different configuration but no field-level record cannot show what prompted the change. Add the old value, new value, and reason.
Overwritten output or manifest. If two execution IDs point to files that were replaced in place, the identifiers alone do not preserve the evidence. Store each execution’s artifacts separately or use an immutable artifact reference.
Unexplained environment drift. A different software build or computing environment can affect reproducibility. Record the difference and do not assert that the executions were identical.
Ambiguous parent in a branch. Point to the actual source execution. “Previous run” is not enough when several executions belong to the same request.

The main alternative to linked execution records is a single mutable record that always describes the latest attempt. It is simpler to browse, but it loses prior configurations and makes it hard to establish which outputs belong to which execution. A separate folder per attempt can help preserve files, but folders alone do not explain whether an attempt was a retry or resubmission, which execution it followed, or why it was launched. Use folders and identifiers as storage and indexing tools, not as substitutes for an explicit lineage record.

A well-maintained history does not establish that a model is scientifically appropriate or that an execution is numerically reliable. It establishes what was run and how the attempts relate. The record can also be incomplete if relevant settings are omitted from the declared specification or if a tool changes behavior without exposing that change. These limitations make a clear scope, accurate manifests, and recorded environments essential.

Common Mistake

Common mistake: “The first attempt failed, so I will reuse its ID and replace its settings and outputs with the successful rerun.” This reasoning confuses a request with an execution and destroys the history needed to understand the result. Keep the request ID stable, create a new execution ID, link it to the actual parent, and preserve both records. If the new attempt changed a declared setting, record it as a resubmission rather than presenting it as an unchanged retry.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

An execution is relaunched with the same model, data, estimation settings, execution controls, and recorded environment after a transient interruption. What relationship should the new execution record show, and what must remain separate?
A new attempt changes an estimation tolerance from 0.001 to 0.0005. Calculate the change and classify the attempt, assuming tolerance is within the declared specification.
Two new executions are launched independently from the same parent: one keeps the specification unchanged and one changes a starting value. Should the second execution be linked to the first new execution?
Why is replacing the original manifest with the latest manifest insufficient, even if the latest execution succeeds?

Solutions. (1) Record a retry linked to the interrupted execution. Give it a new execution ID and preserve its own status, logs, manifest reference, and artifacts separately from the original. (2) The change is \(0.0005-0.001=-0.0005\); the tolerance has been reduced by 0.0005, so this is a resubmission linked to the execution whose specification it changes. Record both values and the reason. (3) No. Both executions should point to the shared parent because each was launched from that parent. The unchanged specification is a retry; the changed starting value is a resubmission. (4) Replacing the manifest removes evidence of the original specification. The latest manifest cannot establish what values or settings the earlier execution used, so retain both manifests and connect the execution records explicitly.