Why Automate Sensitivity Runs?
A sensitivity analysis may require fitting the same model under many configurations: a grid of selection-model parameters, alternative pattern-mixture restrictions, or different imputation settings. Running these configurations manually makes omissions and accidental overwrites likely. A batch runner can reduce those risks, but only if it records what was run, detects what finished, and has a deliberate response to interruptions.
The goal here is execution control, not a new statistical model or a method for interpreting sensitivity results. The workflow assumes that the scenario specifications and analysis code have already been defined and checked. It should preserve enough information to identify an execution and its outputs, expose failures for review, and allow unfinished work to resume without silently replacing a result already verified as complete.
Give Each Job a Stable Identity
A useful runner starts from a configuration table with one row per scenario. Each row should specify the scenario identifier, all analysis parameters, and the requested random seed or seed-generation rule. The run record should also include the input-data version or checksum, code version, relevant software and package versions, random-number generator details where available, start and end times, and the execution status.
The scenario identifier should be stable and unambiguous. For example, a label such as delta_0.5 can be supplemented by a canonical representation of the complete configuration, so that two distinct settings cannot accidentally share a label. If the same scenario is run on changed input data or code, it is a different job, even if its human-readable scenario label is unchanged. Keep the requested seed fixed in the job record; do not rely on a batch position such as “the third job” to determine it, since changing job order could then change which random stream a scenario receives.
The record should distinguish the identity of a job from its attempts. A retry is another attempt at the same job, not a new scenario. Store a separate attempt number, timestamps, and outcome for each attempt. This makes it possible to tell whether a job failed once and later completed, rather than allowing a later success to erase evidence of an earlier failure.
| Record item | Purpose |
|---|---|
| Scenario ID and full configuration | Identifies the requested analysis settings |
| Input and code identifiers | Shows which data and analysis implementation were used |
| Seed and RNG information | Supports traceability and, where possible, reproduction of stochastic steps |
| Software and package versions | Documents the computational environment |
| Attempt number, times, status | Shows execution history and whether work was interrupted |
| Warnings, errors, and output checks | Helps distinguish a completed calculation from a usable, verified result |
Worked Example: Creating a Batch Record
Suppose a hypothetical analysis has three delta scenarios, \(\delta=0\), \(\delta=0.5\), and \(\delta=1.0\). The same input file and code version are used, and each scenario receives a prespecified seed: 4101, 4102, and 4103, respectively. The software environment is recorded as R 4.4.1 with package versions stored in a lockfile.
The batch therefore contains three jobs. The record for \(\delta=0.5\), for example, includes its scenario ID, the exact value \(\delta=0.5\), the input checksum, code commit identifier, seed 4102, RNG details if available, and the software versions. If this job is interrupted and retried, its new attempt number is 2; its seed and job identity remain unchanged unless the analysis plan explicitly defines a new stochastic run as a distinct job.
Before launching, the runner checks that there are three unique scenario IDs and that each configuration has the expected seed. If there are \(3\) requested jobs and \(1\) is already verified complete, the remaining workload is \(3-1=2\) jobs. This count is valid only after the completion checks have been applied; a status label alone does not establish that a result is usable.
Use Explicit States and Structured Logs
A practical state sequence is pending, running, then either complete or failed. A job can also be marked unverified during recovery when the runner cannot establish whether its result was fully published. Record state changes with timestamps. Keep a structured manifest for current job status and append attempt-level log records rather than treating a single overwritten text log as the whole history.
For each attempt, capture the job and attempt identifiers, start time, software details, warnings, errors, exit status, and the location of any temporary or final output. Warnings should be recorded even if the software returns a successful exit status; a warning may signal convergence trouble, boundary estimates, or another condition requiring review. Errors should include enough context to locate the failed operation, while avoiding sensitive data in logs. A run that exits normally is not automatically a statistically acceptable fit: the runner can check required fields and basic output conditions, but model diagnostics still require subject-matter review.
Check required fields, unique job identities, seed assignments, and input availability before starting expensive work.
Mark it running with an attempt number and timestamp. When multiple workers are used, use a lock or equivalent claim mechanism to prevent two workers from writing the same job concurrently.
Run the configured analysis, recording versions, warnings, errors, exit status, and outputs for that attempt.
Check that outputs are readable, identify the expected job, and contain required results. Publish only after these checks pass; otherwise record failure or leave the result unverified.
Inspect both the manifest and artifacts. Resume jobs that are not verified complete, and preserve verified results rather than replacing them automatically.
Safe Recovery and Result Protection
The central recovery rule is that a job is skipped only when its result is positively verified. A manifest entry marked complete is not enough by itself: the artifact may be absent, truncated, unreadable, or associated with a different configuration. Conversely, a final-looking file does not prove completion until it can be read and checked against the expected job identity and required output fields.
Write each attempt to a temporary location, validate it, and then publish it to a final location that is unique to the job. If the platform provides atomic rename, a reader may see either the old name or the new name rather than a partially renamed entry. Atomic visibility is not the same as crash durability. A successful rename alone does not guarantee that file contents or the containing directory entry will survive a machine crash or power loss. Durability depends on the storage system and its guarantees; where required, use the platform’s durability operations, typically syncing the file before rename and syncing the containing directory after rename.
Recovery must still handle uncertainty. If a final file is missing or unreadable after restart, treat the job as unverified, not complete. Apply the retry policy: preserve the damaged or ambiguous artifact for diagnosis when feasible, write a new attempt to a separate temporary location, and publish a replacement only under an explicit policy. Never overwrite a verified successful result merely because a job appears in a restart queue. If a result is already verified, record that the job was skipped on resume.
This policy follows from a simple invariant: the runner may label a job complete only after checking a durable, readable artifact against the job identity and completion criteria. If any check fails, the implication from “artifact found” to “job complete” does not hold. Retrying an unverified job is safer than treating uncertain output as valid; preserving verified output prevents a retry from destroying a known result.
Worked Example: Resuming After an Interruption
A hypothetical batch has four jobs. On restart, the manifest says that jobs A and B are complete, job C is running, and job D is pending. The runner checks the artifacts: A is readable and has the expected job key and required fields; B has no final file; C has a file that cannot be read; D has no output.
Job A passes verification, so it is skipped and its artifact is not overwritten. The manifest claim for B is not sufficient because its final file is absent; B is marked unverified and queued for a new attempt. C is also unverified because its file cannot be read; the runner retains it for diagnosis if possible and retries according to policy. D is pending and runs for the first time.
Thus \(4\) jobs divide into \(1\) verified job and \(3\) jobs requiring action: \(4-1=3\). The runner does not claim that B or C completed, and it does not discard A. If the storage system had guaranteed crash-durable publication, the earlier successful publication would be expected to persist under that guarantee; without such a guarantee, recovery still relies on checking what is actually present and readable.
Batch Size, Diagnostics, and Alternatives
Batching affects the consequences of failure. A single long process can be efficient but may lose a large amount of work if it stops. Very small batches make recovery and review easier but add scheduling and startup overhead. A useful compromise is to run a bounded number of jobs per worker and publish each job’s result independently, so one failure does not invalidate other completed jobs.
Monitor counts by state, attempt number, elapsed time, and warning or error type. Useful signals include jobs remaining in “running” beyond a reasonable timeout, repeated failures for the same configuration, duplicate job keys, missing expected output fields, and a gap between the manifest and the artifacts. These are workflow diagnostics, not evidence that a statistical model is wrong by themselves. Follow the earlier material on diagnosing failed fits and unstable imputations when interpreting convergence or imputation problems.
A simple sequential script may be adequate for a small, short batch, provided it still records configuration and checks outputs. A parallel job scheduler can reduce elapsed time for many independent scenarios, but it requires reliable job claiming and protection against two workers writing the same destination. A workflow or database system can offer stronger state tracking, but adds operational complexity. Whichever approach is chosen, preserve the same core rules: stable job identity, attempt history, captured warnings and errors, explicit validation, and conservative recovery.
Worked Example: Handling a Warning and a Failed Attempt
Suppose job E is run twice. Attempt 1 exits with status 0 but emits a warning that the optimizer reached its iteration limit. Attempt 2 exits with status 1 and records an error before producing a complete result artifact. The runner stores both attempt records; it does not replace the first record with the second.
Assume the required completion check is that the artifact is readable, contains the expected job key and estimate field, and has an acceptable exit status under the prespecified policy. Attempt 1 meets the file checks but has a warning that requires review; the runner can mark it computationally produced while flagging it for statistical diagnostics, rather than treating the warning as invisible. Attempt 2 fails the exit-status and artifact checks, so it is not complete. The job is not declared verified complete merely because one attempt produced a file.
The log should identify the warning text, error text, timestamps, seed, versions, and attempt numbers. If the analysis team decides that the warning makes the fit unacceptable, that decision should be recorded as a review outcome and should trigger an explicitly configured rerun or investigation. The workflow should not silently change model settings or seeds to make the warning disappear.
Common Mistakes and Limits
- Using a single “done” flag as proof. A stale manifest can disagree with the artifact. Verify the artifact and its job identity before skipping work.
- Assuming a seed guarantees identical results everywhere. Reproduction can depend on RNG implementation, software versions, parallel execution, and platform behavior. Record these details, but do not promise bit-for-bit equality without confirming it.
- Overwriting an output on retry. A retry can destroy evidence or a previously verified result. Use attempt-specific temporary outputs and an explicit publication policy.
- Ignoring warnings because the process exited successfully. Preserve warnings and review their implications; computational completion and statistical adequacy are different judgments.
- Assuming rename means durable storage. Atomic visibility does not guarantee survival after a crash. Follow platform durability requirements and treat missing or unreadable output as unverified.
Automation cannot establish that a sensitivity model is scientifically appropriate, that a fit is substantively acceptable, or that an analysis is reproducible on every future platform. It can make execution more traceable and recovery more disciplined. Those protections are valuable precisely because a large batch otherwise makes it difficult to distinguish a missing run, a failed run, and a verified result.
Check Your Understanding
Answer each question using the workflow principles above. Brief solutions follow.
- A manifest says a job is complete, but its output file is absent. Should the runner skip or rerun the job, and why?
- Why should an attempt number be recorded separately from a scenario identifier?
- A process exits successfully but records a convergence warning. What should the log and completion review do?
- What does atomic rename guarantee, and what additional issue remains after a machine crash?
- Two workers are assigned the same job at once. Name one risk and one control that reduces it.
Solutions. (1) Treat the job as unverified and retry according to policy: the manifest alone does not establish that a readable result exists. (2) The scenario identifies the requested analysis, while the attempt number preserves the history of separate execution efforts, including failures and retries. (3) Retain the warning with the attempt record and review it under prespecified criteria; a successful exit is not by itself proof of statistical adequacy. (4) Atomic rename can provide all-at-once visibility of a name change, but does not by itself guarantee crash durability of file contents or directory metadata; use platform durability operations where required and verify output after restart. (5) Workers may race to overwrite or publish conflicting outputs; a lock or equivalent atomic job-claim mechanism helps ensure only one worker owns the job at a time.