Why Orchestration Matters
An end-to-end MICE workflow is not one indivisible command. It consists of stages that transform inputs into outputs: raw data are checked, prepared, imputed, analyzed, and used to create deliverables. Treating these stages as an explicit pipeline makes the order and dependencies visible. It also makes a practical distinction possible: a changed analysis contrast may require a new analysis and table, while a corrected source value may require new imputations as well.
Orchestration is the logic that decides which stages to run, in what order, and whether a prior output can be reused. Its aim is not simply to avoid computation. Reusing an output is safe only when the stage's relevant inputs and settings have not changed. A pipeline that runs quickly but silently combines stale imputations with new inputs is not reproducible.
Represent the Workflow as Dependencies
A useful basic graph is: source data → schema validation → preprocessing → MICE imputation → completed-data checks → analysis → tables and figures. Each arrow records an actual dependency, not just a preferred order. For example, if preprocessing creates a derived variable used in the imputation models, imputation depends on that preprocessing output. If an analysis uses only the saved imputations and a model specification, it need not depend on a later table-formatting choice.
The graph should be a directed acyclic graph (DAG): a stage cannot depend on itself through a chain of stages. A valid execution order is a topological order, meaning every stage runs after all of its upstream dependencies are available. Independent stages can be run separately or in parallel, provided their inputs and outputs do not conflict.
For orchestration, each stage needs a complete description of what determines its result. This generally includes its input data or upstream output, relevant configuration, code, and software environment. For stochastic imputation, the random-number configuration and the other details that determine the generated imputations matter too. The earlier material on configuration files, deterministic preprocessing, random streams, and provenance supplies ways to record these elements; here the focus is how they control stage execution.
A fingerprint is a compact identifier derived from content or settings. The orchestration system need not compare files by their names alone: two files can have the same name but different contents, and different filenames can contain identical content. The important requirement is that the fingerprint cover the information that can affect the stage's output. A missing dependency is more dangerous than an unnecessary one: if an input is omitted from the fingerprint, changing it may leave the cache incorrectly marked valid.
Deriving the Minimal Rerun Set
Suppose a stage \(v\) has an output determined by its declared inputs \(I_v\) and settings \(C_v\). Write this as \(O_v = f_v(I_v,C_v)\). When one input changes, first identify the stage that directly consumes it. Then propagate staleness along outgoing dependency arrows. A stage outside this downstream set need not be rerun, assuming its own inputs and settings remain unchanged.
This gives a simple justification for selective reruns. If a stage is deterministic and every declared input and setting is unchanged, then \(f_v\) receives the same arguments as before, so its saved output remains valid. Applying this reasoning in topological order, unchanged upstream outputs preserve downstream inputs. Conversely, if a declared input changes, the prior output is not established as the result for the new inputs, so that stage must be recomputed. For stochastic stages, the random-stream configuration and execution environment must be part of the conditions for reuse; matching a seed alone does not establish that all other result-determining inputs are unchanged.
Name the data, setting, code, environment, or external resource that differs from the recorded run.
Use the dependency graph to identify stages whose declared inputs include the changed item.
Mark every dependent stage stale, even if its own settings appear unchanged, because an upstream output it consumes has changed.
Before using a cached output, verify its key and integrity checks; then recompute only the stale stages and their affected descendants.
Worked Examples
Worked Example: Changing an Analysis Contrast
Consider hypothetical data from 180 participants. A completed MICE object has already been generated from validated and preprocessed data. The pipeline has six stages: validation (V), preprocessing (P), imputation (I), completed-data checks (C), analysis (A), and output generation (O), with dependencies \(V \to P \to I \to C \to A \to O\). The analyst changes the requested contrast from a 1-unit to a 2-unit difference in a continuous exposure. The imputation specification and analysis model otherwise remain appropriate.
The contrast is an input to analysis, not to validation, preprocessing, or imputation. The direct consumer is A; the downstream stage O also depends on A. Thus the stale set is \(\{A,O\}\), containing 2 of the 6 stages. Recompute the analysis for the new contrast, then regenerate its outputs. Reusing V, P, I, and C is justified only if their recorded inputs and settings still match.
The reason for not rerunning imputation is a dependency claim, not a general rule that analyses never affect imputations. If the analysis change also changes which variables, transformations, or specifications must be included in the imputation models, then that information is an imputation dependency and the graph must reflect it. The stage boundary must represent the actual workflow.
Worked Example: Updating a Preprocessing Rule
In a hypothetical study of 120 patients, preprocessing converts a recorded weight in pounds to kilograms before imputation. The transformation is \(w_{\mathrm{kg}}=w_{\mathrm{lb}}/2.20462\). A changed protocol instead uses the exact factor \(0.45359237\), so the new transformation is \(w_{\mathrm{kg}}=0.45359237w_{\mathrm{lb}}\). For a recorded weight of 176 lb, the old result is \(176/2.20462 \approx 79.83\) kg, while the new result is \(176(0.45359237) \approx 79.83\) kg (79.832 kg before rounding). The close numerical agreement does not make the configuration identical.
Because the preprocessing rule is an input to P, its settings fingerprint changes. If I consumes P's output, then I is stale, followed by C, A, and O. V can be reused if the raw data and validation rules are unchanged. P must be rerun even though the difference at this particular value is tiny: a rule change can have larger effects elsewhere in the data, and the recorded output must correspond to the specified transformation.
The affected set is \(\{P,I,C,A,O\}\), or 5 of 6 stages. This example also checks a key modelling assumption: the conversion rule is deterministic and applied consistently. If the conversion uses an external lookup table, that table is another declared input; a changed lookup table must likewise invalidate P and its descendants.
Worked Example: Correcting a Source Value
Suppose a hypothetical dataset contains 240 records and one patient's age was entered as 420 years instead of 42.0 years. The source data are corrected, and the pipeline uses the same six stages V, P, I, C, A, and O as above. The raw-data fingerprint changes, so V must run on the corrected input. Since P consumes validated data, P is then stale; its descendants I, C, A, and O are stale as well. All 6 stages require execution in dependency order.
A common but unsafe shortcut would be to replace the value in a displayed table while retaining the old imputation object. That object was generated from preprocessing based on the uncorrected record, so the imputed datasets and pooled analysis do not represent the corrected source data. The dependency graph makes the reason explicit: the changed source value reaches every downstream stage.
If the validation stage rejects age 420, the corrected run should record that the earlier input failed the applicable check and that the corrected input passed. Passing validation is not itself proof that the imputation model is statistically adequate; it is evidence only that the declared input checks were satisfied.
Assumptions, Diagnostics, and Failure Modes
Selective rerunning is valid under several operational assumptions. The graph must include every result-determining dependency, stage operations must have clearly defined inputs and outputs, and fingerprints must represent those inputs accurately. Reused outputs must still be readable and pass relevant integrity checks. For stochastic stages, the saved result must be tied to the random-stream and environment information needed to identify the run. These conditions concern workflow validity; they do not replace the statistical assumptions required for a suitable imputation model or a valid analysis.
Useful diagnostics include checking whether each output records its parent fingerprints, comparing current fingerprints with the recorded values, and confirming that every requested output is downstream of the current run's inputs. Inspect the execution plan before launching an expensive imputation: it should show which stages are cached, which are stale, and why. After execution, confirm that no stage consumed a stale parent and that the saved imputation and analysis refer to the same run configuration.
- Undeclared dependencies: A script reads a local lookup file or environment variable not listed among its inputs. Changing that resource leaves the cache falsely valid. Declare and fingerprint it, or make the stage independent of it.
- Overly broad dependencies: Every output is treated as dependent on every setting. This is safe but can force unnecessary full reruns. Refine the graph so each stage depends on what it actually uses.
- Reusing by filename: A familiar file path is mistaken for proof that contents are unchanged. Compare content or configuration fingerprints and check output integrity.
- Manual edits after a stage: Editing a completed dataset outside the pipeline breaks the recorded dependency chain. Treat the edited artifact as a new input only through an explicit, documented stage, or regenerate it through the pipeline.
Incremental Reruns and Full Reruns
A full rerun is conceptually simple: execute every stage from the beginning. It is a useful fallback when dependency information is incomplete, the saved state cannot be trusted, or several changes make the affected set uncertain. Its cost is that it repeats unaffected work and may generate a new stochastic result when imputation is rerun.
An incremental rerun recomputes only stale stages and their downstream dependents. It can save time and preserve valid upstream artifacts, but only when dependencies are accurately declared and cache validation is reliable. When in doubt, use a full rerun rather than silently reuse an output whose provenance or inputs cannot be verified. This is not a statistical test or a guarantee of model quality; it is a control on whether computational results correspond to the intended inputs and specifications.
Check Your Understanding
For each question, decide what should be rerun and compare your reasoning with the solutions.
- A plotting label changes, but the analysis estimate and its inputs do not. Which stages are necessarily affected?
- A new predictor is added to the imputation specification and is also used in the analysis. What is the first stage that must be rerun, and why?
- Why is a matching random seed insufficient, by itself, to establish that a cached imputation result is valid?
- A preprocessing script reads a conversion table that is not recorded in the pipeline. What is the failure, and how should it be corrected?
Solutions. (1) Output generation is affected; analysis and upstream stages can be reused if the label change does not alter their inputs. If the label is produced by a separate formatting stage, rerun that stage and its outputs. (2) Imputation is the first affected stage because its specification changes; rerun its descendants, including completed-data checks, analysis, and outputs. (3) The imputed result can also depend on the input data, preprocessing, imputation methods and settings, software environment, and random-stream setup. Those dependencies must match too. (4) The graph omits a result-determining input, so changing the table can leave preprocessing incorrectly cached. Declare the table as an input, include its content in the preprocessing fingerprint, and invalidate preprocessing and its descendants when it changes.