Why Checkpoints and Cache Invalidation Matter
Single-cell workflows often contain expensive stages: reading large count matrices, filtering cells, fitting a model, or constructing an integrated representation. If a workflow stops after several hours, a well-placed checkpoint can let the next run resume without repeating completed work. A cache can also reuse a previous output when the inputs and computation are unchanged. But reuse is safe only when the cached artifact really corresponds to the computation now requested.
The central design problem is therefore two-sided: save outputs where doing so meaningfully reduces repeated work, and define precisely when each saved output is still valid. A missed invalidation can silently propagate stale results; invalidating too much may waste substantial compute time. The dependency-graph material establishes how pipeline stages depend on one another. Here, that graph is used to reason about saved intermediate artifacts and the conditions for their reuse.
A checkpoint is not automatically a valid cache entry. A file may exist but be incomplete, corrupted, or associated with different inputs. Conversely, a cache can be implemented using checkpoint files, a database, or an artifact store. The important distinction is that checkpointing concerns saving a restartable result, while cache validity concerns whether that result matches the current request.
Choosing Checkpoint Locations
A useful checkpoint usually follows a stage that is expensive to repeat, produces an artifact that can be read reliably, and has a well-defined set of dependencies. Examples include a validated, filtered count matrix; a normalized expression matrix; or a fitted model with its required metadata. Checkpointing every small operation can add storage and bookkeeping without meaningful savings. Saving only the final report, however, may force a rerun of many costly upstream stages after a minor downstream change.
Plan checkpoints with the dependency graph in view. If a filtering stage feeds normalization and dimensionality reduction, a change to a plot label should not require recomputing the filtered matrix. A change to the filtering threshold, by contrast, means that the filtered output and its descendants are no longer valid. Checkpoint boundaries should preserve enough information for downstream work, including essential cell and gene identifiers, data dimensions, and relevant metadata. A numeric matrix without its row and column identities may not be a usable checkpoint.
Worked Example: Placing Checkpoints in a Large Workflow
Suppose a hypothetical analysis starts with a count matrix, applies cell and gene filters, normalizes expression, fits a latent-variable model, and creates plots. On the available compute system, filtering takes 18 minutes, normalization takes 35 minutes, model fitting takes 150 minutes, and plotting takes 8 minutes. The model output is needed for several alternative plots and summaries.
A checkpoint only after plotting would not help when a plot is revised: the entire 211-minute workflow might run again. Saving the filtered matrix and model results creates more useful restart points. If only plot code changes, the 203 minutes of filtering, normalization, and fitting can be avoided. If the model settings change, the model must be refit, but the filtered matrix and normalized data can remain reusable if their own dependencies are unchanged.
A reasonable design is therefore to persist the filtered matrix with cell and gene identifiers, the normalized representation with its parameter record, and the fitted model with its configuration and software provenance. The choice is not based on runtime alone: each artifact must also be sufficient for downstream stages and safely associated with its producing computation.
Constructing a Cache Key
A cache key is a compact identifier for a computation request. It should change whenever a change could alter the stage's output. A general construction is:
Here, \(K_j\) is the key for stage \(j\), \(I_j\) represents the content fingerprints of its inputs, \(P_j\) its parameters, \(V_j\) the relevant software and code versions, \(R_j\) any declared randomness setting required by the computation, and \(U_j\) other documented dependencies. \(C\) denotes a canonical, unambiguous serialization, and \(H\) a cryptographic hash function. For very large inputs, a content checksum can serve as an input fingerprint rather than placing the input itself in the key.
Canonical serialization matters because equivalent configurations should be represented consistently. For instance, the same parameter mapping should not produce different keys merely because its fields were written in a different order. A key should also distinguish meaningful types and values: the integer 10 should not accidentally be treated as the text string "10." Include the actual configuration values used by the stage, not just a filename for a configuration file that might be edited in place.
For a stage that consumes upstream artifacts, their keys or content fingerprints belong in \(I_j\). This makes invalidation propagate through the graph: if an upstream key changes, the downstream stage receives a different input identity and therefore a different key. Relevant software information may include the stage's code revision, the versions of packages that affect its output, and a lockfile or environment image identifier. A package name alone is not enough if different package versions may behave differently.
Worked Example: Testing Which Changes Alter a Key
Consider a hypothetical PCA stage whose input is a normalized matrix. Its cache-key record includes the normalized matrix fingerprint, \(n_{\mathrm{pcs}}=30\), a feature-selection setting of 2,000 genes, the stage code revision, and the locked software-environment identifier. The cache key is the hash of the canonical record, not a hand-written label such as "PCA result."
If a user changes the number of principal components to 50, then \(P_j\) changes. The canonical record differs, so its hash gives a different key and the old 30-component output is not a valid hit. If the locked environment identifier changes, \(V_j\) changes and the output is recomputed even when the data and parameters are the same.
Now suppose only a downstream figure title changes. If the title is not an input to PCA, neither the PCA parameters nor its other dependencies change. The PCA key remains the same, so its cached result can be reused while the plotting stage runs again. This is the intended balance: a key changes for relevant differences, not for unrelated edits.
Invalidation, Costs, and a Justification
A correct key gives a practical invalidation rule: reuse an output only when its stored key equals the key for the current request and the artifact can be read successfully. A changed key means recomputation is required for that stage. Its descendants must also be considered: if a stage's output changes, any downstream result that depends on that output must be treated as a new computation. Unrelated branches of the graph can remain reusable.
Caching is useful when the expected computation saved exceeds the cost of checking and retrieving the artifact. A simple time-based argument makes this explicit. Let \(C\) be the time to recompute a stage, \(L\) the time to look up and retrieve a cached output, and \(r\) the probability that a later run has a valid cache hit. Ignoring storage and maintenance costs, the expected recomputation time avoided is \(rC\), while lookup costs \(L\). Thus the expected time benefit on a later run is:
Caching reduces expected runtime under this simplified comparison when \(rC > L\). The calculation does not include the first-run cost of writing the checkpoint, storage limits, or the risk of defective cache management; these must also inform the design.
Worked Example: Is a Checkpoint Worthwhile?
Suppose fitting a hypothetical model takes \(C=40\) minutes. Checking for and retrieving a matching cached result takes \(L=2\) minutes. Based on expected analysis changes, the team estimates a valid hit on 80% of reruns, so \(r=0.80\).
The expected time benefit on a later run is:
Under these assumptions, cache lookup and reuse save an expected 30 minutes per rerun, before accounting for storage and maintenance. If nearly every run changes a model input, the hit rate would be much lower and the checkpoint might save little. This calculation is a planning aid, not a guarantee of observed runtime.
A short justification explains why dependencies should be included in keys. Suppose the stage output is a function \(Y_j=f_j(I_j,P_j,V_j,R_j,U_j)\). If any argument that can affect \(Y_j\) changes, the new request may produce a different output. A key that omits that argument can remain unchanged, falsely indicating that the old output is valid. Including all output-relevant arguments in the canonical record makes a changed request produce a changed record and, barring a hash collision, a different key. The argument is only as complete as the dependency list: hashing an incomplete record cannot detect an omitted dependency.
Diagnostics and Failure Modes
A cache hit is not evidence that an output is scientifically correct; it indicates only that the cache system considers the request to match. Inspect cache behavior during development by recording, for each stage, the computed key, the inputs and parameters represented by it, whether the decision was a hit or miss, and the artifact location. When a supposedly small edit causes many expensive stages to rerun, look for unnecessarily broad dependencies. When a change that should affect results leaves the key unchanged, look for a missing dependency.
- Paths used as input identities. A file can change while retaining its path. Use a content checksum or another reliable content fingerprint, not the path alone.
- Incomplete parameter records. A threshold read from an environment variable or an untracked configuration file is still a dependency. Make it explicit in the key record.
- Software version omitted. If an output may depend on a package or code change, include an appropriate version or environment identifier.
- Partial checkpoint mistaken for completion. A process may stop while writing an artifact. Write to a temporary location and publish it as complete only after the write succeeds; on retrieval, check that the expected artifact is present and readable.
- Over-invalidation. Including unrelated settings in every key can force recomputation when those settings cannot affect a stage. Scope dependencies to the stage that actually uses them.
For stochastic stages, the declared randomness setting can affect the requested computation and should be represented when it is part of that request. This does not imply that a cache key guarantees identical outputs across different execution environments or implementations. It records declared dependencies; it cannot compensate for unrecorded ones or prove that an algorithm behaves identically everywhere.
Comparing Reuse Strategies
| Strategy | Strength | Main limitation |
|---|---|---|
| Recompute every stage | Simple rule; avoids accidental reuse of stale artifacts. | Can repeat expensive work even when inputs and computation are unchanged. |
| Checkpoint by stage with dependency-aware keys | Allows safe reuse of unchanged stages and targeted recomputation of affected descendants. | Requires complete dependency records, storage, and cache maintenance. |
| Reuse based only on file existence or path | Quick to implement. | Cannot reliably tell whether contents, parameters, or software have changed. |
For a small, inexpensive workflow, recomputation may be a sensible alternative to maintaining a cache. For an expensive workflow with stable inputs and repeated downstream analyses, stage-specific checkpoints and complete keys can save substantial time. A file-existence check is not a safe substitute for either approach.
Common Mistake
The opposite mistake is to erase every cached artifact after any change to the project. Instead, identify the changed dependency, invalidate the stage that uses it, and follow the dependency graph only to its affected descendants. This avoids both stale reuse and unnecessary recomputation.
Worked Example: Tracing Invalidation Through a Workflow
A hypothetical workflow has three stages: filtering produces matrix \(F\), normalization produces \(N\) from \(F\), and a visualization produces \(G\) from \(N\). Each stage's key includes the fingerprints of its direct inputs and its own settings. Initially, all three artifacts are cached.
First, the plotting color palette changes. The palette is a visualization parameter, so the key for \(G\) changes. The keys for filtering and normalization do not; \(F\) and \(N\) remain valid, and only the visualization needs to run again.
Next, the minimum-gene threshold used in filtering changes. The filtering parameters change, so \(F\)'s key changes and \(F\) must be rebuilt. Because \(N\) uses \(F\) as an input, its input fingerprint changes; \(N\)'s key changes as well. Since \(G\) uses \(N\), it too must be rebuilt. In this case the invalidation path is filtering, normalization, then visualization. A separate workflow branch that does not depend on \(F\) can remain cached.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A model-fitting stage takes two hours. Its input matrix, settings, or software environment changes on nearly every run. What should you consider before investing in a cache for this stage?
- A configuration file is edited, but the cache key includes only the configuration file's path. Why can this cause stale reuse, and what should the key represent instead?
- A figure's font changes, but the figure is the only stage using that font. Which stages should be invalidated if the dependency graph is correctly specified?
- Explain why a valid cache hit does not establish that the cached scientific result is correct.
Solutions. 1. Estimate the valid-hit rate and compare expected saved computation with lookup, retrieval, storage, and maintenance costs. With changes on nearly every run, the hit rate may be too low to justify caching. 2. The path may stay the same while the file contents change, leaving the key unchanged. Represent the actual configuration values or a content fingerprint of the configuration, along with other relevant dependencies. 3. Recompute the figure stage; upstream stages that do not use the font remain reusable. 4. A hit establishes a match to the dependencies represented in the key, not scientific validity, completeness of the dependency record, or correctness of the computation. The artifact still needs to be produced by a valid workflow and interpreted appropriately.