Why Separate Code From Large Artifacts?
A single-cell analysis may involve a sparse count matrix, cell and feature annotations, intermediate embeddings, and model outputs. These files can be much larger than the code that creates or analyzes them. Committing every matrix and intermediate file to an ordinary code repository makes routine work harder: repository clones grow, history retains old copies after files are changed or deleted, and collaborators can have difficulty distinguishing the intended input from an accidental local copy.
The useful separation is not “code here, everything else elsewhere.” Code, small configuration files, and concise documentation usually belong in the repository. Large count matrices and derived artifacts generally belong in storage designed for data files. The repository should still tell a collaborator which artifact version to obtain, where to find it, and what it represents. This tutorial focuses on that connection between a compact code repository and separately stored artifacts.
The distinction matters because a file path alone is not necessarily an identity. A path such as data/current_counts.h5 may refer to different contents on different dates. Conversely, a version identifier without a usable location may identify an artifact that no collaborator can retrieve. A reproducible record needs both a sufficiently precise version and an accessible reference.
What Belongs in the Repository?
A code repository is best treated as the home for the instructions and records needed to understand the analysis, not as the default storage location for all its data. A typical repository can include source code, small parameter files, a README, and a manifest describing large artifacts. It should not silently include multi-gigabyte count matrices merely because a script expects them in a particular directory.
| Repository content | Typical treatment | Reason |
|---|---|---|
| Analysis scripts and reports | Version in the code repository | These are the instructions and analysis record to review and change. |
| Small configuration and manifest files | Version in the code repository | They describe analysis choices and identify required artifacts without embedding them. |
| Large raw count matrices | Store separately; record a stable reference | Large files inflate repository clones and their history. |
| Large intermediate outputs | Store separately when they need to be retained or shared | They can be regenerated in some settings, but should not be assumed disposable without considering cost and provenance. |
| Credentials or access tokens | Do not commit; use an approved secure mechanism | Repository history is not a safe place for secrets. |
This is a practical division, not a claim that every large file is unimportant or that every small file belongs in Git. A compact file may contain sensitive information, while a large artifact may be essential evidence for a published analysis. Storage choice should reflect size, access controls, retention requirements, and whether the artifact must remain available. The earlier material on Git Versioning for Analysis Code and Reports covers the role of the code repository; the earlier material on Input Provenance and Checksums for Count-Matrix Workflows covers input identity checks. Here, the emphasis is on organizing and referencing the artifacts themselves.
Stable References and Documented Versions
A stable reference should not depend on a researcher's machine-specific directory, such as /home/alex/Desktop/final2/. It should instead use an agreed storage service or shared location and a clear identifier that collaborators can resolve. Depending on institutional infrastructure, this might be a managed project directory, a data repository accession, or an object-storage URI. Access permissions and retention policies still matter: a syntactically stable reference is not useful if the intended users cannot retrieve it or the storage is later deleted.
Avoid using a mutable label such as latest as the only version record. A label can be useful as a convenience for finding the newest approved release, but its meaning may change. Record a release label together with a non-changing identifier for the exact artifact, such as a versioned object key or a content checksum. A checksum can help distinguish content, but it does not itself preserve a file or grant access. The artifact must also remain stored at an accessible location.
A lightweight manifest can be a plain text, YAML, or tabular file tracked with the code. It should record enough to identify and use an artifact, while avoiding a copy of the artifact itself. Useful fields often include a stable artifact name, version, storage reference, file format, description, creation date, and any relevant sample or feature scope. A checksum may be included to check that a retrieved file matches the recorded content. Do not place private access credentials in the manifest.
Worked Example: Referencing a Count Matrix Without Committing It
Suppose a hypothetical lab has a processed count matrix stored in its managed project storage. The matrix is 18 GB, and the code repository is intended to remain small enough for routine cloning. The team wants all analysts to use the same released matrix.
A weak arrangement is to put a local path in the analysis script, such as /Users/mira/Downloads/counts_final.h5. Another analyst probably does not have that path, and the filename does not indicate which release it represents.
Instead, the team stores the matrix under a version-specific location, for example project-storage://sc-study/counts/release-03/counts.h5, and records that reference in a small manifest tracked with the analysis code. The manifest also gives the format, a short description, and the recorded checksum. The repository contains the manifest and the script, not the 18 GB file.
The storage saving in the repository is 18 GB for this one artifact. More importantly, the script can read the reference from configuration rather than assuming that every collaborator has the same local directory. If the matrix is corrected later, the team creates release-04, records its reference and checksum, and updates the analysis configuration when that new version is intended. Release 03 remains available for reproducing work that used it.
This arrangement assumes the project-storage service supports access by the intended analysts and retains both versions for the required period. If either condition fails, the reference is not sufficient for reproducibility; the storage and retention plan must be fixed as well.
Choosing Storage and Artifact Granularity
No single storage system is ideal for every artifact. A researcher working locally may use a managed shared drive for moderate files. A distributed team may use institutional object storage or a data repository with persistent identifiers. A small, non-sensitive test fixture may reasonably be included in the code repository. The choice should be explicit rather than determined by whichever location happens to be easiest to write in a script.
It is also useful to decide what counts as one artifact. A count matrix might be a single HDF5 file, or a coordinated set of sparse matrix, feature, and barcode files. In the latter case, reference and version the collection as a unit. Recording a version for only one component while allowing the other files to change can produce an internally inconsistent dataset. Describe the components and their relationship in the manifest.
| Approach | Strength | Limitation to consider |
|---|---|---|
| Ordinary code repository | Convenient review and version history for small text files | Large artifacts can make clones and history unwieldy. |
| Shared project storage | Often straightforward for a stable research group | Paths, access rules, and retention may depend on local administration. |
| Object storage or data repository | Can support versioned objects, shared access, or persistent identifiers | Requires appropriate permissions, documentation, and an understanding of service-specific retention. |
| Git Large File Storage or similar extensions | Can connect selected large files to a Git workflow | Has storage, bandwidth, and server-support constraints; it does not remove the need for a documented artifact policy. |
A repository extension for large files can be appropriate for a modest number of files when the hosting service supports it and collaborators understand the setup. It is not automatically a solution for many very large or frequently changing matrices. Likewise, storing everything in shared storage is not enough if files are overwritten without recording versions. The relevant comparison is not simply “which location holds the bytes?” but “which approach lets the team retrieve and identify the intended content over time?”
Worked Decisions for Common Cases
Worked Example: Estimating Repository Growth
A hypothetical analysis currently has a 250 MB repository. The team plans to add three count-matrix artifacts of 6 GB each and two intermediate files of 4 GB each. Ignoring compression and repository overhead, what is the resulting stored content if all five files are committed?
The added content is \(3 \times 6 + 2 \times 4 = 18 + 8 = 26\) GB. The total content would be approximately 26.25 GB if 250 MB is treated as 0.25 GB. If these files are later replaced, ordinary version history can retain earlier contents as well, so the repository's storage burden may exceed this simple total.
A better arrangement for this hypothetical team is to keep scripts, small configurations, and a manifest in the repository, while storing the large matrices and retained intermediate files in managed artifact storage. This example does not imply that every intermediate must be retained; that decision depends on its scientific value, reconstruction cost, and storage policy. It does show why committing large files by default can have a disproportionate effect on routine repository use.
Worked Example: Releasing a Corrected Metadata File
A hypothetical project has a released matrix bundle identified as counts-release-02. During review, the team discovers that a cell annotation file in the bundle has a corrected sample label. The matrix values themselves have not changed.
The team should first decide whether the corrected annotation is part of the released bundle's definition. If it is, they should create a new bundle version, for example counts-release-03, containing the unchanged matrix and corrected annotation, and record the new location and content identifiers in the manifest. They should state that the matrix component is unchanged and the annotation component is corrected.
Replacing the annotation in place while continuing to call the bundle release 02 would make that identifier ambiguous: two users could retrieve different annotations under the same version label. Publishing release 03 preserves the ability to identify which bundle supported each analysis. The correction should not be described as changing the matrix counts, because only the annotation file changed.
A Practical Management Sequence
Before distributing an analysis, use a short, repeatable sequence. It is useful whether the artifact is a raw matrix, a filtered matrix, or a saved model result. The sequence concerns durable identification and access, not the computation that created the artifact.
Decide whether it is code, lightweight documentation, or a data artifact. Note its size, sensitivity, and whether it is an input or a retained output.
Choose an approved location that supports the needed access and retention. Check that intended collaborators can retrieve it.
Use a version-specific identifier for the released content. Do not silently overwrite a released artifact while keeping its old identifier.
Write the stable reference and enough descriptive metadata to tell what the artifact contains. Add a checksum when appropriate for content verification.
From a clean working environment or a second user's account, confirm that the recorded reference resolves to the intended version and that required access is available.
The final retrieval check catches problems that a well-formatted manifest cannot. A stale URI, missing permission, renamed folder, or absent file can all leave the documentation looking plausible while making the analysis impossible to run. Check that filenames and descriptions in the manifest agree with the stored files. For a multi-file bundle, verify that all listed components are present and that the manifest identifies the bundle version unambiguously.
Diagnostics, Pitfalls, and Limits
Several warning signs indicate that artifact management is failing. A script contains a personal absolute path; two files have the same name but no recorded versions; a manifest points to a mutable latest location only; or collaborators report obtaining different inputs. Large files appearing in repository history are another signal to review, although deleting them from the current directory alone may not remove their historical storage cost.
- Confusing a location with a version. A stable folder name does not prove that its contents stayed fixed. Record a release identifier and avoid overwriting its contents.
- Confusing a checksum with preservation. A checksum can help detect content differences, but it cannot recover a missing file or guarantee access.
- Recording too little context. “Matrix file” is not enough. Record what the file represents, its format, and which released collection it belongs to.
- Recording too much in the repository. Avoid copying large files or confidential metadata into a manifest. Keep the repository record descriptive and appropriately non-sensitive.
- Assuming storage lasts forever. Services and institutional policies change. Important artifacts need an explicit retention plan and, when required, a documented migration to a new location.
There are limits to what a manifest can accomplish. It cannot make an inaccessible storage system available, decide whether an intermediate result is scientifically adequate, or guarantee that software can read a file in the future. Artifact records work alongside locked software environments and documented analysis code; they do not replace them. For very large or sensitive datasets, institutional policies and data-governance requirements may determine where storage is permitted.
Check Your Understanding
Answer each question, then compare with the solutions.
- A script reads /home/lee/data/final_matrix.h5. What makes this reference fragile, and what should replace it in the shared analysis record?
- A team updates a released matrix file but keeps the same version label. Explain the problem and give a safer alternative.
- A checksum is recorded, but the artifact has been deleted from storage. What does the checksum establish, and what does it not establish?
- A hypothetical repository is 400 MB and a planned matrix artifact is 9 GB. What should the team consider before committing it, and what is a common alternative?
- A matrix bundle includes a matrix file and a feature table. Why should the manifest identify the bundle components together?
Solutions. (1) The absolute path is specific to one machine and may not exist for collaborators. Replace it with an approved shared-storage reference or persistent repository identifier, plus a recorded artifact version. (2) The same label would refer to different contents over time. Keep the old version unchanged and publish the modified content as a new version, updating the manifest to identify it. (3) The checksum can be used to compare content if a file is obtained; it does not store the artifact, restore it, or make it accessible. (4) A 9 GB addition would greatly increase repository size and may persist in its history. The team should consider access, retention, and hosting support; managed project or object storage with a manifest is a common alternative. (5) The matrix and feature table jointly describe the data. Versioning only one component could let users combine files from different releases, so the manifest should identify the coherent bundle and its component files.