Why Put Analysis Settings in a Configuration File?
A single-cell workflow often applies the same sequence of operations to multiple studies, while each study has different input paths and may require different declared settings. If those values are embedded directly in the analysis code, adapting the workflow can mean editing the code each time. That creates opportunities for accidental changes to the analysis logic, inconsistent parameter updates, and uncertainty about which settings were used.
A configuration file separates values that describe a particular run from the code that defines what the pipeline does. A file might specify where a count matrix and cell metadata are stored, the minimum detected genes for retaining a cell, a mitochondrial-fraction threshold, and a normalization choice. The pipeline reads these values, checks them, and uses them as inputs. When the code is designed to do so, a new study can be processed by supplying a different configuration rather than modifying the pipeline.
This separation is useful only if the file is explicit and checked. A syntactically readable file can still contain an impossible threshold, a misspelled key, or a path to the wrong file. The earlier material on input provenance and checksums addresses identifying input files; a configuration complements that work by recording which paths and analysis settings a run is instructed to use.
Designing a Configuration Contract
A useful configuration is an explicit contract between the analyst and the pipeline. The file should state the values that may legitimately vary by run, while the pipeline should define what each key means and how it is applied. For example, “maximum mitochondrial fraction” is incomplete unless it is clear whether the value is a fraction from 0 to 1 or a percentage from 0 to 100.
A compact hypothetical YAML-style configuration might contain the following entries:
study_id: study_A
counts_path: data/study_A/counts.h5
metadata_path: data/study_A/cell_metadata.csv
min_genes_per_cell: 300
max_mito_fraction: 0.10
normalization: log1p_library_size
The particular file format is less important than the contract. The pipeline must agree with the file about key names, value types, units, permitted values, and how paths are interpreted. In this example, the mitochondrial threshold is a fraction, so 0.10 means 10%, not 0.10%. The identifier distinguishes the run in records produced by the workflow; it does not itself verify that the input files belong to that study.
Validation is best considered at two levels. Structural validation checks that required keys exist, unexpected keys are handled deliberately, and values have the expected types. Semantic validation checks whether values make sense for the analysis: a fraction threshold is between 0 and 1, a minimum gene count is a positive integer, an allowed normalization name is recognized, and required paths resolve to the expected files. A file can pass one level and fail the other.
From Configuration to Analysis
Let \(D\) denote the input data, \(C\) the analysis code, and \(\theta\) the settings supplied by a configuration. The workflow can be represented as \(A(D,C;\theta)\), where \(A\) is the analysis procedure. The configuration does not replace the procedure: it supplies values to it. For reuse to work, the code must actually read those values rather than silently substitute constants embedded elsewhere.
The basic justification for separating settings from code is that it makes the source of a change inspectable. If the same code \(C\) is applied to the same input \(D\), and only one validated component of \(\theta\) changes, then the difference between the resulting analyses can be attributed to that declared setting, subject to the workflow’s other sources of variation. If code, input, and configuration all change at once, that attribution is not possible from the configuration alone.
This is a traceability argument, not a guarantee that two analyses are statistically equivalent or that a particular setting is appropriate. Its assumptions are that the configuration is the one actually read by the run, the pipeline implements the documented meaning of each key, and inputs and code are identified separately. Validation helps enforce the first two assumptions, but does not establish them by itself.
Worked Example: Applying Cell-Filtering Thresholds
Suppose a hypothetical study has four cells with already calculated numbers of detected genes and mitochondrial fractions. A configuration sets min_genes_per_cell to 300 and max_mito_fraction to 0.10. The pipeline’s stated rule is to retain a cell when its detected-gene count is at least the minimum and its mitochondrial fraction is at most the maximum.
| Cell | Detected genes | Mitochondrial fraction |
|---|---|---|
| A | 320 | 0.06 |
| B | 280 | 0.04 |
| C | 450 | 0.12 |
| D | 300 | 0.10 |
For cell A, \(320 \geq 300\) and \(0.06 \leq 0.10\), so A is retained. For B, \(280 < 300\), so B is excluded even though \(0.04 \leq 0.10\). For C, \(450 \geq 300\), but \(0.12 > 0.10\), so C is excluded. For D, both conditions hold at equality: \(300 \geq 300\) and \(0.10 \leq 0.10\), so D is retained.
The retained set is therefore A and D: 2 of the 4 cells. The calculation checks that the pipeline’s boundary conventions match the declared rule. If the implementation instead used strict inequalities, D would be excluded; the rule must be specified rather than inferred from the threshold values alone.
Reuse Across Studies Without Editing the Pipeline
Reusing the same pipeline means that its analysis logic is held fixed while run-specific values are supplied through configurations. It does not require every study to have identical settings. Sometimes the purpose is to apply identical thresholds; in other cases a study-specific threshold is justified and explicitly recorded. Those are different analysis choices, and the resulting outputs should not be treated as directly comparable merely because the same code ran.
Worked Example: Two Study-Specific Configurations
Consider one pipeline and two hypothetical studies. The pipeline expects a count-matrix path, a metadata path, a minimum detected-gene count, and a maximum mitochondrial fraction. Study A’s configuration supplies paths ending in study_A/counts.h5 and study_A/cell_metadata.csv, with thresholds 300 genes and 0.10. Study B’s configuration supplies the corresponding paths under study_B, with thresholds 250 genes and 0.08. Both files use the same key names and types.
For a cell in Study A with 310 detected genes and a mitochondrial fraction of 0.09, the checks are \(310 \geq 300\) and \(0.09 \leq 0.10\), so it passes. For a cell in Study B with 260 detected genes and a fraction of 0.07, the checks are \(260 \geq 250\) and \(0.07 \leq 0.08\), so it also passes.
The pipeline logic was not edited between runs; the study-specific configuration supplied the different paths and thresholds. However, the filtering rules were not identical: Study A used a 300-gene minimum and a 10% maximum, whereas Study B used a 250-gene minimum and an 8% maximum. If the scientific aim requires identical filtering criteria, the two files must instead declare the same thresholds. If differing criteria are justified, their differences must remain visible when interpreting results.
Validation Before the Pipeline Runs
A robust workflow checks a configuration before performing expensive computation. A useful order is to parse the file, validate its structure and values, resolve and check inputs, and only then begin analysis. The order matters: a missing count matrix should produce a clear configuration or input error, not a failure much later during a model-fitting step.
Confirm that the file can be read in the chosen format and that its contents have the expected overall structure.
Require the documented fields, reject or explicitly handle unknown fields, and distinguish integers, fractions, strings, and paths.
Verify allowed choices and conditions such as a positive gene-count threshold and a fraction between 0 and 1.
Resolve paths according to a stated rule and confirm that required files exist and are of the expected kind.
Worked Example: Finding a Configuration Error Before Analysis
A hypothetical configuration contains min_genes_per_cell: 300, max_mito_fraction: 8, and a count path written as data/study_C/counts.h5. The documented contract requires a mitochondrial fraction in the interval from 0 to 1, and paths are resolved relative to the project directory. The project directory is /analysis.
The gene threshold passes the structural and semantic checks: 300 is an integer greater than zero. The mitochondrial value is numeric, so it passes a basic type check, but it fails semantic validation because \(8 > 1\). If the intended threshold was 8%, the configured fraction should be \(8/100 = 0.08\). This correction should be made explicitly; silently dividing values above 1 by 100 would conceal a unit error and could change the meaning of legitimate inputs.
The path resolves to /analysis/data/study_C/counts.h5. The workflow then checks whether that file exists and is the expected count input. If it does not exist, validation stops before analysis. Thus the example identifies two different problems: a value with the wrong unit scale and a path that may not point to an available input. A clear error should identify the field and the failed condition, rather than allowing either problem to produce an obscure downstream failure.
Diagnostics, Alternatives, and Limitations
Warning signs include a configuration key that the pipeline ignores, a default value used without being reported, a fraction stored as a percentage, a relative path resolved from an unexpected working directory, or settings that differ across studies without an analysis reason. Useful diagnostics include a startup summary of the validated settings, explicit errors for unknown keys, and tests that confirm a change to a configuration field changes the corresponding pipeline behavior. Such checks establish that settings are connected to the code; they do not establish that the chosen settings are scientifically optimal.
A common mistake is to assume that a valid file automatically makes an analysis reproducible. Wrong reasoning: “The configuration parses, so the run used the intended data and methods.” Correction: parsing only establishes that the file can be read. The workflow must also check the schema, ranges, units, path resolution, and actual use of the values. Input identity and analysis code must be identifiable as well; a configuration alone does not record every condition that can affect results.
Hard-coded constants can be reasonable for a genuinely fixed rule, but they are awkward for study-specific values and make run-to-run changes less visible. Command-line options are convenient for a small number of one-off overrides, but long commands can be difficult to review and preserve as a coherent collection of settings. Configuration files make groups of related values easier to inspect and reuse. These approaches can also be combined, provided the precedence rule is explicit: for example, a command-line override should either replace a named configuration value or be rejected, not silently compete with it.
Configurations have limits. They cannot make different studies comparable when their inputs, filtering criteria, or analysis choices differ. They cannot demonstrate that a threshold is scientifically justified, and they cannot prevent a pipeline from misinterpreting a correctly written value. A configuration should therefore be treated as a validated, run-specific specification—not as proof of data quality, methodological validity, or reproducibility by itself.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What is the difference between structural validation and semantic validation? Give one example of each for a maximum mitochondrial fraction.
- A configuration sets a minimum detected-gene count of 250. Under a rule that retains cells with counts at least the threshold, does a cell with 250 detected genes pass? Explain.
- Two studies use the same pipeline, but one configuration sets a 300-gene minimum and the other a 200-gene minimum. Can their filtering be described as identical? Why or why not?
- A fraction threshold is intended to represent 6%, but the file contains 6. What value represents the intended threshold under a 0-to-1 fraction convention?
- Why does a configuration file that parses successfully not, by itself, establish that the analysis used the intended count matrix?
Solutions. 1. Structural validation checks form and type; for example, confirming that the threshold is numeric. Semantic validation checks meaning and allowed range; for example, requiring the fraction to lie between 0 and 1. 2. Yes. The stated rule is “at least,” so \(250 \geq 250\). 3. No. The minimum thresholds differ, so the filtering rules differ even though the code is the same. 4. The value is \(6/100 = 0.06\). 5. Parsing only shows the file is readable; the workflow must also resolve the path, check that the expected file exists, and ensure that the pipeline actually reads that configured path.