Why Sensitivity Analyses Need Explicit Specifications
A sensitivity analysis is only interpretable if its assumptions can be identified precisely. A result labelled “scenario 2” is not enough if no one can tell which outcome model was used, what missingness assumption it encoded, how its sensitivity parameter was defined, or which estimand was calculated. These details are easy to lose when settings are scattered across scripts, notebooks, and analyst notes.
A machine-readable scenario configuration brings those choices together in a defined structure. It is a record of the analysis to be run, not a statistical result and not evidence that a missingness assumption is true. The earlier material on longitudinal missingness patterns provides a useful foundation for describing which repeated outcomes are observed; here the focus is how to specify the assumptions and analysis choices that accompany those patterns.
Separate the Pieces Before You Encode Them
A useful configuration distinguishes the target of inference from the models and assumptions used to estimate it. For example, a treatment-group mean difference at a specified visit is an estimand; a Gaussian outcome model is a modeling choice; and a conditional missing-at-random assumption is a missingness assumption. They answer different questions and should not be stored in one vague field such as “analysis type.”
A practical specification gives each concept its own named field. The table shows a compact schema; real projects can add fields, but should retain these distinctions.
| Field | What it records | Example of a useful value |
|---|---|---|
| scenario_id | Stable, unique name for the scenario | mar_primary |
| outcome_model | Outcome type, distribution, link or mean structure, and included predictors | Continuous Gaussian outcome with treatment, visit, and treatment-by-visit terms |
| missingness_assumption | The stated relationship between missingness and observed or unobserved outcomes | MAR conditional on recorded treatment and observed outcome history |
| sensitivity_parameter | Parameter value, meaning, scale, and units, or an explicit declaration that none is used | delta = -3 score points |
| estimand | Population, outcome variable, contrast, and time point or summary | Active-treatment minus control mean score at week 12 |
| randomness | Seed policy and any replicate identifier | Seed 42007; one fixed stream for this scenario |
| provenance | Configuration version and relevant data and software identifiers | Schema version 1; analysis code commit abc123 |
A scenario should also make its analysis population and units unambiguous. “Mean change” is incomplete unless the baseline definition, follow-up time, and population are clear. Likewise, a sensitivity value of \(-3\) is uninterpretable without its parameter name and scale: it could mean three score points, three standard deviations, or a log-odds coefficient.
A compact JSON-style record might read: {"scenario_id":"mar_primary","outcome_model":{"type":"continuous","distribution":"Gaussian","predictors":["treatment","visit","treatment_by_visit"]},"missingness_assumption":{"type":"MAR","conditioning_variables":["treatment","observed_outcome_history"]},"sensitivity_parameter":{"type":"none"},"estimand":{"contrast":"active_minus_control","outcome":"score","time":"week_12","unit":"points"},"randomness":{"seed":42007},"provenance":{"schema_version":"1"}}. JSON keys and allowed values should be defined in a schema or data dictionary; merely writing a JSON-shaped object does not guarantee that another program will interpret it as intended.
Assumptions and Validation Rules
Validation has two distinct jobs. Structural validation checks that the record can be read and contains required fields of the correct type. Scientific consistency checks check whether the declared choices agree with each other. Neither kind of check can prove that an untestable missingness assumption is true.
Before running an analysis, confirm at least the following:
- The outcome type and distribution agree. A binary outcome should not be declared Gaussian merely because the software accepts a numeric column coded 0 and 1.
- The estimand refers to the configured outcome, population, contrast, and time point. If the estimand is a week-12 mean difference, the model and data specification must include the information needed to define that contrast.
- Every sensitivity parameter has a name, a role, a scale, units where applicable, and a value or an explicitly declared range. A null or absent value must not silently mean two different things.
- The missingness assumption names the variables on which it conditions and does not contradict the parameterization. A configuration claiming “no sensitivity parameter” should not also contain an active nonzero sensitivity value.
- Seed values are valid integers under the software’s rules, and the seed policy states whether scenarios and repeated runs share or receive distinct seeds.
- Scenario identifiers are unique, and provenance records identify the code and data versions needed to interpret a saved result.
A configuration can state, for example, that missingness is assumed MAR conditional on recorded treatment and observed outcome history. That declaration does not test MAR: the values that would have been observed after dropout are unavailable for direct verification. Sensitivity parameters describe alternative assumptions to examine; they do not make those assumptions empirically identified.
Worked Example: Specifying a Continuous-Outcome Scenario
Suppose a hypothetical two-arm trial measures a symptom score at baseline and week 12. Higher scores indicate worse symptoms. The planned estimand is the active-treatment minus control difference in mean week-12 score, in score points, among randomized participants. The outcome is treated as continuous and modeled with a Gaussian distribution. The scenario assumes MAR conditional on treatment and observed score history and declares no additional sensitivity parameter.
Estimand check. If the target group means used in a hypothetical calculation were 18 points for active treatment and 21 points for control, the stated contrast would be \(18-21=-3\) points. The negative sign means the active-treatment mean is 3 points lower on the score scale. This arithmetic illustrates the contrast’s direction and units; it does not establish a treatment effect or justify the missingness assumption.
Configuration. A concise record could be: {"scenario_id":"mar_week12","outcome_model":{"type":"continuous","distribution":"Gaussian","predictors":["treatment","visit","treatment_by_visit"]},"missingness_assumption":{"type":"MAR","conditioning_variables":["treatment","observed_score_history"]},"sensitivity_parameter":{"type":"none"},"estimand":{"population":"randomized_participants","contrast":"active_minus_control","outcome":"symptom_score","time":"week_12","unit":"score_points"},"randomness":{"seed":73104}}.
Checks. The outcome and distribution agree; week 12 appears in the model and the estimand; the contrast direction is explicit; and the missingness assumption names its conditioning information. Since the parameter is explicitly “none,” there is no unlabelled numerical sensitivity value to misinterpret. A reviewer can still question whether the assumption is credible, but that is a scientific assessment, not a schema error.
Seed Policies and Reproducibility
A random seed initializes a pseudo-random number generator. For a given generator implementation, software environment, seed, and sequence of random-number calls, it generally permits the same stochastic sequence to be regenerated. The seed alone is not a complete guarantee of identical results: software versions, parallel computation, algorithms, data, and the order of calls may also matter. Record the seed policy and relevant computing context rather than treating a seed as a universal reproducibility certificate.
A project must decide whether scenarios use the same random stream or distinct streams. Using a shared stream can sometimes make comparisons less noisy, but only if the implementation uses the streams in a controlled, documented way. Assigning separate seeds is simpler to audit, but does not itself make two estimates statistically independent. The configuration should document what the program actually does, not what the analyst intended it to do.
Worked Example: Assigning Seeds to Scenario Replicates
A hypothetical workflow needs three scenarios and five stochastic replicates per scenario. The team chooses a documented integer rule: seed = 41000 + 1000 × scenario index + replicate index, where scenario indices are 1, 2, and 3 and replicate indices are 1 through 5.
For scenario 2, replicate 4, the calculation is \(41000 + 1000(2) + 4 = 43004\). For scenario 3, replicate 1, it is \(41000 + 1000(3) + 1 = 44001\). The listed scenario-replicate pairs receive distinct seeds under this rule because each scenario increment is 1000 and each replicate increment is at most 5.
Configuration check. Each run should store its scenario index, replicate index, and resulting seed together. If a run is restarted, it should reuse the stored seed for that exact run rather than generate a new one. To reproduce it fully, the project also records the data version, generator or software environment, and relevant code version. This policy describes reproducible bookkeeping; it does not prescribe how any missing outcomes are modeled.
A Justification for Complete, Typed Specifications
The reason to encode these fields separately can be stated directly. Let \(C\) be a configuration, \(D\) the analysis data, \(V\) the software and code versions, and \(U\) the random-number state. A computational result is produced by a procedure \(A\):
If two records differ in an undeclared choice, such as the estimand’s time point or the units of a sensitivity parameter, they need not represent the same procedure even if both are labelled with the same scenario name. Conversely, if all required choices are explicit and compatible, a reviewer can identify which procedure was intended and compare runs on those declared dimensions. This is a justification for traceability, not a theorem that matching configurations guarantee matching results: software behavior, data processing, and stochastic execution still matter.
Typed fields make this check operational. For instance, a validation rule can require that a binary outcome has a binary-compatible model, that an estimand’s time point is present in the model specification, and that a numerical sensitivity value has declared units or a dimensionless scale. These checks catch common contradictions before expensive analyses run. They cannot decide whether the chosen model is adequate or whether an assumption is plausible.
Worked Example: Catching a Scale and Type Mismatch
Consider a hypothetical study with a binary outcome indicating whether a participant improved by week 8. An analyst enters an outcome distribution of Gaussian, an estimand of the week-8 risk difference, and a sensitivity parameter labelled \(\delta=-0.5\) with units “score points.”
Outcome-model check. Improvement is binary, while “Gaussian” describes a continuous outcome model. A configuration validator should flag the mismatch rather than allow the numeric codes 0 and 1 to disguise it.
Estimand check. A risk difference is a difference in probabilities and is dimensionless, whereas the configured sensitivity parameter is expressed in score points. Unless the parameter has a separately specified meaning compatible with this binary outcome, its scale is inappropriate. The value \(-0.5\) cannot be converted into a risk difference merely by storing both in the same configuration.
Resolution. The analyst must correct the outcome-model declaration and define a sensitivity parameter with a meaningful scale for the specified outcome and assumption, or explicitly set the sensitivity parameter to none. The configuration does not choose the scientifically correct alternative; it makes the inconsistency visible before the analysis is run.
Diagnostics, Alternatives, and Limits
A configuration system is going wrong when it produces ambiguous or contradictory records: duplicate scenario identifiers, missing units, parameter values with unclear meanings, estimands that refer to absent visits, or seeds that are regenerated during restart. A second warning sign is that the saved result cannot be linked back to an exact configuration and data version. These are workflow diagnostics. They do not replace statistical diagnostics of a fitted model, which belong to the analysis itself.
Structured configurations have advantages over common alternatives:
| Approach | Strength | Limitation |
|---|---|---|
| Scattered script constants | Fast for a one-off exploratory run | Choices are difficult to review, compare, or recover later |
| Free-text notes | Can explain scientific rationale and nuance | Values are difficult to validate or use automatically |
| Machine-readable scenario records | Supports systematic validation, reuse, and audit trails | Requires a defined schema and cannot judge scientific plausibility by itself |
A good workflow uses both structured fields and explanatory documentation: the configuration makes choices explicit, while a short rationale can explain why those choices were selected. Do not encode a large collection of poorly named options merely because the file format allows it. Each field should have a defined meaning and allowed values, and changes to the schema should be versioned.
The principal limitation is that machine-readability is not statistical validity. A perfectly formatted configuration can encode a scientifically implausible outcome model, a poorly motivated missingness assumption, or an estimand that does not address the research question. Nor does a seed compensate for undocumented preprocessing or differing software behavior. The configuration is a disciplined specification and audit aid, not a substitute for judgment.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Why should an estimand be stored separately from an outcome model?
- A continuous outcome is measured in millimeters, and a sensitivity parameter is entered as \(-2\) with no scale. What essential information is missing?
- A configuration states that the outcome is binary, the outcome distribution is Gaussian, and the estimand is a risk difference. What should validation flag?
- A stored seed is reused, but the software version and data version are not recorded. Is the run fully reproducible? Explain.
- Can a valid configuration establish that its missingness assumption is true? Why or why not?
Solutions. (1) The estimand defines the target quantity, while the outcome model is a modeling choice used to analyze data; keeping them separate makes it possible to identify the target even when models change. (2) The parameter’s meaning and scale are missing, so \(-2\) cannot be interpreted; the record must say whether it is millimeters, a standardized value, or something else. (3) It should flag the binary-outcome/Gaussian-model mismatch and check that the model and data support the week or population named by the risk-difference estimand. (4) No. Reusing the seed is useful, but data, software, code, and sometimes execution details can change the result. (5) No. Validation can check that the assumption is clearly and consistently stated, but the truth of a missingness assumption involving unobserved outcomes cannot be established by formatting or schema checks.