Report Sensitivity Results in an Auditable Format

Why Sensitivity Results Need an Audit Trail

A sensitivity analysis is useful only if a reader can tell what was changed, what was held fixed, and which result came from which specification. A table of estimates without this information can make materially different assumptions look interchangeable. A reproducible report therefore needs to connect each result to its model, identifying assumptions, sensitivity-parameter values, computational status, and source run.

The goal is not to repeat every technical detail of model fitting in the main text. It is to make the report a reliable map to those details. A reader should be able to start with a displayed estimate, locate the exact scenario that produced it, inspect the settings and diagnostics for that run, and distinguish a valid result from a run that failed or was excluded.

What Each Reported Result Must Identify

Start by defining the quantity being reported. State the estimand in words, the analysis population, the time point, the outcome scale, and the contrast direction. For example, a treatment-minus-control difference in mean symptom score at week 12, measured in score points, has an interpretable direction only if the report also says whether a lower score is better. Keep this definition consistent across the baseline and sensitivity scenarios unless a change is deliberate and clearly explained.

For each result, report the model and assumptions relevant to interpreting that result. At minimum, identify the baseline missing-at-random (MAR) specification and the MNAR sensitivity specification, the parameters that distinguish the scenario, and the fixed choices shared across scenarios. Do not assume that a label such as “conservative” or “delta = −1” fully describes a model: give the parameter’s meaning, units, affected outcomes or groups, and direction of adjustment.

A report also needs enough numerical information to assess estimation uncertainty. Give the estimate, its standard error (SE), and the confidence interval, all on the estimand’s scale and with units. State the interval method and confidence level. Include p-values only when they answer a stated inferential question; they do not replace effect estimates or intervals. The confidence interval describes uncertainty under the specified analysis and does not account automatically for uncertainty about whether that scenario is the right one.

A Compact, Traceable Report Structure

A useful report has two connected layers. The main results table is designed for comparison: it places the baseline beside prespecified MNAR scenarios, uses the same estimand and units, and shows uncertainty and status. The supporting run record contains the exact specification and execution details. Use an unambiguous scenario ID to link the two. The supporting record might be a saved configuration, a run log, or both; the report should specify where it can be found and how its version is identified.

For each run, retain the data version or data-file identifier, analysis code version, software and relevant package versions, scenario ID, model specification, parameter values, random seed where applicable, start and end status, diagnostics, and any warning or error messages. If results are pooled across imputations, retain the number of imputations and the pooling output needed to verify the reported estimate and uncertainty. These details complement, rather than replace, the statement of the scientific assumptions.

Put Baseline and MNAR Results Side by Side

Side-by-side presentation makes it easier to see what changes under different assumptions. It is meaningful only when the estimand, scale, population, and relevant analysis choices are sufficiently aligned for comparison. Explain which assumptions change between rows and which remain fixed. If a sensitivity scenario also changes the estimand, model structure, or analysis population, identify that change rather than presenting the row as a direct comparison of missingness assumptions alone.

The table below illustrates a hypothetical report. For this example, \(\delta\) is a shift in score points applied to missing treatment outcomes relative to their MAR imputations; control imputations are not shifted. The baseline row uses MAR, and the other rows are prespecified MNAR scenarios. The estimates are outputs from refitting and pooling under each scenario; they are not calculated by simply adding \(\delta\) to the baseline contrast.

These numbers are hypothetical and serve to illustrate reporting. In this example, all displayed intervals were computed as estimate \(\pm 1.96SE\), with endpoints rounded to two decimal places. The table makes the assumed shifts visible, but interpretation still depends on the exact run specification, including the imputation and pooling procedures. A delta is not a directly observed quantity: its plausibility must be considered substantively, and the report should give readers enough information to understand its role.

Derivation: Why the Confidence Interval Must Match the Reported Run

For an estimate \(\hat{\theta}\) with estimated standard error \(SE\), a large-sample Wald interval uses the approximate sampling distribution \((\hat{\theta}-\theta)/SE \approx N(0,1)\). The central 95% of a standard normal distribution runs from approximately \(-1.96\) to \(1.96\). Solving the corresponding inequality for \(\theta\) gives:

This justification depends on the stated approximation and on using the \(SE\) from the same analysis run as \(\hat{\theta}\). If the estimate comes from one scenario while the \(SE\) comes from another, the displayed interval does not correspond to either fitted analysis. If the method used for inference is not a normal-based Wald interval, report and use the interval method that actually matches the analysis.

Failed, Excluded, and Recovered Runs

A sensitivity report should account for all prespecified scenarios, not just those that produced convenient estimates. A failed run has no valid estimate to report. Record its scenario and run IDs, the stage at which it failed, the warning or error, and whether a retry occurred. If it was excluded from the main results, give the reason and make clear that the omission was a disposition of the run, not a numerical finding. A recovered run should retain a link to both the original failure and the successful retry.

Distinguish computational failure from a decision to exclude a run. Failure can result from a fitting or convergence problem; exclusion can result from a prespecified rule or a documented data or specification issue. Do not replace a failed fit with an unexplained estimate from a different scenario. If a troubleshooting change alters the model or settings, treat it as a distinct run and describe the change. The earlier material on failed fits and unstable imputations covers diagnosis; the reporting task is to make those diagnoses and dispositions visible and traceable.

For example, a run ledger might include the scenario ID, run ID, status, attempt number, error summary, action taken, and whether it appears in the results table. A concise main-text note can summarize these records, while the ledger preserves the detail. This is more informative than reporting only the number of successful runs, because it shows which planned scenarios did not contribute and why.

Diagnostics, Alternatives, and Limits

Before release, check that each table row agrees with its linked run record and that the record agrees with the stated specification. Confirm that estimates, standard errors, intervals, units, scenario IDs, and statuses match. Check that all planned scenarios are accounted for, that failed runs are not represented as completed results, and that the reported parameter values use the stated scale and direction. A row whose estimate changes but whose run ID or configuration does not is a warning that the report may not reflect the saved analysis.

There are several ways to present results. A narrative-only summary is compact but can leave scenario definitions and run status ambiguous. A results table with no linked specification is easier to scan but may not be reproducible. A linked results table plus a run ledger or saved configuration takes more preparation, but supports both comparison and audit. For many scenarios, show a concise, prespecified set in the main table and provide the full scenario grid and run accounting in supporting material. State the selection rule; do not imply that the displayed scenarios are exhaustive if they are not.

No format can make an unsupported assumption credible or guarantee that another analyst will reproduce a result if essential inputs are missing. Auditability also does not remove uncertainty about the appropriate MNAR assumptions. The report’s role is to expose those assumptions and their consequences, not to choose a scenario by hiding alternatives or treating the baseline as automatically correct.

• Common mistake: “The MNAR estimate is more negative, so MNAR is the correct analysis.” A shift between scenario estimates describes sensitivity to assumptions; it does not identify which assumption is true.
• Correction: State the change in estimate in context, identify the parameter values that generated it, and explain the substantive basis and limits of the scenarios.
• Common mistake: Showing only successful fits because failed runs have no estimates.
• Correction: Include a status and disposition for every planned run. A failure is not a result, but it is part of the analysis record.
• Common mistake: Treating each scenario’s confidence interval as if it includes uncertainty about the choice of scenario.
• Correction: Label intervals as conditional on the specified model and assumptions, and discuss sensitivity across scenarios separately.