Organizing Robustness Analyses Without Selective Reporting

Why Robustness Reporting Needs a Plan

An observational comparative study can produce different estimates when analysts change modeling choices, matching rules, outcome definitions, or the population under study. Some alternatives are planned because they address known analytic uncertainties. Others arise during analysis, after data or results reveal a concern. Both can be informative, but selectively reporting only the analyses that support a preferred conclusion makes the evidence difficult to evaluate.

The purpose of a robustness analysis is not to find one specification that produces a desirable result. It is to show how the reported finding behaves under a transparent set of defensible alternatives, and to explain what each alternative changes. A comparison is interpretable only when readers can identify the analysis, the population it represents, the quantity estimated, and the uncertainty around it.

A robustness analysis does not automatically estimate the same quantity as the primary analysis. For example, changing a restriction can change the target population, while changing a model can change the adjustment specification. Explain those differences before describing estimates as reassuringly similar or meaningfully different.

Inventory Analyses Before Comparing Results

Start by naming the primary analysis and recording the alternatives actually examined. The inventory should include unsuccessful, inconclusive, and unfavorable analyses, not just those eventually highlighted. For each alternative, state the decision changed and whether it was planned or post hoc. Distinguish analyses that were carried out from options that were merely considered.

A useful inventory covers the following where relevant: propensity-score model and covariate specification; matching or weighting rule; restrictions or trimming; outcome definition and follow-up window; missing-data handling; and the population included in the analysis. These are reporting categories, not a recommendation to vary every choice. An alternative should have a substantive rationale, such as testing sensitivity to a defensible modeling choice, rather than being an arbitrary search for a preferred result.

Where an alternative specification was planned, say when it was specified, for example in a protocol or analysis plan. Where it was post hoc, state what prompted it. Avoid suggesting that a post hoc analysis was planned merely because it is scientifically reasonable. Transparency about timing is more informative than trying to label every analysis as confirmatory or exploratory without explanation.

Report Population Changes as Part of the Result

A population sensitivity analysis is not just another row of estimates. Restricting to people with a complete follow-up period, excluding a subgroup with limited overlap, or applying a different trimming rule can alter who contributes to the analysis. That may change both precision and the population to which the estimate pertains. State the number of people in each group, the reason for the restriction, and the resulting analysis population. Do not imply that an estimate from a restricted group answers exactly the same question as an estimate from the full eligible population.

Sample sizes should refer to the actual analysis for that row, not simply the initial cohort. If matching or weighting changes which observations contribute, report the relevant counts and any consequential diagnostics alongside the estimate. In particular, a similar estimate from a much smaller sample may be less precise, and a different estimate from a restricted population may reflect a different population rather than instability in a fixed estimand.

Compare Findings Without Overclaiming

A practical comparison separates three questions: Did the estimated magnitude change? Did the uncertainty change? Did the substantive interpretation change according to a threshold or criterion that matters for the study? These are related but not identical. An estimate can move while remaining imprecise; its interval can widen without a large change in the point estimate; or a modest change can matter if it crosses a prespecified threshold for practical importance.

If a study has a prespecified threshold for a meaningful difference, report whether estimates fall on different sides of it and explain the context. Do not invent such a threshold after seeing the estimates and then present it as planned. When there is no established threshold, describe the numerical changes and avoid declaring a result “material” solely because a p-value crossed a conventional cutoff. Confidence intervals describe uncertainty for each analysis; overlap or non-overlap of separate intervals is not, by itself, a test of the difference between estimates.

For two estimates on the same scale and with the same target quantity, their difference can be written as \(\Delta = \hat{\tau}_{2}-\hat{\tau}_{1}\). This arithmetic difference is descriptive. To make a formal claim about that difference, an appropriate standard error for the contrast is needed; it generally cannot be obtained by treating the two reported confidence intervals as independent, especially when analyses use overlapping participants.

The covariance term matters because estimates from the same cohort are often dependent. Unless the study specifically estimates and reports a contrast between specifications, describe the observed differences without assigning a significance test to them.

Diagnostics, Failure Modes, and Alternatives

An estimate table is not a substitute for checking whether each alternative worked as intended. For each specification, summarize relevant features such as covariate balance, overlap, effective sample size for weighting, or the number retained after matching or restriction. The exact diagnostics depend on the method. A result that appears stable but comes from an analysis with poor balance or very limited overlap should not be presented as strong reassurance. Conversely, a changed estimate may be informative even if the alternative is not preferred; report the diagnostic reason for giving it less weight.

Common reporting failures include omitting analyses that do not support the preferred narrative, calling all alternatives “prespecified,” comparing estimates that target different populations without noting the change, and treating a change in statistical significance as the only meaningful change. Another failure is listing many variations without explaining why they were examined. A concise rationale and a complete inventory are more useful than a long, unexplained menu of results.

A robustness analysis complements, but does not replace, other approaches. A primary analysis provides a clear main estimate; a robustness analysis assesses sensitivity to specified choices. Analyses of unmeasured confounding, negative controls, and subgroup differences address different questions and should retain their own purposes and interpretations. No collection of alternative specifications can establish that all assumptions hold or remove every source of bias. Robustness means stability across the alternatives examined, not proof of correctness.

A Consistent Reporting Sequence

For each alternative, provide enough information for a reader to trace the comparison from decision to result. A compact results table can use one row per analysis, with columns for timing, changed choice, analysis population, group counts, estimate and interval, and a relevant diagnostic. In the text, identify which changes notably affected magnitude, precision, or a prespecified substantive criterion. If no alternative materially changes the finding, describe that pattern without implying that unexamined alternatives would also agree.