Why Report Sensitivity Analyses?
A regression result depends on choices: which predictors to include, how to represent them, and how to quantify uncertainty. A sensitivity analysis checks whether a result changes under one or more plausible, explicitly justified alternatives to the primary analysis. Reporting it helps readers understand which findings are stable across those choices and which depend on them.
The goal is not to try many specifications and highlight whichever produces the smallest p-value. Instead, identify a specific concern, choose an alternative that addresses it, and report the comparison whether it supports the primary result or not. A small change across chosen alternatives is evidence of stability to those particular choices; it does not prove that the model is correct or that other reasonable choices would give the same result.
Make the Comparison Interpretable
A useful comparison starts with a clearly described primary model. State the outcome, the predictor whose estimate is central, the adjustment variables, any important functional forms, and the analysis sample. Give the units and reference groups needed to interpret the estimate. These details let a reader see what changes in the alternative and what stays fixed.
When feasible, change one feature at a time. If the primary and alternative models use different samples, predictors, and standard-error methods all at once, a changed result cannot be attributed to any one choice. Sometimes several changes are needed to address a concern, but then describe them together rather than implying that their individual effects have been isolated.
For a central coefficient, a simple comparison is the difference between the alternative and primary estimates:
A difference should be interpreted in the coefficient's units. If the coefficient is close to zero, a percentage change relative to it can be misleading or undefined; report the absolute difference instead. Also compare standard errors or confidence intervals when uncertainty is relevant. Similar point estimates do not guarantee similar uncertainty, and a changed p-value alone does not show how much the estimated association changed.
Report its model, sample, central estimate, uncertainty, and interpretation in context.
Explain the specific modeling or analysis choice that motivates a check, such as a potentially nonlinear adjustment relationship.
Say exactly what was changed and, where possible, what was held constant.
Give the alternative result, quantify the change, and state what this comparison does and does not support.
Worked Comparisons
Worked Example: Checking an Adjustment Functional Form
Hypothetical analysis. In 100 adults, a linear regression estimates the association between weekly exercise hours and systolic blood pressure in mmHg. The primary model adjusts for age as a straight-line term, sex, and a baseline health score. Its exercise coefficient is \(2.40\) mmHg per additional weekly exercise hour, with standard error \(0.80\). The model's usual interpretation assumes the included terms adequately describe the conditional mean, observations are independent, and the standard error procedure is appropriate for the error variation.
Investigators have a subject-matter reason to consider a curved relationship between age and blood pressure. They fit a sensitivity model replacing the straight-line age term with age and age squared, keeping the outcome, exercise measure, other adjustment variables, and 100-person sample fixed. This makes the exercise coefficient comparable across the two models. The alternative estimate is \(2.10\) mmHg per hour, with standard error \(0.82\).
The estimate change is
For a rough comparison of uncertainty, use the large-sample 95% interval estimate \(\hat{\beta} \pm 1.96SE\) in both fits. The primary interval is
and the alternative interval is
A transparent report would say that allowing a curved age adjustment changed the estimated exercise association by \(-0.30\) mmHg per weekly hour, while both estimates remained positive and the intervals were similar. This comparison suggests limited sensitivity to this particular age specification. It does not establish that exercise causes a change in blood pressure, nor does it show that every possible adjustment model would agree.
Worked Example: Adding a Potentially Important Adjustment Variable
Hypothetical analysis. A study estimates the association between a treatment score and a recovery score measured in points. The primary linear regression adjusts for age and sex. It uses the same 120 people throughout, assumes a suitable linear conditional mean, independent observations, and uses the stated standard-error method. The treatment-score coefficient is \(1.25\) recovery points per treatment-score point, with standard error \(0.48\).
Before examining results, the analysts note that baseline recovery score is related to both the treatment score and subsequent recovery. They therefore fit an alternative that adds baseline recovery score as an adjustment variable. This is a check on the adjustment set, not proof that the added variable removes all confounding. Suppose the alternative coefficient is \(0.92\) recovery points per treatment-score point, with standard error \(0.46\).
The absolute change is
Relative to the primary estimate, the magnitude has decreased by
Here the relative change is interpretable because the primary estimate is not near zero, but the absolute change is still essential and is stated in units. The comparison suggests that the estimated association depends meaningfully on whether baseline recovery is adjusted for. A careful report would present both estimates and explain why the added variable was considered. It would not automatically label either estimate the causal effect: that interpretation requires assumptions beyond this comparison.
Worked Example: Separating an Estimate Change From an Uncertainty Change
Hypothetical analysis. A primary ordinary least-squares model estimates that each additional hour of a training program is associated with \(1.80\) more points on a skill score. There are 80 independent observations, the conditional mean is modeled as linear in the included predictors, and the primary standard error assumes constant error variance. The coefficient's conventional standard error is \(0.70\).
A residual plot suggests that error spread may vary with fitted score. As a sensitivity check, the analysts keep the same regression model and sample but use a heteroskedasticity-consistent standard error of \(1.00\). This alternative changes the estimated uncertainty, not the fitted coefficient. For this illustration, both two-sided tests and intervals use a \(t\) reference distribution with \(df=77\), whose 95% critical value is approximately \(1.991\).
With the conventional standard error, the test statistic for a zero coefficient is
The two-sided p-value is approximately \(0.012\), and the 95% interval is
Using the alternative standard error gives
with an approximate two-sided p-value of \(0.076\), and interval
The point estimate is unchanged, but the uncertainty is greater under the alternative procedure; the interval now includes zero and the p-value differs. A report should say this explicitly rather than suggesting that the estimated association itself changed. The alternative standard error addresses nonconstant variance, but it does not correct a misspecified mean model or dependence between observations.
What to Include in a Report
A short table can make several planned comparisons easy to scan. Include enough information for readers to identify the primary analysis, understand each alternative, and judge the size of any change. A useful table might look like this:
| Analysis | Reason and change | Estimate and uncertainty | Comparison |
|---|---|---|---|
| Primary | Prespecified adjustment and procedure | Coefficient, units, standard error or interval | Reference result |
| Sensitivity A | Alternative age form; same sample and other terms | Coefficient and interval in the same units | Difference from primary |
| Sensitivity B | Alternative standard-error method; same model | Same coefficient; revised uncertainty | Explain inference change |
For each alternative, report the reason it was examined, the exact change, the estimate and its uncertainty, and the practical consequence. If a conclusion depends on a threshold such as a 95% confidence interval excluding zero, describe that threshold and the change directly. Do not treat a p-value just below or above a cutoff as a complete summary of the evidence.
Be equally clear when results are similar. “Robust” by itself is vague: readers need to know which alternatives were tested and how similar the estimates were. For example, say that the coefficient changed by \(0.30\) mmHg per weekly hour and remained positive under the specified alternative, rather than merely calling it robust.
Common Pitfalls
- Choosing alternatives after seeing which results look favorable. This can turn a sensitivity analysis into selective reporting. State the concern and rationale, and report the results of the justified checks even when they weaken the primary finding.
- Changing several things without saying so. If the alternative changes the sample, adjustment set, functional form, and standard-error method together, readers cannot tell which change accounts for the result. Isolate choices where possible and document combined changes explicitly.
- Calling a stable result “proven.” Agreement across a few alternatives only shows stability to those alternatives. It cannot verify all model assumptions or rule out unexamined sources of bias.
- Reporting only whether significance changed. A threshold-crossing p-value can distract from the size and units of the estimate. Report the coefficient and uncertainty alongside any change in the inferential conclusion.
- Failing to distinguish model changes from uncertainty changes. A new predictor or functional form can change the coefficient; a different standard-error procedure can leave it unchanged while changing its interval and p-value. Say which occurred.
Check Your Understanding
Use these questions to test whether a sensitivity comparison is both interpretable and transparent.
- Why should a report state the primary specification before describing an alternative?
- A coefficient changes from \(0.08\) to \(0.04\) units. Why might the absolute change be more useful than a percentage change?
- An alternative standard-error method leaves the coefficient unchanged but widens its confidence interval. What changed, and what did not?
- Why is it important to state the reason for examining an alternative specification?
- What can be concluded if the central estimate is similar across two justified specifications, and what cannot be concluded?