Why Assess Specification Sensitivity?
An RD estimate depends on analytic choices: how wide a neighborhood of the cutoff to use, how to approximate the outcome’s relationship with the running variable, how to weight observations, and whether to adjust for predetermined covariates. A single estimate can be numerically precise under one specification while a reasonable alternative yields a different magnitude or uncertainty. Examining these choices systematically helps show whether the substantive conclusion depends heavily on one defensible analytic decision.
Sensitivity analysis is not a way to find the specification that produces the smallest p-value. It is a comparison of estimates from a set of prespecified, substantively defensible specifications, interpreted in light of the common RD target and the assumptions that make it causal. The goal is to learn how much the conclusion changes when reasonable modeling choices change.
The causal interpretation still requires the RD identification assumptions. In particular, absent treatment, the expected potential outcomes must vary continuously through the cutoff, and treatment assignment must change as specified at the threshold. Sensitivity to bandwidth or modeling choices does not verify these assumptions. It describes how the estimate behaves conditional on the RD design being credible.
What to Vary—and What to Hold Fixed
A useful sensitivity analysis varies choices that could plausibly affect the estimate while keeping the comparison interpretable. A bandwidth determines which observations contribute locally; the polynomial order determines the local functional form; the kernel changes how observations within the bandwidth are weighted; and covariate adjustment changes the regression specification. Change one dimension at a time when the purpose is to diagnose that choice. A compact, prespecified grid can also vary combinations, provided each combination is defensible and reported.
| Choice | What changes | How to interpret a change in the result |
|---|---|---|
| Bandwidth | The neighborhood of the cutoff included | Differences can reflect a bias–variance tradeoff and greater influence of observations farther from the cutoff. |
| Polynomial order | The local shape used to approximate the outcome relationship | Differences may indicate sensitivity to functional-form approximation; high-order global fits are not a substitute for a local RD analysis. |
| Kernel | The relative weights assigned within the chosen neighborhood | Differences show whether the result depends strongly on how near-cutoff observations are emphasized. |
| Covariate set | Which predetermined variables enter the adjustment model | Large changes merit investigation; adjustment does not repair a discontinuity in potential outcomes. |
Keep the inference method, outcome definition, cutoff, and sample rules consistent across comparisons unless one of those is itself the object of a planned sensitivity check. Report the point estimate and its uncertainty for every specification, not just whether a confidence interval excludes zero. When estimates use the same observations, their errors are generally related; a visual comparison of their confidence intervals is not a formal test that the estimates differ.
Covariates used for adjustment should be predetermined rather than affected by treatment, and their inclusion should be justified in advance. As in covariate adjustment more generally, adjustment can improve precision under appropriate conditions, but it is not required to identify the RD effect and should not be used to conceal evidence of a problematic design.
Worked Example: Varying the Bandwidth
Suppose a hypothetical sharp RD study estimates the effect of eligibility on a test score measured in points. The cutoff and the local-linear specification are fixed, as are a triangular kernel, the covariate set, and the inference procedure. The analysts prespecify four bandwidths, measured in score points from the cutoff. The resulting estimates and standard errors are:
| Bandwidth | Local sample size | Estimate (points) | SE (points) |
|---|---|---|---|
| 4 | 110 | 2.1 | 0.80 |
| 6 | 168 | 2.4 | 0.65 |
| 8 | 224 | 2.5 | 0.58 |
| 10 | 275 | 2.7 | 0.56 |
For illustration, use a normal-approximation 95% confidence interval, \(\hat{\tau}\pm1.96SE\). At bandwidth 4, the calculation is \(2.1\pm1.96(0.80)=2.1\pm1.568\), giving \((0.532,\ 3.668)\) points. At bandwidth 6, it is \(2.4\pm1.96(0.65)=2.4\pm1.274\), giving \((1.126,\ 3.674)\) points. The intervals at bandwidths 8 and 10 are \(2.5\pm1.96(0.58)=(1.363,\ 3.637)\) and \(2.7\pm1.96(0.56)=(1.602,\ 3.798)\) points, respectively.
All four estimates are positive, and their intervals exclude zero under this illustrative approximation. The point estimates rise from 2.1 to 2.7 points as the bandwidth increases, while the standard errors decline. This pattern is consistent with increased precision from using more observations, alongside some sensitivity in magnitude to how far from the cutoff the fit reaches. It does not show that the larger-bandwidth estimate is preferable: bandwidth choice involves bias as well as variance, and these results alone do not establish which estimate is closest to the causal effect at the cutoff.
If a 1.5-point effect is the prespecified threshold for a substantively important effect, the estimates are all above that threshold, but the intervals do not all lie entirely above it. For example, the bandwidth-4 interval extends down to 0.532 points. Thus the evidence supports a positive effect across these specifications more clearly than it establishes an effect exceeding 1.5 points.
Comparing a Planned Set of Specifications
Sensitivity is best summarized by looking at estimates and uncertainty together. Ask whether the signs agree, whether magnitudes vary enough to change the substantive interpretation, and whether uncertainty leaves room for materially different conclusions. “Every estimate is statistically significant” is not an adequate summary: it ignores effect size, the range of plausible values, and whether a policy-relevant threshold is supported.
Worked Example: A Prespecified Specification Grid
Consider a hypothetical sharp RD analysis of an outcome measured in points. The primary specification is local linear, bandwidth 6, triangular kernel, and adjustment for two predetermined covariates. The analysts planned a small set of alternatives. The estimates below use the same outcome and cutoff and a consistent inference procedure; the displayed intervals use \(\hat{\tau}\pm1.96SE\) for illustration.
| Specification | Estimate (points) | SE (points) | Approximate 95% interval (points) |
|---|---|---|---|
| Primary: local linear, \(h=6\), triangular, adjusted | 2.4 | 0.65 | (1.126, 3.674) |
| Bandwidth \(h=4\), otherwise primary | 2.1 | 0.80 | (0.532, 3.668) |
| Bandwidth \(h=8\), otherwise primary | 2.5 | 0.58 | (1.363, 3.637) |
| Uniform kernel, otherwise primary | 2.3 | 0.68 | (0.967, 3.633) |
| Local quadratic, otherwise primary | 2.6 | 0.78 | (1.071, 4.129) |
| No covariate adjustment, otherwise primary | 2.5 | 0.67 | (1.187, 3.813) |
For example, the local-quadratic interval is \(2.6\pm1.96(0.78)=2.6\pm1.5288\), or approximately \((1.071,\ 4.129)\) points. The no-adjustment interval is \(2.5\pm1.96(0.67)=2.5\pm1.3132\), or approximately \((1.187,\ 3.813)\) points.
The estimates range from 2.1 to 2.6 points, all positive; the illustrative intervals all exclude zero. That supports a reasonably stable conclusion about direction across this planned grid. The largest point estimate is about 24% greater than the smallest, so the magnitude is not identical across choices. If the substantive question is whether the effect is greater than 1.5 points, none of these intervals lies wholly above 1.5: the results do not establish that stronger claim at the displayed confidence level.
These intervals are marginal intervals for separate specifications. They are not a single interval with 95% coverage for the entire collection, and the table does not make the most favorable row a valid replacement primary analysis. The useful conclusion is narrower: across the stated choices, the estimated direction is consistent, but precision does not support every possible substantive threshold.
Why Differences Between Estimates Need Care
Two estimates from the same RD data are usually correlated, especially when one bandwidth contains most of the observations used by another. Therefore, a difference between point estimates should not be evaluated by treating the estimates as independent. If \(\hat{\tau}_1\) and \(\hat{\tau}_2\) estimate the same target, then
The covariance term matters. Ignoring positive covariance can overstate the uncertainty of the difference. A formal comparison requires a covariance estimate appropriate to the shared data and estimation procedure, for example from joint influence-function calculations or a bootstrap that respects the design and dependence structure. Such a test addresses whether the two estimates differ; it does not by itself establish which specification is better or whether either estimate is unbiased.
Worked Example: Comparing Nested-Bandwidth Estimates
Suppose the hypothetical estimates from the first example at bandwidths 6 and 8 are \(\hat{\tau}_6=2.4\) and \(\hat{\tau}_8=2.5\) points, with standard errors 0.65 and 0.58 points. For this illustration, suppose a valid joint calculation gives their covariance as \(0.300\) points squared. The estimated difference is
Its estimated variance is
Thus the standard error of the difference is \(\sqrt{0.1589}=0.3986\) points, and the normal-approximation statistic is \(0.1/0.3986=0.251\). The observed difference is small relative to its standard error. A corresponding approximate 95% interval for the difference is \(0.1\pm1.96(0.3986)=(-0.681,\ 0.881)\) points. This interval includes zero, so these data do not provide clear evidence that the two estimates differ. It does not prove the estimates are equal; nor would a test of equality determine which bandwidth is preferable.
The covariance was supplied here to make the arithmetic explicit. In an actual analysis it must be estimated using a method justified for the estimators and data. Substituting zero covariance simply because it is unavailable is generally not justified for estimates from overlapping observations.
A Practical Review of Sensitivity Results
A disciplined report separates the design’s identifying assumptions from the stability of the estimator. The latter is informative but cannot establish the former. Use the following sequence to make the analysis transparent:
State the outcome, cutoff, bandwidth, polynomial order, kernel, covariate set, and inference procedure, along with the reason for choosing them.
Choose a limited set of bandwidths and other variations that address plausible modeling concerns. Do not add specifications only after seeing which ones produce favorable results.
For each specification, give the point estimate, standard error or interval, and units. Keep inference choices comparable and explain any necessary differences.
Describe changes in direction, magnitude, and uncertainty, and state whether those changes affect the conclusion relevant to the application.
A result that is stable across reasonable specifications is reassuring about dependence on those particular choices, but it is not proof of causal validity. Conversely, some movement in estimates does not automatically invalidate an RD analysis. It calls for an explanation grounded in the local approximation, the changing sample, and the uncertainty—not a decision to report only the most convenient result.
Common Mistake: Treating Sensitivity as a Significance Search
A common mistake is to run many bandwidths, kernels, polynomial orders, and covariate sets, then report only the row whose interval excludes zero or whose estimate looks largest. The reasoning is wrong because the reported result was selected using the observed outcomes; its apparent certainty does not account for that selection. It also obscures whether ordinary, defensible choices lead to different substantive conclusions.
The correction is to define a primary specification and a limited set of alternatives before interpreting results, report the full planned set, and emphasize effect sizes and uncertainty rather than a count of significant rows. If the analysis was exploratory and choices were made after viewing outcomes, say so. Robustness across specifications is evidence about stability to those choices, not a substitute for the RD assumptions or a guarantee that the effect is precisely known.
Check Your Understanding
Use the definitions and examples above to answer the following.
- In a sharp RD sensitivity analysis, why does changing the bandwidth not automatically define a different causal target, and why might the estimates nevertheless differ?
- A kernel change leaves the point estimate nearly unchanged but increases its standard error. What aspect of the result is stable, and what aspect is not?
- Why is it generally inappropriate to calculate the standard error of the difference between two overlapping-bandwidth estimates as if their covariance were zero?
- All estimates in a planned specification grid are positive, but some intervals include zero. What can you conclude about direction and uncertainty without claiming that the effect is established under every specification?
- Why does stability across bandwidths and covariate sets not demonstrate that the RD identification assumptions hold?