Testing For Sorting At The Threshold

Why Examine the Running Variable’s Density?

In a regression discontinuity (RD) design, treatment assignment changes at a cutoff of a continuous running variable. The design is most credible when units just below and just above the cutoff are comparable apart from the treatment assignment rule. If units can precisely adjust the running variable, however, they may sort to one side of the threshold. The observed units near the cutoff could then differ in ways that also affect the outcome.

A useful diagnostic is to ask whether the distribution of the running variable itself appears to change abruptly at the cutoff. The idea is not that every valid RD design must have identical counts in immediately adjacent intervals. Rather, if the population density changes smoothly through the cutoff, a persistent jump in its one-sided limits can be evidence that the composition of units changes there. This is a diagnostic for a possible threat to the design, not a direct test of whether the treatment effect is identified.

Definition: Let $X$ be the running variable, $c$ its cutoff, and $f_X(x)$ the density of $X$. The density discontinuity at the cutoff compares the one-sided limits $f_X(c+)$ and $f_X(c-)$. When both are positive, the log-density jump is $$ \delta=\log f_X(c+)-\log f_X(c-) =\log\left(\frac{f_X(c+)}{f_X(c-)}\right). $$ The no-discontinuity null hypothesis is $H_0:\delta=0$, equivalently $f_X(c+)=f_X(c-)$. A positive $\delta$ means the density is higher just above the cutoff; a negative $\delta$ means it is lower there.

The ratio form makes the size easier to interpret: $e^\delta$ is the right-side density divided by the left-side density at the cutoff. For example, $\delta=0.2231$ corresponds to a density ratio of about $e^{0.2231}=1.250$, or a right-side density about 25% higher than the left-side density. This describes a relative density difference, not a treatment effect.

Assumptions and the Test in Outline

A density test relies on assumptions about both the running variable and how its density behaves near the cutoff. For the usual continuity-based interpretation, the running variable should be measured on a sufficiently fine scale and have a density that is positive and finite near $c$. Under the no-sorting benchmark, that density should vary smoothly through $c$, rather than making a genuine jump there. The local approximation and its uncertainty calculation also need enough observations on both sides of the cutoff.

A common density-discontinuity procedure, associated with McCrary’s test, uses binned counts to estimate the density on either side and fits local functions to the log density near the cutoff. It then estimates the difference between the two fitted limits and its standard error. The inferential target is the jump in density, not whether the two nearest histogram bars happen to have the same height. Choices such as bin width, bandwidth, and local approximation affect the estimate and its uncertainty; a formal test should use a method designed for density-discontinuity inference rather than treating a visual histogram as a test.

Under the null and the regularity conditions of the chosen estimator, the estimated jump divided by its standard error is approximately standard normal in large samples. For a two-sided test at significance level $\alpha$, reject $H_0$ when the absolute test statistic exceeds the appropriate standard-normal critical value, or equivalently when the two-sided p-value is below $\alpha$. Rejection is evidence against density continuity under the model; it does not establish why the discontinuity arose.

A Count-Based Approximation and Its Derivation

A simplified calculation shows why log density ratios are convenient. Suppose two adjacent intervals, one on each side of $c$, have the same width $w$. Let their observed counts be $N_-$ and $N_+$. The corresponding histogram density estimates are $N_-/w$ and $N_+/w$. Their log ratio is therefore

\widehat{\delta} =\log\left(\frac{N_+/w}{N_-/w}\right) =\log\left(\frac{N_+}{N_-}\right).

For a rough large-count approximation, treat the two counts as independent Poisson variables with means $\lambda_+$ and $\lambda_-$. A Poisson count has variance equal to its mean. Applying the delta method to the log of a count gives an approximate variance of $1/\lambda$ for its log. Replacing each unknown mean by its observed count yields the familiar approximate standard error for the log ratio:

SE(\widehat{\delta})\approx \sqrt{\frac{1}{N_+}+\frac{1}{N_-}}.

This calculation is a teaching approximation for counts in equal-width intervals, not a substitute for the formal local density procedure. It assumes the intervals are comparable and that the density is nearly constant within them. In particular, a smoothly rising or falling density can make counts in adjacent intervals differ even when there is no discontinuity exactly at the cutoff. Formal methods address the density’s local shape rather than relying on a single pair of bins.

Worked Example: A Simple Equal-Width Count Comparison

Suppose a hypothetical RD analysis uses equal-width intervals immediately below and above the cutoff. The below-cutoff interval has 120 observations and the above-cutoff interval has 144. Use the count approximation to test for a density jump.

The estimated log ratio, with the above-cutoff side in the numerator, is $\widehat{\delta}=\log(144/120)=\log(1.20)=0.1823$. Thus the estimated density just above the cutoff is 1.20 times the estimated density just below it in this two-bin comparison.

The approximate standard error is $SE=\sqrt{1/144+1/120}=\sqrt{0.0152778}\approx0.1236$. The test statistic is $z=0.1823/0.1236\approx1.475$.

For a two-sided standard-normal test, the p-value is approximately $2[1-\Phi(1.475)]=0.140$. At $\alpha=0.05$, this simplified test does not reject the no-jump null. The result is not proof that sorting is absent: the calculation is noisy, uses only two intervals, and may have limited power. The conclusion is restricted to this approximate comparison.

Worked Interpretation of a Local Density Test

A formal local density procedure produces a fitted log-density jump and an estimated standard error. The same basic test statistic and interpretation apply, but the estimates incorporate local data on both sides rather than using only a pair of adjacent-bin counts. The following hypothetical result illustrates how to report the magnitude and uncertainty.

Worked Example: Testing a Fitted Log-Density Jump

Suppose a hypothetical local density analysis estimates $\widehat{\delta}=0.2231$ with $SE(\widehat{\delta})=0.1225$. Test $H_0:\delta=0$ against a two-sided alternative.

The statistic is $z=0.2231/0.1225\approx1.821$. Its two-sided standard-normal p-value is approximately $0.0686$. At the 5% significance level, the test does not reject $H_0$. At the 10% level, it would reject. The evidence therefore depends on the stated significance level; the 5% result should not be described as proof of continuity.

An approximate 95% confidence interval for the log jump is $0.2231\pm1.96(0.1225)=0.2231\pm0.2401$, or approximately $[-0.0170,0.4632]$. Exponentiating the endpoints gives an approximate interval $[e^{-0.0170},e^{0.4632}]=[0.9831,1.589]$ for the right-to-left density ratio. Since this interval includes 1, it is consistent with no density jump at the 5% level, while also allowing a meaningfully higher density on the right.

A p-value is the probability, assuming the null and the test’s assumptions, of obtaining a test statistic at least as extreme as the observed one. It is not the probability that sorting occurred, nor the probability that the null is true. The confidence interval describes values of the density ratio compatible with the procedure’s assumptions and sampling uncertainty; it is not a probability statement about the fixed ratio.

Worked Example: A Significant Jump Is a Warning, Not a Verdict

Consider a third hypothetical analysis with an estimated log-density jump of $0.4000$ and standard error $0.1500$. The estimated right-to-left density ratio is $e^{0.4000}\approx1.492$, about 49.2% higher on the right. The test statistic is $z=0.4000/0.1500=2.667$, giving a two-sided p-value of approximately $0.0077$.

At $\alpha=0.05$, reject the null of no density discontinuity for this procedure. This result is a warning that the running-variable distribution differs at the cutoff in a way that may be consistent with sorting. It does not prove intentional manipulation: a data-processing rule, a change in how the running variable was generated, or a misspecified local density approximation could also contribute. The analyst should investigate the assignment process and examine whether the finding is sensitive to defensible density-estimation choices before drawing a substantive conclusion.

Diagnostics, Alternatives, and Limitations

A density test is most informative when treated as one part of an RD assessment. Inspect the running-variable distribution graphically, report the estimated jump and uncertainty, and check whether conclusions are stable under reasonable choices of bins and local bandwidths. A result that changes sharply with minor tuning choices may indicate that the data do not support a precise conclusion. Sparse observations near the cutoff, a small sample, or an imprecise running variable can also make the test weak or distort its approximation.

The density test has an important limitation: it detects a discontinuity in the marginal distribution of $X$, not the mechanism that produced it. Rejection does not establish that units deliberately manipulated their values, and it does not by itself quantify bias in the RD effect estimate. Conversely, failure to reject does not establish that precise sorting is impossible. Manipulation may be too limited to detect, may not produce a visible density jump, or may be offset by other changes in the distribution. The test also cannot verify every condition needed for RD identification.

A histogram is useful for visual exploration but is sensitive to bin placement and width, so it is not a reliable stand-alone inferential procedure. A formal density-discontinuity test provides an estimate and uncertainty under specified assumptions, but it too depends on tuning choices and adequate data. Predetermined covariate continuity checks address a different diagnostic question—whether measured characteristics vary at the cutoff—and do not replace a density test. Neither approach, alone, proves the design valid.

Common pitfall: “The density test has $p>0.05$, so there is no sorting and the RD design is valid.” This is incorrect. A non-significant result means the test did not find sufficiently strong evidence against density continuity at the chosen level; it does not demonstrate continuity, rule out undetected sorting, or establish all RD assumptions. Report the estimate, uncertainty, and limitations rather than treating nonrejection as validation.

A careful report states the direction and estimated size of the jump, the standard error or confidence interval, the test and significance level, and relevant sensitivity checks. If the test rejects, describe the finding as evidence of a density discontinuity that raises a concern about sorting or another change in the running-variable distribution. If it does not reject, describe it as a failure to detect a discontinuity, not as proof that the running variable is unsorted.

Check Your Understanding

Answer each question, then compare with the solutions below.

If $\widehat{\delta}=-0.20$, which side has the higher estimated density at the cutoff, and what is the approximate right-to-left density ratio?
With equal-width bins, counts are 100 below and 121 above the cutoff. Calculate the log ratio and its approximate standard error.
A density test reports $p=0.18$ at $\alpha=0.05$. What is the correct conclusion?
Why does a significant density jump not prove that units deliberately manipulated the running variable?

Solutions. (1) The negative log jump means the density is lower just above the cutoff. The ratio is $e^{-0.20}\approx0.819$, so the right-side density is about 81.9% of the left-side density. (2) The log ratio is $\log(121/100)=\log(1.21)\approx0.1906$. Its approximate standard error is $\sqrt{1/121+1/100}=\sqrt{0.018264}\approx0.1351$. (3) Do not reject the no-discontinuity null at the 5% level. This is not proof that the density is continuous or that sorting is absent. (4) The test detects a discontinuity in the running-variable density, not its cause. Intentional adjustment is one possible explanation, but other features of the assignment or data-generating process may also produce a jump.