What an Outcome Plot Can—and Cannot—Show
In a regression-discontinuity design, treatment assignment changes at a threshold in a running variable. A plot of outcomes against that variable is a useful first look: it can show the broad outcome pattern near the threshold and whether the data appear to have a visible jump. But the picture depends on how observations are grouped and displayed. A plot is a diagnostic and communication tool, not a substitute for the assumptions that identify a causal effect or for an estimator designed to estimate it.
Let \(X\) be the running variable, \(c\) the cutoff, and \(Y\) the outcome. For convenience, center the running variable at the cutoff by defining \(R=X-c\). In a sharp design, treatment assignment is \(D=1(R\geq 0)\): units at or above the cutoff are assigned treatment, and units below it are not. The cutoff effect is defined by the difference between the limits of the conditional mean outcome as \(R\) approaches zero from the right and the left. The earlier material on continuity explains why, under the relevant continuity assumptions, this jump identifies a local causal effect.
To make the design visible, mark the cutoff at zero and show observations on both sides. Report or display how bins are defined, and consider showing the number of observations in each bin. Where feasible, a raw scatterplot can accompany the binned means: the raw points reveal the distribution of observations, while the means make the broad pattern easier to see. A vertical reference line at zero helps readers distinguish the two sides.
Constructing and Reading Binned Means
Choose a set of intervals that covers the displayed range, placing zero at a bin boundary when the goal is to compare patterns on either side. For each bin, count the observations, sum their outcomes, and divide the sum by the count. Do not treat an empty bin as having a zero outcome. Record the bin limits and the plotted location of each mean; bins with different widths should not be represented as though they cover the same range.
Worked Example: Calculating Means on Both Sides
Suppose a hypothetical sharp-threshold study has an outcome measured in points. For four bins, the observed counts and outcome totals are as follows. The intervals are half-open, so each observation belongs to exactly one bin.
| Centered running-variable bin | Count | Sum of outcomes (points) | Binned mean (points) |
|---|---|---|---|
| [-2, -1) | 20 | 900 | 45.0 |
| [-1, 0) | 40 | 1,960 | 49.0 |
| [0, 1) | 50 | 2,750 | 55.0 |
| [1, 2) | 25 | 1,425 | 57.0 |
For the bin immediately to the left of the cutoff, the mean is \(1{,}960/40=49.0\) points. For the bin immediately to the right, it is \(2{,}750/50=55.0\) points. Their difference is \(55.0-49.0=6.0\) points. A plot would show a rise in these neighboring bin means, but 6.0 points is not thereby established as the cutoff effect: each mean averages outcomes across a full one-unit interval, and the outcome may change smoothly within either interval.
The counts also matter when judging precision. The right-hand bin has 50 observations and the left-hand bin has 40, so the plotted means need not be equally precise. If standard error bars are added, the usual within-bin standard error of a mean is \(s_j/\sqrt{n_j}\), where \(s_j\) is the sample standard deviation of outcomes in bin \(j\). Such bars summarize sampling variability under the usual sampling assumptions; they do not account for bias from averaging over a bin or establish a causal interpretation.
Bin Width Can Create or Hide a Visual Gap
Binning trades detail for stability. Narrow bins retain more detail about how outcomes vary with the running variable, but contain fewer observations and can produce noisy means. Wide bins tend to average out noise, but blur changes near the cutoff and combine outcomes from noticeably different running-variable values. Bin placement matters too: shifting boundaries can change which observations contribute to a mean, even when the underlying data are unchanged.
A simple derivation shows why adjacent bin means can differ even when there is no discontinuity. Suppose the conditional mean outcome follows a smooth linear function \(m(r)=a+br\) on both sides, with no jump at zero. In a bin \([u,u+h)\) of width \(h\), the average of this linear function across the interval is its value at the midpoint \(u+h/2\):
For two equally wide bins directly beside the cutoff, \([-h,0)\) and \([0,h)\), the difference between these average values is \(bh\). It shrinks as the bins narrow, even though the true limiting means at zero are equal. With nonuniform observation density, a bin average generally reflects the distribution of observations within the bin rather than a uniform average, but it still averages over a range rather than measuring the cutoff limit.
Worked Example: A Smooth Trend That Looks Like a Jump
Suppose, hypothetically, the conditional mean outcome is \(m(r)=50+4r\) points on both sides of a cutoff at \(r=0\). The function is continuous there: both limiting means equal 50 points, so the cutoff jump is zero. Consider the adjacent bins \([-1,0)\) and \([0,1)\), each of width 1.
The first bin’s midpoint is \(-0.5\), giving its average outcome \(50+4(-0.5)=48\) points. The second midpoint is \(0.5\), giving \(50+4(0.5)=52\) points. The plotted bin means differ by \(52-48=4\) points even though the true cutoff jump is \(50-50=0\) points.
Now use bins of width 0.2 directly beside zero: \([-0.2,0)\) and \([0,0.2)\). Their midpoints are \(-0.1\) and \(0.1\), so the means are \(50+4(-0.1)=49.6\) and \(50+4(0.1)=50.4\) points. Their difference is \(0.8\) points. The narrower bins bring the plotted means closer to the cutoff values in this smooth linear example, but in real data they may also have fewer observations and noisier means. Neither bin choice alone proves that a jump exists or identifies its magnitude.
Axes, Counts, and Fitted Curves
The vertical scale affects how large a discontinuity appears to the eye. A truncated outcome axis can make a small difference look dramatic; a very broad axis can make meaningful variation hard to see. The numerical difference between two plotted values does not change with the axis, but the visual impression does. Label both axes with units, use a scale that makes the range legible, and state clearly if the vertical axis does not start at zero. Avoid changing scales across panels if the reader is meant to compare apparent jump sizes.
A curve or line can help summarize a broad trend, but its appearance depends on the chosen functional form and the range of data used to fit it. In particular, drawing one line through bin means on both sides can smooth over a real discontinuity. Fitting separate descriptive lines on each side can make the contrast visible, but extrapolating those lines to zero is sensitive to the fit and the displayed range. Such fitted curves are visual summaries, not automatically valid estimates of the cutoff effect.
Worked Example: Neighboring Bins Versus Separate Trend Lines
Imagine a hypothetical plot with four equally wide bins centered at \(-2,-1,1,\) and \(2\). Their mean outcomes are 44, 47, 55, and 58 points, respectively. The nearest pair of bins differs by \(55-47=8\) points. If a reader compares only those two means, the visual contrast may seem to suggest an 8-point cutoff jump.
Instead, suppose the two left means lie on the descriptive line \(50+3r\): at \(r=-2\), it predicts 44 points, and at \(r=-1\), 47 points. The two right means lie on \(52+3r\): at \(r=1\), it predicts 55 points, and at \(r=2\), 58 points. Extending these lines to zero gives 50 points from the left and 52 from the right, a fitted vertical gap of \(52-50=2\) points. This difference from the 8-point comparison illustrates how smooth trends within sides can account for much of the difference between neighboring bin means.
The 2-point gap here is only a descriptive extrapolation from the specified lines. With just two plotted means on each side, the example does not establish that a linear form is appropriate, quantify uncertainty, or justify a causal conclusion. The purpose of the lines is to show why the eye should distinguish a side-specific trend from a discontinuity at the threshold.
When adding uncertainty bars, state what they represent and keep them visually distinct from the fitted curves. Standard-error bars for individual bin means do not directly provide a confidence interval for the difference between cutoff limits. Overlapping or nonoverlapping bars are not, by themselves, a formal test of the regression-discontinuity effect. A plot can reveal sparse regions, irregular patterns, or sensitivity to display choices, but it cannot verify continuity of the relevant potential-outcome means or rule out other threats to identification.
Practical Checks and Common Pitfalls
Center the running variable so the cutoff is at zero, mark it clearly, and show which observations are represented on each side.
State the bin limits or width, indicate where the plotted means are located, and report counts so sparse bins are not mistaken for stable patterns.
Check whether the broad impression changes with plausible bin widths or placements, and consider a raw scatterplot alongside the binned means. A changed picture is a warning that the visual summary is sensitive to choices.
Describe the observed pattern first. Treat a fitted curve or apparent gap as suggestive, and do not label it a causal effect without the design assumptions and appropriate estimation and inference.
A frequent mistake is to call the difference between the nearest left and right bin means “the regression-discontinuity estimate.” That reasoning confuses averages over intervals with limits at a point. The correction is to describe the plotted difference as a difference in bin averages, then use a method appropriate to estimating the cutoff limits if a numerical causal estimate is needed. Another mistake is to choose bins after looking for the arrangement that makes a jump most striking. Since bin choices can alter the display, report them transparently and assess whether the overall message is robust rather than selecting the most persuasive picture.
Compared with a raw scatterplot, binning reduces clutter and makes broad patterns easier to see, but hides within-bin variation. Compared with a fitted curve, binned means impose fewer visual assumptions but can depend strongly on bin construction and can be noisy. Neither is universally superior: displaying both can make their strengths and limitations clearer. Plotting is most useful as an exploratory and explanatory step, while formal estimation must address the behavior of outcomes sufficiently close to the cutoff and the uncertainty in that estimate.
Check Your Understanding
Answer each question, then compare with the solutions.
- In a hypothetical dataset, a bin contains 30 observations with a total outcome of 1,350 points. What is its binned mean?
- If the conditional mean is \(m(r)=20+2r\), what are the expected average outcomes in bins \([-1,0)\) and \([0,1)\), assuming a uniform distribution of \(r\) within each bin? What is the true cutoff jump?
- Why might a plot with narrow bins look less stable than one with wider bins?
- What does a difference between the nearest left and right bin means describe, and why is it not automatically the cutoff effect?
Solutions. (1) The binned mean is \(1{,}350/30=45.0\) points. (2) The bin midpoints are \(-0.5\) and \(0.5\), so the average outcomes are \(20+2(-0.5)=19\) and \(20+2(0.5)=21\) points. Their difference is 2 points, while both limiting means at zero equal 20 points, making the true jump zero. (3) Narrow bins generally contain fewer observations, so their means can vary more from sampling noise and may have larger standard errors. (4) It describes the difference between averages over two intervals. Smooth variation within those intervals can produce a difference even when the limiting means at the cutoff are equal.