Placebo Cutoffs And Outcomes

Why Use Placebo Checks?

A regression discontinuity (RD) estimate is credible only if, near the true assignment threshold, the outcome would otherwise vary smoothly with the running variable. A visible jump at the threshold is consistent with a treatment effect, but it could also reflect another change that happens at the same location, sorting, measurement problems, or an unsuitable model. Placebo checks probe some of these alternative explanations by asking whether apparent jumps also appear where the treatment rule should not produce one.

There are two common checks. A placebo cutoff is a false threshold at which treatment assignment does not change. A placebo outcome is an outcome that the treatment cannot plausibly affect. Either can reveal patterns that cast doubt on a simple treatment-effect interpretation. Neither can prove that the true-cutoff estimate is unbiased.

Definition: Let $X_i$ be the running variable, $c$ the true cutoff, $D_i=1(X_i\geq c)$ the sharp assignment rule, and $Y_i$ the outcome. The RD discontinuity in the outcome at $c$ is the difference between its right- and left-hand limits: $$ \tau_Y(c)=\lim_{x\downarrow c}E[Y_i\mid X_i=x]-\lim_{x\uparrow c}E[Y_i\mid X_i=x]. $$ A placebo-cutoff check estimates an analogous jump at a value $c^*\ne c$ where assignment does not change. A placebo-outcome check estimates the jump at the true cutoff for an outcome $W_i$ that treatment cannot plausibly affect.

The sharp RD setup and the local interpretation of its cutoff effect are established in the earlier material on sharp regression discontinuity and the local estimand in regression discontinuity. A placebo check does not change that estimand; it tests whether the observed data show discontinuities in places or variables where the design gives no reason to expect them.

When Is a Placebo Check Informative?

For a placebo cutoff $c^*$, the treatment rule must not change there. Usually, $c^*$ is chosen in a region entirely below or entirely above $c$, so treatment status is constant in a neighborhood of $c^*$. If outcomes are also smooth at $c^*$, then the expected outcome jump there is zero. A jump at that false threshold suggests that some background feature of the outcome, another policy, or the chosen analysis could generate apparent discontinuities independently of the treatment transition at $c$.

For a placebo outcome, the key requirement is that the outcome is genuinely unaffected by treatment, including through indirect or anticipatory pathways relevant to the study. Let $W_i(d)$ denote the potential value of that outcome under treatment status $d$. The no-effect condition is $W_i(1)=W_i(0)$ for the population under consideration. In addition, the untreated—or equivalently, under this condition, observed—placebo outcome should have no other discontinuity at $c$. If these conditions hold, its expected jump at the true cutoff is zero.

These requirements are substantive, not merely statistical. A variable measured after treatment begins may still be a poor placebo outcome if treatment could change it. Conversely, a baseline measurement may be unaffected by treatment but still jump at the cutoff because of sorting or another threshold-linked process. Predetermined covariate continuity checks address related questions, but a placebo outcome is selected specifically for its lack of a plausible causal response to treatment.

Expected placebo pattern: Under a valid placebo setup, the expected discontinuity is zero. For a placebo cutoff, this requires no treatment or other threshold-linked change at $c^*$ and smooth outcomes there. For a placebo outcome, it requires no treatment effect on that outcome and continuity of its background relationship at $c$. An estimate near zero is consistent with these conditions, but does not establish them.

How to Estimate and Interpret the Checks

Use an RD estimator appropriate to the question, such as the local polynomial methods and inference procedures established earlier in the course. Keep the comparison local to the relevant cutoff and report the estimate, standard error, confidence interval, and test result. A placebo test with $H_0:\tau=0$ evaluates whether the data are inconsistent with a zero discontinuity under the stated model and inference procedure. A small p-value is evidence against that zero-jump model; it does not identify the cause of the jump.

Placebo checks are most interpretable when cutoffs, outcomes, bandwidths, and analysis choices are selected for a clear design reason rather than after searching for favorable results. Repeating many checks raises the chance of at least one apparently unusual result by chance. Report the full set of checks, or otherwise account for the multiplicity in the inferential claims. Also consider whether a placebo bandwidth reaches the true cutoff or another policy boundary: in that case, the estimate may mix different regimes and cease to be a clean falsification check.

Worked Example: Comparing Two False Thresholds

Hypothetical data. A program is assigned sharply when a test score reaches $c=50$. Investigators estimate local outcome jumps at scores 45 and 55, where treatment status does not change. Suppose the estimated jumps and standard errors are 0.30 and 0.40 outcome points at 45, and $-0.20$ and 0.35 points at 55. For illustration, use a normal approximation and two-sided tests.

At 45, the test statistic is $$ z=\frac{0.30}{0.40}=0.75. $$ The two-sided normal p-value is approximately 0.453. The estimated jump is only 0.75 standard errors from zero, so these data do not provide strong evidence of a discontinuity at that false threshold.

At 55, the statistic is $$ z=\frac{-0.20}{0.35}\approx -0.571. $$ The two-sided p-value is approximately 0.568. This estimate, too, is compatible with a zero jump at the false threshold.

Both results are reassuring in the limited sense that these checks do not reveal detectable discontinuities at the selected false thresholds. They do not prove that the outcome is smooth at every non-treatment location, rule out sorting at 50, or validate the main RD estimate by themselves. The calculation is meaningful only if treatment really is constant near each false cutoff and the analysis windows do not cross 50 or another relevant threshold.

A Placebo Outcome at the True Cutoff

A placebo outcome uses the actual treatment cutoff but changes the variable being analyzed. This differs from testing a predetermined covariate simply because it was measured before treatment: the placebo outcome is chosen to have no plausible causal response to treatment. Its purpose is to check whether the same assignment threshold is associated with a jump in something that treatment cannot have caused.

Worked Example: Testing a Treatment-Invariant Outcome

Hypothetical data. The program is assigned at a score of 50. Researchers use a recorded pre-program administrative measure, in units of points, as a placebo outcome. Subject-matter knowledge supports that the program cannot change this already-recorded value. A local RD estimate of its jump at 50 is $-0.18$ points, with standard error 0.12 points.

The test statistic is $$ z=\frac{-0.18}{0.12}=-1.50. $$ The two-sided normal p-value is approximately 0.134. A 95% normal-approximation confidence interval is $$ -0.18\pm1.96(0.12)=-0.18\pm0.2352=[-0.4152,\ 0.0552]\text{ points}. $$

At the 5% significance level, the test does not reject a zero discontinuity. The interval indicates that the data are compatible with jumps from about $-0.415$ to $0.055$ points under the stated approximation. This is not evidence that the probability of a zero jump is 95%, nor does it prove there is no imbalance. The result is a useful diagnostic only if the measure is truly unaffected by treatment and has no independent reason to jump at the cutoff.

What a Suspicious Placebo Result Means

A placebo estimate that is statistically distinguishable from zero deserves investigation, but it does not automatically invalidate the design. It may reflect a real background discontinuity, a second policy threshold, sorting, an inappropriate bandwidth, chance variation, or a misspecified local approximation. A placebo outcome might also fail its basic premise: treatment may affect it after all. The diagnostic identifies a discrepancy between the expected placebo pattern and the observed estimate; further design knowledge is needed to explain it.

Worked Example: A False Cutoff Reveals a Competing Threshold

Hypothetical data. A benefit begins at score 70. Investigators choose 65 as a placebo cutoff, believing that no rule changes there. Their estimate of the outcome jump at 65 is 1.40 points with standard error 0.50 points.

The test statistic is $$ z=\frac{1.40}{0.50}=2.80, $$ which gives a two-sided normal p-value of approximately 0.0051. The approximate 95% confidence interval is $$ 1.40\pm1.96(0.50)=1.40\pm0.98=[0.42,\ 2.38]\text{ points}. $$

This is evidence of a positive discontinuity at the proposed false cutoff under the model and approximation. Suppose further investigation reveals that a separate tutoring policy also begins at score 65. Then 65 was not a valid placebo cutoff: the outcome could jump there for a real policy reason. The result warns that a competing threshold may matter; it does not show that the benefit beginning at 70 caused the jump at 65. The analyst should clarify the policy environment and avoid describing 65 as a location where nothing changes.

Why the Checks Matter—and Where They Fall Short

The main value of placebo checks is diagnostic triangulation. A well-chosen false threshold asks whether the analysis finds jumps away from the assignment boundary. A well-chosen placebo outcome asks whether the true boundary is associated with a jump in a variable the treatment cannot change. If both show no notable discontinuity, the evidence is more consistent with the design’s smoothness claims than if either shows a clear jump.

The checks are not substitutes for the assumptions that identify the RD effect. A non-significant result may reflect low precision, especially with few observations near a threshold or a noisy outcome. It is a failure to detect a jump, not proof that the jump is exactly zero. A statistically significant result may arise by chance, particularly when many cutoffs and outcomes are searched. Report estimates and uncertainty rather than reducing the evidence to a yes-or-no label.

The main alternatives answer different questions. Predetermined covariate continuity checks examine whether observed baseline characteristics change at the threshold; placebo outcomes additionally require a defensible claim that treatment cannot affect the outcome. Sorting diagnostics examine whether the distribution of the running variable changes around the cutoff, rather than testing outcome jumps at false locations. Bandwidth and specification sensitivity analyses assess how the main estimate responds to analytic choices. These diagnostics complement one another; none alone establishes the RD assumptions.

Specify the falsification question.
For a placebo cutoff, identify a location where assignment and other relevant rules do not change. For a placebo outcome, explain why treatment cannot plausibly affect it.

Check local comparability.
Confirm that the analysis window does not include the true cutoff or another threshold when testing a false cutoff, and use an appropriate local RD procedure.

Report estimates with uncertainty.
Give the jump estimate, standard error, interval, and test result, using units that make the magnitude interpretable.

Investigate discrepancies cautiously.
Consider competing rules, sorting, measurement, model choices, limited precision, and chance. Do not treat a placebo result as a direct estimate of the cause of a discontinuity.

Common Mistake

A common mistake is to argue, “The placebo test is not significant, so the RD assumptions are proven.” The reasoning is wrong because a non-significant test may simply have low power, and it tests only one particular outcome or location under a particular analysis. The correction is to describe the result as no detectable placebo discontinuity in this check, report its uncertainty, and assess it alongside the design’s substantive assumptions and other diagnostics. The opposite mistake is also possible: one significant placebo result does not automatically establish that the treatment effect estimate is false; first verify that the placebo itself was valid and investigate alternative explanations.

Key takeaway: Placebo cutoffs and placebo outcomes are falsification checks, not proofs. Their value depends on a credible reason to expect no jump at the tested location or in the tested outcome, and their results must be interpreted with uncertainty and design context.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

A placebo-cutoff estimate is 0.24 outcome points with standard error 0.30. Calculate the normal-approximation test statistic and approximate two-sided p-value.
A treatment-invariant placebo outcome has an estimated cutoff jump of 0.70 points and standard error 0.25 points. Calculate the test statistic and interpret the result using a 5% level.
Why does failing to reject a zero placebo jump not establish that the true jump is exactly zero?
A proposed false cutoff is two score units from the real cutoff, and the analysis uses a window extending three units on either side of the false cutoff. What is the problem?

Solutions. (1) $z=0.24/0.30=0.80$, with a two-sided normal p-value of approximately 0.424. This does not provide strong evidence of a jump at that placebo cutoff. (2) $z=0.70/0.25=2.80$, with a two-sided p-value of approximately 0.0051. Under the stated assumptions, this is evidence against a zero jump in the placebo outcome and calls for investigation; it does not identify the cause. (3) A non-significant test may have low power, and its uncertainty can include nonzero jumps. It indicates insufficient evidence to reject the specified null, not proof of exact equality. (4) The window crosses the real cutoff, so some observations may have different treatment assignment. The resulting estimate is not a clean check at a location where assignment remains unchanged.

What You'll Learn