When Does a Causal Conclusion Change?
A calibrated selection-bias limit is most useful when it answers a decision-relevant question: how large would the bias have to be before the conclusion changed? For example, an estimated treatment effect may remain positive across plausible bias values, but eventually its identified interval may include zero. Or a policy may require an effect to exceed a prespecified minimum; the relevant tipping point is then the bias limit at which that benchmark first becomes possible.
This tutorial works with the average treatment effect on the treated. Let \(D\) indicate treatment, with \(D=1\) for treated units and \(D=0\) for untreated units. Let \(Y(1)\) and \(Y(0)\) denote the potential outcomes under treatment and no treatment, respectively. The observed outcome is \(Y=DY(1)+(1-D)Y(0)\). The target is the treated-group average effect, and the observed difference in outcome means is decomposed into that effect plus selection bias.
The decomposition is algebraic: the observed difference is \(E[Y(1)\mid D=1]-E[Y(0)\mid D=0]\). Adding and subtracting \(E[Y(0)\mid D=1]\) gives the treated effect plus the difference in untreated potential outcomes across treatment groups. A bound on \(B\), rather than an assumption that it equals zero, therefore yields an interval for \(\tau\).
Deriving a Tipping-Point Rule
Suppose the lower bias limit \(b_L\) is fixed and the upper limit is varied as \(\lambda\), so \(B\in[b_L,\lambda]\), with \(\lambda\geq b_L\). The induced identified interval is
As \(\lambda\) increases, the lower endpoint of the effect interval decreases, while its upper endpoint stays fixed. A target value \(c\) belongs to the interval exactly when
The second condition matters: expanding the upper bias limit cannot admit a bias value below the fixed lower limit. If \(c\leq\Delta-b_L\), the first value of \(\lambda\) that admits \(c\) is \(\max(b_L,\Delta-c)\), provided that value is in the allowed domain. If \(c>\Delta-b_L\), the threshold can never enter this interval as \(\lambda\) increases. At the tipping value, the interval touches \(c\); for a strictly larger \(\lambda\), it extends below \(c\). It strictly crosses \(c\) only when the upper endpoint is also greater than \(c\).
For zero, set \(c=0\). Thus zero enters when the allowed bias range first includes \(B=\Delta\). If \(\Delta\) is already in the bias range at its starting limit, zero is included from the start. If \(\Delta\) is below the fixed lower limit, zero never enters under this one-sided expansion. These boundary cases are important: a tipping point is not always a positive finite number.
Worked Example: The Bias Limit for Including Zero
Suppose a hypothetical evaluation has an observed treated-minus-untreated difference of \(\Delta=5\) percentage points in a favorable outcome. The calibrated restriction is \(B\in[-2,\lambda]\) percentage points, where \(\lambda\geq0\). This restriction allows selection bias as low as \(-2\) percentage points and allows its positive upper limit to vary; it does not assert that bias is nonnegative.
The identified interval is
At \(\lambda=4\), the interval is \([1,7]\) percentage points, so it remains strictly above zero. At \(\lambda=5\), it is \([0,7]\) percentage points and first includes zero. At \(\lambda=6\), it is \([-1,7]\) percentage points and contains effects on both sides of zero. Equivalently, zero requires \(B=\Delta=5\) percentage points, so an upper bias limit of 5 percentage points is exactly the tipping point. This is a sensitivity calculation, not evidence that the actual selection bias equals 5 percentage points.
Policy Thresholds and Direction of Change
A policy conclusion often depends on a benchmark other than zero. Let \(c\) be a prespecified effect threshold in the same units as \(\tau\). For example, a decision rule might regard an increase of at least 4 percentage points as practically meaningful. To find when the interval first reaches \(c\), calculate the bias value that would make the effect equal to that threshold: \(B=\Delta-c\). Then check that this value is not below the maintained lower bias limit. This direct calculation is usually safer than trying several candidate intervals.
Worked Example: Tipping at a Policy Benchmark
Suppose a hypothetical program has \(\Delta=8\) percentage points, and selection bias is restricted to \(B\in[-1,\lambda]\) percentage points, with \(\lambda\geq0\). A policy benchmark is \(c=4\) percentage points: the question is when the identified interval first allows an effect of at most 4 percentage points.
The interval at any limit \(\lambda\) is
The threshold requires \(B=\Delta-c=8-4=4\) percentage points. The lower bias limit is \(-1\), so this required bias is allowed once \(\lambda\geq4\). At \(\lambda=3\), the interval is \([5,9]\) percentage points and lies above the benchmark. At \(\lambda=4\), it is \([4,9]\) percentage points and touches the benchmark. At \(\lambda=5\), it is \([3,9]\) percentage points and strictly crosses it. Therefore, the first-entry tipping point is 4 percentage points; strict crossing occurs for limits greater than 4 percentage points.
The direction of the policy statement must match the interval. At the tipping point, the interval no longer guarantees an effect above 4 percentage points, but it does not imply that the effect is below 4. It merely makes the benchmark compatible with the maintained bias range.
Worked Example: When Zero Has No Tipping Point
Consider a hypothetical outcome for which the observed difference is \(\Delta=-3\) percentage points and the bias restriction is \(B\in[-2,\lambda]\) percentage points, with \(\lambda\geq0\). The implied interval is
For every allowed \(\lambda\), the upper endpoint is \(-1\) percentage point, so zero is outside the interval. The bias needed to make the effect zero would be \(B=\Delta-0=-3\) percentage points, which is below the maintained lower limit of \(-2\). Increasing only the upper bias limit cannot admit that value. Consequently, there is no zero-effect tipping point within this bias class. This is a substantive consequence of the restriction, not a computational failure.
Assumptions, Diagnostics, and Alternatives
The calculation is exact for the stated model and bias range, but its causal interpretation depends on the assumptions defining them. The observed difference and the target effect must refer to the same population and outcome scale; \(B\) must be defined consistently with the decomposition; and the proposed range must be a defensible restriction on that selection bias. If the bias range is only an informal calibration, the resulting interval is conditional on that calibration. It is not a confidence interval and does not account for sampling uncertainty in \(\Delta\) or in the calibrated limits.
A useful check is to verify the interval endpoints directly. A larger allowed upper bias must lower or leave unchanged the effect interval's lower endpoint, while the fixed lower bias determines its upper endpoint. The interval must also have nonnegative width: \((\Delta-b_L)-(\Delta-\lambda)=\lambda-b_L\geq0\). If any of these checks fail, the bias range or endpoint arithmetic has likely been reversed. Keep the units and sign convention explicit: changing from treated-minus-untreated to untreated-minus-treated changes every sign.
| Approach | What it answers | What it does not establish |
|---|---|---|
| Point identification under a zero-bias assumption | A single effect value when \(B=0\) is maintained | Whether conclusions survive nonzero selection bias |
| Worst-case bounds | Effects compatible with broad restrictions and observed information | A tipping point for a calibrated selection-bias limit unless that limit is varied |
| Tipping-point calculation | The first specified bias limit at which a target value enters the effect interval | The actual size of bias or the probability that the target effect is true |
A common mistake is to say, “zero is included at the tipping point, so the treatment has no effect.” Inclusion means only that zero is compatible with the maintained data and bias restrictions; other effects in the interval remain compatible as well. Another mistake is to label the largest bias limit examined as the tipping point without checking the boundary equation. The tipping point is where the endpoint first equals the target, not merely where a chosen grid happens to show it.
A single tipping point summarizes one change in the conclusion for one specified interval construction. Its value depends on the chosen bias range, the fixed endpoint, the target effect, and the direction in which the range is expanded. Comparing alternative calibrations can be informative, but the calibration itself must be justified rather than selected to obtain a desired conclusion. The earlier material on choosing and calibrating a bias limit addresses that substantive input; the calculation here translates it into a threshold statement.
Check Your Understanding
For each question, identify the relevant bias value and interpret the boundary carefully. Solutions are provided below.
- If \(\Delta=6\) percentage points and \(B\in[-1,\lambda]\) percentage points, what is the interval and when does it first include zero?
- If \(\Delta=10\) percentage points, \(B\in[-2,\lambda]\) percentage points, and the threshold is \(c=7\) percentage points, find the first-entry limit. Does the interval strictly cross the threshold at that limit?
- If \(\Delta=-4\) percentage points and \(B\in[-3,\lambda]\) percentage points, can increasing \(\lambda\geq0\) make the interval include zero? Explain using the required bias.
- At a tipping limit, does inclusion of a policy benchmark prove that the true effect equals the benchmark? State what it does establish.
Solutions. (1) The interval is \([6-\lambda,7]\) percentage points. Zero requires \(B=6\), so it first enters at \(\lambda=6\); the interval then touches zero. (2) The required bias is \(\Delta-c=10-7=3\) percentage points, which is above the lower limit. The first-entry limit is \(\lambda=3\), giving \([7,12]\) percentage points. It touches, but does not strictly cross, 7 at that limit. (3) Zero requires \(B=\Delta=-4\) percentage points, below the lower limit \(-3\). Increasing the upper limit cannot include it, so no zero tipping point exists. (4) No. Inclusion establishes only that the benchmark is compatible with the observed difference and the maintained bias range; it does not identify the effect as equal to that benchmark.