Choosing And Calibrating A Bias Limit

From a Bias Judgment to an Effect Interval

A sensitivity bound is only as useful as the limit placed on the unmeasured bias. A limit that is too narrow can make a conclusion look more secure than the evidence warrants; one that is too wide may leave little information about the effect. Choosing and calibrating a limit therefore requires more than substituting a number into a formula. The limit should be stated in the outcome’s units, connected to a defensible substantive judgment or benchmark, and interpreted as an assumption rather than as a quantity established by the data alone.

We will consider the average treatment effect among treated units. Let $D$ equal 1 for treated units and 0 for untreated units, and let $Y(1)$ and $Y(0)$ be the potential outcomes under treatment and no treatment. Assume treatment and outcomes are well-defined, consistency holds so that $Y=Y(1)$ when $D=1$ and $Y=Y(0)$ when $D=0$, and both treatment groups have positive probability. Assume also that the relevant means are finite. These conditions define the quantities below; they do not make treatment assignment unconfounded.

Definition: The average treatment effect among the treated is $\tau_{ATT}=E[Y(1)-Y(0)\mid D=1]$. The mean selection bias is $B=E[Y(0)\mid D=1]-E[Y(0)\mid D=0]$: the difference between the treated group’s mean outcome if untreated and the observed untreated group’s mean outcome. The observed difference in means is $\Delta=E[Y\mid D=1]-E[Y\mid D=0]$.

Selection bias compares two groups’ untreated potential outcomes. It is not the treatment effect, and it need not be zero even when the observed difference in outcomes is estimated precisely. The untreated potential outcome for treated units is not observed; a sensitivity analysis limits how far its mean may differ from the untreated group’s observed mean.

Why a Bias Limit Produces an Interval

By consistency, the observed mean for treated units is $E[Y(1)\mid D=1]$, and the observed mean for untreated units is $E[Y(0)\mid D=0]$. Add and subtract $E[Y(0)\mid D=1]$ in the observed contrast:

\begin{aligned} \Delta &=E[Y(1)\mid D=1]-E[Y(0)\mid D=0]\\ &=\bigl(E[Y(1)\mid D=1]-E[Y(0)\mid D=1]\bigr) +\bigl(E[Y(0)\mid D=1]-E[Y(0)\mid D=0]\bigr)\\ &=\tau_{ATT}+B. \end{aligned}

Thus $\tau_{ATT}=\Delta-B$. This identity is a bookkeeping decomposition, not an assumption of no confounding. If substantive knowledge supports only the symmetric restriction $|B|\leq\Gamma$, where $\Gamma\geq0$, then $B\in[-\Gamma,\Gamma]$. Subtracting that range from $\Delta$ gives the following identified interval under the restriction:

Result: If $|B|\leq\Gamma$, then $$ \tau_{ATT}\in[\Delta-\Gamma,\ \Delta+\Gamma]. $$ The interval is centered at the observed difference $\Delta$, and its width is $2\Gamma$. It describes the values of the treated-group average effect compatible with the assumed bias limit; it does not by itself account for sampling uncertainty.

The direction of the subtraction matters. If treated units would, on average, have had higher outcomes even without treatment, then $B>0$, and the observed contrast overstates $\tau_{ATT}$. If they would have had lower untreated outcomes, then $B<0$, and the observed contrast understates it. More generally, if the defensible restriction is asymmetric, $B\in[b_L,b_U]$, then

\tau_{ATT}\in[\Delta-b_U,\ \Delta-b_L].

The upper bias limit is subtracted to obtain the lower effect endpoint. This reversal is a frequent source of sign errors.

Worked Example: Choosing a Limit From a Substantive Judgment

Suppose a hypothetical program has an estimated treated-minus-untreated outcome difference of 4.8 additional symptom-free days over a fixed follow-up period. A subject-matter panel judges that the treated group’s mean untreated outcome could plausibly exceed or fall below the comparison group’s mean by at most 2.0 days. For this calculation, treat the observed contrast as $\Delta=4.8$ days and encode the judgment as $|B|\leq\Gamma=2.0$ days.

The interval is $[\Delta-\Gamma,\Delta+\Gamma]$. Substituting the stated values gives $[4.8-2.0,\ 4.8+2.0]=[2.8,\ 6.8]$ days.

Under this bias restriction, the compatible average treatment effect among treated units ranges from 2.8 to 6.8 additional symptom-free days. This is conditional on the panel’s limit being credible. It is not a claim that the bias is known to be no larger than 2.0 days, nor does it include uncertainty from estimating the observed contrast.

Calibrating the Limit With a Benchmark

A benchmark can make a proposed limit easier to explain: compare the possible unmeasured selection bias with the bias contribution of a measured characteristic. The comparison is a calibration device, not a proof that unmeasured factors are no stronger than the benchmark. The benchmark must be chosen and calculated on a scale that makes the comparison meaningful.

For example, suppose a binary characteristic $Z$ is associated with an untreated outcome through an additive difference of $\delta$ outcome units between $Z=1$ and $Z=0$. If that association is constant across treatment groups, the mean untreated-outcome difference attributable to the prevalence imbalance in $Z$ is

B_Z=\delta\{P(Z=1\mid D=1)-P(Z=1\mid D=0)\}.

The formula multiplies an outcome difference by a difference in proportions, so its units are outcome units. In an actual application, the association and prevalence difference may need adjustment for other measured covariates. The simple calculation below isolates the benchmark’s contribution to show how a scaling judgment can be translated into $\Gamma$.

Worked Example: Scaling a Limit From a Measured Benchmark

In a hypothetical study, a measured binary risk factor is present in 60% of treated units and 30% of untreated units. Suppose its untreated-outcome association is 5 percentage points, with higher risk-factor prevalence associated with a higher outcome. Its benchmark contribution is $5\text{ percentage points}\times(0.60-0.30)=1.5$ percentage points.

Investigators judge that unmeasured selection bias could be as large as twice this benchmark in either direction. Their stated calibration is therefore $\Gamma=2(1.5)=3.0$ percentage points. If the observed treatment-group contrast is $\Delta=6.0$ percentage points, the effect interval is $[6.0-3.0,\ 6.0+3.0]=[3.0,\ 9.0]$ percentage points.

Under the chosen calibration, the treated-group average effect is compatible with values from 3 to 9 percentage points. The factor of two is a substantive choice, not something determined by the benchmark calculation. Moreover, the benchmark calculation represents the bias due to $Z$ only under the stated additive and constant-association description; it does not establish a bound on bias from unmeasured factors.

A credible calibration should explain both the benchmark and the multiplier. A benchmark may be a strong observed confounder, a factor considered important by subject-matter experts, or a plausible range elicited from experts about untreated outcomes. State what it represents, why it is relevant to the unmeasured factors, and whether the chosen limit is symmetric. A purely numerical choice with no such explanation is difficult to assess or reproduce.

Asymmetric Limits and Scale Checks

Substantive knowledge may support different upper and lower bias limits. For example, clinical knowledge might make a positive bias more plausible than a negative one. In that case, retain the signed range rather than forcing it into a symmetric bound. If $B\in[-1.0,2.5]$ days and $\Delta=3.0$ days, then $[\Delta-2.5,\Delta-(-1.0)]=[0.5,4.0]$ days. The lower endpoint subtracts the largest positive bias; the upper endpoint subtracts the most negative bias.

Worked Example: Applying an Asymmetric Bias Range

Suppose a hypothetical evaluation measures an observed difference of $\Delta=1.7$ points on a 0-to-100 outcome scale. Experts judge that selection bias is unlikely to be less than $-0.4$ points but could be as high as $1.1$ points. Thus $B\in[-0.4,1.1]$.

Using $[\Delta-b_U,\Delta-b_L]$, the interval is $[1.7-1.1,\ 1.7-(-0.4)]=[0.6,\ 2.1]$ points. Under this asymmetric restriction, the treated-group average effect lies between 0.6 and 2.1 points. The interval does not have to be centered at the observed contrast because the bias range is not centered at zero.

Check that the bias limit and observed contrast use the same outcome scale and population. A limit of 2 points cannot be added to a contrast measured in days without a justified conversion. If an outcome is rescaled—for example, from proportions to percentage points—rescale both $\Delta$ and the bias limit. Also check the definition of the target: the decomposition here concerns the effect among treated units, not automatically the average effect in the entire population.

Diagnostics, Alternatives, and Limits

Several simple checks can reveal an arithmetic or conceptual problem:

Check the center and width. A symmetric limit must produce endpoints equally far from $\Delta$, with width $2\Gamma$. If $\Gamma=0$, the interval collapses to $\Delta$; this is the zero-selection-bias assumption.
Check signs. A positive $B$ reduces the effect relative to the observed contrast. For asymmetric limits, the lower effect endpoint uses $b_U$, not $b_L$.
Check units and calibration. Both the benchmark contribution and $\Gamma$ must be in the outcome units. A multiplier should have a substantive rationale.
Check what the data can establish. The observed contrast and measured benchmark can be calculated from data, but the claim that unmeasured bias falls within the chosen limit requires external or substantive justification.

This approach differs from assuming $B=0$, which gives a point estimate equal to $\Delta$ but requires no mean selection bias. It also differs from worst-case bounds based only on the outcome’s known range: those bounds may require fewer restrictions on selection, but can be much wider. Adjustment for measured covariates can address differences captured by those covariates, but it does not automatically limit residual bias from unmeasured factors. A sensitivity bound makes that residual limitation explicit rather than claiming it has disappeared.

The interval is an identification statement conditional on the chosen restriction. In practice, $\Delta$ is estimated from a sample, so the displayed endpoints calculated with a sample contrast are also subject to sampling error. Treating them as if they were population endpoints known exactly would overstate precision. The bias limit itself may also be uncertain: reasonable alternative calibrations can yield different intervals. Report the chosen limit, its rationale, the effect and outcome scale, and the resulting interval together. Do not present the calibration as an empirically verified maximum unless the evidence truly supports that interpretation.

Key takeaway: First define the target effect and selection bias, then justify a numerical range for the bias in outcome units. The effect interval follows by subtracting the bias range from the observed contrast; the credibility of that interval depends on the credibility of the restriction.

Check Your Understanding

For each question, calculate the requested interval and check how the bias restriction affects its interpretation.

A hypothetical study has an observed difference of 7.0 minutes and a symmetric bias limit of 2.5 minutes. Find the effect interval.
An observed contrast is 0.5 percentage points, and the bias is restricted to $[-0.8,1.2]$ percentage points. Find the effect interval.
A measured binary benchmark has a 25-percentage-point prevalence difference between treatment groups and an untreated-outcome association of 4 percentage points. Investigators choose a symmetric limit 1.5 times the benchmark contribution. If $\Delta=3.0$ percentage points, calculate the limit and effect interval.
For a symmetric bias limit, what happens to the interval’s center and width if $\Gamma$ increases by 1 outcome unit?

Solutions. (1) $[7.0-2.5,\ 7.0+2.5]=[4.5,9.5]$ minutes. (2) The lower endpoint is $0.5-1.2=-0.7$, and the upper endpoint is $0.5-(-0.8)=1.3$, giving $[-0.7,1.3]$ percentage points. (3) The benchmark contribution is $4(0.25)=1.0$ percentage point. Thus $\Gamma=1.5(1.0)=1.5$ percentage points and the interval is $[3.0-1.5,\ 3.0+1.5]=[1.5,4.5]$ percentage points. (4) The center remains $\Delta$; each endpoint moves outward by 1 unit, so the width increases by 2 units.