Diagnosing Whether Bounds Are Informative

What Makes an Identified Interval Informative?

A partially identified effect is not summarized by one value, but by a set of values compatible with the observed data and the stated assumptions. That set can still answer useful questions. It may establish the sign of an effect, rule out a decision threshold, or show that every compatible effect is practically small. It can also be too wide to distinguish materially different conclusions.

In this tutorial, write the identified interval for an effect \(\tau\) as \([L,U]\), where \(L\) and \(U\) are its lower and upper endpoints and \(L\leq U\). The interval is taken to be closed: both endpoints are included. Its width is \(W=U-L\). The endpoints and width have the same units as \(\tau\), so a width is meaningful only when interpreted on the outcome’s scale and in the decision context.

There is no universal numerical cutoff for a “narrow” interval. A width of 2 may be small for an outcome measured in thousands of dollars and large for an outcome measured in millimeters. Even after units are fixed, the relevant comparison depends on what effect sizes would matter. Assess width alongside the interval’s location and a prespecified substantive scale, such as a minimally important effect or a decision threshold.

Width, Location, and a Short Justification

The width gives the range of effects still compatible with the analysis. If \(W\) is small relative to the range of effects considered substantively important, the interval is compact for that purpose. If \(W\) spans effects with different practical implications, the interval may be uninformative for a fine-grained decision even if its endpoints are estimated precisely.

Location answers a different question. An interval can be wide but entirely positive, which settles the sign, or narrow but close to zero, which may not establish an important effect. To see why endpoint comparisons answer these questions, let \(\tau\) denote any compatible effect. Since \(L\leq\tau\leq U\), comparisons of both endpoints with zero or a threshold determine what all compatible values can do.

For example, if \(L>0\), then \(\tau\geq L>0\) for every compatible effect, so the effect is strictly positive throughout the interval. If \(L\leq 0\leq U\), zero is among the interval’s values, so the interval does not establish a strictly positive or strictly negative effect. The same endpoint logic applies to a decision threshold \(c\), but equality must be handled according to how the threshold is defined.

Thus, when \(L<c\leq U\), the interval includes values below the threshold and values at or above it; it does not settle whether the threshold is met. If \(L<c<U\), it has values strictly on both sides. At the boundary \(U=c\), it includes values below \(c\) and the value \(c\), but no values above \(c\). That boundary still leaves the “at least \(c\)” question unsettled: some compatible effects meet the threshold and some do not.

Reading Intervals Against Decision Thresholds

A decision threshold should be stated in the same units as the effect and with its direction made explicit. A minimum beneficial effect might be “at least 3 mmHg reduction,” while a safety criterion might be “increase in risk less than 1 percentage point.” These are not interchangeable questions: “at least” includes equality, while “greater than” does not.

Threshold analysis can also reveal useful negative conclusions. If \(U<c\), the data and assumptions rule out meeting an “at least \(c\)” criterion. If \(L\geq c\), they establish that every compatible effect meets it. If \(L<c\leq U\), the bounds leave the decision unresolved. “Unresolved” does not mean that the effect is zero, that the threshold is equally likely to be met or missed, or that no decision can be made using other considerations.

Data-Driven Precision Versus Assumption-Driven Precision

A narrow interval is not automatically evidence that the observed data strongly determine the effect. Bounds are calculated relative to a set of maintained assumptions. Additional restrictions can remove values from the identified set and narrow the interval, even when the observed data have not changed. That narrowing may be useful, but its source is the restriction, not new empirical information.

One way to diagnose the source of informativeness is to compare nested analyses using the same data. Let \(S_0\) be the set of effects compatible with the data and a baseline set of assumptions. Suppose an additional assumption retains only effects in \(S_1\), with \(S_1\subseteq S_0\). The interval implied by \(S_1\) cannot be wider than the interval implied by \(S_0\), but its smaller width alone does not establish that the extra restriction is credible.

This endpoint comparison explains why added restrictions can tighten bounds: they exclude some previously compatible values. Whether the resulting interval deserves greater substantive confidence depends on whether the restrictions are defensible. A transparent report distinguishes information coming from the observed distribution from narrowing that depends on maintained restrictions, rather than presenting a narrow interval without explaining its basis.

Diagnostics, Alternatives, and Limitations

A practical diagnostic is to ask three questions separately: how wide is the interval in meaningful units, what values does it include relative to zero or a decision threshold, and which features of the data and assumptions produce that width and location? Keeping these questions separate prevents a compact interval from being mistaken for a substantively decisive one, or an assumption-driven result from being mistaken for a data-driven result.

• Check the scale. State \(U-L\) in the effect’s units and compare it with a meaningful range or decision margin. A numerical width without context is hard to interpret.
• Check endpoints, not just the midpoint. The midpoint can lie on one side of a threshold even when the interval includes values on the other side. It does not summarize the uncertainty represented by the bounds.
• Check the question’s boundary convention. “At least \(c\)” and “greater than \(c\)” differ at equality. State which criterion is being assessed.
• Check estimation uncertainty. An estimated interval’s endpoints are themselves estimated. Do not treat them as known population endpoints when sampling uncertainty could change whether an endpoint is above, below, or equal to a threshold. Use the appropriate confidence-set methods from the earlier material on confidence sets for partially identified effects and bootstrap diagnostics for estimated bounds.
• Check what narrowed the interval. Compare results under clearly described assumption sets when possible. Narrowness that appears only after an added restriction should be attributed to that restriction.

A point estimate can be easier to act on, but it selects one value rather than representing all effects compatible with partial identification. A conventional confidence interval for a point-identified parameter addresses sampling uncertainty about that parameter; it is not a substitute for an identified interval when the parameter is only partially identified. Sensitivity curves, discussed in the earlier material on sensitivity curves for treatment effect bounds, show how bounds change as a sensitivity parameter varies. They can help reveal what drives width, but do not by themselves turn an assumption-dependent result into a data-only conclusion.

Bounds have limits. An interval may be too wide to guide a particular decision even though it is mathematically valid and informative about the range of compatible effects. Conversely, a narrow interval may be conditional on strong assumptions. And if the reported endpoints are estimated, apparent separation from zero or a threshold may not persist once sampling uncertainty is accounted for. The right conclusion is therefore specific: state what the interval does establish, what it leaves unresolved, and what assumptions that conclusion depends on.

Common Mistake

A common mistake is to call an interval informative because its midpoint is above zero, or because its width looks small in isolation. For example, an interval \([-0.01,0.05]\) has midpoint \(0.02\), but it includes zero and negative values; its midpoint does not establish a positive effect. Another mistake is to call a threshold met whenever the upper endpoint reaches the threshold. For a rule requiring at least \(c\), \(U=c\) means the endpoint meets the rule, but lower values in the interval do not. The correction is to compare both endpoints with the specific boundary and state whether the criterion includes equality.