Why Audit Assumptions One by One?
A narrow identified interval is not automatically persuasive. It may be narrow because the data are informative, because substantive restrictions rule out many potential-outcome configurations, or because the analysis quietly relies on both. An assumption audit makes that distinction visible. For each restriction, it records what the restriction says, why it might be credible, how it changes the feasible values of the causal effect, and which conclusions rely on it.
The audit is especially useful when different readers find different assumptions plausible. Instead of presenting one interval as if it followed from the observed data alone, an analyst can show which bounds follow under minimal restrictions and how they change under additional, explicit claims. This is a disciplined way to interpret bounds; it does not make an untestable restriction true.
From Restrictions to Bounds
Let \(D\in\{0,1\}\) denote treatment, and let \(Y(1)\) and \(Y(0)\) be an individual's potential outcomes under treatment and no treatment. The observed outcome is \(Y=DY(1)+(1-D)Y(0)\). The target here is the population average treatment effect, \(\tau=E[Y(1)-Y(0)]\). Because only one potential outcome is observed per person, assumptions constrain the unobserved parts of the potential-outcome distribution.
A useful audit separates the restrictions into two broad groups. Design and measurement restrictions include consistency, a well-defined treatment, and a specified target population. Substantive restrictions that can tighten bounds include limits on the outcome range, restrictions on treatment-effect heterogeneity, limits on selection differences, and restrictions supplied by auxiliary data. Each must be stated precisely enough to determine which potential-outcome configurations remain possible.
The general logic can be stated without committing to a particular bound calculation. Let \(\mathcal{F}_0\) be the set of potential-outcome distributions compatible with the observed data and baseline assumptions. An added restriction \(A\) leaves the subset \(\mathcal{F}_A=\{F\in\mathcal{F}_0:F\text{ satisfies }A\}\). The sharp bounds are the smallest and largest values of \(\tau\) over the relevant feasible set. Thus, if \(A\) is compatible with the data, \(\mathcal{F}_A\subseteq\mathcal{F}_0\), and the new sharp interval cannot be wider. It can stay the same if the restriction removes no configurations that attain the old endpoints. If the restriction conflicts with the data or other assumptions, the feasible set is empty; that is a specification warning, not evidence for an especially precise effect.
This nesting argument is the basic justification for the audit's “what it tightens” column. It applies to sharp bounds: each endpoint is optimized over all distributions allowed by the stated restrictions. A reported interval that is not sharp may fail to show the full tightening, or may be invalid if its calculation does not correctly enforce the restrictions.
Worked Example: Auditing Outcome Support
Suppose a hypothetical study has treatment prevalence \(P(D=1)=0.4\), with observed mean outcome 8 among treated people and 5 among untreated people. The outcome is measured in points. Start with only consistency and the observed group means, and then consider the restriction that both potential outcomes lie in \([0,10]\).
Define \(a=E[Y(0)\mid D=1]\), the unobserved mean untreated potential outcome among treated people, and \(b=E[Y(1)\mid D=0]\), the unobserved mean treated potential outcome among untreated people. By the law of total expectation, \(\tau=0.4(8-a)+0.6(b-5)\). With no outcome-support restriction, these missing means are not bounded by the observed means, so the group means alone do not give finite bounds for \(\tau\).
Under the proposed support restriction, \(0\leq a\leq10\) and \(0\leq b\leq10\). The expression for \(\tau\) decreases as \(a\) increases and increases as \(b\) increases. Therefore its minimum uses \(a=10,b=0\), while its maximum uses \(a=0,b=10\):
The resulting bounds are \([-3.8,6.2]\) points. The restriction makes the bounds finite, but it does not identify the sign of the effect. Its substantive justification must concern the scale and possible values of both potential outcomes, not merely the observed outcomes. If 0 and 10 are arbitrary coding limits rather than credible limits on outcomes under either treatment condition, this tightening is not justified.
Make the Link Explicit
An audit should not stop at listing assumptions. It should connect each restriction to the particular unknowns or feasible distributions it constrains. For example, a range restriction may bound missing counterfactual means; a restriction on selection differences may restrict how treatment groups differ in their potential outcomes; an auxiliary-data moment may rule out distributions inconsistent with a separate benchmark. State the resulting bound change, rather than saying only that the assumption “improves identification.”
The next calculation shows how a restriction on selection differences translates directly into a treatment-effect interval. Let \(m_1=E[Y\mid D=1]\), \(m_0=E[Y\mid D=0]\), \(p=P(D=1)\), \(a=E[Y(0)\mid D=1]\), and \(b=E[Y(1)\mid D=0]\). Define \(\delta_0=a-m_0\) and \(\delta_1=m_1-b\). These are differences in potential-outcome means between the treated and untreated groups, under the indicated treatment state. Writing \(\Delta=m_1-m_0\), substitute \(a=m_0+\delta_0\) and \(b=m_1-\delta_1\) into the group decomposition of the ATE:
This identity is an audit tool: the selection restrictions act on \(\delta_0\) and \(\delta_1\), with weights determined by treatment prevalence. It does not assert that either selection difference is zero. Assuming both are zero would make \(\tau=\Delta\), a much stronger claim that requires its own justification.
Worked Example: Quantifying a Selection-Bias Restriction
Continue with the hypothetical means \(m_1=8\), \(m_0=5\), and \(p=0.4\). Suppose subject-matter knowledge motivates \(|\delta_0|\leq1\) point and \(|\delta_1|\leq2\) points. Assume these limits are compatible with the outcome support and the observed data. Since \(\Delta=8-5=3\), use the identity above to find the implied range for \(\tau\).
The smallest effect occurs at the largest allowed values of both selection differences, because each is subtracted. The largest effect occurs at their smallest allowed values:
The restriction gives \([1.4,4.6]\) points, which is narrower than the support-based interval \([-3.8,6.2]\). Under these restrictions, the average effect is positive. That conclusion depends on the stated limits on both selection differences: if either limit is relaxed, the endpoints must be recalculated. The limits cannot be justified by the observed difference \(\Delta=3\) alone, since \(\Delta\) combines treatment effects and selection differences.
Build an Audit That Can Be Reviewed
A useful audit can be organized as a short sequence. It should be specific enough that another analyst can identify exactly which assumptions support each reported conclusion.
Name the target population, treatment contrast, outcome scale, observed-data facts, and assumptions required to interpret the potential outcomes.
Specify which potential outcomes, conditional means, moments, or treatment effects are restricted, and for which population or covariate strata.
Give a scientific or design-based reason for the restriction. Distinguish that rationale from evidence that merely shows the restriction is not contradicted by observed data.
Show which feasible configurations are removed and report the resulting endpoints. Check that the restriction leaves at least one feasible configuration.
Record whether claims about sign, magnitude, or a decision threshold hold under baseline restrictions, added restrictions, or both.
A compact table helps expose assumptions that otherwise disappear in prose. “Diagnostic” should be read carefully: many restrictions concern unobserved potential outcomes and cannot be tested directly from the observed data. Diagnostics can find conflicts, assess implications, or compare predictions; they generally cannot establish the truth of an untestable restriction.
| Restriction | What it constrains | Justification and diagnostic | Conclusion depending on it |
|---|---|---|---|
| Potential outcomes lie in a specified range | Missing counterfactual outcome values | Use a defensible measurement or scientific range; check coding and whether the limits apply under both treatment conditions | Finite support-based endpoints |
| Selection differences have specified limits | Differences in potential-outcome means across treatment groups | Ground limits in substantive knowledge or auxiliary evidence; observed data alone do not reveal the missing means | Whether the interval excludes zero or a decision threshold |
| Treatment effects obey a specified restriction | Possible individual or group treatment-effect patterns | Explain the mechanism; assess compatibility with data and other restrictions | The endpoints after excluding disallowed effect patterns |
Worked Example: Separating a Robust Claim From a Conditional One
Worked Example: Does the Effect Exceed a Practical Threshold?
Suppose a hypothetical analysis reports baseline bounds \([-0.5,5.0]\) points. An additional, explicitly stated restriction on potential-outcome differences yields bounds \([1.4,4.6]\) points. A decision-maker regards an average increase greater than 1 point as practically important. Audit the conclusion under each assumption set.
Under the baseline bounds, zero lies inside the interval, so the sign of the average effect is not determined. The value 1 also lies inside, so the baseline restrictions do not establish that the effect exceeds the practical threshold. It would be incorrect to summarize these bounds as evidence that the effect is positive or practically important.
Under the added restriction, the lower endpoint is 1.4 points, and \(1.4>1\). Thus every effect value allowed by that restricted model exceeds the specified threshold. The conclusion “the effect exceeds 1 point” is conditional on the added restriction, not a conclusion from the baseline information alone. The upper endpoint, 4.6 points, also shows that the restriction does not identify one precise effect size.
The audit should record the threshold, the interval supporting the claim, and the restriction responsible for excluding values at or below 1. It should also report whether that restriction is substantively credible and how the result changes if its limits are relaxed. No sampling uncertainty is included in these hypothetical population bounds; if endpoints are estimated from data, inferential uncertainty must also be addressed separately.
Diagnostics, Failure Modes, and Alternatives
Useful checks include verifying that all restrictions are written on the same outcome scale and apply to the stated target population; confirming that the joint restrictions are feasible; recomputing bounds after removing each added assumption; and checking how endpoints move as numerical limits vary. When restrictions are imposed within covariate strata, check that their definitions and target weights are consistent across strata. A constraint can be individually plausible yet incompatible with another constraint when imposed jointly.
- Calling a restriction “data-driven” because it fits observed outcomes. Compatibility is not verification when the restriction concerns missing potential outcomes.
- Reporting only the narrowest interval. This hides which assumptions did the tightening and can make a conditional conclusion sound unconditional.
- Assuming each restriction tightens the bounds strictly. It may remove no endpoint-attaining configuration, so the sharp bounds can remain unchanged.
- Ignoring the joint feasible set. Separate restrictions may each be compatible with the data but incompatible with one another.
- Treating a sensitivity limit as an estimated fact. A chosen limit is an assumption unless supported by an appropriate source of information; label its status explicitly.
The main alternatives differ in what they ask the analyst to defend. A point-identifying model supplies enough restrictions to select one causal value; it can be informative, but its identifying assumptions must be audited just as carefully. Partial identification instead reports the range left by stated restrictions. Sensitivity analysis varies one or more restrictions to show how the range and conclusions respond. Auxiliary data can constrain the feasible set, but their relevance, measurement, and transport to the target population require justification. These approaches are complements: a bounds analysis can show whether a conclusion survives weaker restrictions than a point estimate requires.
The central limitation is that an audit improves transparency, not truth. A mathematically correct tightening can still be scientifically unconvincing if its restriction lacks a credible rationale. Conversely, broad bounds are not a failure when they accurately reflect limited information. The earlier material on diagnosing whether bounds are informative concerns the content of the interval; the audit adds a record of which assumptions support that content.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If an additional restriction is compatible with the observed data, can it make the sharp identified interval wider? Explain.
- In the selection-bias identity, \(\tau=\Delta-p\delta_0-(1-p)\delta_1\), let \(\Delta=2\), \(p=0.25\), \(|\delta_0|\leq2\), and \(|\delta_1|\leq1\). Find the implied bounds.
- Why does agreement between an observed outcome and a proposed range restriction not prove that the restriction applies to the unobserved potential outcome?
- Baseline bounds are \([-1,3]\), and bounds under an additional restriction are \([0.4,2.5]\). Is a positive effect established by the baseline assumptions? Is it established under the additional restriction?
Solutions. 1. No. A compatible restriction makes the feasible set a subset of the original feasible set; minimizing over a subset cannot lower the minimum, and maximizing over it cannot raise the maximum. The interval can stay unchanged or become narrower. 2. The minimum uses \(\delta_0=2,\delta_1=1\): \(2-0.25(2)-0.75(1)=0.75\). The maximum uses \(\delta_0=-2,\delta_1=-1\): \(2+0.25(2)+0.75(1)=3.25\). The bounds are \([0.75,3.25]\). 3. Only one potential outcome is observed for each person. Observed values can be consistent with a restriction while the missing potential outcomes violate it; compatibility is not confirmation. 4. The baseline interval contains zero, so it does not establish a positive effect. The restricted interval has a lower endpoint of 0.4, so it establishes positivity conditional on the additional restriction.