What Can the Observed Data Tell Us?
A causal analysis depends on assumptions about how treatment was assigned, how treatment is defined, and what outcomes would have occurred under alternatives that each person did not receive. Study design can make those assumptions more credible, and observed data can expose some serious problems. But a diagnostic is not a certificate that the assumptions hold: some concern outcomes that are never jointly observed and cannot be checked directly.
Consider a binary treatment \(A\), an observed outcome \(Y\), and pretreatment covariates \(X\). Write \(Y(1)\) and \(Y(0)\) for the potential outcomes under treatment and control. The following assumptions are central to identifying causal effects from observed comparisons.
These assumptions differ in how evidence bears on them. Study records may clarify which intervention was delivered, while outcome and covariate data may reveal a lack of observed treatment overlap. Neither a careful protocol nor a reassuring balance table directly reveals both potential outcomes for the same person. In particular, the observed distribution alone generally cannot establish conditional exchangeability.
Evidence, Warnings, and Assumptions
| Issue | What observed data or records can show | What they cannot establish by themselves |
|---|---|---|
| Consistency and treatment definition | Whether the protocol describes treatment versions, timing, and delivery; whether records show major deviations or ambiguous coding. | That all relevant versions are causally equivalent, or that the recorded treatment captures every consequential difference. |
| Positivity | Whether both treatment groups appear across observed covariate patterns; whether some strata contain few or no people in one group. | That the population probability is strictly positive at every relevant covariate value, especially where data are sparse or absent. |
| Conditional exchangeability | Whether measured pretreatment covariates differ between groups, and whether important candidate confounders were measured before treatment. | That no unmeasured common causes remain, or that measured covariates capture all relevant selection into treatment. |
The distinction is between diagnosing observable implications and verifying an identification assumption. For example, if a covariate stratum contains no treated individuals, that is direct evidence of a sample overlap problem. If treatment groups have similar measured covariate averages, that is evidence about those measured averages—not proof that their potential outcomes are exchangeable.
Worked Example: A Missing Treatment Group Within a Stratum
Suppose a hypothetical observational study records treatment and a baseline severity category for 100 patients. The table shows 36 patients with lower severity (20 treated, 16 untreated) and 64 with higher severity (0 treated, 64 untreated). What can the data say about positivity?
Check the observed treatment proportions. In the lower-severity stratum, the treated proportion is \(20/36=0.5556\), and the untreated proportion is \(16/36=0.4444\). Both treatment groups are represented.
In the higher-severity stratum, the treated proportion is \(0/64=0\), and the untreated proportion is \(64/64=1\). There are no observed treated patients in this stratum. Thus, the study provides no treated outcomes there with which to compare the observed untreated outcomes. The sample shows a serious positivity warning for an analysis that aims to compare treatment groups within both severity strata.
The conclusion must be limited to what was observed. If these 100 patients are a sample from a broader population, the zero treated count does not by itself prove that the population probability of treatment is exactly zero among all higher-severity patients. It does show that the study has no empirical support for estimating that comparison in this sample. A design that recruits or assigns treatment across relevant severity levels could improve support; changing the target population to exclude unsupported levels is another possibility, but changes the population to which the conclusion applies.
Balance Is Useful but Not a Test of Exchangeability
Measured covariate balance describes how similar treatment groups are on recorded pretreatment variables. A simple diagnostic for a quantitative covariate is the standardized mean difference: the difference in group means divided by a pooled standard deviation. It puts the difference on a scale that is easier to compare across variables with different units. This is a descriptive diagnostic, not a test of whether all confounding has been removed.
Worked Example: Interpreting a Baseline Imbalance
In a hypothetical study, the 40 treated patients have a mean baseline age of 54 years and standard deviation 10 years. The 60 untreated patients have a mean age of 49 years and standard deviation 12 years. Calculate the standardized mean difference using the average of the two variances as the pooled variance.
The pooled standard deviation is
The standardized mean difference is
The treated group’s mean age is about 0.453 pooled standard deviations higher than the untreated group’s mean age. This flags a measured baseline difference worth examining in the causal analysis. It does not, by itself, show how much the estimated treatment effect is biased: that depends on how age relates to treatment assignment and outcome, as well as on the analysis. Nor would a value near zero prove exchangeability. A similar pattern could occur for measured age while an unrecorded determinant of treatment and outcome remains imbalanced.
Balance summaries should be interpreted alongside the study’s timing and measurement decisions. Covariates used to assess pretreatment comparability should be measured before treatment begins. A variable measured after treatment may itself be affected by treatment; its apparent balance is not evidence about baseline comparability, and conditioning on it can change the causal question.
Why Exchangeability Cannot Be Verified From the Observed Distribution
For each person, only one of \(Y(1)\) and \(Y(0)\) is observed. The missing potential outcome is not recovered merely by collecting more detailed summaries of the same observed treatment-outcome distribution. The following construction shows why two different causal situations can produce identical observed data.
Imagine a hypothetical set of 1,000 people: 500 receive treatment and have observed outcome 1; 500 receive control and have observed outcome 0. In one possible world, every person has \(Y(1)=1\) and \(Y(0)=0\). The average treatment effect is then \(1-0=1\). In another possible world, the treated people have \((Y(1),Y(0))=(1,1)\), while the control people have \((Y(1),Y(0))=(0,0)\). The observed treatment and outcome records are identical, but every person’s treatment effect is zero, so the average treatment effect is 0.
The observed data cannot distinguish these worlds. In the second world, treatment status is associated with the potential outcomes; in the first, the treatment effect is constant. Since both worlds produce the same observed records, the causal effect—and whether exchangeability holds—cannot generally be determined from those records alone. Additional design information and subject-matter knowledge are needed to argue which causal story is credible.
Design Features That Strengthen an Assumption
Design can improve the credibility of assumptions by changing what is measured, when it is measured, or how treatment is delivered. These features provide a rationale for an assumption; they do not turn an untestable assumption into a directly observed fact.
Specify treatment content, timing, and relevant versions in the protocol and data records. This makes consistency more plausible and makes departures easier to identify.
Use subject-matter knowledge to identify baseline factors that could influence both treatment choice and the outcome, then measure them with appropriate timing and quality. This supports, but cannot prove, conditional exchangeability.
Set eligibility, recruitment, or assignment procedures so that each relevant covariate pattern can include both treatment levels. Examine the resulting data for sparse or empty groups.
State who the conclusion is intended to describe. A design or analysis restricted to well-supported groups may make comparisons more defensible, but does not automatically justify conclusions for people outside those groups.
For consistency, records can reveal that a named treatment includes substantially different doses or delivery schedules. If those versions are likely to have different effects, a single treatment label may not correspond to one well-defined intervention. For exchangeability, a prospective baseline assessment can ensure that covariates are measured before treatment and reduce reliance on incomplete historical records. Still, an investigator must judge whether important causes were omitted or measured poorly. For positivity, planned recruitment can avoid a study in which one type of patient almost always receives the same treatment.
These considerations differ from the evidence provided by a randomized assignment procedure, which can support exchangeability through the way treatment is allocated. The details of assignment and its consequences require separate attention; merely labeling a study “randomized” does not settle questions about implementation, missing outcomes, or the target population.
Failure Modes and Appropriate Conclusions
- Calling balance proof of no confounding. Balance concerns recorded covariates. It cannot reveal whether an unmeasured cause still relates to both treatment and outcome.
- Treating a nonsignificant difference as equivalence. A failure to detect a difference may reflect limited sample size. It is not evidence that covariate distributions are identical.
- Ignoring sparse support. A group with very few observations can make estimates unstable even when the observed treatment proportion is not exactly zero. Report sparse cells and avoid pretending the data support comparisons they barely contain.
- Using data to claim population positivity everywhere. Observations cover only a finite set of sampled covariate values. Diagnostics can identify observed gaps and warn about thin support; they cannot certify positive treatment probabilities at every population value.
- Assuming a detailed protocol proves consistency. A protocol clarifies the intended intervention, but actual delivery and meaningful treatment versions must also be considered.
When evidence is weak, the correct response is calibrated language: report what is observed, explain which assumptions are supported by design or subject-matter reasoning, and state what remains uncertain. Sensitivity analysis can examine how conclusions change under specified departures from assumptions, but it does not establish which departure is true.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A study has no treated participants in one observed covariate stratum. What does this establish, and what does it not establish about population positivity?
- Why does similar mean age between treatment groups not prove conditional exchangeability?
- A treatment label combines two delivery versions that may have different effects. Which identification assumption is in question, and what design feature could help?
- What does a standardized mean difference of \(0.10\) say about the measured covariate? What does it not say about unmeasured confounding?
- Why can identical observed treatment and outcome records be compatible with different average treatment effects?
Solutions. (1) It establishes that the sample has no treated observations in that stratum, so a within-stratum comparison is unsupported there. It does not prove that the population treatment probability is exactly zero. (2) Similar means describe one measured feature; they do not establish balance in every relevant covariate or rule out unmeasured causes of treatment and outcome. (3) Consistency is in question because the treatment may not be a single well-defined intervention. Specify and record treatment versions and delivery in the protocol. (4) It describes a difference of one-tenth of a pooled standard deviation for that measured covariate; it does not show that unmeasured confounding is absent. (5) Each person’s outcome under the treatment not received is unobserved. Different assignments of those missing potential outcomes can preserve every observed record while changing the causal effect.