Why Matching Does Not End the Confounding Question
Matching is a design strategy: it pairs or groups treated and untreated people who are similar on recorded covariates. That can make treatment groups more comparable on those measured characteristics and can clarify which population the analysis compares. But matching cannot guarantee that the groups are comparable on characteristics that were not measured, were measured poorly, or were not included in the matching design. A matched estimate may therefore remain vulnerable to unmeasured confounding.
This is why sensitivity analysis follows matching. Balance checks ask whether measured covariates are similar after matching. Sensitivity analysis asks a different question: How much could an unmeasured difference in treatment selection change the evidence or conclusion? Rosenbaum bounds provide a framework for assessing that question in matched observational studies. They do not reveal whether hidden bias exists or identify the true causal effect in its presence.
For a matched analysis targeting an average treatment effect among treated units, the target is typically the average contrast between each treated unit’s outcome under treatment and its outcome without treatment, for the treated units represented by the matched sample. Matching changes which units are compared and may restrict the target population; it does not make the unobserved outcomes observable.
What Matching Can—and Cannot—Establish
Let \(Z\) indicate treatment, \(X\) denote measured pretreatment covariates, and \(Y\) denote the outcome. In potential-outcome notation, \(Y(1)\) and \(Y(0)\) are the outcomes a unit would have under treatment and no treatment. A causal comparison requires assumptions, not just a matching algorithm.
- Consistency: the observed outcome equals the potential outcome under the treatment actually received.
- No interference: one unit’s treatment does not alter another unit’s outcome, as required for the potential outcomes being used.
- Conditional exchangeability given measured covariates: within levels of \(X\), treatment assignment is independent of the potential outcomes. This is the no-unmeasured-confounding assumption for the measured adjustment set.
- Overlap for the target comparison: units relevant to the target have a nonzero chance of appearing in either treatment group at the covariate values being compared. In practice, matching also requires adequate common support for the chosen design.
These assumptions are not proved by close matches or small post-match standardized differences. A balance table can show whether the observed distribution of \(X\) became more similar. It cannot establish conditional exchangeability, because the distribution of unobserved covariates is not in the table. Nor does a good propensity-score match ensure that every important measured covariate is balanced; balance should be checked directly.
Worked Example: Good Measured Balance Is Not a Test for Hidden Bias
Suppose a hypothetical matched study compares a treatment with usual care. After matching, the treated and control groups have the following recorded covariate summaries.
| Covariate | Treated | Matched control | Difference (treated minus control) |
|---|---|---|---|
| Mean age | 52 years | 51 years | 1 year |
| Prior condition present | 30% | 20% | 10 percentage points |
For a binary covariate, one commonly used standardized mean difference divides the difference in proportions by the pooled binary standard deviation:
Here, \(\hat p=(0.30+0.20)/2=0.25\). Thus the pooled standard deviation is \(\sqrt{0.25(0.75)}=\sqrt{0.1875}\approx 0.4330\), and the standardized difference is \(0.10/0.4330\approx 0.231\). This indicates a remaining measured imbalance in prior condition that merits attention; it is not evidence about any unrecorded factor. For age, an SMD cannot be calculated from the means alone: the group standard deviations are also needed.
The table therefore supports a limited conclusion: the reported summaries show a 10-percentage-point difference in prior condition and a one-year difference in mean age. They do not show that all measured covariates are balanced, and even perfect balance on every recorded covariate would not demonstrate balance on unrecorded severity, motivation, or access to care.
How an Unmeasured Difference Can Distort a Matched Contrast
A simple additive model illustrates the issue. Let \(U\) be a binary, unmeasured risk factor. Suppose, for illustration, that the outcome follows \(Y=\tau Z+\beta U+\varepsilon\), where \(\tau\) is a constant treatment effect, \(\beta\) is the outcome difference associated with \(U\), and the mean of \(\varepsilon\) is the same between treatment groups after matching. The expected treated-minus-control outcome contrast is then:
The second term is the contribution of the difference in \(U\) prevalence to the observed contrast. Matching on \(X\) does not make this term zero unless it also removes the relevant difference in \(U\), or \(U\) does not predict the outcome (\(\beta=0\)). This is a simplified illustration, not a general bias formula for every outcome model; its purpose is to show why matching observed variables cannot, by itself, settle the hidden-confounding question.
Worked Example: A Hidden Risk Factor in a Matched Sample
Consider a hypothetical matched study with a continuous outcome measured in days. Assume, just for this illustration, that treatment has no effect, that an unmeasured binary severity factor increases the outcome by 10 days, and that all other mean outcome differences are absent. Suppose the severity factor is present in 60% of treated units and 20% of matched controls, despite balance on the recorded covariates.
The treatment effect is \(\tau=0\), the outcome association is \(\beta=10\) days, and the prevalence difference is \(0.60-0.20=0.40\). The contribution of the hidden factor to the observed contrast is:
Under these assumptions, the expected treated-minus-control contrast is \(0+4=4\) days even though the treatment effect is zero. The example does not assert that any particular matched study has this bias. It shows why good measured balance and a causal interpretation are separate claims.
What Rosenbaum Bounds Add
A matched observational study is not a randomized experiment. Within a matched set, units can still differ in their probabilities of receiving treatment because of hidden characteristics. Rosenbaum sensitivity analysis asks how inference would change under a specified degree of departure from the assignment pattern expected if matched units were equally likely to receive treatment, conditional on the matching design.
The sensitivity parameter commonly denoted by \(\Gamma\) represents the allowed degree of hidden treatment-selection bias. At the benchmark \(\Gamma=1\), matched units are treated as equally likely to receive treatment under the model. Larger allowed values permit greater differences in treatment odds within matched sets due to hidden factors. The exact assignment model and calculation of bounds are developed separately; the key point here is that the analyst evaluates inference over a range of allowed hidden-bias levels rather than assuming that matching eliminates all such bias.
For a test of a sharp null hypothesis of no treatment effect, sensitivity analysis can report a worst-case p-value: the largest p-value compatible with the chosen sensitivity model. A small worst-case p-value means that the null remains difficult to reconcile with the data even under the specified hidden-bias allowance. If that worst-case p-value exceeds the chosen significance level, the result is not robust to that allowance. Neither outcome proves the presence or absence of hidden bias.
Worked Example: Reading a Hypothetical Sensitivity Analysis
A hypothetical matched analysis tests a sharp null of no treatment effect at \(\alpha=0.05\). Suppose a sensitivity procedure reports the following upper bounds on the p-value. These are stipulated outputs of a hypothetical analysis, not values derived from the preceding examples.
| Allowed hidden-bias level | Worst-case p-value bound | Conclusion at \(\alpha=0.05\) |
|---|---|---|
| \(\Gamma=1.00\) | 0.018 | Below 0.05 |
| \(\Gamma=1.25\) | 0.041 | Below 0.05 |
| \(\Gamma=1.50\) | 0.083 | Above 0.05 |
| \(\Gamma=2.00\) | 0.190 | Above 0.05 |
At \(\Gamma=1.25\), the worst-case p-value bound is 0.041, still below 0.05, so the null would be rejected at the 5% level under that specified allowance. At \(\Gamma=1.50\), the bound is 0.083, which is above 0.05, so rejection is no longer guaranteed across the allowed assignment patterns. The result therefore has a tipping region between these two displayed levels. This is a statement about robustness of the test under the model, not a probability that the null is true or a measure of how likely a hidden confounder is.
Diagnostics, Alternatives, and Limits
Sensitivity analysis complements design diagnostics; it does not replace them. Before interpreting bounds, check that the matching design is coherent for the target population, inspect overlap and matched-set construction, and report balance for important measured covariates before and after matching. Large measured imbalances may indicate that the design needs improvement. A sensitivity analysis cannot repair poor measured balance.
Adjustment for measured covariates and sensitivity analysis also answer different questions. Regression adjustment, propensity-score methods, or other adjustment strategies can address measured confounders when their assumptions and implementation are appropriate. Rosenbaum bounds assess how a conclusion might change if hidden selection remains after that design and adjustment. Applying both is useful: first make the observed groups as comparable as possible, then examine robustness to plausible hidden bias.
| Approach | Primary question | What it cannot establish |
|---|---|---|
| Balance checks | Are recorded covariates distributed similarly after matching? | That unrecorded confounders are balanced |
| Adjustment for measured covariates | Can observed confounding be controlled under the chosen model or design? | That all relevant confounding was measured or modeled correctly |
| Rosenbaum sensitivity analysis | How robust is inference to a specified allowance for hidden treatment-selection bias? | The actual presence, identity, or magnitude of hidden confounding |
A bound is conditional on the sensitivity model, the matched design, the outcome, and the test being used. It is not a universal guarantee: a particular allowance may be too small or too large for the substantive setting, and a sensitivity model does not cover every way the data-generating process could fail. Sensitivity analysis also does not automatically correct measurement error, selection into the study, interference, or misspecification of the target population.
The practical value of Rosenbaum bounds is not that they certify a causal result. They make an otherwise implicit vulnerability explicit: readers can see whether a conclusion persists under increasing allowances for hidden bias. A careful report gives the matching design and target comparison, the measured-balance diagnostics, the sensitivity parameter values examined, the inferential quantity bounded, and the point at which the substantive conclusion changes.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why do small standardized differences after matching not establish that treatment groups are exchangeable on potential outcomes?
- In the additive illustration, let \(\tau=2\) outcome units, \(\beta=5\) units, and the prevalence of \(U=1\) be 0.50 in treated units and 0.30 in controls. What is the expected observed contrast under the stated equal-error-means condition?
- A sensitivity analysis gives an upper p-value bound of 0.07 at a stated hidden-bias level. At \(\alpha=0.05\), what does this imply about rejecting the sharp null?
- Which question is answered by a post-match balance table, and which is answered by Rosenbaum bounds?
- Does a result that remains significant over a range of sensitivity values prove there is no hidden confounding? Explain.
Solutions. (1) Standardized differences describe measured covariates; they provide no direct information about unrecorded characteristics, so they cannot establish exchangeability. (2) The hidden-factor contribution is \(5(0.50-0.30)=1\) unit, and the expected contrast is \(2+1=3\) units. (3) Since the worst-case bound \(0.07\) exceeds \(0.05\), rejection is not guaranteed across the allowed assignment patterns; this does not establish that the null is true. (4) A balance table asks whether recorded covariates are similar after matching. Rosenbaum bounds ask how inference changes under a specified allowance for hidden treatment-selection bias. (5) No. It indicates robustness within the sensitivity model and range examined, not absence of hidden confounding or validity under every possible data-generating process.