From an Adjusted Estimate to a Calibrated Conclusion
A regression result can be numerically precise and still be described too strongly. In an observational study, people receive different exposures for reasons that may also be related to the outcome. Adjustment can address differences in measured covariates, but it does not by itself show that the adjusted contrast is causal. A careful conclusion therefore says what contrast was estimated, for whom, how the analysis adjusted, how uncertain the estimate is, and what assumptions would be needed to interpret it causally.
This tutorial focuses on communicating that conclusion. It uses an exposure with two levels: \(A=1\) for the exposure or treatment of interest and \(A=0\) for the comparison. Let \(Y\) be the outcome and \(X\) the measured covariates used in adjustment. The outcome may be continuous, binary, or otherwise defined, but the estimate and its units must match the stated outcome scale.
What a Causal Interpretation Requires
A regression-adjusted contrast is not automatically the causal contrast in the definition. The following conditions describe what would be needed for a causal interpretation in a typical observational comparison. They are substantive assumptions about the study and its data, not conclusions that follow from a small p-value.
- Consistency and a well-defined exposure: each unit's observed outcome equals the potential outcome under the exposure actually received. Exposure versions must be sufficiently well-defined for the comparison to be meaningful.
- Conditional exchangeability: after conditioning on the measured baseline covariates \(X\), exposure assignment is independent of the potential outcomes. In practical terms, there are no remaining common causes of exposure and outcome that distort the comparison.
- Positivity: for covariate profiles in the target population, there is a nonzero chance of receiving either exposure level. Without this overlap, the data cannot support a reliable comparison for those profiles without extrapolation.
- No interference: one unit's outcome is not changed by another unit's exposure, unless such spillovers are explicitly part of the estimand and analysis.
- Adequate estimation: the adjustment procedure estimates the relevant conditional outcome means or contrast well enough for the intended target population. For regression standardization, this includes an adequately specified outcome model over the covariate values being averaged.
A short identification argument shows why these conditions matter. For either exposure level \(a\), average the potential outcome over covariate profiles. Exchangeability permits replacing the mean potential outcome among units with exposure \(a\) by the corresponding mean among units observed with \(A=a\), within each \(X\) profile. Consistency then equates that observed outcome with \(Y\). Positivity ensures those observed conditional means are available throughout the target covariate distribution.
The last expression is the population average of the outcome regression prediction under exposure level \(a\). This argument does not establish that exchangeability or the other conditions hold in a particular study. It explains why a causal conclusion must be conditional on their credibility. Measurement error, selection into the analysis, and model misspecification can also undermine the adjustment; a reported estimate should not imply that these concerns have been eliminated merely because covariates were included.
The Components of a Useful Report
A useful conclusion is specific enough that a reader can tell what was estimated and what the estimate does not establish. Include the following elements, in the text or in a nearby table:
- Population and outcome: name the eligible or sampled population, the outcome, its time frame, and its scale.
- Contrast and target: state which exposure conditions were compared and whether the target is the full eligible population, a defined subgroup, or a population with adequate overlap.
- Adjustment strategy: name the measured baseline covariates and the approach used, such as an outcome regression standardized over the target covariate distribution. Do not describe a list of covariates as though it proves that all confounding has been controlled.
- Estimate and uncertainty: give the estimate with units and a confidence interval. A p-value may be included, but it does not replace the interval or convey the magnitude of the contrast.
- Overlap and assumptions: summarize whether both exposure levels were represented across the covariate profiles relevant to the target. State that causal interpretation depends on assumptions such as exchangeability, consistency, and positivity.
- Limits: identify important threats that remain, such as unmeasured confounding, limited overlap, measurement error, selection, or uncertainty in model specification.
A confidence interval describes sampling uncertainty under the analysis and its statistical assumptions. It does not include every source of uncertainty: for example, it does not automatically account for unmeasured confounding or a wrong outcome model. A p-value is calculated under a specified null hypothesis and model; it is not the probability that the null is true, nor is it a measure of whether the causal assumptions are credible.
Worked Example: A Positive Adjusted Contrast
Hypothetical setting. An observational evaluation compares adults who enrolled in a six-month exercise program (\(A=1\)) with otherwise eligible adults who did not enroll (\(A=0\)). The outcome is change in a fitness score, measured in points, from baseline to six months. The target is all eligible adults in the participating clinics. A regression model uses age, baseline fitness score, chronic-condition count, and clinic; predictions under each exposure condition are averaged over the eligible adults' covariate profiles. The estimated mean difference is 4.2 points, with \(SE=1.5\) points.
For a large-sample 95% confidence interval using the normal critical value 1.96, the calculation is:
The estimate is 4.2 points higher for program enrollees in the adjusted, standardized comparison. The interval ranges from 1.26 to 7.14 points higher. If a two-sided test of a zero contrast is appropriate under the model, the test statistic is \(z=4.2/1.5=2.80\), giving a p-value of about 0.0051. This is evidence against a zero adjusted contrast under that model; it is not evidence that the causal assumptions have been proven.
Suppose propensity-score diagnostics also show that 94% of eligible adults have estimated exposure probabilities between 0.10 and 0.90, with both exposure groups represented across most of that range. This is useful evidence that overlap is not obviously absent for much of the target population, though it does not guarantee positivity for every relevant covariate profile. Baseline covariates were measured before enrollment, and the analysts found no major lack of fit in checks of the outcome model. These checks support the analysis but cannot establish that unmeasured causes of enrollment and fitness change are absent.
A calibrated conclusion would be: “Among eligible adults in the participating clinics, exercise-program enrollment was associated with a 4.2-point higher adjusted mean six-month change in fitness score than non-enrollment (95% CI 1.26 to 7.14). The estimate was standardized over the eligible adults' measured baseline covariates, with generally good observed overlap. A causal interpretation would require no important unmeasured confounding, adequate model specification, and the other stated assumptions.” This wording reports the finding without turning an observational association into an unconditional causal claim.
Uncertainty Does Not Settle the Causal Question
An interval that includes zero does not demonstrate that the exposure has no effect. It indicates that, under the model and sampling assumptions, the data are compatible with a range of contrasts that includes zero. Likewise, an interval that excludes zero does not show that the contrast is causal. Statistical uncertainty and causal validity are different questions: a narrow interval can accompany substantial bias, while a wide interval can accompany a carefully designed and appropriately adjusted analysis.
Worked Example: An Interval That Includes Zero
Hypothetical setting. A health-system analysis compares a care-coordination service with usual care for eligible patients. The outcome is the number of unplanned hospital visits in the following year. The target is all eligible patients in the system. An outcome regression adjusts for age, prior-year visit count, baseline illness severity, and clinic, then standardizes predictions to the eligible patients. The adjusted mean difference, service minus usual care, is \(-1.2\) visits per patient, with \(SE=1.1\) visits.
The 95% interval is:
The corresponding test statistic is \(z=-1.2/1.1\approx-1.09\), with a two-sided p-value of about 0.275. The data do not provide strong evidence against a zero adjusted mean difference at conventional significance levels. But it would be incorrect to conclude that the service has “no effect”: the interval includes both a reduction and a possible increase in visits. It is also incorrect to say there is a 95% probability that the fixed true contrast lies in this particular interval. In repeated use of this interval procedure, 95% of intervals would cover the fixed contrast if the procedure's assumptions hold.
Suppose both groups were represented across the observed propensity-score range and the model checks did not reveal major problems. These facts support using the estimate for the stated target, but they do not rule out unmeasured confounding or other bias. The appropriately limited conclusion is that the adjusted estimate is \(-1.2\) visits per patient, with a 95% interval from 3.36 fewer to 0.96 more visits, and that the observational analysis alone does not establish a causal effect.
When Overlap Changes the Target
Overlap is not a cosmetic diagnostic. If people with some covariate profiles almost always receive one exposure level, the data contain little information about what their outcomes would have been under the other level. Regression predictions for these profiles may then rely heavily on extrapolation. Restricting analysis to a region with adequate overlap can make a comparison more defensible, but it changes which population the estimate describes. The conclusion must name that new target rather than presenting it as an estimate for everyone initially eligible.
Worked Example: Restricting to a Population With Overlap
Hypothetical setting. A study compares a new medication with standard treatment among 500 eligible patients. The target initially proposed is all 500 patients. Estimated exposure probabilities are near 0.02 for some very low-risk patients, who almost always receive standard treatment, and near 0.98 for some high-risk patients, who almost always receive the new medication. After defining an overlap population in advance, 312 patients remain; both treatments are represented across their covariate profiles. A standardized regression contrast for symptom-score improvement in this group is 3.1 points in favor of the new medication, with \(SE=1.4\) points.
The 95% confidence interval is:
Rounded to three significant figures, the interval is 0.356 to 5.84 points higher. The estimate concerns the 312 patients in the defined overlap population, not all 500 eligible patients. Its causal interpretation still depends on consistency, no important unmeasured confounding within that population, adequate estimation, and the other assumptions. The improvement in observed overlap addresses one difficulty but cannot validate all assumptions.
A suitable conclusion is: “Among the 312 eligible patients in the prespecified overlap population, the new medication was associated with 3.10 points greater adjusted mean symptom-score improvement than standard treatment (95% CI 0.356 to 5.84). Because treatment assignment was strongly separated for some eligible patients, this estimate should not be generalized to all 500 patients; causal interpretation also requires adequate control of confounding and the other stated assumptions.” Reporting the original population as the target here would hide a material change in the question being answered.
A Practical Reporting Sequence
Before writing a causal-sounding sentence, check that its target and evidential strength match the analysis. The sequence below helps keep the conclusion complete without burying the main result in qualifications.
Identify who the estimate is intended to describe, and whether overlap restrictions or exclusions changed that population.
Name exposure levels, outcome scale, and the baseline covariates and adjustment strategy used.
Give the point estimate with units and a confidence interval; add a p-value only as supplementary information.
Use “associated with” for the observed adjusted contrast, and state the assumptions required to interpret it causally.
Briefly identify consequential concerns such as limited overlap, unmeasured confounding, selection, measurement error, or model dependence.
Common Mistake: Treating Adjustment as Proof
Wrong reasoning: “The regression included all important covariates, and the p-value was below 0.05, so the exposure caused the outcome.” This confuses a model-based test of an adjusted contrast with evidence that all causal assumptions hold. Covariate adjustment only addresses the variables measured and represented adequately in the analysis; statistical significance does not remove residual confounding or repair poor overlap.
Correction: report the adjusted contrast as an association in the named target population, describe the adjustment and overlap evidence, and make causal wording conditional on the required assumptions. If substantive knowledge supports those assumptions, a causal interpretation may be reasonable, but it should still be stated with the relevant limits. A result should not be called causal solely because a regression model was fitted.
Check Your Understanding
For each question, focus on the match between the numerical result, the target population, and the strength of the conclusion.
- An adjusted mean difference is 2.0 hours with a 95% confidence interval from \(-0.5\) to 4.5 hours. What can and cannot be concluded from this interval?
- Why does a regression adjustment for several measured covariates not prove that confounding has been controlled?
- An analysis excludes patients with estimated exposure probabilities below 0.05 or above 0.95. What must be clarified about the target population in the report?
- Give two pieces of overlap evidence that could be reported, and explain why neither one alone proves exchangeability.
- Write one sentence that reports an observational adjusted contrast while making clear what assumptions a causal interpretation would require.