Start With How Treatment Was Chosen
An observational comparison can look straightforward: compare outcomes for people who received a treatment with outcomes for people who did not. Its credibility depends, however, on how those groups came to differ. If patients with worse underlying health are more likely to receive a treatment, their outcomes may differ even if the treatment itself has no effect. Conversely, patients who are healthier, more motivated, or better able to access care may be more likely to receive it and also more likely to have good outcomes.
The first diagnostic task is therefore not to choose a statistical adjustment. It is to reconstruct the treatment-selection process in the setting and at the time the treatment decision was made. Ask who made the decision, what information they used, what alternatives were available, and whether patients could opt in or out. A list of recorded variables cannot answer those questions on its own.
Let \(D\) indicate treatment (\(D=1\) for treatment and \(D=0\) for comparison), \(Y\) be the outcome, \(Y(1)\) and \(Y(0)\) the potential outcomes, and \(X\) a set of pretreatment covariates. A conditional comparison has a causal interpretation only with suitable assumptions. In particular, conditional exchangeability requires that, within levels of \(X\), treatment assignment be unrelated to the potential outcomes: \(Y(d)\perp D\mid X\), for \(d=0,1\). Consistency connects the observed outcome to the potential outcome under the treatment actually received, and positivity requires a nonzero chance of each treatment option at the covariate values being compared.
These assumptions are not confirmed by a favorable-looking table or by a small p-value. The practical question is whether the study’s design and subject-matter knowledge make them plausible. The earlier material on conditional exchangeability and identification establishes why these assumptions matter; here the emphasis is on inspecting a study scenario for threats to them.
Work From the Timeline and the Clinical Process
A useful assessment begins by putting events in order: eligibility, baseline measurement, treatment decision, treatment start, follow-up, and outcome measurement. A variable is not a pretreatment covariate merely because it appears in a database under a familiar label. Its measurement date and relationship to the actual treatment decision matter. For instance, a laboratory result recorded after treatment starts may reflect treatment response, even if the same laboratory test is usually part of baseline assessment.
For each plausible variable, ask three questions: Was it determined before treatment? Could it influence treatment selection? Could it influence the outcome, apart from treatment? Variables with plausible answers of yes to all three are strong candidate confounders. Also record plausible causes of the outcome that influence treatment selection indirectly through a clinician’s judgment or a patient’s preferences, even if they are difficult to measure.
Describe who selected treatment, when selection occurred, and what information or preferences shaped it.
Separate information fixed or measured before treatment from consequences of treatment and from factors that determine inclusion or outcome observation.
For baseline variables, consider whether they influence both treatment and outcome. For later variables, ask whether treatment can affect them or whether conditioning on them could create selection bias.
Identify unmeasured causes, limited treatment overlap, measurement problems, and any gap between the study sample and the causal comparison being discussed.
Worked Example: Treatment Selection in a Diabetes Program
Consider a hypothetical health system comparing adults who enrolled in an intensive diabetes-management program with adults receiving usual care. The outcome is hospitalization within 12 months. Clinicians recommend the program partly based on recent glucose measurements and prior hospitalizations. Patients also choose whether to enroll; transportation access and willingness to attend frequent visits influence that choice.
Treatment-selection process. Treatment was not assigned by chance. Clinician recommendations use clinical history, and enrollment depends partly on patient choice and access. These mechanisms create plausible differences between the groups before treatment begins.
Candidate pretreatment confounders. Baseline glucose level and prior hospitalization are candidates because they may influence a clinician’s recommendation and predict future hospitalization. Transportation access may affect enrollment and, through access to routine care, affect hospitalization. Baseline disease severity and relevant comorbidities are also candidates if they influence either the treatment decision or outcome. These are candidates, not automatically proven confounders; their timing and causal relevance need confirmation.
Post-treatment and selection variables. Attendance at program visits happens after enrollment. It may be affected by the program and by patient circumstances, so it should not automatically be included as a baseline adjustment variable. Whether a 12-month outcome is recorded may also depend on continued enrollment in the health system. If follow-up availability is related to both treatment and health, restricting analysis to patients with observed outcomes can create selection concerns.
The comparison becomes more credible if the baseline clinical history is measured before enrollment, captures the information used in treatment decisions, and is measured comparably in both groups. It remains vulnerable if motivation, access, or clinical judgment is poorly recorded. Adjustment can address measured baseline differences only under assumptions; it cannot establish that every relevant selection factor has been captured.
What a Covariate-Adjusted Contrast Assumes
The role of baseline adjustment can be seen by writing the observed outcome mean among treated units as a mixture over their covariate values. Even if the outcome comparison is calculated within covariate groups, the treated and untreated groups may contain different proportions of those groups. A standardized contrast compares treatment-specific outcome means using the same distribution of \(X\).
For discrete \(X\), this expression is a covariate-standardized contrast. If consistency holds, conditional exchangeability holds given \(X\), and positivity holds for the covariate values included, then \(E(Y\mid D=d,X=x)=E\{Y(d)\mid X=x\}\). Substituting those equalities shows that the standardized contrast compares the two potential-outcome means over the same covariate distribution. Without those assumptions, standardization is still a descriptive adjustment, but the causal interpretation is not secured.
The formula also clarifies why “we adjusted for several variables” is not a complete credibility argument. The chosen variables need to be measured before treatment and sufficiently capture the common causes relevant to treatment and outcome. The treatment groups also need to include both options across the covariate patterns used in the comparison. A model cannot recover a comparison for a subgroup in which, for example, everyone receives treatment and no one receives usual care without relying on extrapolation.
Worked Example: How Baseline Severity Changes a Comparison
Suppose a hypothetical cohort has 200 patients in each group. “Severe” and “mild” describe baseline disease severity. The table gives the number hospitalized within 12 months.
| Baseline severity | Program: hospitalized / total | Usual care: hospitalized / total |
|---|---|---|
| Severe | 20 / 100 | 18 / 60 |
| Mild | 6 / 100 | 18 / 140 |
The crude program risk is \(26/200=0.130\), or 13.0%. The crude usual-care risk is \(36/200=0.180\), or 18.0%. Thus the crude risk difference, program minus usual care, is \(0.130-0.180=-0.050\), a 5.0 percentage-point lower observed hospitalization risk in the program group.
Within the severe group, the risks are \(20/100=0.200\) and \(18/60=0.300\), giving a difference of \(-0.100\). Within the mild group, the risks are \(6/100=0.060\) and \(18/140\approx0.1286\), giving a difference of approximately \(-0.0686\). Both within-severity differences favor the program in this hypothetical data.
For illustration, standardize to the combined cohort’s severity distribution: \(160/400=0.40\) severe and \(240/400=0.60\) mild. The standardized program risk is \(0.40(0.200)+0.60(0.060)=0.116\). The standardized usual-care risk is \(0.40(0.300)+0.60(0.1286)\approx0.1971\). The standardized difference is \(0.116-0.1971=-0.0811\), or about 8.11 percentage points lower.
Here, the treatment groups have different severity distributions: half the program group is severe, compared with 30% of the usual-care group. Standardization changes the descriptive comparison by giving both groups the same severity mix. It supports a causal interpretation only if severity is measured adequately, the relevant exchangeability and positivity conditions hold, and the other assumptions are appropriate. The arithmetic alone cannot verify those conditions.
Post-Treatment Variables, Selection, and Other Failure Modes
A common diagnostic error is to adjust for every variable that predicts the outcome. Prediction alone does not make a variable appropriate for adjustment. If treatment changes a variable, and that variable then affects the outcome, it may be a mediator. Adjusting for it changes the question by removing part of the treatment’s pathway. If a variable is caused by treatment and by another cause of the outcome, conditioning on it can also introduce bias. The earlier material on mediators and collider adjustment explains these mechanisms; in a study review, the key step is recognizing the timing and plausible causes before fitting a model.
Selection deserves separate attention. Suppose a comparison includes only people who completed follow-up. Completion may depend on treatment burden and on worsening health. Among completers, treatment and health can then become associated through the shared requirement of completion, even if the original comparison did not have that association. Ask who was excluded, why, and whether outcome observation differs between treatment groups. Also distinguish missing outcome values from genuine absence of the outcome; the two may need different interpretations.
Worked Example: Why Visit Attendance Is Not a Baseline Confounder
In a hypothetical rehabilitation comparison, patients choose an exercise program or usual care. At baseline, pain severity and prior mobility are recorded. Program participants are then offered weekly visits. Attendance is recorded after treatment begins, and the outcome is walking ability at six months.
Baseline pain and mobility may affect both the decision to enroll and later walking ability, so they are plausible pretreatment confounders. Weekly attendance occurs after program assignment. The program can affect attendance by offering visits; baseline mobility, motivation, or worsening symptoms may also affect attendance and the outcome.
Therefore, treating attendance as if it were a baseline confounder is not justified. Adjusting for it could remove part of the program’s effect operating through participation, or create bias if attendance is influenced by treatment and other causes of the outcome. A sound diagnostic report would classify attendance as post-treatment, describe its possible causal role, and avoid claiming that simple adjustment for it solves confounding.
Other warning signs include inconsistent definitions of treatment, covariates measured at different times across groups, and proxies that do not capture the factors clinicians actually used. Check whether all important baseline characteristics are available before treatment and whether their measurement process is comparable. Examine whether both treatment options occur across relevant covariate patterns. Severe lack of overlap makes an adjusted result depend heavily on modeling assumptions rather than direct comparisons.
Comparing Designs and Stating Limits
| Approach | What it can contribute | What it does not establish by itself |
|---|---|---|
| Randomized assignment | Creates treatment groups by a chance mechanism, supporting baseline comparability on average. | Does not guarantee identical groups in a finite sample, correct outcome measurement, or complete follow-up. |
| Observational comparison with baseline adjustment | Can account for measured pretreatment differences when the covariates and model are appropriate. | Does not remove unmeasured confounding or prove exchangeability and positivity. |
| Unadjusted observational comparison | Describes the observed outcome difference between the groups as assembled. | Does not separate treatment effects from differences that influenced treatment selection. |
A credible report should say what is known about treatment assignment, why each adjusted variable is included, and which threats remain. It should not turn “the groups look similar on recorded covariates” into “the groups are exchangeable.” Similarity on measured variables is useful evidence about those variables; it says little directly about unmeasured motivation, undocumented clinical judgment, or other omitted causes.
Nor does adjustment automatically make an observational study equivalent to a randomized trial. Randomization is a design feature; observational adjustment is an analysis relying on measured information and assumptions. The right conclusion is calibrated: state the observed or adjusted contrast, describe why it may or may not represent a causal effect, and name the most consequential remaining uncertainties.
Common Mistakes
- Starting with the model instead of the assignment process. A complex adjustment cannot explain why treatment groups differed. First describe the clinical or behavioral decision that produced treatment receipt.
- Adjusting for every available variable. Post-treatment variables and selection variables can introduce bias or change the causal question. Classify variables by timing and role before adjustment.
- Treating covariate balance as proof. Balance on recorded covariates does not demonstrate balance on unmeasured causes or establish conditional exchangeability.
- Ignoring limited overlap. If treatment is nearly determined by a covariate pattern, the data offer little direct comparison there; a fitted model may be extrapolating.
- Calling an association a treatment effect. The adjusted contrast is causal only under assumptions such as consistency, conditional exchangeability, and positivity, plus an appropriate analysis.
Check Your Understanding
For each question, focus on the study process before deciding what adjustment can support.
- A hypothetical study compares patients who accept a new smoking-cessation program with those who decline. The groups differ in baseline nicotine dependence, and dependence predicts quitting. Why is dependence a candidate confounder?
- In the diabetes example, researchers adjust for weekly program attendance. Give one reason this may be inappropriate.
- A study reports similar measured baseline averages in the treatment groups. Does this establish conditional exchangeability? Explain.
- Suppose no patients with a particular severe baseline condition receive usual care. What assumption is threatened for a comparison that includes this condition, and why?
- A cohort comparison finds a crude risk of 10% under treatment and 15% under usual care. What can be concluded from this difference without further assumptions?
Solutions. (1) Dependence is measured before treatment, may affect acceptance of the program, and predicts quitting; it is therefore a plausible common cause of treatment receipt and outcome. (2) Attendance occurs after treatment begins and may be affected by the program and by health or motivation. Adjusting for it may block a treatment pathway or create bias, so it is not an ordinary baseline confounder. (3) No. Similarity on measured variables does not establish that all common causes were measured or that potential outcomes are conditionally independent of treatment. (4) Positivity is violated for that condition: there is no usual-care comparison in that covariate group, so the treatment contrast there cannot be estimated directly from observed units. (5) The observed risk difference is \(0.10-0.15=-0.05\), or 5 percentage points lower in the treated group. Without further assumptions, this is a descriptive association, not necessarily a treatment effect.