Why a Workflow Matters
A propensity-score analysis can involve several reasonable modeling choices: how to code treatment, which baseline covariates to include, and whether to use a simple or flexible model. Making these choices ad hoc can produce results that are difficult to reproduce and propensity scores that fit treatment assignment well but do not support the intended causal comparison. A deliberate workflow makes each choice explicit and checks what the fitted scores do before they are used downstream.
The target is the propensity score \(e(X_i)=P(D_i=1\mid X_i)\), where \(D_i\) indicates treatment and \(X_i\) is a vector of measured covariates. The score model estimates this probability from the observed data. Its role is not to prove that treatment assignment is unconfounded, nor to guarantee that covariates will be balanced in a particular finite sample.
For a causal interpretation, the modeling workflow sits within a larger set of assumptions. The treatment and outcome must be defined consistently; measured covariates should precede treatment; conditional exchangeability requires that, given \(X_i\), treatment assignment is independent of the potential outcomes; and positivity requires a nonzero probability of either treatment condition for covariate profiles in the target population. These assumptions are substantive, not consequences of fitting a logistic regression. The estimated score is a tool for addressing measured differences, not a remedy for unmeasured confounding or a substitute for an appropriate study design.
A Reproducible Sequence
A useful workflow keeps the causal definition, model fitting, diagnostics, and reporting distinct. The following sequence is a practical default. Some decisions may require iteration, but each iteration should have a recorded reason.
State what treatment and comparison conditions mean, which units are eligible, and the population to which the later analysis is intended to refer.
Code \(D_i=1\) for the specified treatment and \(D_i=0\) for the specified comparison condition. Check counts, missingness, dates, and any exclusions against the definition.
Use subject-matter knowledge and the causal question to choose measured pre-treatment variables relevant to treatment assignment and outcome. Do not select variables merely because they predict treatment or because their observed associations look convenient.
Use logistic regression as a transparent starting point when appropriate, with justified transformations and interactions. Consider flexible candidates if functional form may be inadequate.
Use out-of-fold predictions when comparing or tuning flexible models. Inspect fitted-score distributions, extreme values, and covariate balance under the planned use of the scores.
Document the data, variables, model, tuning, diagnostics, and decisions. Save scores at full available precision and preserve the analysis code and software details.
From a Logistic Model to Estimated Probabilities
In logistic regression, a specified linear predictor \(\eta_i=X_i^\mathsf{T}\beta\) is mapped to a probability by the inverse-logit function. The model assumes that the log odds of treatment are represented by that predictor; it does not require every covariate to enter linearly on its original scale. Transformations and interactions can be included when they are justified and estimable.
The conversion follows by exponentiating the log-odds equation: \(e(X_i)/(1-e(X_i))=\exp(X_i^\mathsf{T}\beta)\). Solving for \(e(X_i)\) gives \(e(X_i)=\exp(X_i^\mathsf{T}\beta)/(1+\exp(X_i^\mathsf{T}\beta))\). In practice, the unknown parameters are replaced by their fitted estimates \(\hat\beta\). This calculation produces model-based probabilities; whether the model is useful for the intended causal analysis still requires diagnostics.
Worked Example: Coding Treatment and Calculating a Fitted Score
Suppose a hypothetical observational data set contains 240 eligible adults comparing a new care program with usual care. The intended treatment is enrollment in the new program during a specified month. The data contain 96 enrolled adults and 144 adults receiving usual care. Define \(D_i=1\) for enrollment in the new program and \(D_i=0\) for usual care. The counts sum to \(96+144=240\), consistent with the stated sample size.
Age, smoking status, and a baseline comorbidity measure were recorded before that month. Assume, for illustration, that a logistic model using these covariates was specified as \(\eta_i=-0.50+0.04(\text{age}_i-50)+0.50(\text{smoker}_i)+0.30(\text{comorbidity}_i)\), where smoking and comorbidity are coded 1 when present and 0 otherwise. Consider a hypothetical adult aged 60 who smokes and has no recorded comorbidity. The linear predictor is \(\eta_i=-0.50+0.04(60-50)+0.50(1)+0.30(0)=-0.50+0.40+0.50=0.40\).
The fitted probability is therefore \(\hat e_i=\exp(0.40)/(1+\exp(0.40))=1.4918/2.4918\approx 0.5987\). The model assigns this profile an estimated probability of enrollment of about 59.87%. This is a fitted probability conditional on the included covariates and model specification, not a statement that this individual’s treatment was caused by those covariates or that the model has achieved adequate balance.
For this specification to be usable, the recorded covariates must genuinely be measured before treatment, the treatment coding must match the stated comparison, and the model must be estimable in the observed data. The counts and one calculated probability do not establish positivity or conditional exchangeability; those require substantive assessment and diagnostics.
Choosing and Tuning Candidate Models
Logistic regression is an interpretable starting point, but its functional form may be too restrictive. A nonlinear age association, for example, can be represented with a spline or a generalized additive model; trees and other flexible estimators provide additional alternatives. The earlier material on these model classes explains their individual mechanics. In a workflow, the central task is to specify plausible candidates and evaluate their performance for the propensity-score purpose, rather than choosing a method because it is fashionable or maximizes treatment classification accuracy.
When tuning model complexity, use a prespecified procedure and, where feasible, out-of-fold predictions: each unit’s prediction is generated by a model that was not fitted using that unit’s treatment indicator. This reduces the misleading optimism that can arise when evaluating predictions on the same observations used to fit a flexible model. Prediction metrics such as log loss can compare how well candidate models represent treatment probabilities, but they do not alone establish covariate balance or causal validity. Inspect balance diagnostics relevant to the planned downstream analysis as well.
Worked Example: Comparing Candidate Specifications
In a hypothetical data set, investigators compare logistic regression, a generalized additive model, and a random forest using the same pre-treatment covariates. They use five-fold out-of-fold predictions and compute mean log loss, with lower values indicating better probabilistic prediction. They also record the largest absolute standardized mean difference among the measured covariates after applying the planned score-based weighting, along with the range of predicted probabilities.
| Candidate | Out-of-fold log loss | Largest absolute standardized difference | Predicted-score range |
|---|---|---|---|
| Logistic regression | 0.612 | 0.18 | 0.08 to 0.91 |
| Generalized additive model | 0.574 | 0.07 | 0.05 to 0.96 |
| Random forest | 0.590 | 0.05 | 0.01 to 0.99 |
The generalized additive model has the lowest log loss and a smaller maximum standardized difference than logistic regression. The random forest has the smallest reported difference, but its predictions include values very near 0 and 1, which may signal weak overlap or unstable weighting for some units. These summaries do not establish that any candidate is universally best. The investigators should examine which covariates drive the differences, verify that the diagnostics are computed as intended, and consider whether the extreme predictions are supported by the data.
For this example, choosing the generalized additive model as the working candidate is defensible if its smoother complexity was selected without using outcome values and its diagnostics remain acceptable under closer inspection. The choice should be reported as a reasoned modeling decision, not as proof that all measured confounding has been removed. If no candidate gives adequate diagnostics, the appropriate response is to investigate the specification and target population, not simply to select the most complex model.
Diagnostics, Failure Modes, and Limits
Diagnostics should ask whether the score model is useful for the next step, not merely whether it predicts \(D_i\). Plot or summarize estimated scores separately for treated and comparison units. Extensive separation, clusters of scores near 0 or 1, or a lack of comparable score ranges can indicate limited overlap. Examine balance in the measured covariates after applying the planned score-based procedure; a good treatment-prediction metric does not ensure that the distributions of important covariates have been balanced.
If balance is poor, check coding and data construction first, then consider whether key nonlinearities or interactions are missing or whether the chosen estimator is overfit. If scores are extreme, inspect influential observations, sparse combinations of covariates, separation, and whether the target population includes profiles with little treatment support. The earlier material on separation and overfit is relevant to diagnosing these problems. Changing a model until a desired result appears, without recording decisions or acknowledging the resulting uncertainty, is not a reliable remedy.
The main alternatives differ in flexibility and transparency. A prespecified logistic model is comparatively easy to report and inspect, but can miss nonlinearities if specified too simply. Flexible methods can capture more complex patterns, but may require careful tuning and can yield unstable or extreme predictions. Model comparison should consider out-of-fold prediction and balance diagnostics together, while retaining the same treatment definition, eligible sample, and covariate set unless a documented scientific reason supports changing them.
No score-estimation workflow can guarantee exact sample balance, overlap, or causal identification. A model can be fitted successfully even when positivity is implausible, and a score that predicts treatment accurately can still be unsuitable for the target analysis. Also, variables measured after treatment should not be included as baseline adjustment covariates merely because they improve treatment prediction: they may be consequences of treatment, and adjusting for them can change or bias the causal comparison.
Common Mistakes and Good Reporting
- Reversing or blurring treatment coding. A coefficient or score is uninterpretable unless the analyst states exactly which condition is coded \(D_i=1\).
- Including post-treatment variables. Strong prediction is not a sufficient reason to adjust for a variable measured after treatment.
- Choosing by classification accuracy alone. Classification can reward threshold decisions while obscuring probability quality, overlap, or covariate balance.
- Reporting rounded scores as the only saved values. Display a sensible number of digits, but retain the full-precision fitted scores for computation and reproducibility.
- Equating a fitted score with causal validity. The model does not verify no unmeasured confounding, consistency, or positivity.
A concise report should identify the treatment and comparison conditions, the analysis sample, the timing and rationale for covariates, the model family and functional forms, the fitting and tuning procedure, and the diagnostics reviewed. It should state how extreme scores and poor balance were handled, if applicable, and distinguish decisions made before analysis from decisions made after inspecting results. These details allow another analyst to reproduce the estimated probabilities and understand what they do, and do not, support.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A model gives a unit a linear predictor of \(-0.20\). Calculate the estimated propensity score to four decimal places.
- An analyst codes \(D_i=1\) for usual care, although the stated treatment is enrollment in a new program. What must be corrected or clarified before interpreting the fitted scores?
- A flexible model has the best out-of-fold log loss but produces scores near 0 and 1 and poor covariate balance. Is it automatically the preferred model? Explain.
- A variable measured one month after treatment strongly predicts treatment status. Should it be included as a baseline propensity-score covariate? Why or why not?
Solutions. (1) \(\hat e_i=\exp(-0.20)/(1+\exp(-0.20))=0.8187/1.8187\approx 0.4502\). (2) The coding must be aligned with the stated comparison, or the report must explicitly redefine which condition is \(D_i=1\). Scores refer to the probability of the condition coded 1, so silently reversing it changes their meaning. (3) No. The analyst should investigate overlap, extreme predictions, and balance alongside the prediction metric; the best log loss alone does not make the model suitable. (4) Generally, no. A post-treatment variable is not a baseline covariate and could be affected by treatment. Its predictive strength does not justify conditioning on it as though it preceded treatment.