Why Model-Choice Reporting Matters
A propensity-score analysis depends on decisions about which pretreatment variables enter the model and how they are represented. “We estimated a propensity score using logistic regression” does not tell a reader whether age was entered linearly, whether disease severity was grouped, or whether interactions were included. Those details can change the estimated scores and the subsequent analysis. A clear report therefore makes the specification understandable and, where possible, reproducible.
The aim here is to report those choices—not to claim that a particular specification has succeeded in controlling confounding. Model specification is one part of an observational comparison. It does not, by itself, show that measured covariates are balanced after the planned analysis, that treated and comparator participants are comparable in the relevant population, or that unmeasured confounding is absent. These are separate questions requiring separate evidence.
What a Reproducible Specification Includes
Begin by identifying the outcome of the model: treatment assignment, not the clinical outcome. Name the treatment and comparator as defined in the study, identify the analysis population used to fit the model, and state which covariates were included. For each variable, say enough to recover its model representation. For example, report age in years as a continuous term, rather than merely listing “age”; for a categorical variable, give the categories and reference level. If a continuous variable was categorized, provide the cut points and their rationale.
Also report any nonlinear terms, interactions, or transformations. “Age and severity” is not equivalent to a model with a quadratic age term and an age-by-severity interaction. If variables were centered or scaled, state how. If there were indicator variables, state their coding. Include the model equation or a compact table when that is clearer than prose.
Describe estimation: for example, an unpenalized logistic regression fitted by maximum likelihood. If the procedure used a penalty, a correction for separation, variable selection, or a flexible prediction algorithm, name it and report the important settings. Give the software and version, and identify relevant packages or procedures. These details matter because different estimation procedures, defaults, and tuning choices may produce different fitted scores from the same covariates.
Finally, explain how the specification was chosen. Distinguish prespecified terms from terms added after examining the data. If data-driven choices were made, describe what variables or terms were considered, what criterion was used, and whether the treatment-assignment model was selected without using outcome information. Avoid vague phrases such as “the best-fitting model” without saying what “best” meant.
State that the model predicts treatment assignment from measured pretreatment covariates, and identify the treatment, comparator, and population used for fitting.
List variables, coding, reference levels, transformations, nonlinear terms, and interactions. An equation can make dependencies explicit.
Name the estimator or algorithm, relevant settings, software, version, and packages or procedures.
Separate prespecified choices from data-driven changes and give the selection criterion and information used.
Worked Specifications and Score Calculations
Worked Example: Reporting a Linear Logistic Model
Suppose a hypothetical cohort compares a new treatment with usual care. The fitted model uses age in years, centered at 60, an indicator for high baseline severity, and an indicator for prior treatment use. Suppose maximum-likelihood logistic regression gives the following illustrative fitted equation:
Here, \(I(\cdot)\) equals 1 when the condition is true and 0 otherwise, and \(\operatorname{logit}(p)=\log\{p/(1-p)\}\). Consider a hypothetical participant aged 65 with high severity and no prior treatment use. The linear predictor is
Converting from the logit scale gives
The fitted model assigns this participant an estimated treatment probability of about 0.426. A transparent report would state the age centering, severity and prior-use definitions, the equation, the estimation method, and the software. The illustrative coefficient values do not represent results from a real study, and the estimated probability is not evidence that this participant is adequately adjusted against any particular comparator.
Worked Example: Making Nonlinearity and Interaction Explicit
In a second hypothetical analysis, investigators prespecify a quadratic age term and an interaction between high severity and female sex. The fitted model is
For a hypothetical 60-year-old woman with high severity, the centered age value is \(60-50=10\). The age terms contribute \(0.030(10)-0.0005(10)^2=0.30-0.05=0.25\). The full linear predictor is
The interaction contributes 0.40 because both indicators equal 1. For a woman without high severity, that product would be 0; for a man with high severity, it would also be 0. A report that lists only age, severity, and sex would therefore omit a defining part of this model. It should state that age was centered at 50 and entered quadratically, and that the severity-by-sex product term was included. The terms describe the treatment-assignment model; they are not treatment-effect estimates.
Why the Exact Form Matters
The fitted probability is obtained by applying the inverse-logit function to a linear predictor. Changing a term changes that predictor for participants whose covariate values activate it, and therefore can change their fitted probabilities. For example, adding a product term with coefficient \(\gamma\) changes the linear predictor by \(\gamma x_1x_2\); if both indicators equal 1, the change is \(\gamma\), while if either equals 0, the change is 0. Because the inverse-logit is increasing, a positive change in the linear predictor increases the fitted probability.
This justifies why “included an interaction” is not a cosmetic reporting detail: it can change the assigned scores. It also explains why the same variable names can conceal materially different specifications. Centering a linear term changes the intercept but not the fitted values if the model contains the corresponding intercept; centering in a model with higher-order terms also affects how those terms are expressed, so the centering point should still be reported.
Worked Example: Reporting a Data-Driven Choice
Suppose investigators of a hypothetical study prespecified age, baseline severity, and prior treatment as covariates. Before fitting the final model, they compared that model with a version adding a quadratic age term and a severity-by-prior-treatment interaction. They report that the comparison used five-fold cross-validated log loss for treatment assignment, with lower values preferred; the simpler model had a mean loss of 0.612 and the expanded model 0.587. They selected the expanded model. These values are hypothetical.
A reproducible account would state that the three main effects were prespecified, identify the two additional terms, define the folds and the loss criterion sufficiently to reproduce the comparison, and say whether the comparison used treatment assignment only or also considered outcome information. It would name the final estimation procedure, software, version, and any settings that affect the result. If the folds were randomly assigned, it would report the random seed or other reproducible fold-generation procedure.
The arithmetic comparison is \(0.587<0.612\), so the expanded model had the lower reported cross-validated log loss under this criterion. That supports only the stated predictive-model selection decision. It does not establish that the resulting propensity scores achieve adequate covariate balance, resolve limited overlap, or control confounding. Those require distinct assessment and cannot be inferred from the selected model or its loss value.
Estimation Choices and Alternatives
A conventional logistic regression is a common choice because its equation, covariate terms, and fitted probabilities can be described directly. Its use does not mean that every covariate must enter only as a straight-line term: nonlinear terms and interactions can be included and should be explicitly named. If the model uses penalization or another method to address estimation difficulties, report the penalty or method and its settings rather than describing it simply as logistic regression.
More flexible approaches can represent complex patterns without specifying every term in a single linear predictor. They may require reporting additional choices, such as the algorithm, tuning procedure, and settings used to generate predictions. A prespecified logistic specification and a flexible learner are alternatives for estimating treatment probabilities; neither should be described as successful adjustment solely because it was used. The central reporting standard is to make the actual procedure and its data-dependent choices clear.
| Approach | What to report | What the description does not establish |
|---|---|---|
| Logistic regression | Covariates, coding, functional forms, interactions, estimator, and software | That the covariates are balanced in the final comparison |
| Penalized or corrected regression | Penalty or correction, tuning or settings, covariate representation, and implementation | That the chosen penalty guarantees adequate confounding control |
| Flexible prediction algorithm | Algorithm, candidate inputs, tuning or validation procedure, and software details | That predictive performance alone demonstrates causal validity |
Diagnostics, Limitations, and Common Mistakes
A report can be incomplete even when the model fitted successfully. Warning signs include undefined categories, unexplained variable selection, a claim of “nonlinear adjustment” without naming the terms, or software details too vague to identify the implementation. If estimation encountered separation, convergence problems, or other instability, disclose the issue and the action taken. Do not silently remove terms or switch methods without reporting the change.
The model is based on measured covariates and on the treatment and covariate definitions actually used. A correctly described formula cannot repair a covariate measured after treatment initiation when the intended adjustment requires pretreatment information, nor can it guarantee that important confounders were measured. Causal interpretation also depends on assumptions such as conditional exchangeability given the measured covariates, positivity for the treatment strategies in the target population, and consistency. A model specification is not proof that these assumptions hold.
A frequent mistake is to write, “We included all clinically relevant variables in the propensity-score model; therefore confounding was controlled.” The first clause reports a rationale for selection, not evidence of the result. The correction is to report the variables and exact model forms, explain why they were chosen, and keep that description separate from assessments of the resulting analysis. Likewise, a higher treatment-prediction score or lower prediction loss should not be presented as proof of better causal adjustment.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A report says, “The propensity score was estimated by logistic regression using age and severity.” Name three additional details that would improve reproducibility.
- In a model with \(I(\text{severe})\) and \(I(\text{female})\), what does including their product term mean for the linear predictor when both indicators equal 1?
- A model has cross-validated log loss 0.48, while a comparison model has 0.52. What conclusion is supported, and what conclusion is not supported?
- Why should a report identify the reference category for a categorical covariate?
- Does a detailed account of a propensity-score specification establish conditional exchangeability? Explain briefly.
Solutions. (1) Report the variable definitions and coding, age’s functional form and units, the severity categories and reference level, any interactions or transformations, the estimation procedure, and software details; any three of these are improvements. (2) The product equals \(1\times1=1\), so its coefficient is added to the linear predictor for participants in both categories. (3) The first model had lower reported cross-validated log loss under the stated comparison; this does not establish adequate adjustment or causal validity. (4) The reference category determines how indicator coefficients and the model coding are interpreted and helps reconstruct the design matrix. (5) No. Conditional exchangeability is a substantive assumption about treatment assignment given measured covariates; documenting the model does not prove that assumption.