Reporting High-Dimensional Propensity Analyses

Why Reporting an HDPS Analysis Requires More Than Naming the Method

A high-dimensional propensity score (HDPS) analysis turns a large collection of recorded variables into candidate features, selects or prioritizes some of them, estimates propensity scores, and uses those scores in a causal analysis. “We used HDPS” does not tell a reader which records generated the features, what counted as a feature, how selection worked, or whether the resulting comparison had adequate overlap and balance.

Reporting should let a knowledgeable reader reconstruct the important analytic decisions and assess their consequences. It cannot, by itself, establish that the design is free of bias. In particular, a long feature list does not prove that all important confounders were measured, and good measured-covariate balance does not demonstrate the absence of unmeasured confounding.

The analysis still relies on the usual causal conditions for the stated estimand: a well-defined treatment and outcome, consistency, no unmeasured confounding conditional on the measured covariates used, and positivity in the target population. Feature selection does not guarantee these conditions. Covariates used for adjustment must be measured before treatment assignment for the intended comparison; conditioning on post-treatment variables or colliders can introduce bias. The report should say how feature timing and causal structure were handled, not imply that automation resolved those issues.

Describe the Data-to-Feature Pipeline

A reader needs to know where the candidate information came from and how raw records became analysis variables. Identify each data source by its relevant content, population, and observation period. State how sources were linked and how eligibility, exclusions, and the final analysis sample were determined. Give the dates or relative time windows that define the data available before treatment assignment. Do not rely on a database name alone: two analyses using the same source can construct materially different cohorts and covariates.

For every feature block, describe the underlying record type and the generation rules. For diagnosis, procedure, medication, or other codes, report the coding system and version, code grouping or hierarchy, and any minimum frequency threshold. Explain whether a feature indicates any occurrence, a count, a timing category, or another summary. State how missing, invalid, or duplicate records were handled. Explain the lookback window and whether features required one or multiple occurrences. These choices affect which signals can enter the selection step.

Report the number of candidates at each stage: raw variables or codes considered, features generated, features removed by prespecified filters, and features passed to selection. If a block was capped or grouped, state the rule. A concise flow diagram or table can make this funnel auditable.

The feature-selection description should identify the selection algorithm and its inputs, ranking or screening criteria, number of features retained, and any restrictions based on causal structure. Report how prognostic variables or instrumental features were treated if those considerations affected selection. State explicitly whether outcome data entered any selection or tuning step. An outcome-informed procedure can change the interpretation of downstream uncertainty, so it should never be obscured by the general label “automated selection.”

For the propensity model, give the model class, included terms, any transformations or interactions, regularization, and the method used to choose tuning parameters. If cross-fitting or sample splitting was used, report the fold construction and how predictions were combined. Include software and package versions, relevant defaults that were overridden, random seeds where applicable, and code or implementation details sufficient to reproduce the fit. Reporting the selected feature count without these choices is not enough to reproduce the analysis.

Make Overlap and Balance Inspectable

Show the propensity-score distributions for the treatment groups before and after any restriction or matching, and state the score range or other overlap rule used. Report the number and characteristics of people outside the retained region and how their exclusion changes the target population. A plot can reveal separation or sparse regions that a single summary obscures. If weights were used, describe their construction and report diagnostics such as the effective sample size alongside the nominal sample size.

For balance, provide a table or plot that allows readers to inspect covariates or feature blocks before and after adjustment. Standardized mean differences (SMDs) are useful because they express differences relative to a scale rather than in the original units. Report their direction and magnitude, how they were calculated, and any threshold used as a diagnostic. A threshold is a convention, not a guarantee of no confounding. With thousands of features, summarize the distribution of imbalance while also identifying important residual imbalances and examining blocks that may hide poorly balanced individual features.

A short justification clarifies why the scale matters. The numerator is the difference in group proportions; the denominator is a pooled scale based on the two Bernoulli variances. Thus the SMD expresses the difference relative to within-group variability, making it less dependent on sample size than a significance test of equal proportions. It remains a diagnostic summary, not a test that proves exchangeability.

Report Sensitivity Analyses as Comparisons

A useful sensitivity analysis states the concern, the changed choice, and the resulting estimate. For HDPS, relevant variations may include coding granularity, lookback-window length, feature-frequency thresholds, the number of selected features, or the propensity-model tuning rule. Explain which choices were prespecified and which were explored after seeing results. Do not describe a set of analyses as “robust” without showing how their results compare and whether they target the same population and estimand.

Common Reporting Failures and Alternatives

• Reporting only the algorithm’s name. “HDPS was applied” leaves feature construction, filtering, ranking, and tuning unspecified. Give the rules and counts at each stage.
• Reporting only the final selected features. A list does not show what was eligible or excluded. Summarize all feature blocks and the selection funnel, and make code or code lists accessible where possible.
• Using a balance threshold as proof. A small maximum SMD among reported features is not evidence that unmeasured confounding is absent. Show the diagnostics and state their limits.
• Calling a different model a sensitivity analysis without explaining the change. Identify the changed decision and disclose any resulting change in sample, overlap, or target population.
• Ignoring non-automated alternatives. A prespecified covariate set can improve interpretability and reflect subject-matter knowledge; regularized or flexible learners may offer different modeling choices. Comparing these approaches is informative only when the target estimand, analysis sample, and diagnostics are made clear. The method label alone does not rank their validity.

Automated feature selection is especially limited when recorded data are incomplete or distorted by healthcare use, coding practice, and access to care. A reproducible pipeline helps expose those choices but cannot recover variables the data do not contain. Reports should therefore separate measured-data diagnostics from claims about causal identification and describe uncertainty about data quality or unmeasured confounding.