Why EHR HDPS Analysis Needs Extra Care
Electronic health records (EHRs) can contain many diagnosis, medication, and health-care-utilization variables. Their detail is useful for confounding adjustment, but the records are generated by care and documentation processes, not by complete surveillance of every patient’s health. A missing diagnosis code may mean that a condition was absent, but it may also mean that the patient was not observed, did not seek care, or received care elsewhere. A high-dimensional propensity score (HDPS) analysis must therefore address both which features are selected and how reliably those features represent the baseline health of each person.
The analysis below considers initiation of a new medication at an index date. It uses diagnoses, medication records, and utilization recorded before that date to estimate the probability of treatment and match treated patients to comparable untreated patients. All data and results are hypothetical.
A clear target trial-like setup helps keep the analysis coherent. Specify eligibility, treatment strategies, the index date, the outcome, and the baseline lookback period before constructing features. For example, eligibility might require continuous enrollment or a minimum period of observable EHR activity before initiation. The comparison group must have a corresponding index date, defined by a comparable rule. Otherwise, treated patients may have a richer or longer record simply because their index dates were chosen differently.
Assumptions and EHR-Specific Feature Construction
A causal interpretation of an adjusted treatment contrast requires more than a well-fitting treatment model. The key assumptions are consistency (the observed outcome under the received strategy equals the corresponding potential outcome), conditional exchangeability (given measured baseline covariates, treatment assignment is independent of the potential outcomes), and positivity (each treatment strategy has positive probability for covariate profiles in the target population). The analysis also requires a well-defined time ordering, no interference between individuals, and sufficiently valid measurement of treatment, covariates, and outcomes.
In EHR data, these assumptions require attention to the observation process. The baseline window should end before treatment initiation. Define whether a feature means “ever recorded in the window,” “recorded on at least two dates,” or a count or rate. If observation length varies, a raw count can encode time at risk of being recorded rather than health status. Available observation time, visit frequency, and data-system participation may therefore need to be represented, restricted, or handled in a sensitivity analysis. Adjusting for utilization can help capture differences in documentation opportunity, but utilization itself may reflect illness severity and access to care; its causal role should be considered rather than assumed.
A code’s absence should not automatically be interpreted as evidence of absence of disease. For each patient and feature, distinguish at least conceptually among “not recorded during observable care,” “not observable during the window,” and “recorded.” Depending on the data structure, this can mean requiring adequate baseline observation, representing availability separately, or defining a denominator such as observed days. Feature definitions should also avoid information recorded after treatment starts, including diagnoses that could be consequences of treatment or early manifestations of the outcome.
Worked Example: Comparing Features Under Irregular Observation
Suppose two hypothetical patients have a 180-day calendar lookback before the index date. Patient A has 180 days of observable records and six visits. Patient B has only 90 observable days and three visits. A diagnosis appears once for A and not at all for B.
The crude visit rates are \(6/180=0.0333\) visits per calendar day for A and \(3/180=0.0167\) for B. But these rates treat B’s unobserved 90 days as if records could have been generated then. Using observed days as the denominator gives \(6/180=0.0333\) visits per observed day for A and \(3/90=0.0333\) for B. Thus, the apparent difference in crude visit rates is entirely explained by the unequal observed time in this simplified example.
The single diagnosis code for A is evidence that it was recorded for A; the missing code for B is not enough to establish that B did not have the condition. A defensible analysis would set and apply an observation rule, then consider whether observation time or visit intensity should be represented in the covariates. Dividing by observed time does not by itself solve all measurement problems: a code count still depends on clinical practice, care-seeking, and coding. The example shows why observation opportunity must be examined before “no code” becomes a clinical category.
Prioritizing Candidate Features
An EHR may generate thousands of candidate features, such as diagnosis indicators, medication indicators, and utilization summaries. The analyst should define candidate domains, coding rules, and the prioritization procedure before fitting the final propensity model. A common HDPS strategy uses how prevalent a feature is and how strongly it is associated with treatment and outcome to prioritize potential confounders. The exact implementation varies; any chosen score is a screening device, not a proof that the retained features are the correct adjustment set.
For a transparent illustration, define \(p_j\) as the prevalence of binary feature \(j\), \(RR_{T,j}\) as its observed risk ratio for treatment, and \(RR_{Y,j}\) as its observed risk ratio for the outcome. Use the following prioritization score:
This particular product is a declared heuristic for the example; it should not be confused with a uniquely required HDPS formula. It gives higher priority to features that are reasonably prevalent and associated with both treatment and outcome. It does not establish that a feature is a confounder, and estimates for rare features can be unstable. The feature-selection stage must also respect causal structure: a proxy for a collider or a variable affected by prior treatment may introduce bias. Use baseline timing and subject-matter knowledge to constrain automated selection.
Worked Example: Ranking Three Hypothetical EHR Features
In a hypothetical eligible cohort, consider three binary features. The displayed risk ratios are assumed to have been calculated from baseline features and the specified treatment and outcome definitions. Apply \(B_j=p_j|\log(RR_{T,j})||\log(RR_{Y,j})|\).
| Feature | \(p_j\) | \(RR_{T,j}\) | \(RR_{Y,j}\) |
|---|---|---|---|
| Chronic respiratory diagnosis | 0.20 | 1.8 | 2.0 |
| High prior utilization | 0.40 | 1.2 | 2.5 |
| Rare medication marker | 0.08 | 2.5 | 1.1 |
For chronic respiratory diagnosis, \(B_1=0.20|\log(1.8)||\log(2.0)|=0.20(0.5878)(0.6931)=0.0815\). For high prior utilization, \(B_2=0.40|\log(1.2)||\log(2.5)|=0.40(0.1823)(0.9163)=0.0668\). For the rare medication marker, \(B_3=0.08|\log(2.5)||\log(1.1)|=0.08(0.9163)(0.0953)=0.00699\).
The ranking is chronic respiratory diagnosis, high prior utilization, then rare medication marker. This ranking is not a causal conclusion: it only applies the stated score to the stated quantities. In particular, the small score for the rare marker may reflect its weak observed outcome association, noisy estimates, or both. Clinical knowledge and the prespecified causal structure remain relevant when deciding whether and how to retain features.
Fit the Score, Match, and Diagnose Balance
After feature construction and selection, estimate the propensity score using a model appropriate to the treatment and design. For a binary treatment, a logistic model can be written as
where \(S\) is the selected feature set and \(X_{ij}\) is patient \(i\)’s value for feature \(j\). This model estimates treatment probabilities; it is not an outcome model. A good prediction metric alone does not demonstrate adequate confounding control. Match treated and untreated patients using the estimated scores and a specified matching rule, then assess overlap and balance in the matched sample. The earlier material on overlap and balance diagnostics gives the general principles; here the challenge is to apply them across many EHR features and domains.
For a continuous covariate, one common balance measure is the standardized mean difference (SMD), the difference in group means divided by a pooled standard deviation. For a binary feature, use the difference in proportions divided by the corresponding pooled standard deviation. An SMD near zero indicates similar measured distributions on that feature; a threshold such as 0.1 is a diagnostic convention, not a guarantee of no confounding. Examine individual features as well as summaries by diagnosis, medication, and utilization blocks. A block average can conceal a badly imbalanced feature.
The SMD calculation follows directly from standardizing the observed mean difference. If \(s_1\) and \(s_0\) are the group standard deviations, the pooled standard deviation used here is \(s_p=\sqrt{(s_1^2+s_0^2)/2}\). Dividing the mean difference by \(s_p\) removes measurement units, allowing descriptive comparisons across features measured on different scales. It does not turn the SMD into a hypothesis test or adjust for unmeasured variables.
Worked Example: Checking Balance and Interpreting a Matched Contrast
Suppose a hypothetical EHR cohort has 400 initiators and 600 comparators. After applying a prespecified score and matching rule, 160 pairs are retained. Before matching, mean age is 68 years among initiators and 64 years among comparators, with \(s_1=s_0=10\) years. The age SMD is \( (68-64)/\sqrt{(10^2+10^2)/2}=4/10=0.400\). After matching, mean ages are 67.1 and 66.8 years, with a pooled standard deviation of 9.5 years. The SMD is \( (67.1-66.8)/9.5=0.0316\). The measured age difference is substantially smaller after matching.
For a high-utilization indicator, the pre-match proportions are 0.60 and 0.35. Its pooled standard deviation is \(\sqrt{\{0.60(0.40)+0.35(0.65)\}/2}=\sqrt{0.23375}=0.4835\), so its SMD is \((0.60-0.35)/0.4835=0.517\). After matching, suppose proportions are 0.51 and 0.49. The pooled standard deviation is \(\sqrt{\{0.51(0.49)+0.49(0.51)\}/2}=0.500\), and the SMD is \(0.02/0.500=0.0400\). This is an improvement on this feature, but it does not establish balance on all utilization measures or all candidate covariates.
Suppose an outcome is recorded for every patient in the 160 pairs. There are 29 outcome events among initiators and 43 among comparators, giving risks \(29/160=0.1813\) and \(43/160=0.2688\). The matched risk difference, initiators minus comparators, is \(0.1813-0.2688=-0.0875\), or 8.75 percentage points lower in the initiator group. This is an observed matched contrast, not automatically a causal effect. A causal interpretation depends on the assumptions above, including exchangeability for the matched target population, adequate measurement, and correct handling of the matched design in uncertainty estimation.
Failure Modes, Alternatives, and Limits
A useful diagnostic review should look beyond the final propensity-score model. Check that baseline windows and index dates are comparable, that feature prevalence and missingness are plausible, and that diagnosis, medication, and utilization features are temporally eligible. Review the score distributions and how many treated patients are discarded by matching; poor overlap changes the population to which the result applies. Inspect the SMDs of selected and clinically important unselected features, feature-block summaries, and tails or rare categories. Revisit coding and observation definitions if balance appears to be driven mostly by visit intensity.
- Unequal observation treated as disease absence. A patient with little observable care can have fewer recorded diagnoses without being healthier. Apply the observation rule consistently and assess observation opportunity.
- Good treatment prediction mistaken for good adjustment. A model can distinguish treatment groups well and still leave important covariates imbalanced. Diagnose covariate balance directly.
- One summary hiding a problem. Mean SMDs can look small while a rare or clinically important feature remains imbalanced. Review feature-level results and blocks.
- Automated selection overriding causal timing. A feature recorded after initiation or caused by treatment is not a baseline confounder. Enforce time ordering before selection.
- Discarding patients without reconsidering the estimand. Matching may retain only a subset with adequate overlap. Describe the population represented by retained pairs rather than generalizing automatically to all eligible patients.
HDPS is not the only approach. Prespecified clinical covariate adjustment can prioritize a smaller set based on substantive knowledge, but may omit useful record-based proxies. A conventional claims-based HDPS uses claims-derived features; EHR analyses must additionally confront variable observation and documentation. Flexible propensity learners can represent nonlinearities and interactions, but still require careful feature timing, overlap assessment, and balance diagnostics. These approaches are not interchangeable guarantees: each relies on measured information and a defensible design.
The central limitation is that a large number of codes cannot compensate for systematic unmeasured confounding, poor treatment overlap, or misleading measurement. EHR records may be rich but selectively observed; coding practices may differ across clinicians and settings; and a feature’s statistical association does not establish its causal role. The strongest analysis combines a prespecified design, transparent feature definitions, appropriate treatment of observation opportunity, and direct high-dimensional balance checks.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A patient has no diagnosis code in a baseline window with only 20 observable days. Why is it unsafe to classify the patient as having no condition?
- For a binary feature, treatment risk ratio is 1.5, outcome risk ratio is 2.0, and prevalence is 0.10. Calculate \(B_j\) using the stated heuristic.
- After matching, the mean SMD across 2,000 features is 0.04, but one important diagnosis feature has SMD 0.24. Is the analysis adequately balanced on the diagnosis feature?
- A matched analysis estimates an initiator-minus-comparator risk difference of \(-0.05\). State a correct interpretation and one condition required for a causal interpretation.
Solutions. (1) The patient had limited opportunity for a diagnosis to be recorded, so absence of a code is not reliable evidence of absence of disease. Use the observation rule and consider observation opportunity. (2) \(B_j=0.10|\log(1.5)||\log(2.0)|=0.10(0.4055)(0.6931)=0.0281\). This is a prioritization score, not a causal effect. (3) No. The overall mean conceals substantial residual imbalance in that feature; inspect and address it or explain why it cannot be resolved. (4) The observed matched risk of the outcome was 5 percentage points lower among initiators than comparators. A causal interpretation additionally requires, among other conditions, conditional exchangeability given measured baseline covariates and adequate overlap in the target population.