Sensitivity to Coding Granularity and Lookback Windows

Why Coding and Time Windows Deserve Sensitivity Checks

In high-dimensional propensity-score analyses, the covariates are not simply handed to the analyst as a fixed, unambiguous list. A diagnosis or procedure may be represented by a broad parent code or by several more specific child codes. A recorded condition may be counted if it occurred during the previous six months, two years, or another pre-treatment period. These choices affect which features exist in the analytic data, how frequently they occur, and how they relate to treatment and outcome.

Consequently, changing code granularity or the lookback window can change more than a feature ranking. It can change the fitted propensity scores, the degree of overlap between treatment groups, measured covariate balance, and the resulting effect estimate. A sensitivity analysis makes these dependencies visible. It does not establish which specification is true, nor does stability across specifications prove that confounding has been eliminated.

What Must Stay Fixed, and What May Vary

Let \(A\) denote treatment, \(Y\) the outcome, and \(X\) the measured pre-treatment covariates. The propensity score is \(e(X)=P(A=1\mid X)\). The choice of coding and lookback window changes the operational construction of \(X\), and therefore may change the fitted score. For a causal interpretation using adjustment, the relevant assumptions include consistency, conditional exchangeability given the adjusted covariates, and positivity: for covariate profiles in the target population, treatment probabilities must not be zero or one. These assumptions are substantive; comparing specifications cannot verify them.

The time origin matters. Covariates should be measured before treatment assignment or initiation, according to the design. Including a code recorded after treatment can condition on a consequence of treatment, a mediator, or a collider, and can introduce bias. A longer lookback is not automatically better: it can capture more history but also include stale information, changes in clinical status, or records that are less relevant to treatment at time zero. A shorter window may better reflect current status but miss important prior conditions.

For an interpretable comparison, keep the eligibility criteria, index date, treatment and comparator, outcome definition, follow-up, estimand, and estimation procedure fixed where possible. State each alternative feature definition explicitly. If changing a window also changes eligibility or the available follow-up, report that as a design change rather than attributing all differences to covariate construction.

Why Feature Rankings Change

An illustrative high-dimensional ranking heuristic is the product of the absolute log odds ratio for the feature-treatment association and the absolute log odds ratio for the feature-outcome association:

Here \(OR_{A,j}\) and \(OR_{Y,j}\) are odds ratios for feature \(j\) with treatment and outcome, respectively, as defined by the analysis. The log transformation expresses odds-ratio associations on a symmetric scale: an odds ratio of 2 and an odds ratio of \(1/2\) have equal absolute log magnitude. The product is a ranking device, not an effect estimate, causal criterion, or universal rule for selecting covariates. It may be sensitive to sparse cells and to how features are defined.

Aggregation can change both odds ratios. A parent code combines people with different child-code patterns; the resulting association depends on the mixture of those patterns and their frequencies. It is generally not a simple average of child-code odds ratios. Conversely, finer codes can create rare indicators with unstable estimates or separation. Thus a parent feature may rank above its child features, below them, or differently from each of them. A longer window can also increase prevalence or combine episodes that have different associations with treatment and outcome.

The basic calculation makes the ranking's sensitivity clear. If a coding decision changes either odds ratio, it changes one or both absolute log terms, and hence can change the product and rank. Even unchanged odds ratios do not guarantee unchanged rank if other features' values change. Ranking stability should therefore be evaluated across the full feature set, not inferred from one prominent code.

Compare Scores, Balance, and Estimates Together

A useful comparison includes more than the top-ranked features. For each coding and window specification, record the number of features, their prevalence, and the stability of rankings. For example, if the top five lists share three features, their Jaccard similarity is \(3/(5+5-3)=3/7\approx0.429\). This describes overlap between lists, not whether either list is causally adequate.

Also examine propensity-score distributions by treatment group, including their ranges or selected quantiles, and assess covariate balance after the intended matching or weighting procedure. A propensity model can have good apparent prediction while producing poor overlap or residual imbalance. A changed score distribution is a warning to investigate, not by itself proof that one model is preferable. In high-dimensional settings, inspect important clinical features and feature blocks as well as aggregate summaries.

For a binary covariate, a standardized mean difference before or after adjustment can be calculated as:

where \(\hat p_1\) and \(\hat p_0\) are the covariate prevalences in the treated and control groups, using the appropriate weighted or matched summaries for a post-adjustment diagnostic. Comparing the same diagnostic across specifications helps identify whether a window or coding choice improves one feature while worsening another. No single threshold certifies adequate balance in every application.

Separate Specification Sensitivity From Model Failure

Differences across specifications are informative only when the analysis is checked for failure within each specification. A comparison should flag features that become extremely rare after code splitting, highly prevalent after aggregation, or differently timed across windows. Inspect separation or unstable coefficient estimates, extreme propensity scores or weights, effective sample size, and residual imbalance. If a feature is absent because the required historical data are unavailable, distinguish that from a true zero; otherwise the comparison can conflate data availability with clinical history.

For effect estimates, report the same estimand and units across specifications. Present the primary estimate and uncertainty interval, alongside prespecified sensitivity estimates and diagnostics. If estimates vary materially, investigate which codes, patient subgroups, or overlap regions changed. Do not select the coding level or window because it yields the most favorable estimate or the smallest p-value. That practice uses the outcome to tune the adjustment set and can make uncertainty statements misleading.

Clinically specified covariates provide an important reference point: they make clear which known confounders the feature-generation process is intended to capture. Automated ranking can help prioritize a large set but does not replace causal judgment. Likewise, a flexible propensity model may accommodate complex relationships, but it cannot recover information absent from the chosen covariate definitions or repair lack of positivity. Sensitivity analysis complements these approaches by showing how conclusions depend on plausible operational choices.

Common Pitfalls and Practical Limits

• Calling a longer window more complete and therefore better. It may add relevant history, but it may also include stale or poorly timed records. Justify windows clinically and respect the pre-treatment time boundary.
• Treating a top-ranked feature as a confounder by definition. Ranking measures associations under a particular representation; it does not determine causal role. Consider the causal structure and avoid adjusting for post-treatment variables or inappropriate colliders.
• Comparing only the final effect estimates. Similar estimates can conceal worsening balance or overlap, while different estimates can arise from legitimate changes in measured covariates. Compare feature lists, score distributions, balance, and uncertainty together.
• Choosing the specification with the preferred result. This makes the reported analysis outcome-dependent. Predefine a primary specification and report defensible alternatives transparently.
• Assuming stability proves no unmeasured confounding. All compared specifications may omit the same important confounder. Robustness to coding choices is not a test of exchangeability.

The most defensible conclusion is usually conditional: describe which estimates are stable across plausible definitions, which diagnostics change, and what clinical or data limitations remain. A sensitivity analysis maps dependence on analytic choices; it cannot turn imperfectly measured observational data into randomized evidence.