Constructing an HDPS Feature Matrix

From Claims Records to a Person-by-Feature Matrix

An HDPS analysis may begin with millions of claim lines, but its analysis-ready input is usually organized with one row per person and one column per feature. Constructing that matrix is not merely a software task. Decisions about time windows, duplicate records, code aggregation, and missingness determine what each entry means. A matrix can be sparse and computationally convenient while still representing the clinical history incorrectly.

This tutorial follows a hypothetical claims-based study comparing people who initiate a treatment with eligible people who do not initiate it at the index date. The feature matrix represents recorded history before that date. Treatment and outcome are kept in separate person-level variables; future outcome information must not enter the pre-treatment feature matrix.

For person \(i\), let \(t_i\) denote the index date and define the lookback window as \([t_i-L,t_i)\), where \(L\) is a prespecified duration. The right endpoint is excluded so that claims on or after the index date cannot leak into baseline history. The exact window, code system, coding level, and feature definitions must be fixed before constructing the matrix and applied consistently to treated and comparison people.

The earlier material on defining the feature space and encoding diagnosis and procedure histories motivates which code groups and representations may be considered. The present task is to implement those choices reliably. It does not choose the number of covariates or fit a propensity model.

Deduplicate Carefully, Then Aggregate

A claims file can contain repeated rows because of exact administrative duplication, resubmissions, multiple claim lines for one encounter, or genuinely separate services. These are not interchangeable. Removing every repeated person-code pair can erase repeat utilization; retaining exact duplicate records can inflate counts. Define a record key using the fields needed to identify a duplicate in the data source, such as person, claim identifier, line identifier, service date, code, and code position. Apply the source-specific rule for exact administrative duplicates, and preserve distinct services or dates when they represent separate records under that rule.

After record-level cleaning, map raw codes to prespecified feature categories. A feature might indicate any occurrence of a diagnosis category during the lookback window, a procedure category, or a utilization category. The mapping should be stable across people and documented: each matrix column needs a definition, code set or rule, time window, and value scale. Aggregating codes to broader categories can reduce dimension, but overly broad categories may combine clinically different histories.

The example also shows why deduplication and aggregation are separate operations. Deduplication determines which source records are retained; aggregation maps those records into person-level features. A binary indicator is unchanged by repeated qualifying records, but a count may change substantially.

Missingness Is Not Automatically Zero

In claims data, no recorded code can mean that a person did not have a recorded service in that category, but it can also mean that the person was not observable, enrollment information is incomplete, or the relevant data source was unavailable. A zero therefore has a defensible interpretation only when the study's observation rules support it. In particular, lack of a claim is not proof that a condition was absent.

A common approach is to define eligibility and continuous enrollment requirements that establish observation during the lookback window. For people meeting those requirements, a feature with no qualifying code can be encoded as zero, meaning no qualifying code was recorded in the observable claims. For an unobservable interval or unavailable source, the value is unknown rather than zero. Depending on the data and analysis plan, such people may be excluded by eligibility rules, handled with a prespecified missing-data strategy, or represented with an explicit missingness feature. Those choices have different interpretations and should not be made silently during matrix creation.

An explicit missingness indicator can be useful when missingness itself is informative and the modeling plan allows it. It does not recover the missing clinical history, and it should not be used to disguise a data-quality failure. Keep the underlying feature's unknown status distinguishable during preprocessing, even if the later analysis uses a particular imputation or missingness-indicator scheme. In sparse storage, an unstored numeric zero is not the same thing as an unknown value: missingness requires its own representation or mask.

Build and Audit the Sparse Matrix

Claims-derived occurrence features often have many zeros: most people have no record for most code groups in a limited window. A sparse matrix stores nonzero entries and their row and column indices rather than allocating storage for every zero. It should represent the same \(N \times p\) matrix as a dense version; sparsity changes storage, not the feature definitions or statistical meaning. For binary features, a stored entry commonly represents 1. For counts, it stores the positive count. Unknown values must be tracked separately rather than silently treated as unstored zeros.

Before export, maintain an explicit row map linking each row index to one eligible person and an explicit column dictionary linking each column index to a feature definition. Sort or otherwise fix these mappings so that treatment, outcome, and any person-level variables align with the correct rows. A matrix with correct values but mismatched row order is still wrong.

A practical audit checks at least the following: the number of rows equals the number of eligible people; each person appears once; the number and order of columns match the dictionary; all values are in their permitted range; the lookback dates precede the index dates; and the nonzero count and zero fraction are plausible. Check duplicate rates before and after cleaning, code frequencies by feature, missingness by person and feature, and whether treated and comparison groups have unexpectedly different observability. Such differences may reveal data-processing problems or real utilization differences; the audit identifies them but does not by itself decide their causal meaning.

A short construction justification is useful. If a binary feature is defined by a set \(R_j\) of qualifying retained records and person \(i\)'s records are \(R_i\), then its value is \(X_{ij}=\mathbb{1}(R_i \cap R_j \ne \varnothing)\). Thus one or more qualifying records yield exactly one, while no qualifying record yields zero only when the relevant observation window is established. For a count feature, \(X_{ij}=|R_i \cap R_j|\) when records are the counting unit. This makes the deduplication rule consequential: changing the retained record set can change the intersection size and therefore a count. The same principle applies if the count uses distinct dates instead, with the set defined as qualifying dates rather than rows.

Diagnostics, Alternatives, and Limitations

Construction diagnostics should look for implausible patterns, not just software errors. Examples include a feature that is one for nearly everyone because codes were mapped too broadly; a feature that is always zero because of a code-version mismatch; counts that are much larger than expected because duplicate claim lines remain; or apparent zeros concentrated among people with incomplete enrollment. Verify date boundaries, code versions, code-to-feature mapping, and the unit of counting. Also verify that a feature uses only information available before the index date. A post-index diagnosis or a code created during outcome follow-up can leak outcome-related information into baseline features.

The main representation alternatives answer different questions. Binary occurrence indicators are robust to repeated billing within a window and straightforward to interpret, but discard frequency. Counts retain some utilization intensity but depend on a defensible counting unit and can be influenced by repeated billing. Distinct-date counts reduce within-day repetition but still measure recorded service frequency rather than underlying disease severity. A carefully defined enrollment or observation indicator can distinguish incomplete coverage, but it cannot make unobserved claims complete. Dense storage is conceptually simple and may be appropriate for small matrices; sparse storage is more efficient when most entries are zero. None of these choices removes the need to define and audit the data.

A well-constructed matrix does not guarantee that all confounding is controlled. Claims codes are imperfect proxies for health and may reflect access, billing practices, or provider behavior. Different code systems and enrollment patterns can produce differential measurement. Sparse construction also does not decide which features should be retained, how a propensity model should be fit, or whether causal assumptions are credible. Those are separate steps in an analysis.

A frequent mistake is to say, “There is no row in the claims file for this diagnosis, so the person did not have it,” then fill the cell with zero. The correction is to ask first whether the person was observable for the complete prespecified window and whether the source reliably captures the relevant claims. Only an observed absence of qualifying records supports a zero under the stated feature definition. When observability is incomplete, preserve unknown status or apply an explicit eligibility or missing-data rule.