Selecting the Number of Covariates

Why the Number of Covariates Is a Design Choice

High-dimensional propensity score (HDPS) methods can produce a long ranked list of candidate features. Ranking helps decide which features to consider first, but it does not decide how many to include. A rule that selects too few may leave important confounding uncontrolled. A rule that selects many may add little useful information, make treatment assignment harder to model, or weaken overlap. The number selected therefore affects the adjustment design, not just the length of a variable list.

This tutorial focuses on choosing a count from a set of candidate features already prioritized by an HDPS procedure. It does not replace causal reasoning about which variables are eligible for adjustment. Features must be measured before the treatment they are intended to help control, and the timing and causal role of each feature still matter. In a longitudinal application, the adjustment set must also be appropriate to the treatment and outcome times being studied.

A priority score summarizes information used to rank candidate features; it is not itself a causal effect or a certificate that a feature is a confounder. For example, high-ranked features may be strongly associated with treatment and outcome, while low-ranked features may still matter through a less well-measured confounding process. The ranking is a heuristic for allocating attention. The count rule determines where to stop.

What a Selected Set Must Accomplish

For a treatment \(A\), outcome \(Y\), and potential outcomes \(Y(1)\) and \(Y(0)\), causal adjustment using a selected set \(X_S\) requires substantive conditions. In a simple two-treatment setting, consistency links observed outcomes to potential outcomes, conditional exchangeability requires \(Y(a)\perp A\mid X_S\) for each treatment level \(a\), and positivity requires \(0<P(A=1\mid X_S=x)<1\) for covariate profiles in the target population. Choosing a larger \(K\) does not establish any of these conditions.

In particular, if a necessary confounder is absent from the candidate features, or is omitted from the selected set, the treatment groups may remain confounded after adjustment. Conversely, adding a feature does not automatically improve causal adjustment: a feature can be redundant, noisy, or inappropriate for adjustment. The appropriate set depends on the causal structure and target estimand, not only on how well treatment can be predicted.

The practical effect of changing \(K\) depends on the adjustment procedure. In propensity-score matching, changing the set can change estimated scores, which individuals find matches, and which individuals remain in the matched sample. In weighting, it can change the weights and their variability. Thus, results for different values of \(K\) may not represent the same realized sample or have the same precision. Balance and overlap should be examined for the actual design produced by each choice.

Three Ways to Choose the Count

Fixed Top-K Selection

A fixed top-\(K\) rule selects exactly the first \(K\) features in the priority ranking. Its appeal is procedural clarity: investigators can state the count before examining the resulting treatment-effect estimate, and others can reproduce the selection. It is especially useful when a convention or prior design decision specifies a count. The chosen value should nevertheless be justified for the application rather than presented as a universal optimum.

A fixed count can be insensitive to the shape of the ranking. If scores fall sharply after the first few features, a large fixed \(K\) may add many low-priority features. If scores are nearly equal, the same \(K\) may omit several features close in priority to those included. A useful practice is to examine nearby counts as a sensitivity analysis, while keeping the primary selection rule explicit.

Cumulative Priority-Score Rules

A cumulative rule selects a target fraction \(\gamma\), where \(0<\gamma\leq1\), of the total priority score. It chooses the smallest \(K\) such that the selected scores reach that fraction:

This rule changes the count in response to the concentration of the ranking. A sharply concentrated score distribution can reach the target with a small \(K\); a diffuse distribution generally requires a larger one. The fraction \(\gamma\) is a design choice and should be stated before looking at treatment-effect results.

The cumulative rule is not meaningful if all scores are zero, because its denominator is zero. In that case it cannot define a fraction of total score; another prespecified rule or a reassessment of the ranking is needed. When scores are nonnegative but tied, tied features may straddle the threshold. A reproducible implementation must specify a tie rule, such as a fixed secondary ordering, or include all features tied at the boundary and report the resulting count.

Data-Adaptive Counts

A data-adaptive rule evaluates several candidate counts and chooses one using information about the resulting adjustment design. For example, investigators can specify acceptable balance and overlap criteria, then choose the smallest count that meets them. The criteria should be selected in advance and should assess the design rather than choose the count that produces the most favorable estimated treatment effect. This distinction helps prevent outcome-informed selection from making the reported estimate look better by construction.

Balance summaries describe measured covariate differences between treatment groups after the proposed adjustment. One common summary for a quantitative feature is the absolute standardized mean difference, which divides the absolute difference in group means by a chosen standard deviation scale. The scale and calculation should be applied consistently across candidate counts. A small value indicates little measured mean difference on that feature; it does not establish that all confounding is controlled.

Why More Features Do Not Guarantee Better Adjustment

A useful justification for prioritization is to imagine that each feature contributes some amount to residual bias when omitted. If \(b_j\) denotes a signed contribution in a simplified additive approximation, the omitted contribution after selecting the first \(K\) features is

If priority scores were exact values of \(|b_{(j)}|\), selecting the largest scores first would minimize the sum of the magnitudes of the omitted contributions for any fixed \(K\). To see why, suppose a selected feature has magnitude \(u\) and an omitted one has magnitude \(v>u\). Swapping them into the selected set removes \(v\) from the omitted magnitudes and adds only \(u\), reducing the omitted total by \(v-u>0\). Repeating such swaps leaves the largest \(K\) magnitudes selected.

That argument explains the appeal of ranking, but its ideal condition is rarely met: an HDPS priority score is a heuristic, not a known absolute bias contribution. Contributions may also have opposite signs, features may be correlated, and the adjustment estimator can change as the selected set changes. Consequently, neither a top-\(K\) rule nor a cumulative score threshold guarantees a particular bias reduction. Empirical balance can help assess measured differences, but it cannot verify the untestable exchangeability assumption.

Diagnostics, Comparisons, and Limitations

For each serious candidate count, evaluate the resulting design rather than relying only on the score ranking. Check whether important measured features remain imbalanced, whether matching or weighting retains a usable target population, and whether the result changes substantially across nearby counts. A pattern in which balance improves while many treated individuals lose matches signals a trade-off, not an automatic reason to choose the largest count. Report the chosen rule, threshold or criteria, and how many features it selected.

The three approaches answer different questions. Fixed top-\(K\) asks, “How does a predetermined number perform?” Cumulative selection asks, “How many features are needed to capture a chosen share of this score total?” Data-adaptive selection asks, “Which candidate count meets specified design requirements in these data?” They can produce the same count, but their justification is different. A cumulative rule is especially dependent on score scale and concentration; a fixed rule is easiest to reproduce; an adaptive rule is responsive to observed design quality but most vulnerable to post hoc decision-making.

All three rules inherit limitations of the candidate feature set and ranking. They can miss confounders not represented in the data, and a good fit for treatment assignment is not the same as adequate confounding control. A selected feature can also be unsuitable for adjustment on causal grounds. Eligibility and timing should therefore be assessed before ranking-based count selection, not treated as problems that a larger \(K\) will solve.

Common Mistake

Wrong reasoning: “The cumulative rule selected features accounting for 80% of the priority score, so it removed 80% of confounding.” The score total is not a calibrated measure of causal bias, and some relevant confounding may be absent from the candidates altogether.

Correction: State the result as a property of the ranking: the selected set reached at least 80% of the defined priority-score total. Then assess measured balance and feasibility for the resulting design, and discuss the remaining causal assumptions separately.