High-Dimensional Logistic Propensity Estimation

Why High-Dimensional Propensity Models Need Care

A propensity model may include many candidate covariates: baseline measurements, indicator variables for categories, transformations, and interactions. The relevant number is the number of model columns, not just the number of named variables. With a modest sample and many columns, ordinary logistic regression can be unstable, may have no finite maximum-likelihood estimate, or may return fitted probabilities that are too extreme for useful adjustment. Penalization can stabilize estimation, but it does not automatically produce a good propensity score or identify which variables cause treatment assignment.

Let \(A_i\) be the binary treatment indicator for unit \(i\), with \(A_i=1\) for treated and \(A_i=0\) for control, and let \(X_i\) denote measured pretreatment covariates. The propensity score is \(e(X_i)=P(A_i=1\mid X_i)\). A logistic specification represents the log odds of treatment as a linear combination of covariate columns. When there are many such columns relative to \(n\), regularization is one way to constrain the fitted coefficients.

A propensity model is not chosen to maximize treatment prediction for its own sake. Its purpose is to support adjustment for measured confounding, so the included information should be guided by the causal question and the covariate-selection principles already established in the earlier material on choosing covariates. Regularization addresses estimation instability; it does not replace that reasoning.

Penalized Logistic Estimation

For a coefficient vector \(\beta=(\beta_1,\ldots,\beta_p)\), write the fitted propensity for unit \(i\) as \(\hat e_i=\operatorname{logit}^{-1}(\hat\beta_0+Z_i^\mathsf{T}\hat\beta)\), where \(Z_i\) is the vector of modeled covariate columns and \(\operatorname{logit}^{-1}(u)=1/(1+\exp(-u))\). A common fitting objective is the average negative log-likelihood plus a penalty:

In this convention, \(\lambda\geq 0\) controls the strength of regularization, and the intercept is not penalized. Ridge uses \(P(\beta)=\frac12\sum_{j=1}^p\beta_j^2\); lasso uses \(P(\beta)=\sum_{j=1}^p|\beta_j|\). Ridge continuously shrinks slopes toward zero, usually retaining all columns. Lasso can set some estimated slopes exactly to zero, producing a sparse fitted model. Neither penalty determines whether a selected covariate is a confounder, nor whether a zero coefficient means a variable is causally irrelevant.

The exact numerical value of \(\lambda\) depends on the objective’s scaling and software conventions. For example, multiplying the likelihood term by \(n\) without changing the penalty changes the corresponding numerical penalty strength. State the objective when reporting a penalty, and tune it using an appropriate procedure rather than interpreting a software’s raw \(\lambda\) in isolation.

Why Scaling Matters

A penalty acts on coefficient magnitudes, and coefficient magnitudes depend on measurement units. A slope expressed per one year of age may be numerically small, while the slope for a measurement recorded in fractions of a unit may be numerically large, even when their fitted contributions over typical ranges are similar. Penalizing these raw coefficients equally can therefore make the result depend on arbitrary units.

A common preparation is to center and scale continuous columns using their sample means and standard deviations, then fit the penalized model to the transformed columns. A standardized column is \(Z_{ij}=(X_{ij}-\bar X_j)/s_j\), with the transformation estimated from the analysis sample. Binary indicator columns are often left as zero-one variables; factor coding and the treatment of groups of indicator columns should be stated. Constant columns cannot be standardized by division by \(s_j=0\), and must be removed or handled explicitly. The intercept remains unpenalized. Scaling improves comparability of penalty effects; it does not make the covariates equally important for confounding control.

What High Dimensionality Does to Estimation

For finite fitted linear predictors, let \(W\) be the diagonal matrix with entries \(\hat e_i(1-\hat e_i)\). The slope-block Hessian of the average negative log-likelihood, conditional on the intercept, is \(Z^\mathsf{T}WZ/n\). Its rank cannot exceed \(\min(n,p)\). Therefore, when \(p>n\), the slope-block Hessian is rank deficient. When \(p=n\), it may be full rank or rank deficient depending on the design and weights; equality alone does not establish singularity.

This rank fact helps explain why the unpenalized objective can be flat in some slope directions when \(p>n\). Ridge adds \(\lambda I_p\) to the slope-block Hessian, which is positive definite for \(\lambda>0\) when the design entries are finite and the intercept is handled separately. This stabilizes the slope solution. Lasso can also constrain estimates and encourage sparsity, but its nonsmooth objective does not guarantee a unique coefficient vector in every design. In either case, regularization does not create information in directions unsupported by the data; it selects estimates according to the penalty.

Interpreting Selection and Checking the Fit

A selected lasso coefficient is a coefficient in a regularized treatment-assignment model, conditional on the other columns and on the chosen coding and scaling. It is not an estimate of a causal effect on an outcome. It is also not a general measure of a variable’s importance: correlated covariates can substitute for one another, so one may be selected and another omitted with little change in fitted predictions. Changing the sample, coding, scale, or penalty can change which member of a correlated group is retained.

Likewise, a coefficient set to zero by lasso is not proof that the covariate is unrelated to treatment or irrelevant for confounding adjustment. The selection reflects the penalized optimization problem and the data at hand. Standard errors and confidence intervals from an ordinary unpenalized logistic regression refit after lasso selection do not automatically account for the selection step. Treat selected coefficients as descriptive components of the fitted model unless a valid inferential procedure for the selection process has been used.

After fitting, inspect more than the number of selected columns or the treatment model’s predictive performance. Examine the distribution of fitted scores by treatment group, look for scores near zero or one, and assess covariate balance using the intended downstream adjustment. A model can predict treatment well yet leave important covariates imbalanced, while a less accurate classifier may support better balance. Cross-validation can tune predictive loss, but the selected tuning value should still be evaluated against the causal adjustment objective. Extreme scores may signal limited overlap; penalization can soften fitted probabilities without creating overlap where treated and control units are absent.

The main alternatives are not interchangeable. A smaller, substantively justified logistic model is easier to interpret, but can still be unstable if it has too many terms for the data. Ridge is useful when many columns may carry modest information and retaining them is acceptable; it shrinks without selecting. Lasso offers a sparse model, but selection can be unstable among correlated covariates. Elastic-net penalties combine ridge and lasso behavior and can retain correlated groups more readily, though they still do not guarantee balance or causal validity. Dimension reduction may compress many columns, but can obscure the role of confounders and must be justified for the adjustment goal. The choice should reflect the data structure and the aim of adjustment, not a preference for a short coefficient list.

Common Pitfalls and Practical Limits

• Using raw scales without checking the penalty: a coefficient’s units determine its magnitude. Scale appropriate columns and document how indicators and groups of columns are handled.
• Calling selected variables “the confounders”: selection by a penalized treatment model is not a causal classification. Use subject-matter reasoning to decide what should be controlled.
• Equating prediction with balance: a low treatment-prediction error does not establish adequate covariate balance after adjustment. Check balance directly.
• Assuming regularization solves overlap: shrinkage changes estimates, not the observed support of treatment assignment. If comparable treated and control units are lacking, the causal estimand may not be supported by the data.
• Ignoring model construction: a nominal count of original variables can hide a much larger \(p\) after dummy coding, transformations, and interactions. Monitor the actual design matrix.

Penalization also depends on modeling assumptions. The logistic likelihood treats the observed treatment indicators as independent conditional on the modeled covariates, as appropriate for the sampling design. Its fitted probabilities represent \(P(A=1\mid X)\) only to the extent that the conditional model is adequate. For causal use, measured covariates must support the required exchangeability and positivity conditions; penalization cannot repair unmeasured confounding, incorrect timing of covariates, or structural lack of overlap. High-dimensional fitting is therefore a stabilization strategy within a causal design, not a substitute for one.