Uncertainty in Estimated Propensity Scores

Why a Fitted Score Is Not Known

For a treatment indicator \(T\) and observed covariates \(X\), the propensity score is the conditional treatment probability \(e(X)=P(T=1\mid X)\). In practice, this probability is not usually known. An analyst specifies an estimation procedure, uses the observed sample to fit it, and obtains a fitted score \(\hat e(X)\). A different sample could produce different fitted model parameters and therefore a different score for the same covariate profile.

This tutorial focuses on that sampling variation. It is distinct from the question of whether the chosen model represents the treatment mechanism adequately. A narrow interval or a small bootstrap standard error describes variability under the estimation procedure and its assumptions; it does not establish that the model is correctly specified or that the score equals the true conditional probability.

A single fitted score is therefore a point estimate, not a known probability. Conditional on the observed data and the fitted model, \(\hat e(x)\) is a fixed number. The uncertainty summary asks how much that estimate might vary across comparable samples, subject to the model and sampling assumptions used to calculate it.

Coefficient-Based Uncertainty for a Logistic Score Model

Consider a logistic propensity model with a fixed set of covariates and independent sampled units:

Here \(x\) includes the model's intercept and covariate values, \(\beta\) is the coefficient vector, and \(\hat\beta\) is its estimate. A standard coefficient-based summary assumes the model is correctly specified, the observations are independent (or the covariance estimator correctly accounts for their dependence), and the estimator is sufficiently regular for its large-sample distribution to be approximately normal. Under these conditions, \(\hat\beta\) is approximately distributed around \(\beta\) with estimated covariance matrix \(\widehat{\operatorname{Var}}(\hat\beta)\).

The uncertainty in \(\hat e(x)\) can be approximated by the delta method. First calculate the estimated linear predictor \(\hat\eta=x^\top\hat\beta\). Its estimated variance is \(x^\top\widehat{\operatorname{Var}}(\hat\beta)x\). Since the derivative of the expit function is \(e(x)\{1-e(x)\}\), the delta method gives:

Equivalently, the approximate standard error is \(\hat e(x)\{1-\hat e(x)\}\) times the standard error of the estimated linear predictor. This makes the source of uncertainty explicit: both the fitted coefficient covariance and the profile \(x\) matter. A profile with large coefficient-weighted variance can have a relatively uncertain score even when its point estimate is similar to another profile's.

Worked Coefficient-Based Calculation

Why the Delta-Method Formula Holds

The delta method approximates a smooth function of an estimator by its first-order Taylor expansion. Let \(g(\eta)=\operatorname{expit}(\eta)\), with \(\eta=x^\top\beta\) and estimate \(\hat\eta=x^\top\hat\beta\). Around \(\hat\eta\), a small change in the linear predictor produces approximately this change in the fitted score:

Taking the approximate variance on both sides gives:

Substitution yields the score-variance formula above. This is a local, large-sample approximation. It can be poor when the coefficient estimates are highly uncertain, the sampling distribution is strongly non-normal, or the fitted probability is near a boundary and the approximation behaves poorly on the probability scale. Transforming an interval from the linear-predictor scale respects the score's range from zero to one, although it does not remove model or approximation problems.

Bootstrap Uncertainty

A bootstrap estimates sampling variation by repeatedly resampling the observed units and refitting the score model. For independent observations, a nonparametric bootstrap draws \(n\) rows with replacement from the original \(n\) rows. The treatment and covariates must remain together in each resampled row. For each bootstrap sample, refit the specified procedure and calculate the score for the covariate profile of interest.

For \(B\) bootstrap replicates, their standard deviation is:

A percentile interval uses the corresponding lower and upper quantiles of the bootstrap scores. It does not require a symmetric distribution, but its quality depends on having enough successful, representative replicates and on the bootstrap procedure being appropriate for the data and estimator. If model selection or tuning is part of the analysis, omitting it from the bootstrap estimates uncertainty for a different, more restricted procedure.

Same Fitted Score, Different Uncertainty

Choosing and Checking an Uncertainty Summary

Coefficient-based summaries are convenient when the model has a stable, estimable parameterization and the covariance estimator is credible. They make the role of individual coefficients and covariate profiles transparent. Bootstrap summaries can be useful when the score estimator is complicated or when a normal approximation for the coefficients is questionable, provided the resampling scheme represents the sampling design and the full estimation procedure is repeated.

Neither approach is universally superior. Coefficient-based intervals can be misleading under strong nonlinearity in the sampling distribution or unstable estimation. Bootstrap results can also mislead: resampling cannot repair confounding-model misspecification, unmeasured covariates, poor covariate measurement, or a resampling scheme that ignores dependence. A bootstrap that holds the selected model fixed also does not account for uncertainty from the selection process.

Check an uncertainty analysis for signs that its assumptions or computation may be failing:

• Large or unstable coefficient covariance. Nearly redundant covariates, weak information, or numerical problems can make standard errors very large or unreliable.
• Intervals that behave implausibly. A probability-scale Wald interval extending below zero or above one signals the limitations of that approximation. A transformed interval stays within the allowable range, but may still be wide or poorly calibrated.
• Unusual bootstrap distributions. Strong skewness, multiple peaks, extreme values, or a substantial fraction of failed fits deserves investigation rather than automatic summarization.
• Too few successful replicates. A standard deviation or tail quantile computed from a small number of usable resamples can be unstable. Report the number of attempted and successful fits.
• Mismatch between method and sampling design. Resampling individual rows is not appropriate when the independent sampling units are clusters, unless the dependence is otherwise handled.

An important distinction is between uncertainty in a score at a fixed profile and variation in scores across the observed sample. The latter includes real differences in covariates across units as well as estimation uncertainty. It should not be reported as though it were the standard error of one estimated conditional probability.

Common Mistake

Likewise, a small score standard error is not evidence by itself that the model has achieved good covariate balance or that a causal effect estimate is unbiased. Those are separate questions requiring their own diagnostics and assumptions.