Sparse Treatment Data and Small-Sample Bias

Why Sparse Treatment Patterns Matter

In a sequential treatment model, a probability is estimated for each treatment decision given the history available at that time. Write \(A_t\in\{0,1\}\) for treatment at time \(t\), \(H_t\) for the observed covariate and treatment history before that decision, and \(p_t(H_t)=P(A_t=1\mid H_t)\). A treatment pattern is sparse when only a few observations, or none, have a particular treatment value within a history or covariate region. Sparse data can make estimated probabilities unstable even when the model has been specified correctly.

The practical consequence is especially important for inverse probability treatment weights. For a person observed to receive \(A_t\), the contribution to the inverse probability of the observed treatment is \(p_t(H_t)^{-1}\); for \(A_t=0\), it is \(\{1-p_t(H_t)\}^{-1}\). Across time, these contributions multiply. A small estimated probability at one decision can therefore create a large weight, and several small contributions can compound along a treatment history.

Sparsity is not defined by one universal sample-size cutoff. A cell with two treated people may be adequate for a simple model in a large, well-supported design, but inadequate for a model with many parameters and a sequence of treatment decisions. The relevant question is whether the observed data contain enough information to estimate the conditional probabilities needed for the target population and histories.

What Ordinary Maximum Likelihood Can Do

Ordinary logistic maximum likelihood chooses coefficients that maximize the observed Bernoulli likelihood. Under standard regularity conditions—including a correctly specified model, identifiable parameters, and a nonsingular information matrix—the estimator is consistent and asymptotically normal. Those large-sample properties do not guarantee small finite-sample bias, and they do not apply when the likelihood has no finite maximizer.

In sparse samples, fitted probabilities can be close to zero or one, coefficient estimates can be extreme, and standard errors can be very large. The direction and size of finite-sample bias depend on the design; there is no general rule that ordinary maximum likelihood always overestimates, or always underestimates, a treatment probability. The following calculation illustrates one source of small-sample bias in the logit scale.

To derive the approximation, let \(\hat p=X/m\) and \(g(u)=\operatorname{logit}(u)\). The derivatives are \(g'(p)=1/\{p(1-p)\}\) and \(g''(p)=(2p-1)/\{p^2(1-p)^2\}\). Since \(E[\hat p]=p\) and \(\operatorname{Var}(\hat p)=p(1-p)/m\), the second-order Taylor term in the expectation is

For \(p<1/2\), the leading term is negative, so the estimated logit is pushed downward on average in this approximation; for \(p>1/2\), it is positive. This calculation concerns the logit estimate, not a claim that every fitted probability or every inverse-probability weight has bias in a fixed direction.

Separation and Nonexistence of a Finite Estimate

A more severe problem occurs when treatment status is completely predicted by covariates. For logistic regression with covariate vector \(x_i\) and coefficient vector \(\beta\), let \(\eta_i=x_i^\mathsf{T}\beta\). If there is a vector \(b\) such that \((2y_i-1)x_i^\mathsf{T}b>0\) for every observation, the data are completely separated. The strict inequality means that moving in direction \(b\) makes every observed outcome more likely.

Proof. The Bernoulli log-likelihood is \(\ell(\beta)=\sum_i\{y_i\eta_i-\log(1+\exp(\eta_i))\}\). Set \(\beta=tb\), where \(t>0\). If \(y_i=1\), separation gives \(x_i^\mathsf{T}b>0\), so the fitted probability \(\operatorname{expit}(t x_i^\mathsf{T}b)\) tends to one as \(t\) tends to infinity. If \(y_i=0\), separation gives \(x_i^\mathsf{T}b<0\), so the fitted probability of zero also tends to one. Consequently, every observed outcome probability tends to one, and \(\ell(tb)\) tends to zero. For every finite \(\beta\), each fitted Bernoulli probability is strictly between zero and one, so every log probability is negative and \(\ell(\beta)<0\). Thus zero is a supremum but is not attained at any finite coefficient vector. \(\square\)

With quasi-separation, some observations may lie on the separating boundary. Coefficients can still diverge while the likelihood approaches its supremum, even though not every observation is driven toward probability one in the same strict way. In practice, signs of trouble include convergence warnings, very large coefficients or standard errors, fitted probabilities extremely close to zero or one, and predictions that change sharply when a small number of observations are removed.

Sequential Sparsity and Weight Diagnostics

Even if each treatment model separately returns finite estimates, histories can be sparse across time. For a person with observed treatment path \((A_1,\ldots,A_K)\), the inverse probability of that path, conditional on the modeled histories, is

This expression shows why moderate-looking probabilities can produce a large product. It also indicates what to inspect: counts of treated and untreated observations within important histories; the distributions of fitted probabilities; the largest weights; and whether a few people dominate weighted summaries. These are diagnostics of support and model stability, not tests that can prove a model is correct.

A useful summary of weight concentration is the effective sample size, defined for nonnegative weights \(w_1,\ldots,w_n\), not all zero, by

This quantity is at most \(n\): by the Cauchy–Schwarz inequality, \((\sum_i w_i)^2\leq n\sum_i w_i^2\). It is at least one because \((\sum_i w_i)^2=\sum_i w_i^2+2\sum_{i<j}w_iw_j\geq\sum_i w_i^2\). Equal weights give \(\operatorname{ESS}=n\), while a single nonzero weight gives \(\operatorname{ESS}=1\). A low value signals concentration, but ESS does not identify its cause, and a reassuring ESS cannot establish that the treatment model is correctly specified or that all relevant histories have support.

Responding to Sparse Data

Ordinary maximum likelihood is attractive when the model is identifiable, the sample contains adequate information, and the fit is stable. It has familiar likelihood-based interpretation and standard large-sample theory. When treatment patterns are sparse, however, its finite-sample behavior may be poor or the estimate may not exist. Bias-reduced probability estimation is an alternative class of methods intended to improve estimation in such settings; a method that returns finite estimates is not automatically unbiased, correctly specified, or adequate for weighting.

Other responses address different parts of the problem. Simplifying a model can reduce the number of parameters, but removing important history terms may introduce confounding bias. Collecting more data in underrepresented histories directly improves information when feasible. Bayesian estimation can stabilize estimates through an explicit prior, but the result may depend on that prior. Truncating extreme weights can reduce the influence of a few observations, but changes the weighting procedure and does not repair a poorly estimated treatment model. No numerical adjustment creates information about a history that is absent from the data.

A disciplined assessment therefore combines model-fit diagnostics with design knowledge. Check whether important treatment and covariate-history combinations are observed; examine coefficient estimates, standard errors, fitted probabilities, and convergence; then inspect the resulting weights and their concentration. If estimates are unstable, report that instability and consider whether a simpler scientifically defensible model, more data, or a bias-reduced estimator is appropriate. The specific Firth logistic approach is treated separately; here the key distinction is between ordinary maximum likelihood and the broader goal of reducing small-sample instability without mistaking finite output for guaranteed validity.

Common Mistakes

• Wrong: “The model converged, so the treatment probabilities are reliable.” Correction: Convergence is a numerical fact, not evidence of adequate support or stable estimates. Inspect the data patterns, coefficients, fitted probabilities, and weights.
• Wrong: “A bias-reduced estimate is always closer to the truth.” Correction: Bias reduction targets particular finite-sample behavior; it cannot guarantee lower error in every sample or correct a misspecified treatment model.
• Wrong: “A large weight proves positivity is violated.” Correction: A large estimated weight may reflect a genuinely small probability, model instability, or misspecification. It calls for investigation; it is not conclusive by itself.
• Wrong: “Truncating the largest weights fixes sparse treatment data.” Correction: Truncation can limit influence but does not add support or recover the underlying probabilities, and it changes the estimator.