Density-Ratio Estimation and Overlap Diagnostics

Why Estimate a Density Ratio?

Under covariate shift, a source sample and a target population have different covariate distributions, while the conditional distribution of the outcome given the covariates is assumed to remain the same. Importance-weighted risk uses source observations to represent the target covariate distribution. Its central weight is therefore the target-to-source covariate density ratio. Estimating this ratio is not just a computational step: large or unstable weights can reveal that the source data provide little information about parts of the target population.

Let \(X\) denote the covariates, with source distribution \(P_s\) and target distribution \(P_t\). When both have densities with respect to a common measure, write them as \(f_s(x)\) and \(f_t(x)\). The ratio is defined only where the source distribution can represent the target distribution.

If \(P_t\ll P_s\) and \(g(X)\) is integrable under the target distribution, the change-of-measure identity is

Indeed, in the density case, \(E_s[r(X)g(X)]=\int [f_t(x)/f_s(x)]g(x)f_s(x)\,dx=\int g(x)f_t(x)\,dx=E_t[g(X)]\). The identity explains why source observations receive weights \(r(X)\), and why a very large ratio can make an estimate sensitive to a few observations.

Estimating Ratios With a Domain Classifier

A practical approach is to pool source and target covariates and predict which sample an observation came from. Define a domain label \(D=1\) for a target observation and \(D=0\) for a source observation. Let \(\pi_t=P(D=1)\) and \(\pi_s=P(D=0)\) be the proportions in the pooled population. Write \(q(x)=P(D=1\mid X=x)\). Bayes' rule gives a direct relationship between domain-classification probabilities and the density ratio.

To verify the relationship, the joint densities for the two domain labels are \(\pi_t f_t(x)\) and \(\pi_s f_s(x)\). Therefore

Dividing the first expression by the second gives the stated odds identity. With \(n_t\) target and \(n_s\) source observations, the sample proportions are typically \(\pi_t=n_t/(n_s+n_t)\) and \(\pi_s=n_s/(n_s+n_t)\). Hence the multiplier \(\pi_s/\pi_t\) is \(n_s/n_t\). Omitting it generally gives ratios on the wrong scale.

A fitted logistic regression is one way to estimate \(q(x)\); a more flexible classifier may be used when the relationship is nonlinear. The odds transformation is reliable only to the extent that the estimated domain probabilities are reliable, particularly in their tails. Probabilities near one produce very large estimated ratios. Use held-out or cross-fitted domain predictions when feasible, and examine calibration as well as classification performance: accurate domain labels do not by themselves guarantee accurate probability estimates.

Overlap Diagnostics and Effective Sample Size

The overlap condition is a population statement, but available data can only reveal warning signs. Inspect the estimated weights on source observations: their histogram or quantiles, the largest values, and the fraction of total weight carried by the largest few observations. Also compare source and target covariates or domain scores to identify regions that appear in the target sample but are rare in the source sample. These checks are complementary: a single summary cannot establish that all target-relevant regions are represented.

For positive weights \(w_1,\ldots,w_n\), the weight-based effective sample size is

This number summarizes weight concentration. It equals \(n\) if all weights are equal, and decreases as weight is concentrated in fewer observations. It is a diagnostic, not a guarantee that the weighted estimator has the precision of an independent sample of that size.

For the upper bound, Cauchy–Schwarz gives \((\sum_i w_i)^2\leq n\sum_i w_i^2\), and division by the positive denominator yields \(\operatorname{ESS}\leq n\). For the lower bound, expanding the square gives \((\sum_i w_i)^2=\sum_i w_i^2+2\sum_{i<j}w_iw_j\geq\sum_i w_i^2\), so \(\operatorname{ESS}\geq1\). Equality in the upper bound requires equal weights; equality in the lower bound requires all pairwise products to be zero, so at most one weight is positive. \(\square\)

Truncation: Stability at a Cost

A common response to extreme estimated weights is truncation. At a threshold \(c>0\), replace \(r(x)\) by \(r_c(x)=\min\{r(x),c\}\). This caps the influence of individual observations, but it changes the weighting scheme. For a target quantity \(E_t[g(X)]\), the unnormalized truncated expectation has an exact discrepancy:

where \((u)_+=\max(u,0)\). This follows pointwise from \(r-r_c=(r-c)_+\), followed by taking expectations. If \(0\leq g\leq1\), the absolute discrepancy is at most \(E_s[(r-c)_+]\). The bound shows why the mass and size of the tail above the cutoff matter. If truncated weights are then normalized to sum to one, the resulting estimate targets a normalized, altered weighting distribution; it is not simply the unnormalized expression above.

A Practical Diagnostic Sequence

The most useful assessment combines weight summaries with direct inspection of the covariate regions that generate high ratios. A large weight can indicate genuine target mass in a source-rare region, estimation error in the domain model, or both. Compare estimates under several pre-specified caps, and report the cap, the fraction of observations affected, the resulting ESS, and how the substantive estimate changes. A stable estimate across reasonable choices is reassuring but does not prove overlap; a sharp change is evidence that the conclusion depends on the tail of the weighting distribution.

Common mistake: “The ESS is close to the source sample size, so overlap must be adequate.” This reasoning is not valid. ESS is computed from the estimated weights on observed source cases; it cannot detect target regions absent from both the source sample and the fitted model’s effective view of the covariate space. Conversely, a low ESS is a warning about concentration, not proof that the ratio model is wrong. Use ESS together with support and covariate diagnostics, and state that empirical checks cannot certify population-level overlap.

Another mistake is to treat truncation as a harmless numerical repair. Capping weights can reduce variance, but it also changes the represented target distribution. The appropriate report makes this trade-off visible rather than presenting the capped result as if it were an exact target-weighted estimate.