Overlap, Positivity, And Weight-Tail Diagnostics

Why Overlap Matters

Importance weighting transports a risk from a source population to a target population by giving more influence to source observations that represent target covariate patterns. The transport works only where the source population provides information. If the target assigns positive probability to covariate values absent from the source, no source observation can reveal the loss there. Even when every target pattern is represented, a few extremely large weights can make an estimate depend heavily on very few observations.

These are two related but distinct concerns. Overlap is about whether the target distribution is covered by the source distribution. Weight-tail diagnostics describe how unevenly the covered target distribution is represented in a particular sample. Strong overlap is necessary for identification by weighting; it does not, by itself, guarantee a stable finite-sample estimate.

Overlap and Positivity

Let \(P_X\) and \(Q_X\) denote the source and target distributions of covariates \(X\). For a loss \(L\), suppose the conditional mean loss given \(X=x\) is the same in the two populations. Write it as \(m(x)=E[L\mid X=x]\). The target risk is then \(R_Q=E_{Q_X}[m(X)]\), while source observations are drawn from \(P_X\).

For distributions with densities relative to a common measure, overlap requires \(q(x)=0\) wherever \(p(x)=0\), apart from sets of measure zero. Where \(p(x)>0\), the ratio is \(w(x)=q(x)/p(x)\). A large ratio indicates that the target puts much more density in that region than the source does. It does not mean the source has no observations there; a zero source density is a different, more fundamental failure.

Overlap is a statement about distributions, not about whether two finite scatterplots appear to cover the same area. A target sample may contain a value not seen in a finite source sample even when that value has positive source probability. Conversely, failure to observe a target-only region in a small target sample does not establish that the region has zero target probability. Sample diagnostics can raise concerns but cannot prove population overlap.

When Risk Is Unidentified

If overlap fails, the change-of-measure formula cannot recover the target risk from source observations alone. This is not merely a numerical inconvenience: target risk can be unidentified. The issue remains even if the source conditional loss is known exactly everywhere the source has support.

Proof. Since \(Q_X\not\ll P_X\), there is a measurable set \(A\) such that \(P_X(A)=0\) and \(Q_X(A)>0\). Consider two possible conditional mean loss functions, \(m_0(x)=0\) everywhere and \(m_1(x)=\mathbf{1}\{x\in A\}\). On the source distribution, \(A\) occurs with probability zero, so these functions produce the same source observations almost surely. Their target risks differ: \(E_{Q_X}[m_1(X)]-E_{Q_X}[m_0(X)]=Q_X(A)>0\). Thus the source distribution cannot distinguish two models with different target risks. Additional assumptions or target information are needed to identify the risk. \(\square\)

Weight Tails, Variance, and Effective Sample Size

Even under overlap, the density ratio may be highly variable. The maximum observed weight reports the most extreme ratio in a source sample. The second moment \(E_{P_X}[w(X)^2]\) summarizes the population weight tail and is directly related to the variance of an importance-weighted risk estimate.

Proof. The finite second-moment assumption gives \(\operatorname{Var}(Z_i)<\infty\). Independence implies that the variance of a sum is the sum of its variances, so \[ \operatorname{Var}(\hat R_Q) =\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n Z_i\right) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(Z_i) =\frac{1}{n}\operatorname{Var}(Z_1). \] Using \(\operatorname{Var}(Z_1)=E[Z_1^2]-(E[Z_1])^2\), \(E[Z_1]=R_Q\), and \(Z_1^2=w(X_1)^2L_1^2\) gives the stated expression. \(\square\)

If \(0\leq L\leq1\), then \(E[w^2L^2]\leq E[w^2]\), so a finite \(E[w^2]\) is sufficient for finite variance. If \(E[w^2]\) is infinite, that alone does not prove that every particular loss has infinite variance: the loss may be small exactly where the weights are large. But it removes this general second-moment guarantee, and weighted estimates can be especially sensitive to tail observations.

The ESS ranges from 1 to \(n\) when all weights are positive. To see why, set \(\bar w=n^{-1}\sum_i w_i\) and define the empirical squared coefficient of variation as \(\widehat{\mathrm{CV}}^2=n^{-1}\sum_i(w_i-\bar w)^2/\bar w^2\). Since \(\sum_i(w_i-\bar w)^2=\sum_i w_i^2-n\bar w^2\), it follows that \[ \widehat{\mathrm{CV}}^2=\frac{n\sum_iw_i^2}{(\sum_iw_i)^2}-1, \qquad \operatorname{ESS}=\frac{n}{1+\widehat{\mathrm{CV}}^2}. \] Thus equal weights give ESS \(n\); increasing weight dispersion lowers it. If some weights are zero, ESS is at most the number of positive weights. The largest weight provides a complementary warning: a single unusually large value can dominate even when a summary of average dispersion seems less striking.

Practical Diagnostics and Common Pitfalls

Interpret diagnostics in relation to both the source and target samples. Examine whether target covariate patterns appear represented in the source, and whether source observations that receive large ratios are isolated or influential. If weights are available, report their maximum, a measure of their spread, and ESS alongside the nominal source sample size. These summaries describe the observed sample; estimated ratios and finite samples bring uncertainty of their own.

A target sample can help reveal regions of covariate space that matter for the target, but it does not supply source outcomes in those regions. Conversely, seeing source observations across the observed target range is evidence to inspect, not proof that \(Q_X\ll P_X\). Population overlap concerns sets of probability zero, which finite samples cannot verify directly.

• Mistake: treating a finite maximum as proof of safe tails. The largest observed weight is finite in any finite sample. A rare region with very large ratios may simply not have appeared. Check the full weight distribution and recognize that sample summaries cannot certify population moments.
• Mistake: treating a low ESS as proof of non-identification. A low ESS signals concentration and potential instability, not necessarily a positivity violation. A ratio can be defined everywhere relevant yet vary greatly.
• Mistake: treating overlap as a guarantee of precision. Absolute continuity permits a change of measure, but does not ensure small variance. Large or heavy-tailed weights can make a valid estimate noisy.
• Mistake: using weighting to fill a support gap. When the target puts mass outside source support, no finite ratio can create the missing source information. Identification requires additional assumptions or data.

The key distinction is therefore between identification and stability. Overlap determines whether the source distribution can, in principle, identify the target risk by reweighting. Weight tails and ESS indicate how much the available sample may struggle to estimate that risk. Neither a reassuring ESS nor an observed overlap pattern substitutes for the underlying assumptions.