Clipping Density Ratios: Bias And Variance

Why Clip Density Ratios?

Under covariate shift, the source and target populations have different covariate distributions but share the same conditional distribution of the response given the covariates. A target risk can then be written as a source-population expectation weighted by a density ratio. If that ratio is large for some observations, a few weighted losses can strongly affect an empirical estimate. Clipping limits the size of each weight, but it also changes the expectation being estimated. A careful analysis must keep these two effects separate.

Let \(P_X\) and \(Q_X\) denote the source and target covariate distributions. Assume \(Q_X\) is absolutely continuous with respect to \(P_X\), and write \(w(x)=dQ_X/dP_X(x)\). Thus \(w\geq 0\) and \(E_P[w(X)]=1\). Assume the conditional distribution of \(Y\) given \(X\) is the same under source and target. For a fixed prediction rule, let \(L=\ell(f(X),Y)\) be its loss, with \(0\leq L\leq M\) almost surely for some finite \(M\). Expectations below are under the source distribution unless specified otherwise.

Because \(w_\tau\leq w\) and the loss is nonnegative, \(R_\tau\leq R\). The difference is a downward bias relative to the target risk, whereas the variance of \(\hat R_\tau\) describes its sampling variability around \(R_\tau\). These are distinct quantities.

Bias and Sampling Variance

The bias has an exact expression. It depends on the loss as well as on the tail of the ratio: large ratios create no bias at points where the loss is zero. A useful worst-case bound follows directly from \(L\leq M\).

To verify the identity, subtract the two expectations and use \(w-w_\tau=w-\min\{w,\tau\}=(w-\tau)_+\). Nonnegativity of \(L\) gives the lower bound, and \(L\leq M\) gives the upper bound. Thus clipping has a nonnegative downward bias, but its size is not determined by the maximum ratio alone.

For the empirical clipped risk, let \(Z_\tau=w_\tau(X)L\). Independence of the observations gives \(\operatorname{Var}(\hat R_\tau)=\operatorname{Var}(Z_\tau)/n\). Since \(0\leq Z_\tau\leq M\tau\), clipping bounds the contribution of every observation. This can be useful even when the unclipped weighted loss has a very large or infinite second moment. However, a bound on each contribution does not imply that the actual variance decreases as \(\tau\) decreases.

Risk Bounds and Threshold Choice

For independent observations, the mean squared error around the target risk decomposes exactly into the variance around the clipped risk and the squared clipping bias:

Indeed, write \(\hat R_\tau-R=(\hat R_\tau-R_\tau)+(R_\tau-R)\), square, and take expectations. The cross-term vanishes because \(E[\hat R_\tau-R_\tau]=0\). This decomposition exposes the threshold trade-off: increasing \(\tau\) generally decreases the clipping bias, while the elementary upper bound on the weighted observations becomes larger. It does not assert that the actual variance changes monotonically.

For the variance term, \(\operatorname{Var}(Z_\tau)\leq E[Z_\tau^2]\leq M^2 E[\min(w,\tau)^2]\). Pointwise, \(\min(w,\tau)^2\leq \tau^{1-\delta}w^{1+\delta}\): if \(w\leq\tau\), this follows from \(w^{1-\delta}\leq\tau^{1-\delta}\); if \(w>\tau\), it follows from \(\tau^{1+\delta}\leq w^{1+\delta}\). Hence the variance divided by \(n\) is at most \(M^2 C\tau^{1-\delta}/n\).

For the bias, \((w-\tau)_+\leq w\,\mathbf{1}\{w>\tau\}\leq w^{1+\delta}/\tau^\delta\). The first inequality follows by considering separately \(w\leq\tau\) and \(w>\tau\); on the latter event, \(w\leq w^{1+\delta}/\tau^\delta\). Thus \(R-R_\tau\leq MC\tau^{-\delta}\), and its square is at most \(M^2C^2\tau^{-2\delta}\). Combining these bounds with the exact mean-squared-error decomposition proves the result. \(\square\)

When ratios are known to satisfy \(w\leq W\) almost surely, clipping at any \(\tau\geq W\) introduces no bias, since \((w-\tau)_+=0\). In that case the un-clipped weighted loss is bounded by \(MW\), and its variance is finite. A lower threshold may still be chosen for practical stability, but it sacrifices exact target-risk estimation and should be assessed through the bias expression.

Practical Interpretation and a Common Pitfall

Clipping is useful because it imposes a deterministic cap on each weighted loss, and under finite-moment assumptions it yields explicit bounds even when the un-clipped second moment is not controlled. Its cost is that the clipped expectation is generally not the target risk. In particular, clipping does not preserve \(E_P[w]=1\): typically \(E_P[w_\tau]<1\), so even a constant positive loss can have a lower clipped risk. Renormalizing the clipped weights would define a different estimator and require a separate bias analysis.

Common mistake: “Every observation has a smaller weight after clipping, so the sample variance must be smaller.” The premise only gives a pointwise comparison of weights, not a comparison of variances. Variance depends on both the size and the distribution of the weighted losses around their respective means. In the first worked example, the un-clipped weighted loss was constant, while clipping made it vary. The correction is to say that clipping bounds individual contributions and may reduce sampling variability or a variance bound; it does not guarantee a reduction in actual variance.