Target-Risk Bounds With Estimated Weights

Why Estimated Weights Require Care

Under covariate shift, the distribution of predictors changes between a source population, where labelled data are available, and a target population, where performance is to be assessed. If the conditional distribution of the outcome given the predictors remains the same, a density ratio can translate source expectations into target expectations. In practice, however, that ratio is estimated. The resulting risk error can reflect sampling noise, error in the estimated conditional loss, error in the estimated ratio, or interactions between those errors.

This tutorial makes those contributions explicit. It first gives the decomposition for ordinary importance weighting, then derives a bound for an augmented estimator that uses both a conditional-loss estimate and estimated weights. The augmented bound has a product remainder: if either nuisance estimate is accurate, that remainder is small. This does not remove the need for adequate samples or overlap, but it clarifies which errors a guarantee must control.

Setup and Definitions

Let \(X\) denote predictors and \(Y\) an outcome. Write \(P\) for the source distribution of \((X,Y)\), and \(Q\) for the target distribution. Assume covariate shift: \(P(Y\mid X)=Q(Y\mid X)\). Let \(P_X\) and \(Q_X\) be the corresponding predictor distributions, and assume \(Q_X\) is absolutely continuous with respect to \(P_X\). The density ratio is \(w(x)=dQ_X/dP_X(x)\), so \(E_P[w(X)]=1\).

For a prediction rule \(h\), let \(\ell(h(X),Y)\) be its loss, assumed measurable and between zero and one. Its target risk is \(R_Q(h)=E_Q[\ell(h(X),Y)]\). Define the conditional loss \(m(x)=E_P[\ell(h(X),Y)\mid X=x]\). By covariate shift, this is also the target conditional loss, up to sets of predictor values having probability zero under \(Q_X\).

The change-of-measure identity follows from the definition of \(w\): for any integrable function \(g\) of \(X\), \(E_Q[g(X)]=E_P[w(X)g(X)]\). Applying it to the conditional expectation of the loss gives the risk identity. No claim that the ratio is known or accurately estimated follows from this identity; those are separate statistical issues.

Ordinary Importance Weighting: An Additive Error Decomposition

Suppose an evaluation sample \((X_i,Y_i)_{i=1}^n\) is drawn independently from \(P\), and a fitted weight function \(\tilde w\) is fixed before this sample is observed. The direct weighted estimate is \(\hat R_{\mathrm{IPW}}=n^{-1}\sum_{i=1}^n\tilde w(X_i)\ell(h(X_i),Y_i)\). Its error has an exact decomposition:

Here \(P_ng=n^{-1}\sum_{i=1}^n g(X_i,Y_i)\), and \(P g=E_P[g(X,Y)]\). The first term is evaluation-sample fluctuation with the fitted weights held fixed. The second is the population error caused by using \(\tilde w\) instead of \(w\). To verify the decomposition, add and subtract \(E_P[\tilde w\ell]\), then use \(E_P[\tilde w\ell]=E_P[\tilde w m]\) and \(R_Q(h)=E_P[wm]\).

Since \(0\leq m\leq1\), the absolute value of the ratio-error term is at most \(E_P[|\tilde w-w|]\). This bound shows why a good average fit to the ratio can matter. But it also hides which parts of the loss are affected by ratio error: if the conditional loss is near zero where the ratio is inaccurate, the signed expectation can be much smaller than this general bound.

An Augmented Estimator and a Product Remainder

A conditional-loss estimate can be used to reduce sensitivity to ratio error. Take a labelled source evaluation sample of size \(n\), and an independent target sample of predictors \(X'_1,\ldots,X'_M\). Let \(\hat m\) and a nonnegative fitted weight \(\tilde w\) be trained using data independent of both evaluation samples. Define

The first term predicts conditional loss over the target predictor distribution. The second corrects that prediction using source outcomes, weighted to represent the target. For the following bound, assume \(\hat m\) takes values in \([0,1]\), and that \(0\leq\tilde w\leq\tau\) for some finite \(\tau\). Define \(\|g\|_{L_2(P_X)}=(E_P[g(X)^2])^{1/2}\).

Proof. First take the conditional expectation of the estimator over fresh evaluation samples, keeping the trained functions fixed. Conditional expectation of the source loss given \(X\) is \(m(X)\), so

Subtract \(R_Q(h)=E_P[wm]\), and use \(E_Q[\hat m]=E_P[w\hat m]\). The difference between this conditional mean and the target risk is exactly

By the Cauchy–Schwarz inequality, its absolute value is at most \(\|\tilde w-w\|_{L_2(P_X)}\|\hat m-m\|_{L_2(P_X)}\).

It remains to bound the two centered sample averages. Conditional on the training data, each target term \(\hat m(X'_j)\) lies in \([0,1]\). Hoeffding's inequality therefore bounds its average's deviation from its mean by \(\sqrt{\log(4/\delta)/(2M)}\), except on an event of probability at most \(\delta/2\). Each source correction term \(\tilde w(X_i)(\ell-\hat m(X_i))\) lies in \([-\tau,\tau]\), an interval of width \(2\tau\). Hoeffding's inequality bounds its average's deviation by \(\tau\sqrt{2\log(4/\delta)/n}\), also except on an event of probability at most \(\delta/2\). A union bound makes both bounds hold together with probability at least \(1-\delta\). Adding these deviations to the product remainder proves the claim. \(\square\)

The product structure is important: the population remainder is zero if \(\tilde w=w\), and it is also zero if \(\hat m=m\). With both estimates imperfect, the theorem bounds their interaction rather than treating either error as though it were absent. The sampling terms remain, so the result does not assert that nuisance estimation alone guarantees an accurate risk estimate.

Why Sample Splitting and Truncation Matter

Sample splitting means fitting the prediction rule and nuisance functions on training data, then evaluating the risk using independent observations. In the theorem, that separation lets us condition on the fitted functions: the evaluation terms are then averages of independent variables with fixed, bounded ranges. If a weight or loss estimate is fitted using the same observations that appear in the evaluation average, this conditional argument does not apply as written. More elaborate dependence-aware arguments, such as suitable cross-fitting analyses, require their own conditions; independence should not be silently assumed.

Truncation replaces a nonnegative estimated ratio by \(\tilde w(x)=\min\{\hat w(x),\tau\}\). This enforces the upper bound needed for the source concentration term and limits the influence of individual observations. Truncation can increase ratio error, so it is not a free improvement. For direct importance weighting with a true ratio clipped at \(\tau\), the population bias is

where \((u)_+=\max(u,0)\), using \(0\leq m\leq1\). With estimated ratios, there can also be error from \(\hat w\) itself. In the augmented estimator, clipping enters the product remainder through \(\|\tilde w-w\|_{L_2(P_X)}\). If the conditional-loss estimate is exact, the population remainder is zero even when the weight is clipped; the finite-sample correction term still has variability.

Common Pitfall and Practical Interpretation

A common mistake is to treat estimated weights as if they were fixed and correct merely because a weighted average has the right form. The change-of-measure identity is exact for \(w\), not for an arbitrary estimate \(\tilde w\). Replacing \(w\) by \(\tilde w\) introduces a population error; an ordinary concentration bound for the evaluation sample does not remove it. The augmented estimator makes this explicit: its finite-sample fluctuations and its product remainder are different parts of the guarantee.

A second mistake is to conclude that a small product remainder guarantees a good estimate in a finite sample. The theorem also requires its two sampling terms to be controlled, and it relies on bounded losses, bounded fitted weights, and independent evaluation samples. In applications, report how the truncation threshold was chosen and assess whether the estimated ratios are unstable in regions that contribute to target risk. A formal bound is informative only when its assumptions and error terms are relevant to the data at hand.