Importance-Weighted Risk Under Covariate Shift

Why Reweight Classification Risk?

A classifier may be evaluated in a population whose feature distribution differs from the population that supplied its labeled training or assessment data. For example, a diagnostic rule may be assessed in a clinic with a different mix of patient measurements. If the relationship between features and labels remains stable, the changed feature mix still changes the average error rate. Importance weighting expresses that target-population risk using labeled source observations.

The key distinction is between a change in the distribution of features and a change in the conditional distribution of labels given features. This tutorial considers the former: the class posterior is invariant, while the marginal distribution of the feature vector can differ. The resulting risk identity applies to a fixed classifier, including a fitted classifier when its evaluation observations are independent of the data used to fit it.

Absolute continuity says that any feature region assigned zero probability by the source must also have zero probability in the target. When both feature distributions have densities \(f_S\) and \(f_T\) with respect to the same measure, the ratio is \(r(x)=f_T(x)/f_S(x)\) wherever \(f_S(x)>0\). The ratio changes the contribution of each source feature value according to how common it is in the target relative to the source.

The Target-Risk Identity

Let \(g\) be a measurable classifier mapping features to labels, and let \(L(g(X),Y)\) be a measurable loss. For misclassification risk, \(L(g(X),Y)=\mathbf{1}\{g(X)\ne Y\}\). More generally, the argument works for any nonnegative loss whose target expectation is finite.

To prove the identity, define \(h(x)=E_S[L(g(x),Y)\mid X=x]\). Conditional invariance implies that the same conditional expectation is obtained under \(P_T\), for \(P_T^X\)-almost every \(x\). The law of iterated expectation gives

The third equality is the defining change-of-measure property of the Radon–Nikodym derivative \(r\); the last follows by conditioning on \(X\) under the source distribution. Finiteness of the target risk ensures the integrals are well-defined. Thus source labels can identify target risk through a weighted expectation, without requiring target labels. \(\square\)

Estimating Risk from Labeled Source Observations

Suppose \((X_i,Y_i)\), \(i=1,\ldots,n\), are independent and identically distributed source observations. If \(g\) and the ratio \(r\) are fixed and the ratios are known, the empirical importance-weighted estimator is

For zero-one loss, each observed error is multiplied by the relative target frequency of that observation’s feature value. The estimator uses the source labels and feature ratios; it does not require a labeled target sample.

For unbiasedness, linearity of expectation and the risk identity give

For consistency, the summands \(Z_i=r(X_i)L(g(X_i),Y_i)\) are independent and identically distributed, with \(E_S[|Z_i|]=R_T(g)<\infty\) for a nonnegative loss. The strong law of large numbers therefore gives \(n^{-1}\sum_i Z_i\xrightarrow{a.s.}E_S[Z_i]=R_T(g)\). \(\square\)

When the summands have finite variance, independence also gives

Thus large or highly variable ratios can make a risk estimate imprecise even when it remains unbiased. In particular, a finite target risk does not by itself guarantee finite estimator variance: that additionally requires \(E_S[(r(X)L(g(X),Y))^2]<\infty\). The existence of the weighting identity and the practical stability of its estimate are separate issues.

Conditions and Common Pitfalls

Importance weighting is not an automatic correction for every difference between populations. It specifically relies on the assumption that the conditional label distribution is invariant. If \(P_T(Y\mid X)\) changes, reweighting source features cannot repair the changed relationship between features and labels. This is different from changing only the feature marginal, which is the setting considered here.

• Using an unweighted source error. This estimates source risk, not target risk, unless the feature distributions agree or the particular losses happen to average the same way.
• Ignoring overlap. If a target region has zero source probability, no finite feature ratio exists there, and its conditional error cannot be recovered from source labels by this identity.
• Treating unbiasedness as precision. Large ratios can make individual observations influential and increase variance. An unbiased estimate can still be unstable in a finite sample.
• Evaluating a fitted classifier on its own training observations. The fixed-classifier unbiasedness argument does not generally apply when the same observations were used to select or fit \(g\). Use independent evaluation observations, or an evaluation design that separates fitting from assessment.
• Assuming the invariance condition follows from reweighting. The ratio changes the distribution of \(X\); it does not test or enforce equality of \(P_S(Y\mid X)\) and \(P_T(Y\mid X)\).

If \(g\) is trained on a separate data set, it can be treated as fixed when conditioning on that training data. An independent source evaluation sample then supports the same identity and estimator for the fitted rule, conditional on the rule. By contrast, when \(g\) is trained using the evaluation observations, its prediction errors and those observations are dependent; a separate justification is needed.