Sensitivity To Conditional Drift

Why Conditional Drift Needs Its Own Error Term

Reweighting source observations can account for a change in the distribution of features, but it does not by itself account for a change in the outcome distribution given those features. For example, the relationship between a clinical risk factor and an outcome may differ between the source and target populations. Even if the target feature distribution is well represented in the source, that conditional change can alter prediction risk.

The useful distinction is between marginal shift, which changes how often feature values occur, and conditional drift, which changes the outcome law at a given feature value. The latter contributes a transfer discrepancy between target risk and the risk obtained by evaluating source conditional outcomes at target feature values. This discrepancy is sometimes called conditional-drift bias; it is a transfer error, not necessarily the sampling bias of a particular estimator.

Risks and Conditional Drift

Let $P_S$ and $P_T$ be the source and target joint distributions of features $X$ and outcome $Y$. Write $P_S^X$ and $P_T^X$ for their feature marginals, and $P_S(dy\mid x)$ and $P_T(dy\mid x)$ for conditional outcome distributions. Assume the feature and outcome spaces are standard Borel spaces, so regular conditional distributions exist. Let $h$ be a measurable predictor and let $\ell(h(x),y)$ be a jointly measurable loss.

Definition: The source risk and target risk are $$ R_S(h)=\int\!\int \ell(h(x),y)\,P_S(dy\mid x)\,P_S^X(dx), \qquad R_T(h)=\int\!\int \ell(h(x),y)\,P_T(dy\mid x)\,P_T^X(dx). $$ The target-feature, source-conditional risk is $$ R_{S\to T}(h)=\int\!\int \ell(h(x),y)\,P_S(dy\mid x)\,P_T^X(dx). $$ When $P_T^X$ is absolutely continuous with respect to $P_S^X$, this last risk is the ideal covariate-reweighted source risk.

The absolute-continuity condition $P_T^X\ll P_S^X$, often expressed as an overlap or positivity requirement, ensures that target feature values are represented under the source feature distribution and that the density ratio $w(x)=dP_T^X/dP_S^X(x)$ is defined $P_S^X$-almost surely. Under this condition, whenever the relevant integrals exist,

R_{S\to T}(h) = \int w(x)\ell(h(x),y)\,P_S(dx,dy).

This change of measure adjusts the feature marginal while retaining the source conditional outcome law. Conditional drift is precisely the part that remains uncorrected. The definitions and decomposition below only require source conditional distributions to be defined $P_T^X$-almost surely; the overlap condition is additionally needed for the density-ratio representation.

The Risk Decomposition

Add and subtract $R_{S\to T}(h)$ to separate two distinct differences. The first compares target and source conditional outcomes, while holding the feature distribution at its target value. The second compares feature marginals while holding the conditional outcome law at its source value.

Theorem (risk decomposition): Suppose the risks in the following expression are finite. Then $$ R_T(h)-R_S(h) = \underbrace{\bigl[R_T(h)-R_{S\to T}(h)\bigr]}_{\text{conditional-drift contribution}} + \underbrace{\bigl[R_{S\to T}(h)-R_S(h)\bigr]}_{\text{feature-marginal contribution}}. $$ In particular, the conditional-drift contribution is exactly $$ B_{\mathrm{cond}}(h) = \int \left[ \int \ell(h(x),y)\,P_T(dy\mid x) - \int \ell(h(x),y)\,P_S(dy\mid x) \right]P_T^X(dx). $$

Proof. By the definitions, $R_T(h)$ and $R_{S\to T}(h)$ integrate the target and source conditional outcome laws, respectively, against the same feature marginal $P_T^X$. Subtracting their integral expressions gives the displayed formula for $B_{\mathrm{cond}}(h)$. For the full decomposition, write

R_T(h)-R_S(h) = \bigl[R_T(h)-R_{S\to T}(h)\bigr] + \bigl[R_{S\to T}(h)-R_S(h)\bigr],

because the two occurrences of $R_{S\to T}(h)$ cancel. This is an exact identity; it does not require either contribution to have the same sign. $\square$

The decomposition matters because the two contributions can offset each other. A small difference between raw source and target risks does not establish that conditional drift is small: a substantial conditional-drift contribution may be cancelled by a feature-marginal contribution of the opposite sign.

Worked Example: Binary Outcomes at Two Feature Values

Let $X$ take values $a,b$. Suppose $P_S^X(a)=P_S^X(b)=0.5$, while $P_T^X(a)=0.75$ and $P_T^X(b)=0.25$. The outcome $Y$ is binary. A fixed classifier predicts $h(x)=0$ at both feature values, and the loss is zero-one loss, $\ell(h(x),y)=\mathbf{1}\{h(x)\ne y\}$. The source and target conditional probabilities of $Y=1$ are

\begin{array}{c|cc} & a & b\\ P_S(Y=1\mid X=x) & 0.1 & 0.4\\ P_T(Y=1\mid X=x) & 0.3 & 0.2 \end{array}

Because the classifier always predicts zero, its conditional error probability is $P(Y=1\mid X=x)$. The target risk is therefore $R_T(h)=0.75(0.3)+0.25(0.2)=0.225+0.050=0.275$. The source risk is $R_S(h)=0.5(0.1)+0.5(0.4)=0.050+0.200=0.250$.

The target-feature, source-conditional risk is $R_{S\to T}(h)=0.75(0.1)+0.25(0.4)=0.075+0.100=0.175$. Thus the conditional-drift contribution is $0.275-0.175=0.100$, and the feature-marginal contribution is $0.175-0.250=-0.075$. Their sum, $0.100-0.075=0.025$, agrees with $R_T(h)-R_S(h)=0.275-0.250=0.025$. The raw risk difference is small because the two contributions partly cancel.

Bounding Conditional Drift With Total Variation

For probability distributions $P,Q$ on the same measurable space, use the convention

d_{\mathrm{TV}}(P,Q)=\sup_A |P(A)-Q(A)|,

where the supremum is over measurable sets. With this convention, if $g$ takes values in an interval of width $M$, then its expectation difference is at most $M d_{\mathrm{TV}}(P,Q)$, not $2M d_{\mathrm{TV}}(P,Q)$. The factor of two sometimes seen in this bound corresponds to a different convention for total variation distance.

Theorem (bounded-loss drift bound): Suppose that for a fixed predictor $h$, the loss satisfies $a\leq \ell(h(x),y)\leq b$ for all $x,y$, with $a,b$ finite. Then $$ |B_{\mathrm{cond}}(h)| \leq (b-a)\int d_{\mathrm{TV}}\!\left(P_T(\,\cdot\mid x),P_S(\,\cdot\mid x)\right)\,P_T^X(dx). $$ In particular, if the conditional total-variation distance is at most $\delta$ for $P_T^X$-almost every $x$, then $|B_{\mathrm{cond}}(h)|\leq (b-a)\delta$.

Proof. First consider any measurable function $g$ taking values in $[0,M]$. For a probability distribution $P$, the layer-cake identity gives

\int g\,dP=\int_0^M P\{g>t\}\,dt.

This follows by writing $g(z)=\int_0^M \mathbf{1}\{g(z)>t\}\,dt$ for every $z$, then interchanging the integrals by Tonelli's theorem; the integrand is nonnegative and measurable. Applying the identity to $P$ and $Q$, taking an absolute value, and using the definition of total variation yields

\left|\int g\,dP-\int g\,dQ\right| \leq \int_0^M |P\{g>t\}-Q\{g>t\}|\,dt \leq M d_{\mathrm{TV}}(P,Q).

For a function taking values in $[a,b]$, apply this result to $g-a$, which takes values in $[0,b-a]$; subtracting the same constant from both expectations leaves their difference unchanged. Now fix $x$ and apply the resulting inequality to $g(y)=\ell(h(x),y)$, with $P=P_T(\cdot\mid x)$, $Q=P_S(\cdot\mid x)$, and $M=b-a$. Integrating the pointwise bound with respect to $P_T^X$ and using $\left|\int q\,dP_T^X\right|\leq\int |q|\,dP_T^X$ proves the theorem. $\square$

The bound uses the full range of the loss, so it can be conservative. A sharper assessment for a particular predictor comes from looking directly at how its loss changes under the two conditional laws.

Definition: The loss-specific conditional discrepancy at $x$ for predictor $h$ is $$ \Delta_{\ell}(h,x)= \left| \int \ell(h(x),y)\,P_T(dy\mid x) - \int \ell(h(x),y)\,P_S(dy\mid x) \right|. $$ Consequently, $|B_{\mathrm{cond}}(h)|\leq\int\Delta_{\ell}(h,x)\,P_T^X(dx)$. If this discrepancy is at most $\varepsilon$ for target-almost every $x$, then $|B_{\mathrm{cond}}(h)|\leq\varepsilon$.

Worked Example: A Large Distribution Change With No Risk Change

At a single feature value, let $Y$ be binary, let $h(x)=0.5$, and use squared-error loss $\ell(h(x),Y)=(h(x)-Y)^2$. Suppose $P_S(Y=1\mid x)=0.2$ and $P_T(Y=1\mid x)=0.8$. The conditional distributions are Bernoulli, so their total-variation distance is $|0.8-0.2|=0.6$.

Under either conditional law, the squared error is $0.25$: if $Y=0$, then $(0.5-0)^2=0.25$; if $Y=1$, then $(0.5-1)^2=0.25$. Thus the source conditional risk is $0.25$, the target conditional risk is $0.25$, and the loss-specific discrepancy and conditional-drift contribution are both zero. The loss lies in $[0,1]$, so the total-variation theorem gives only the upper bound $1(0.6)=0.6$. The distributional bound is valid but not tight; this predictor's particular loss is insensitive to this change in outcome probabilities.

Why Boundedness Cannot Be Omitted

A total-variation bound controls expectation differences uniformly over bounded functions. Without a bounded loss, a small probability assigned to an extreme outcome can have a large effect on risk. Finite risks alone do not give a bound proportional to total variation.

Worked Example: Small Total Variation, Large Squared-Loss Drift

Consider a single feature value, predictor $h(x)=0$, and squared-error loss $\ell(h(x),Y)=Y^2$. Under the source conditional law, let $Y=0$ with probability one. Under the target conditional law, let $Y=100$ with probability $0.01$ and $Y=0$ with probability $0.99$. The total-variation distance is $0.01$: the only probability mass moved is $0.01$, from $Y=0$ to $Y=100$.

The source risk is $E_S[Y^2\mid x]=0$. The target risk is $E_T[Y^2\mid x]=0.01(100^2)+0.99(0^2)=0.01(10{,}000)=100$. So the absolute conditional-drift contribution is $100$, even though total variation is only $0.01$. Replacing $100$ by any $M>0$ gives total variation $0.01$ and risk difference $0.01M^2$, which grows without bound as $M$ increases. Both risks are finite for each fixed $M$; there is simply no bound of the bounded-loss form based on total variation alone.

Common Pitfall and Practical Use

Wrong reasoning: “The target feature distribution is covered by the source, and I have used density-ratio weights, so the weighted source risk equals target risk.” The weighting changes the feature marginal in the integral, but it leaves $P_S(dy\mid x)$ in place. It therefore gives $R_{S\to T}(h)$, not $R_T(h)$, unless the conditional outcome laws agree in the risk-relevant sense.

Correction: Separate the target risk from the ideal weighted source risk. Their difference is the conditional-drift contribution. It is zero if the source and target conditional outcome distributions agree $P_T^X$-almost everywhere. More generally, it can be zero for a particular predictor and loss even when those distributions differ, as the squared-loss example shows. When the loss is bounded, average conditional total variation gives a distribution-level bound; when the loss is unbounded, additional control on the loss or outcome tails is necessary.

In applications, the decomposition clarifies what a weighting analysis can and cannot address. It separates feature-marginal transfer from changes in the outcome mechanism, and makes explicit which discrepancy assumptions support a risk claim. A small estimated marginal-shift error does not, on its own, establish small conditional drift.

Check Your Understanding

Use the definitions and bounds above to answer the following questions.

In the two-feature example, what are the conditional-drift and feature-marginal contributions, and why can they not be replaced by the raw source-to-target risk difference?
Under the stated total-variation convention, what upper bound applies to the difference in expectations of a function taking values in $[2,7]$?
For a fixed predictor, how can conditional laws differ while the loss-specific conditional discrepancy is zero?
Which part of the risk decomposition is represented by ideal density-ratio weighting of the source distribution?
Why does finite squared-error risk under both domains not imply a useful total-variation bound when outcomes are unbounded?