Why the Type of Change Matters
A predictive model is built from data collected under some set of conditions and then used under operational conditions that may differ. A change in the kinds of cases presented to the model is not the same as a change in how those cases relate to the outcome. That distinction matters: it determines whether a model’s learned conditional predictions may still be useful, whether aggregate risk may change anyway, and whether a calculation such as importance weighting has a sound basis.
Let \(X\) denote the predictors and \(Y\) the outcome. Use \(s\) for the source setting represented by the training data and \(t\) for the target setting where the model is used. The distribution \(P_s(X)\) describes the predictor mix in the source setting; \(P_s(Y\mid X)\) describes the outcome distribution conditional on those predictors. The corresponding target distributions are \(P_t(X)\) and \(P_t(Y\mid X)\).
Here, “concept drift” refers specifically to a change in the conditional relationship between predictors and outcome. Terminology can vary across applications, so a careful analysis should state which distribution changed rather than relying on a drift label alone. A changed predictor mix does not, by itself, prove that the conditional relationship changed. Conversely, unchanged predictor frequencies do not guarantee that the relationship is still valid.
Covariate Shift: The Cases Have Changed Mix
Under covariate shift, the types of observations reaching a model have changed, but the outcome distribution for a given predictor value is assumed unchanged. For a classifier, for example, the event probability for a specified risk profile could remain the same even as that profile becomes more or less common. The model’s conditional predictions may therefore still describe the same relationship, while averages across all cases change because the cases are weighted differently.
This distinction is important for aggregate quantities. If a model predicts an event probability \(f(X)\), its average prediction in a setting is \(E[f(X)]\), which depends on that setting’s predictor distribution. Even if \(f(X)\) is unchanged, a different distribution of \(X\) can produce a different average. It is therefore incorrect to infer that a model’s aggregate risk estimate is unchanged merely because its conditional relationship is believed to be stable.
Worked Example: A Changing Mix of Risk Bands
A hypothetical insurer uses a model to estimate the probability of a claim within one year. There are two predictor profiles, Low and High. In the training population, 80% of applicants are Low and 20% are High. In a later target population, 40% are Low and 60% are High. Assume the claim probabilities conditional on profile remain 0.05 for Low and 0.20 for High. Calculate each population’s aggregate claim probability and classify the change.
Method. Multiply each profile’s conditional claim probability by its proportion in the population, then add. The source aggregate is \(0.80(0.05)+0.20(0.20)=0.04+0.04=0.08\), or 8%.
The target aggregate is \(0.40(0.05)+0.60(0.20)=0.02+0.12=0.14\), or 14%. The target population’s aggregate probability is higher by \(0.14-0.08=0.06\), or 6 percentage points.
The conditional probabilities did not change, but the predictor mix did. This is covariate shift under the stated assumption, not evidence of concept drift. Common mistake avoided: treating unchanged conditional probabilities as proof that the aggregate claim probability must also remain unchanged. The two aggregate calculations show that conclusion is false.
Concept Drift: The Relationship Has Changed
Concept drift means the conditional outcome distribution has changed. In a binary-outcome setting, this can mean that the probability of an event for the same predictor profile is different in the target setting. A model fitted to source data may then produce systematically stale conditional predictions, even if the predictor mix has not changed.
Potential causes depend on the application. A new intervention can alter outcomes for people with the same recorded predictors. A change in operating rules can affect the relationship between characteristics and a decision outcome. Changes in outcome definitions or measurement can also create an apparent change in the conditional distribution. The earlier material on outcome definition, data quality, and assumptions about the prediction setting is relevant when deciding whether an observed difference reflects a real process change or a change in what was recorded.
Worked Example: Changed Event Probabilities with a Stable Mix
A hypothetical service provider predicts whether a customer will miss a scheduled appointment. Customers are grouped as Routine or Complex. Both the source and target populations are 70% Routine and 30% Complex. In the source setting, the missed-appointment probabilities are 0.10 for Routine and 0.30 for Complex. In the target setting, they are 0.25 and 0.50, respectively. Calculate the aggregate probabilities and identify the type of change.
Source aggregate. The Routine contribution is \(0.70(0.10)=0.07\). The Complex contribution is \(0.30(0.30)=0.09\). Thus the source aggregate is \(0.07+0.09=0.16\), or 16%.
Target aggregate. The Routine contribution is \(0.70(0.25)=0.175\). The Complex contribution is \(0.30(0.50)=0.15\). Thus the target aggregate is \(0.175+0.15=0.325\), or 32.5%.
The predictor mix is unchanged, but both conditional probabilities increased. This is concept drift. The aggregate increased by \(0.325-0.16=0.165\), or 16.5 percentage points. Common mistake avoided: attributing a changed aggregate to a changing mix without checking the conditional probabilities. Here, the mix is identical; the conditional relationship changed.
Why Importance Weighting Can Work Under Covariate Shift
Importance weighting is a way to estimate target-setting prediction risk from source data when the covariate-shift assumptions hold. Let \(f(X)\) be a fixed model’s prediction and let \(L(Y,f(X))\) be a loss function, such as squared error or classification loss. The target risk is the expected loss in the target setting. If the conditional outcome distribution is invariant and target cases occur only where source data provide support, target risk can be expressed as a source expectation with weights.
Under covariate shift, the conditional distribution of \(Y\) given \(X=x\) is the same in source and target, so the conditional expected loss is also the same. If the target density is zero wherever the source density is zero, the density ratio \(w(x)=p_t(x)/p_s(x)\) is defined wherever the target has positive density. Substituting this ratio gives the identity below.
The justification is a change of integration weights: the ratio converts source predictor frequencies into target predictor frequencies. It does not correct a change in \(P(Y\mid X)\). The identity requires both conditional invariance and adequate support in the source data for the target cases. If target cases occupy predictor regions absent from training data, their conditional outcomes cannot be recovered by this weighting formula. Very large weights can also make an estimate unstable because a small number of source observations then contribute disproportionately.
Worked Example: Reweighting Prediction Losses
A hypothetical model is assessed on three profiles, A, B, and C. Their source proportions are 0.50, 0.30, and 0.20; their target proportions are 0.20, 0.30, and 0.50. Assume the conditional outcome distributions are unchanged and the model’s expected loss conditional on each profile is 0.10, 0.20, and 0.40, respectively. Calculate source risk, target risk, and the target risk by importance weighting.
Direct source calculation. The contributions are \(0.50(0.10)=0.05\), \(0.30(0.20)=0.06\), and \(0.20(0.40)=0.08\). Thus \(R_s(f)=0.05+0.06+0.08=0.19\).
Direct target calculation. The contributions are \(0.20(0.10)=0.02\), \(0.30(0.20)=0.06\), and \(0.50(0.40)=0.20\). Thus \(R_t(f)=0.02+0.06+0.20=0.28\).
Importance weights. The profile-specific weights are target proportion divided by source proportion: \(w(A)=0.20/0.50=0.40\), \(w(B)=0.30/0.30=1\), and \(w(C)=0.50/0.20=2.50\). Weight each source contribution by its profile’s ratio: \(0.50(0.40)(0.10)+0.30(1)(0.20)+0.20(2.50)(0.40)=0.02+0.06+0.20=0.28\). The reweighted result agrees with the direct target calculation. As a check, the source-average weight is \(0.50(0.40)+0.30(1)+0.20(2.50)=0.20+0.30+0.50=1\).
The change is covariate shift under the given assumptions, and the target loss is higher because the target setting contains more profile C, where expected loss is largest. Common mistake avoided: applying importance weights as if they repair concept drift. They only adjust the predictor mix; if profile-specific outcome behavior changed, this calculation would not generally estimate target risk.
Separate the Two Questions
When comparing settings, ask two distinct questions: has the distribution of predictors changed, and has the outcome distribution conditional on the predictors changed? The answers yield four possibilities. Neither may have changed; only the predictor distribution may have changed; only the conditional relationship may have changed; or both may have changed. The first two can be consistent with a stable conditional model, but only the first leaves the overall population mix unchanged. The last two raise a direct concern that source conditional predictions may no longer apply.
| Predictor distribution | Conditional outcome distribution | Interpretation |
|---|---|---|
| Unchanged | Unchanged | No change of these two kinds is identified |
| Changed | Unchanged | Covariate shift; aggregate risk can still change |
| Unchanged | Changed | Concept drift; source conditional predictions may be stale |
| Changed | Changed | Both kinds of change; predictor reweighting alone is insufficient |
A practical interpretation should not overstate what is known. A changed predictor distribution is evidence that the mix differs, not proof that the conditional relationship is stable. Establishing conditional invariance requires outcome information and appropriate comparison; when outcomes are not yet available, the relationship may simply be unknown. Likewise, an apparent conditional change can be distorted by immature labels, changed measurement, or differing selection into the observed data.
Worked Example: A Change That Exceeds Source Support
A hypothetical equipment service model was trained on machines with vibration readings from 0 to 8 units. In a target facility, some machines have readings of 10 units. An analyst proposes to estimate target prediction loss by assigning density-ratio weights to the training observations. Explain whether this is justified for the machines with readings of 10 units.
The target has positive probability for a predictor value, \(X=10\), for which the source training distribution has no support: \(p_t(10)>0\) but \(p_s(10)=0\). Therefore the ratio \(p_t(10)/p_s(10)\) is undefined. More fundamentally, the source data do not identify the conditional outcome distribution at that value. The analyst cannot use source observations to estimate target loss there by importance weighting.
This does not by itself prove concept drift. It shows that the target includes a predictor region unsupported by the training data, so the covariate-shift reweighting identity cannot cover that region. The model’s predictions there rely on extrapolation or additional assumptions, which should be stated as limitations. Common mistake avoided: treating any distribution change as correctable by weighting, even when the target includes cases the source never represented.
Common Interpretation Traps
- “The predictor mix changed, so the relationship changed.” This conclusion does not follow. Covariate shift is specifically the case where the predictor distribution changes while the conditional outcome distribution is assumed stable.
- “The relationship is stable, so aggregate risk is stable.” This is also false. A different mix of profiles can change an average, as the first worked example demonstrates.
- “The observed outcome rate changed, so concept drift must have occurred.” An aggregate rate can change solely because the predictor mix changed. Compare conditional outcomes at comparable predictor values before attributing the change to concept drift.
- “Importance weighting fixes drift.” The weighting identity relies on the same conditional outcome distribution. It adjusts predictor frequencies, not the conditional relationship.
- “A predictor value outside training support is just a large-weight case.” If its source probability is zero, its density ratio is not finite or defined. That is a support problem, not an ordinary weighting problem.
The distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus because sound model use depends on describing the changed part of the prediction setting precisely. In a report, state whether the concern is a changed case mix, a changed predictor–outcome relationship, both, or insufficient evidence to tell. Then explain the consequences: aggregate risk can move under covariate shift, conditional predictions can become stale under concept drift, and reweighting is defensible only under its assumptions.
Check Your Understanding
For each problem, identify the type of change and show the key calculation or reasoning.
- A hypothetical warranty portfolio has Low- and High-usage customers. The source proportions are 0.75 and 0.25; the target proportions are 0.50 and 0.50. Conditional claim probabilities remain 0.04 and 0.16. Calculate both aggregate probabilities and identify the change. (Answer: Source \(0.07\); target \(0.10\); covariate shift only.)
- A service team’s customer mix is 60% Group A and 40% Group B in both periods. The conditional missed-deadline probabilities change from 0.10 and 0.20 to 0.10 and 0.35. Calculate the aggregate probability in each period and identify the change. (Answer: Source \(0.14\); target \(0.20\); concept drift only.)
- A model’s conditional expected losses for profiles A and B remain 0.05 and 0.25. Source proportions are 0.80 and 0.20; target proportions are 0.50 and 0.50. Calculate target risk directly and using source density-ratio weights. (Answer: Target risk \(0.15\); weights are 0.625 for A and 2.5 for B; the reweighted risk is \(0.80(0.625)(0.05)+0.20(2.5)(0.25)=0.025+0.125=0.15\).)
- A model’s source predictor mix is 50% A and 50% B, with conditional event probabilities 0.10 and 0.30. In the target, the mix is 25% A and 75% B, with probabilities 0.20 and 0.40. Calculate target aggregate probability and classify the changes. (Answer: \(0.25(0.20)+0.75(0.40)=0.35\); both covariate shift and concept drift.)
- Training data contain predictor values only from 1 to 5, while a target population includes values of 7. What prevents ordinary importance weighting from estimating risk for those target cases? (Answer: The target has positive mass outside source support, so the source density is zero there, the density ratio is undefined, and the source data do not identify the conditional outcome distribution for those cases.)