Why Transportability Matters
A predictive model is developed using data from a particular time, population and operating environment. Its predictions may later be used for a different group, under a revised product, or after a change in business practice. The model can still produce numbers in those circumstances, but that does not establish that the numbers remain accurate or useful.
Transportability is the extent to which a model developed in one setting can provide reliable predictions in a different target setting. The question is not simply whether the model software runs or whether its inputs have the same names. The analyst must assess whether the target setting preserves the relationships and definitions on which the predictions depend. This assessment is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Transportability is always relative to a purpose. A model might remain useful for ordering cases from lower to higher expected risk while no longer providing accurate numerical probabilities. Or its predictions might remain statistically accurate but cease to support a decision because the costs or available actions have changed. State what “adequate” means for the intended decision rather than treating transportability as an all-or-nothing property.
What Must Be Comparable?
A transfer assessment begins by describing what changed and why the model’s predictions might be affected. Several dimensions are worth checking together:
- Population: Are the people, policies, claims or other units in the target setting similar to those represented in the development data? A different mix of known risk groups can change average performance even if predictions within each group remain accurate.
- Predictor measurement: Do variables have the same definitions, timing, units and data-collection process? A field called “prior incidents” is not comparable if one setting counts incidents over six months and the other over twelve.
- Outcome definition and maturity: Is the predicted outcome measured in the same way and over the same horizon? A model trained on reported events may not predict a newly defined event or one observed after a different follow-up period.
- Products and terms: Have deductibles, limits, eligibility rules, coverage or benefit designs changed? A model predicting a quantity tied to the old terms may not predict the corresponding quantity under the new terms.
- Practices and processes: Do underwriting, claims handling, referral, review or customer-service practices alter the outcome being predicted? A model may be sensitive to how the organization acts, not only to the characteristics of the cases.
- Predictor support: Does the source data include cases comparable to the target cases? Predictions for combinations of characteristics never represented in the source data rely on extrapolation.
The dimensions can interact. For example, a new sales channel may attract a different population and also change what information is recorded. A careful assessment documents the changes, explains how they could affect predictions, and identifies what evidence could address the uncertainty.
A Useful Calculation: Target-Setting Prediction Loss
To make the issue precise, let \(\ell(Y,f(X))\) be a chosen prediction loss, such as squared error for a numerical prediction or an indicator of incorrect classification. The target risk is the expected loss for cases drawn from the target setting. For a discrete predictor \(X\), the law of total expectation gives:
This identity separates two ingredients: how common each predictor pattern is in the target population, and the model’s average loss within that pattern. It follows by partitioning the expectation across the mutually exclusive values of \(X\), then weighting each conditional expectation by the probability of its group.
If the conditional outcome behaviour and measurement are unchanged between settings, then the conditional loss at each \(x\) can be estimated from source data. In that case, target risk can be estimated by weighting those source conditional losses using the target distribution of \(X\). But this calculation requires comparable predictors and outcomes, and source observations for the relevant target patterns. If either the conditional relationship or the definitions change, the source conditional losses need not apply. If the target includes predictor values absent from the source, the corresponding conditional losses cannot be recovered from source data alone.
This formula is a way to organize the assessment, not a guarantee of transfer. In practice, \(X\) may contain many variables or continuous measurements, so estimating each conditional loss precisely can be difficult. Target data with outcomes, when available and comparable, provide more direct evidence than assumptions alone.
Worked Example: A Changed Mix of Risk Groups
A hypothetical insurer uses a model to predict whether a policy will have a claim during the next year. In the source setting, the model predicts a claim probability of 0.08 for the low-risk group and 0.30 for the high-risk group. Source validation found observed rates of 0.08 and 0.30, respectively. The source population is 70% low-risk and 30% high-risk. In a proposed target setting, the group mix is 40% low-risk and 60% high-risk. Assume, for this calculation only, that the two group-specific outcome rates and the outcome definition remain unchanged. Find the source and target average predicted probabilities and explain what the difference means.
Method. Calculate each setting’s average prediction as the weighted average of its group-specific predictions. The stated assumption allows the source group-specific rates to serve as target group-specific rates for this illustration.
Source average. The low-risk contribution is \(0.70 \times 0.08=0.056\). The high-risk contribution is \(0.30 \times 0.30=0.090\). Therefore, the source average prediction is \(0.056+0.090=0.146\), or 14.6%.
Target average. The low-risk contribution is \(0.40 \times 0.08=0.032\). The high-risk contribution is \(0.60 \times 0.30=0.180\). Therefore, the target average prediction is \(0.032+0.180=0.212\), or 21.2%.
The target average is \(0.212-0.146=0.066\), or 6.6 percentage points higher. Under the stated assumption, that difference is explained by the changed group mix; it does not by itself show that the model’s group-specific predictions have become inaccurate. The source average of 14.6% should not be carried over as the target average.
Check. The target weights sum to 1, and its 60% high-risk group receives the larger weight on the 30% prediction. The target average of 21.2% is therefore above the source average, as expected. The common mistake this method avoids is treating the source aggregate rate as fixed when the target population has a different composition.
Products, Practices and Outcomes Can Change the Prediction Task
A model’s outcome is part of the task it learned. When products or procedures change, the quantity being predicted may change as well. A model that predicts gross loss, for instance, does not automatically predict an insurer’s payment after a new deductible and limit. Similarly, a change in review practice can alter whether an event is recorded, prevented or escalated. These are not merely changes in the number of cases in each group; they may affect the meaning or generation of the outcome.
Where a change is a clearly specified transformation, it may be possible to apply that transformation to sufficiently detailed model outputs. That is different from assuming that an old prediction remains valid without adjustment. The analyst should establish that the inputs contain the detail needed for the transformation and that the relevant terms are correctly specified.
Worked Example: A New Deductible and Payment Limit
Consider a hypothetical product with an equal probability of three gross claim amounts: $400, $1,000 and $2,000. A model provides these three amounts as its gross-loss predictions for the corresponding cases. The source product pays the gross loss. The proposed target product applies a $500 deductible and a maximum payment of $1,000 per claim. Calculate the average predicted gross loss and the average predicted target payment. Then show why applying the contract terms only to the average gross loss gives a different answer.
Method. Apply the target contract terms to each claim amount separately, then average the resulting payments. For gross loss \(X\), the payment is \(Y=\min(\max(X-500,0),1000)\).
Gross-loss average. The three amounts total \(\$400+\$1{,}000+\$2{,}000=\$3{,}400\). With equal probabilities, the mean gross loss is \(\$3{,}400/3=\$1{,}133.33\), rounded to the nearest cent.
Target payments. For a gross loss of $400, the amount above the deductible is \(\max(400-500,0)=0\), so the payment is \(\min(0,1000)=\$0\). For a gross loss of $1,000, the amount above the deductible is \(\max(1000-500,0)=\$500\), so the payment is \(\min(500,1000)=\$500\). For a gross loss of $2,000, the amount above the deductible is \(\max(2000-500,0)=\$1{,}500\), so the payment is \(\min(1500,1000)=\$1{,}000\). The average target payment is \((0+500+1000)/3=\$500.00\).
Why transforming the average fails. Applying the contract terms to the mean gross loss gives \(\min(\max(1{,}133.33-500,0),1000)=\$633.33\), not $500.00. The deductible and limit act on each claim amount, and averaging after applying them is not equivalent to applying them after averaging. The common mistake is to transform an average when the contract adjustment must be made at the individual-claim level.
Check. The three target payments sum to $1,500, and dividing by the three equally likely outcomes gives $500.00. This calculation assumes the model’s three gross-loss predictions are appropriate for these cases; it does not establish that the model will remain accurate for all losses under the new product.
Use Target Evidence and State the Limits
A transfer decision should identify what is known, what is assumed and what remains unverified. Source validation establishes performance in the source setting; it does not by itself establish performance in a changed target setting. When feasible, compare predictions with outcomes from a representative target sample collected using the target definitions and practices. If outcomes are not yet available, a pilot or staged use can help gather relevant evidence before broader reliance.
Before approving a transfer, check that the target cases are represented in the source data, that inputs and outcomes are comparable, and that the model’s intended use still fits the decision. When only some target cases are well represented, restrict the claim about transportability to those cases rather than making a blanket statement. If the target setting differs in an important unmeasured way, acknowledge that source data cannot resolve the uncertainty.
For communication, describe both the basis for transfer and its limits. A useful conclusion might state that the model’s inputs and outcome definitions are unchanged, that target cases fall within the source range, and that target validation is still needed. Avoid saying simply that a model “works in the new setting” when evidence supports only a narrower claim, such as plausible ranking for a particular group.
Worked Example: A Change in Referral Practice
A hypothetical service uses a model to predict whether an application will require referral for additional review. In the source setting, the model predicts referral probabilities of 0.10 for low-risk applications and 0.40 for high-risk applications. The target setting has a new digital intake process. In a target pilot, 70% of applications are low-risk and 30% are high-risk; observed referral rates are 0.12 and 0.50, respectively. Calculate the model’s target average prediction, the pilot’s observed average rate, and the difference. State what this evidence suggests.
Method. Standardize both the predictions and observed rates to the target group mix, then compare the two averages.
Target average prediction. The low-risk contribution is \(0.70 \times 0.10=0.070\). The high-risk contribution is \(0.30 \times 0.40=0.120\). The target average prediction is \(0.070+0.120=0.190\), or 19.0%.
Target pilot rate. The low-risk contribution is \(0.70 \times 0.12=0.084\). The high-risk contribution is \(0.30 \times 0.50=0.150\). The observed target average rate is \(0.084+0.150=0.234\), or 23.4%.
Difference. The observed rate exceeds the average prediction by \(0.234-0.190=0.044\), or 4.4 percentage points. For 1,000 applications with this group mix, the model predicts \(1{,}000 \times 0.190=190\) referrals, while the pilot rates imply \(1{,}000 \times 0.234=234\), a difference of 44 referrals.
This pilot provides evidence that the model’s predictions do not match the observed target rates under the new process. The group-specific gaps are \(0.12-0.10=0.02\) for low-risk applications and \(0.50-0.40=0.10\) for high-risk applications. The average difference is not explained by the target group mix alone, because both predictions and pilot rates were weighted using that same mix. The common mistake is to compare the pilot’s 23.4% rate with an unweighted or source-population average rather than the model prediction standardized to the target population.
Check. The weighted pilot contributions sum to \(0.084+0.150=0.234\); subtracting the model’s \(0.190\) gives 0.044. The pilot identifies a transfer concern, but further investigation is needed to establish how the process change, case mix and data collection contributed to it.
Common Transferability Traps
- Equating source validation with target validity. Good source performance is evidence about the source setting, not proof about a later period or another operation.
- Confusing a changed aggregate with changed within-group accuracy. A different group mix can change the overall average even when group-specific predictions remain appropriate.
- Ignoring changed definitions. If the target outcome has a different meaning, a numerical comparison with source performance may not be meaningful.
- Extrapolating without acknowledging it. A model can return predictions for unfamiliar cases, but those predictions are not supported by source evidence merely because they are computable.
- Overstating limited evidence. A small pilot or a validation sample covering only some target cases supports a correspondingly limited conclusion.
Check Your Understanding
For each problem, calculate the requested quantity and consider what the result does or does not establish about transfer to the target setting.
- A model predicts event probabilities of 0.06 for Group A and 0.22 for Group B. In the target population, 25% are in Group A and 75% are in Group B. Assuming the group-specific predictions apply, calculate the target average prediction. (Answer: 18.00%)
- A hypothetical target contract pays \(\min(\max(X-400,0),700)\). Gross losses of $300 and $900 are equally likely. Calculate the average target payment by applying the contract to each loss. (Answer: $250.00)
- In a target setting, Group A accounts for 60% of cases and Group B for 40%. The model’s mean squared errors are 0.04 in Group A and 0.16 in Group B. Assuming these errors apply in the target setting, calculate target mean squared error and root mean squared error. (Answer: Mean squared error 0.088; root mean squared error 0.2966)
- A source data set contains ages from 20 through 60, while the target population includes people aged 65 and older. What can source data alone establish about prediction performance for that unsupported age group? (Answer: It cannot establish performance for ages 65 and older without additional target evidence or defensible assumptions.)
- A churn model predicts probabilities of 0.05 for Segment A and 0.20 for Segment B. In the target setting, 70% are in Segment A and 30% in Segment B; a target pilot observes churn rates of 0.08 and 0.25, respectively. Calculate the target average prediction, the observed average rate, and the difference. (Answer: Prediction 9.50%; observed rate 13.10%; observed rate exceeds prediction by 3.60 percentage points.)