Validation Under Covariate And Case-Mix Shift

Why Case Mix Matters in Transition Validation

A transition prediction model can perform differently in two validation populations even when its conditional predictions have not changed. If one population contains more people with covariates associated with a particular transition, its event rate, average predicted risk, and prediction-error metrics may differ simply because the population composition differs. This is a case-mix effect, not by itself evidence that the model has become less accurate for comparable individuals.

The reverse distinction matters too. A similar overall event rate in two populations does not establish that predictions work equally well within relevant covariate groups. Conditional errors can be hidden by averaging, or offset one another. The central task is therefore to state which population a validation metric describes, then examine whether differences arise from composition, conditional prediction, or both.

Consider individuals who are in state \(r\) at a specified landmark. Fix a prediction horizon \(\tau\), and define the outcome categories as the first destination reached from \(r\) by \(\tau\), together with “no exit from \(r\) by \(\tau\).” For destination \(s\), let \(Y_{is}=1\) if individual \(i\)'s first exit by \(\tau\) is to \(s\), and \(Y_{is}=0\) otherwise. Let \(p_{is}(\tau)\) be the model's predicted probability for that destination. The target population is the population to which validation results are intended to apply.

Definition: Let \(X\) denote covariates measured at the landmark and let \(T\) denote the target population. A transition-specific performance measure is a population expectation, such as \(M_T=E_T\{L(Y_{is},p_{is}(\tau))\}\), where \(L\) is a specified loss and the expectation is over people in the target population who are eligible at the landmark. A case-mix shift occurs when the distribution of \(X\) differs between the validation and target populations. A conditional model failure occurs when predictions are systematically inaccurate within relevant covariate values or groups.

The horizon, origin state, eligibility rule, outcome definition, and treatment of censoring must be the same when comparing populations. If follow-up ends before \(\tau\), the outcome may be unknown for some people; a suitable censoring-adjusted estimator is then needed. The comparisons below use fully observed outcomes for clarity. They do not justify treating censored outcomes as “no transition.”

Composition Effects and Conditional Performance

Suppose a prediction is conditionally correct: for each covariate value \(x\), the probability of transition to \(s\) by \(\tau\) is \(p_s(\tau\mid x)\). The target event probability is an average over the target covariate distribution:

P_T(Y_s=1)=\int p_s(\tau\mid x)\,dF_T(x).

Changing \(F_T(x)\) can therefore change the marginal event probability even if the conditional risk function is unchanged. Metrics that average individual losses can change for the same reason: different people, with different risk levels, contribute different proportions of the average. Thus a difference in a marginal Brier score or mean predicted probability does not, by itself, diagnose conditional failure.

A useful separation is to report both the covariate distribution and conditional performance summaries, and to estimate performance standardized to a common target distribution. This is related to subgroup calibration, but the purpose here is specifically to make comparisons across populations with different covariate mixes. The result depends on having measured covariates that adequately describe the relevant differences.

Worked Example: Correct Conditional Risks, Different Marginal Scores

Suppose a hypothetical model predicts the probability of a first transition to destination \(s\) by \(\tau\). In a low-risk group, the prediction and true event probability are both \(0.10\); in a high-risk group, both are \(0.40\). The validation population is 80% low-risk and 20% high-risk, while the target population is 30% low-risk and 70% high-risk. Assume outcomes are fully observed.

The validation event rate is \(0.80(0.10)+0.20(0.40)=0.16\), or 16%. The target event rate is \(0.30(0.10)+0.70(0.40)=0.31\), or 31%. The 15-percentage-point difference follows from the changed mix, although predictions remain correct within both groups.

For a binary outcome predicted with probability \(p\), the expected Brier loss under event probability \(q\) is \(q(1-p)^2+(1-q)p^2\). When \(q=p\), this equals \(p(1-p)\). The validation Brier score is therefore \(0.80(0.10)(0.90)+0.20(0.40)(0.60)=0.120\). In the target population it is \(0.30(0.10)(0.90)+0.70(0.40)(0.60)=0.195\). The target score is higher because the target contains more high-risk individuals, not because either group's conditional prediction is wrong.

The conclusion is limited to this stated loss and these groups. A change in a metric is not automatically “only case mix”; conditional calibration and other diagnostics still need to be checked.

Standardization and Weighting

Direct standardization estimates what performance would be if the validation population had the target distribution of covariates. For covariate groups \(g=1,\ldots,G\), let \(w_g^T\) be the target proportion and let \(\widehat{M}_g\) be the validation estimate of the metric within group \(g\). The standardized estimate is:

\widehat{M}_T^{\,\mathrm{std}}=\sum_{g=1}^{G}w_g^T\widehat{M}_g, \qquad \sum_{g=1}^{G}w_g^T=1.

For continuous or high-dimensional \(X\), a corresponding approach uses density-ratio weights. If \(f_T(x)\) and \(f_V(x)\) are the covariate densities in the target and validation populations, respectively, define \(w(x)=f_T(x)/f_V(x)\). Under the assumptions below, a weighted validation average estimates a target-population average:

\widehat{M}_T^{\,\mathrm{wt}} = \frac{\sum_{i=1}^{n_V}w(X_i)L(Y_{is},p_{is}(\tau))} {\sum_{i=1}^{n_V}w(X_i)}.

In practice the density ratio is usually estimated, for example from a model distinguishing target from validation membership. Weighting and standardization change how validation observations contribute; neither changes their observed outcomes or repairs an incorrect conditional prediction model.

Assumptions for transport by covariate adjustment: The outcome and prediction are defined comparably in both populations, including the landmark, origin state, horizon, and destination. Conditional on measured \(X\), the outcome mechanism relevant to the metric is transportable from validation to target; unmeasured differences that affect outcomes or prediction errors can violate this condition. The validation population must have support wherever the target has covariate values (positivity). The target covariate distribution must be estimable. If outcomes are censored before \(\tau\), censoring must also be handled under appropriate assumptions rather than silently coded as no transition.

For example, if the target has individuals with covariate profiles absent from validation, no reweighting can recover their conditional performance from these data. Very large weights are a warning that practical overlap is weak, even if formal positivity is not exactly zero. Weighted uncertainty estimates must account for estimated weights and the effective information loss from unequal weights.

Worked Example: Standardizing Conditional Calibration Errors

In a hypothetical validation sample, the low-risk group has mean predicted transition probability \(0.10\) and observed transition proportion \(0.12\); the high-risk group has corresponding values \(0.30\) and \(0.42\). The validation mix is 40% low-risk and 60% high-risk. The target mix is 70% low-risk and 30% high-risk. Assume complete outcomes and comparable outcome definitions.

The validation mean prediction is \(0.40(0.10)+0.60(0.30)=0.22\). The observed event proportion is \(0.40(0.12)+0.60(0.42)=0.30\). The crude observed-minus-predicted difference is \(0.30-0.22=0.08\), or 8 percentage points.

Standardize each quantity to the target mix. The target mean prediction is \(0.70(0.10)+0.30(0.30)=0.16\). The standardized observed proportion is \(0.70(0.12)+0.30(0.42)=0.21\). Their difference is \(0.21-0.16=0.05\), or 5 percentage points.

The group-specific discrepancies are \(0.12-0.10=0.02\) in the low-risk group and \(0.42-0.30=0.12\) in the high-risk group. Both are positive, especially the high-risk discrepancy, so this example contains evidence of conditional underprediction as well as a case-mix change. Standardization changes the population being summarized; it does not eliminate the within-group discrepancies.

Importance Weights and Diagnostics

For discrete groups, the density-ratio weight is the target group proportion divided by the validation group proportion. In the preceding example, the low-risk weight is \(0.70/0.40=1.75\), and the high-risk weight is \(0.30/0.60=0.50\). These weights give the target composition when applied to validation observations and normalized. For continuous covariates, inspection of the estimated weight distribution is essential: a small number of very large weights can dominate the result.

A common summary of weight concentration is the effective sample size, \(n_{\mathrm{eff}}=(\sum_i w_i)^2/\sum_i w_i^2\). It is not a replacement for the actual sample size or a complete variance calculation, but a much smaller effective size signals that the weighted estimate relies heavily on a few observations. Also compare covariate distributions before and after weighting; balance on a few means alone may not establish adequate overlap or capture nonlinear differences.

Worked Example: Weighted Target Brier Score and Overlap

Consider a hypothetical validation sample of 100 people: 80 are low-risk and 20 are high-risk. The target proportions are 30% and 70%, respectively. Predicted transition probabilities are \(0.10\) in the low-risk group and \(0.30\) in the high-risk group; observed event proportions are \(0.12\) and \(0.42\). Outcomes are complete.

The empirical Brier score in a group with event proportion \(q\) and constant prediction \(p\) is \(q(1-p)^2+(1-q)p^2=q-2pq+p^2\). Thus, for the low-risk group, it is \(0.12-2(0.12)(0.10)+0.10^2=0.106\). For the high-risk group, it is \(0.42-2(0.42)(0.30)+0.30^2=0.258\).

The unweighted validation score is \(0.80(0.106)+0.20(0.258)=0.1364\). Target weights are \(0.30/0.80=0.375\) for each low-risk person and \(0.70/0.20=3.5\) for each high-risk person. The normalized weighted score is \(0.30(0.106)+0.70(0.258)=0.2124\). Its increase reflects both the target's heavier high-risk mix and the high-risk group's poorer observed score; the within-group calibration discrepancy in that group also warrants investigation.

The effective sample size is \((80(0.375)+20(3.5))^2/[80(0.375^2)+20(3.5^2)]=100^2/256.25=39.02\). Although the validation sample has 100 people, its weighted information is concentrated enough to correspond to an effective size of about 39 by this summary. The high-risk group is represented by only 20 validation individuals, so uncertainty and overlap deserve particular attention.

How to Diagnose the Source of a Difference

Fix the estimand.
Specify the eligible origin-state population, prediction horizon, transition outcome, metric, and target population. A change in any of these can make two numbers incomparable.

Describe the mix.
Compare important covariate distributions and transition-risk groups. Identify regions represented in the target but poorly represented or absent in validation.

Compare within covariate strata.
Examine observed and predicted transition frequencies or losses within relevant groups. Persistent discrepancies within groups point toward conditional failure rather than composition alone.

Standardize and assess stability.
Re-express validation performance under target covariate weights, then inspect balance, extreme weights, effective sample size, and uncertainty.

Interpret what remains.
A standardized difference can still reflect conditional prediction failure, unmeasured population differences, outcome-definition changes, or censoring problems. Weighting alone cannot identify which explanation is correct.

Stratified summaries are transparent but can be imprecise when groups are small; direct standardization depends on the chosen strata and may be coarse. Weighting can handle richer covariate distributions but becomes unstable with poor overlap. Outcome-regression standardization models conditional outcomes and averages predictions over target covariates, but then relies on that regression model. These approaches are alternatives with different vulnerabilities, not automatic guarantees of transportability.

Recalibration may address systematic conditional over- or underprediction in a new population, but it is not a substitute for diagnosing case mix. A baseline-risk adjustment can improve average calibration while leaving errors in particular groups, and changed covariate effects may require more than an intercept or baseline update. Calibration and discrimination summaries should be interpreted in their target population and alongside transition-specific uncertainty.

Common Pitfall and Limitations

Wrong reasoning: “The target Brier score is worse, so the model has deteriorated.” Correction: The score can change when the target contains a different distribution of risk levels. Compare conditional performance and, when appropriate, standardize both populations to a common case mix.
Wrong reasoning: “After weighting, the populations are equivalent.” Correction: Weighting balances measured covariates only to the extent that the weighting model and overlap support it. It cannot correct unmeasured conditional differences or incompatible outcome definitions.
Wrong reasoning: “The overall event rate matches, so predictions are valid.” Correction: Overprediction in one group can cancel underprediction in another. Examine conditional calibration and losses, not only a marginal average.

Case-mix adjustment is most informative when the target population is explicit and the validation sample contains adequate information across that target's covariate range. When overlap is poor, report that limitation and avoid presenting an unstable weighted estimate as definitive. A changed conditional outcome mechanism, such as a new care pathway that alters transition risks at the same \(X\), is not merely covariate shift and requires fresh validation evidence.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A model is conditionally correct in two risk groups. The target has more people in the high-risk group than validation. Can the target event rate and Brier score change without conditional model failure?
Validation group proportions are 75% and 25%; target proportions are 50% and 50%. What are the two group density-ratio weights?
Within a target-standardized analysis, the mean prediction is 0.18 and the observed event proportion is 0.24. State the observed-minus-predicted difference and what it does not establish.
Why can a very large importance weight make a target-performance estimate unreliable?

Solutions. (1) Yes. Target averages use different group proportions, so marginal event rates and loss can change even when risk predictions are correct within each group. (2) The weights are \(0.50/0.75=0.667\) for the first group and \(0.50/0.25=2.00\) for the second. (3) The difference is \(0.24-0.18=0.06\), or 6 percentage points of underprediction on average under the standardized target mix. It does not establish that every subgroup is underpredicted or identify why the discrepancy occurs. (4) A large weight lets a small number of validation observations dominate the estimate, increasing sensitivity to their outcomes and reducing effective information; it can also signal weak overlap.