Selection Bias in Observed Data

Why Selection Into the Data Matters

A predictive model is developed from records that were collected and retained, not from every case that could matter operationally. Eligibility rules, referral practices, reporting behavior, and follow-up procedures can all determine which cases have usable predictors and outcomes. If inclusion is related to the outcome, the observed outcome patterns may differ from those in the population to which the model is meant to apply.

The central question is not simply whether the development data are large. It is how a case came to be observed. A large sample assembled through a selective process can give a precise description of that selected sample while still giving a misleading answer for the intended population.

Selection is not automatically harmful in every respect. If cases are selected according to observed predictors, the predictor mix can change even while outcome rates within predictor groups remain the same. Conversely, if inclusion depends on the outcome even among cases with the same predictors, the observed outcome rate within those groups can be distorted. These situations have different implications and should not be conflated.

This discussion is aligned with the SOA Exam PA (Predictive Analytics) syllabus. The earlier material on representativeness of the modeling population motivates the target-population comparison; here the emphasis is on the mechanism that determines which records and outcomes are actually observed.

Selection Rules and Missing Outcomes

A case can be absent from a development data set in several ways. A policy application may be excluded because it did not meet an eligibility rule. A claim may be observed only if it was reported. A customer may enter a study only after referral. Or the predictors may be available while the outcome is not yet known because follow-up ended too soon. Each process defines, explicitly or implicitly, who has \(S=1\).

For outcome analysis, it is useful to distinguish inclusion of a case from observation of its outcome. Let \(R=1\) mean that the outcome is observed and \(R=0\) mean that it is missing. An outcome may be missing for reasons related to recorded predictors, the outcome itself, or both. For example, a follow-up outcome might be less likely to be recorded for cases with a particular service channel, or cases with an adverse outcome might be more likely to leave a program before the outcome is measured.

These descriptions concern the outcome-observation mechanism, not whether the recorded predictors themselves are accurate. Even under missingness at random conditional on predictors, simply analyzing observed outcomes can change the mix of predictors in the sample. If the objective is an overall population rate, that changed mix matters. For conditional predictions, the key question is whether, at a given predictor value, observed outcomes represent the outcomes that would have been seen for all cases at that value.

How Selection Changes an Outcome Rate

Consider a binary outcome \(Y\), where \(Y=1\) denotes an event and \(Y=0\) denotes no event. At a given predictor value \(X=x\), define \(s_1(x)=P(S=1\mid X=x,Y=1)\) and \(s_0(x)=P(S=1\mid X=x,Y=0)\). If both selection probabilities are positive, the event probability among selected cases at that predictor value is

The expression follows directly from conditional probability. The numerator is the probability of both an event and selection, given \(X=x\). The denominator is the probability of selection given \(X=x\), which includes selected event cases and selected non-event cases. Dividing the first quantity by the second gives the event proportion among selected cases.

If \(s_1(x)=s_0(x)\), the common selection factor cancels, so the observed event probability at \(x\) equals the target event probability at \(x\). This is the key case in which selection does not alter the conditional outcome rate at that predictor value. If the two probabilities differ, the observed rate generally changes. If either type of case has zero probability of selection in a region, the observed data cannot directly reveal the outcome rate there.

Selection on Predictors Can Change the Mix

Now suppose the chance of inclusion depends on a recorded predictor \(X\), but, within each value of \(X\), does not depend on the outcome. In that case the outcome rate conditional on \(X\) is preserved, but the distribution of \(X\) in the selected sample can differ from its distribution in the target population. An overall event rate, which averages conditional rates over the predictor distribution, can therefore change.

This distinction matters for interpretation. A model may estimate useful conditional risks for predictor groups represented in the data, while a summary of the selected sample’s overall risk does not describe the target population. A conditional model can also fail to generalize if it is misspecified or if the intended prediction cases have predictor values not represented in development data. The selection mechanism alone does not guarantee that a fitted model is valid.

When Weighting Can Help—and When It Cannot

If outcome observation depends only on recorded predictors, and the observation probabilities are known or can be credibly estimated, inverse-probability weighting can restore the target predictor mix for an overall rate. A case with observation probability \(r(X)\) receives weight \(1/r(X)\). Cases that were less likely to be observed then represent more target-population cases. This adjustment requires that observation probabilities be positive for every relevant predictor pattern and that the stated conditional observation assumption be reasonable.

Weighting does not manufacture outcomes for cases that are never observed. Nor does it correct selection related to the unobserved outcome after conditioning on the recorded predictors unless further assumptions or information are available. A weighted estimate is therefore not automatically unbiased just because weights were calculated; its interpretation depends on the selection assumptions and the quality of the observation-probability estimates.

Limits on Predictions for Excluded Cases

A model’s predictions are supported by evidence only to the extent that the relevant cases and outcomes are represented in the development data, or that a defensible selection adjustment connects the data to the target. If a class of cases was excluded by design, predictions for that class may require extrapolation. If no outcomes were recorded for a group, the model-development data cannot establish whether its predictions are calibrated for that group.

A limitation should specify the affected cases and the consequence for the intended decision. For example, if an underwriting model was developed only on contracts meeting a particular eligibility rule, its measured performance does not by itself establish performance on contracts that failed that rule. The model might still be applied there as a provisional assumption, but the evidence does not validate that use. This is different from saying the model is certainly wrong; it means the available data do not settle the question.

A common reporting error is to describe a model as applicable to “all applicants” when the development records contain only applicants who completed a particular process. A more defensible statement names the observed population and the unsupported extension. Selection bias is not resolved by a high sample size, a strong validation score on the selected records, or precise wording about model performance; those facts do not show that excluded cases follow the same outcome process.