Why Observed Outcomes May Not Represent the Population
A predictive model is often evaluated using outcomes that are available only for a selected subset of people. A lender may observe repayment only for approved applicants; an insurer may have recorded claim amounts only for reported claims; a service provider may receive follow-up data only from customers who respond. In each case, the observed outcomes describe people who entered the data, not automatically everyone about whom a decision is being made.
This matters when comparing groups. If selection is related to group membership and to the outcome, the selected records can make groups look more alike, more different, or even reverse their ordering. The problem is not fixed by having a large number of selected records: a large, systematically selected sample can estimate the wrong quantity very precisely.
The key distinction is between the target quantity and the quantity the data directly describe. For example, the average outcome among all applicants in group \(g\) is \(E[Y\mid G=g]\). The average among applicants whose outcomes are observed is \(E[Y\mid G=g,S=1]\). These are generally different quantities. A report should identify which one it estimates.
How Selection Changes an Observed Outcome Rate
For a binary outcome, such as default or no default, the selected event rate in group \(g\) is determined by both the population event rate and the chance of selection for each outcome. Applying the definition of conditional probability gives the following expression.
The denominator includes both people with and without the event. Expanding it shows exactly how selection affects the observed rate:
This relationship follows from the law of total probability: among group \(g\), selection can occur either for people with \(Y=1\) or for people with \(Y=0\). The observed event rate is the selected event mass divided by the total selected mass. If selection is independent of \(Y\) within group \(g\), then the two selection probabilities are equal and cancel, so the selected event rate equals the population event rate. When they differ, the observed rate is reweighted by who is selected.
Worked Example: A Reversal in Observed Default Rates
Consider a hypothetical applicant population. The true default rate is 20% in group A and 30% in group B. In the historical data, the selection process has an effective inclusion probability of 80% for group A applicants who default and 80% for those who do not. For group B, the corresponding probabilities are 10% and 80%. Calculate the selected default rate for each group and compare the conclusions.
Method. Use the conditional-probability formula, multiplying each population outcome proportion by its inclusion probability and dividing by the total inclusion probability. These probabilities summarize how selection is associated with outcomes; in practice, the process may act through risk indicators rather than directly using future outcomes.
For group A, the population nondefault rate is \(1-0.20=0.80\). The selected default mass per applicant is \(0.20(0.80)=0.16\), and the selected nondefault mass is \(0.80(0.80)=0.64\). Thus the total selected mass is \(0.16+0.64=0.80\), and the observed default rate is \(0.16/0.80=0.2000\), or 20.00%.
For group B, the population nondefault rate is \(1-0.30=0.70\). The selected default mass is \(0.30(0.10)=0.03\), and the selected nondefault mass is \(0.70(0.80)=0.56\). The total selected mass is \(0.03+0.56=0.59\). The observed default rate is \(0.03/0.59=0.050847\ldots\), or 5.08% to two decimal places.
In the full population, group B has the higher default rate: 30% versus 20%. Among selected applicants, group B instead appears to have the lower rate: 5.08% versus 20.00%. The comparison reverses because the selected group B records contain a much smaller share of people who default.
Check. The selected event and nonevent masses for group B sum to \(0.03+0.56=0.59\), and its selected event rate is \(0.03/0.59\), as used above. Common mistake to avoid: comparing the population rates of 20% and 30% with selected rates as if they had the same denominator. Each rate must be labeled by its population: all applicants or selected applicants.
Adjustment Is Possible Only Under Assumptions
One adjustment is inverse-probability weighting. The idea is to give greater weight to observed records that represent many similar people in the target population. If a person with measured characteristics \(X\) has probability \(\pi(X)=P(S=1\mid X)\) of having an observed outcome, that record receives weight \(1/\pi(X)\). For example, a record with a 50% observation probability receives weight 2.
This method requires more than a formula. The selection probabilities must be known or estimated adequately, and the observed characteristics used for adjustment must account for the reasons that outcomes are missing. In particular, the usual justification requires that, conditional on \(X\), selection is independent of \(Y\). It also requires positivity: every relevant type of person in the target population must have a nonzero chance of selection. If a subgroup has no observed outcomes, weighting cannot recover its outcome distribution from the data alone.
Worked Example: Correcting for Different Follow-Up Rates
A hypothetical program serves 1,000 customers in one group. Of these, 800 are classified as low-risk and 200 as high-risk using information recorded for everyone. Outcomes are observed for 50% of low-risk customers and 100% of high-risk customers. Among observed customers, 4% of the low-risk group and 30% of the high-risk group experience the event. Estimate the event rate for all 1,000 customers, using the stated assumption that within each risk stratum outcome observation is independent of the event.
Method. First find the observed counts and events. Then weight each observed stratum by the inverse of its observation probability, or equivalently apply each stratum’s observed event rate to its full-population count.
There are \(800(0.50)=400\) observed low-risk customers. Their observed event count is \(400(0.04)=16\). There are \(200(1.00)=200\) observed high-risk customers, with event count \(200(0.30)=60\). The unweighted observed rate is therefore \((16+60)/(400+200)=76/600=0.1267\), or 12.67%.
For the weighted calculation, the low-risk observation probability is 0.50, so its inverse weight is \(1/0.50=2\). The estimated full-population event count for this stratum is \(16(2)=32\). The high-risk observation probability is 1.00, so its inverse weight is 1; its estimated event count remains \(60(1)=60\). The estimated total event count is \(32+60=92\). Dividing by the full population of 1,000 gives \(92/1{,}000=0.0920\), or 9.20%.
The equivalent stratum calculation is \(800(0.04)+200(0.30)=32+60=92\) estimated events. The answer is 9.20% for the target population, compared with 12.67% among those whose outcomes were observed.
Check. The two inverse weights reconstruct the stratum sizes: \(400(2)=800\) low-risk customers and \(200(1)=200\) high-risk customers. Common mistake to avoid: treating the 600 observed customers as if they had the target population’s risk mix. The low-risk customers are underrepresented in the observed data, so the unweighted rate overstates the target rate in this example.
This adjustment is justified only under the stated conditions. If outcome observation still depends on the event after conditioning on measured risk, the weighted estimate can remain biased. If the observation probabilities are estimated from the same incomplete data, their uncertainty also matters. Weighting is not a general remedy for unobserved outcomes; it is a way to use explicit assumptions about the selection process.
Claim Data and Access to Outcomes
A related issue arises when the outcome itself is observed only after an event is reported or a person reaches a particular stage. Claim severity among reported claims is not automatically average loss among all insured policies. Nor is repayment among approved borrowers automatically repayment risk among all applicants. The denominator defines the population described.
The following example distinguishes severity among reported claims from severity among all incurred claims. It assumes the full claim amounts and reporting process are known for purposes of illustrating the arithmetic; real analyses may not have that information.
Worked Example: Reported Claims and Average Severity
A hypothetical portfolio has 20 incurred claims: 10 claims of $1,000 each and 10 claims of $9,000 each. All $9,000 claims are reported, but only 20% of the $1,000 claims are reported. Calculate the average severity among reported claims and among all incurred claims. Then show how inverse-probability weights recover the full-claim average under the stated reporting probabilities.
Method. Calculate the two averages using their correct denominators, then weight observed claims by the inverse of their reporting probabilities. The correction assumes reporting is independent of the amount within each stated severity category and that those probabilities are known.
Among reported claims, \(10(0.20)=2\) of the $1,000 claims are observed, and all 10 of the $9,000 claims are observed. The reported total amount is \(2($1{,}000)+10($9{,}000)=$2{,}000+$90{,}000=$92{,}000\). There are \(2+10=12\) reported claims, so average reported severity is \($92{,}000/12=$7{,}666.666\ldots\), or $7,666.67 to the nearest cent.
Across all incurred claims, the total amount is \(10($1{,}000)+10($9{,}000)=$10{,}000+$90{,}000=$100{,}000\). There are 20 claims, so the full-claim average is \($100{,}000/20=$5{,}000.00\).
For the weighted calculation, each reported $1,000 claim receives weight \(1/0.20=5\), giving an estimated \(2(5)=10\) low-severity claims and estimated amount \(2($1{,}000)(5)=$10{,}000\). Each $9,000 claim receives weight \(1/1.00=1\), giving 10 claims and $90,000. The weighted total is $100,000 across 20 estimated claims, again yielding $5,000.00.
Check. The weights reconstruct 10 claims in each category, matching the stated 20-claim population. Common mistake to avoid: calling $7,666.67 the average severity of all incurred claims. It is the average among reported claims; the claim-reporting process has overrepresented the higher-severity claims here.
What Selected Data Can and Cannot Establish
A selected-outcome analysis can support a statement about the selected population, provided the data and calculation are sound. For example, it may describe the observed default rate among approved applicants. By itself, it cannot establish the default rate among all applicants, the outcome that rejected applicants would have had if approved, or the effect of changing an approval policy. Those questions concern outcomes that are not observed for the relevant people and require additional assumptions or data.
Similarly, a group difference among observed claimants does not necessarily describe a group difference in the full insured population. The difference may reflect actual outcome differences, differences in who is observed, or both. A responsible interpretation states the selection rule, the population represented by the denominator, and any assumptions used to generalize beyond the observed records.
This distinction is especially important when results inform consequential decisions. A model or report based only on selected records may appear to describe a broad population while actually describing people who passed a prior decision, reported an event, or completed follow-up. Making that restriction explicit is part of communicating limitations and responsible use, aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify the population represented by the observed records and apply the stated assumptions carefully.
- A hypothetical review observes outcomes for 300 of 600 applicants. The observed event rate is 8%. What is the event count among observed applicants, and can the event rate for all applicants be determined from these facts alone? (Answer: 24 observed events; no, the outcomes for the other 300 applicants are not identified.)
- In a hypothetical group, 25% experience an event. Inclusion probabilities are 60% for people with the event and 60% for people without it. Find the event rate among selected people. (Answer: 25.00%.)
- A hypothetical customer population has 500 low-risk and 500 high-risk customers. Outcomes are observed for 80% of low-risk and 40% of high-risk customers. Observed event rates are 2% and 10%, respectively. Assuming observation is independent of the event within risk group, estimate the event rate in the full population. (Answer: \(500(0.02)+500(0.10)=60\) events per 1,000, or 6.00%.)
- A hypothetical review observes 40 events among 200 selected people. The selection probability is 0.50 for both event and nonevent cases, and the selection process is independent of the outcome. What is the estimated event rate in the target population? (Answer: 20.00%.)
- A hypothetical claims study observes 3 of 15 minor claims, each of $2,000, and all 5 major claims, each of $8,000. Assume reporting probabilities are 20% for minor claims and 100% for major claims, and are independent of amount within each category. Estimate the average severity across all incurred claims. (Answer: Weighted amount is \(3(5)($2{,}000)+5(1)($8{,}000)=$70{,}000\); weighted claim count is \(3(5)+5(1)=20\); estimated average severity is $3,500.00.)