Why Observed Outcomes Can Distort a Dataset
A predictive dataset often contains only records for which an outcome has been recorded. If the chance that an outcome is observed differs between people who experience the outcome and people who do not, the records that remain are not a simple miniature of the target population. For example, a hypothetical service provider might receive outcome reports more promptly for customers who make a claim than for customers who do not. An analysis restricted to records with reported outcomes could then show a claim frequency that differs from the frequency in the population the analysis is meant to describe.
The key question is not just how many records have observed outcomes. It is whether inclusion is related to the outcome, or to predictors of the outcome. This tutorial focuses on the consequences of that relationship when analysts use only records with observed outcomes. The earlier material on sampling designs and case-control sampling gives related context; here, the emphasis is on diagnosing and calculating the distortion created by outcome-dependent availability.
Write \(p=P(Y=1)\), \(q_1=P(R=1\mid Y=1)\), and \(q_0=P(R=1\mid Y=0)\). These probabilities describe the target population and its inclusion process. To calculate an observed prevalence, there must be some included records: \(P(R=1)>0\). For an interior population prevalence, \(0<p<1\), this condition holds whenever at least one of \(q_1\) and \(q_0\) is positive.
How Outcome-Dependent Inclusion Changes Prevalence
Among the target population, the proportion that both experiences the event and has an observed outcome is \(P(Y=1,R=1)=p q_1\). The overall proportion with observed outcomes is the sum of the included event and non-event proportions: \(P(R=1)=p q_1+(1-p)q_0\). Dividing the included-event proportion by the total included proportion gives the observed prevalence.
This is generally not \(p\). If records with events are more likely to be included than records without events, the event group makes up a larger share of the analyzed data. Conversely, if event records are less likely to be included, the observed share can be lower.
The boundary cases matter. If \(p=0\), no one in the population experiences the event, so any included record has \(Y=0\); provided at least one such record is included, the observed prevalence is also zero. If \(p=1\), the same reasoning gives observed prevalence one whenever at least one event record is included. A comparison that says the observed prevalence is strictly higher or lower therefore requires an interior population prevalence.
Deriving the Direction of the Distortion
The formula shows the observed prevalence, but a difference calculation makes the direction of the distortion explicit. Subtract \(p\) from the observed prevalence and combine the terms over their common denominator:
For \(0<p<1\), the factor \(p(1-p)\) is positive. The denominator is also positive when the observed prevalence is defined. Thus, the sign of the difference is exactly the sign of \(q_1-q_0\). This establishes the stated direction without assuming a particular sample size. If \(q_1=q_0>0\), inclusion is equally likely for the two outcome groups, and the formula reduces to \(p\). Equal outcome-specific inclusion probabilities prevent this particular prevalence distortion, though they do not rule out other forms of selection related to predictors.
Worked Example: More Complete Reporting for Event Records
In a hypothetical population of 10,000 service contracts, 10% have a specified event during the measurement period. The outcome is observed for 80% of event contracts and 50% of non-event contracts. Calculate the event prevalence among records with observed outcomes. Name the method and identify the common mistake to avoid.
Method. Use the outcome-specific inclusion probabilities to count the included event and non-event records, then divide the included event count by all included records.
Step 1: Population counts. The event count is \(10{,}000(0.10)=1{,}000\). The non-event count is \(10{,}000-1{,}000=9{,}000\).
Step 2: Included counts. The included event count is \(1{,}000(0.80)=800\). The included non-event count is \(9{,}000(0.50)=4{,}500\). Therefore, the total included count is \(800+4{,}500=5{,}300\).
Step 3: Observed prevalence. The event prevalence among included records is \(800/5{,}300=0.150943\ldots\), or 0.1509 to four decimal places. The population prevalence is 0.10, so the observed prevalence is higher, as expected because event records are more likely to be included.
Check and common mistake. Substitution in the probability formula gives \(0.10(0.80)/[0.10(0.80)+0.90(0.50)]=0.08/(0.08+0.45)=0.150943\ldots\), the same result. A common error is to report \(0.10(0.80)=0.08\) as the observed prevalence. That is the fraction of the entire target population that is both an event case and included, not the fraction of included records that are event cases.
Selection Can Also Change Predictor Representation
Outcome-dependent inclusion makes the observed outcome mix unrepresentative when \(q_1\ne q_0\). Whether it also changes a predictor-outcome relationship depends on how inclusion varies across predictor groups. If inclusion depends only on \(Y\), then the predictor distribution within each outcome group is preserved, provided the relevant groups have positive inclusion probabilities. But the overall outcome prevalence changes, so probabilities calibrated to the target population cannot simply be read from the selected records.
If inclusion also depends on a predictor, even the relationship between that predictor and the outcome can be distorted. A useful diagnostic is to compare inclusion probabilities across the joint groups defined by outcome and important predictors, rather than checking only the overall rate of observed outcomes.
Worked Example: Inclusion Depends on Outcome and Risk Group
A hypothetical study population has 1,000 members. There are 200 in a high-risk predictor group and 800 in a low-risk group. In the high-risk group, 80 experience the outcome and 120 do not. In the low-risk group, 80 experience it and 720 do not. Outcomes are observed for 90% of high-risk event records, 50% of high-risk non-event records, 50% of low-risk event records, and 90% of low-risk non-event records. Compare the population and observed event rates by risk group, and check the overall observed prevalence.
Method. Calculate the population rates first, then apply each cell's inclusion probability to obtain selected counts. This cell-by-cell approach avoids assuming that one inclusion rate applies to every group.
Step 1: Population event rates. The high-risk rate is \(80/(80+120)=80/200=0.40\). The low-risk rate is \(80/(80+720)=80/800=0.10\). Overall, there are \(80+80=160\) events among 1,000 people, so the population prevalence is \(160/1{,}000=0.16\).
Step 2: Included counts. In the high-risk group, the included event count is \(80(0.90)=72\), and the included non-event count is \(120(0.50)=60\). In the low-risk group, the included event count is \(80(0.50)=40\), and the included non-event count is \(720(0.90)=648\).
Step 3: Observed rates by group. The high-risk observed rate is \(72/(72+60)=72/132=0.54545\ldots\), or 0.5455 to four decimal places. The low-risk observed rate is \(40/(40+648)=40/688=0.05814\ldots\), or 0.0581 to four decimal places. These differ from the population rates of 0.40 and 0.10.
Step 4: Overall check. There are \(72+40=112\) included events and \(72+60+40+648=820\) included records. The overall observed prevalence is \(112/820=0.136585\ldots\), or 0.1366 to four decimal places, compared with the population prevalence of 0.16.
Check and common mistake. The cell counts sum to \(72+60+40+648=820\), matching the selected denominator. A common mistake is to use a single outcome-only inclusion probability for the whole population, even though the stated probabilities differ by risk group. The group-specific calculations show why that shortcut would miss the altered predictor representation and group rates.
When Weighting Can Help
If inclusion probabilities are known or can be estimated from suitable information, inverse-probability weighting can help represent the target population. A record with inclusion probability \(q\) receives weight \(1/q\): records that were less likely to be included represent more members of the target population. For outcome-only inclusion, event records receive weight \(1/q_1\), and non-event records receive weight \(1/q_0\).
Here \(j\) indexes included records and \(q_{Y_j}\) is the inclusion probability for that record's outcome group. This weighted ratio estimates the target prevalence under an appropriate inclusion model. The related weighted totals use the same principle: each included record contributes \(1/q_{Y_j}\) to its estimated population group total. The probabilities must be positive for the groups being represented, and the weights must correspond to the actual selection process. Weighting cannot recover information for a group that has zero chance of inclusion.
Worked Example: Correcting an Observed Prevalence With Weights
A hypothetical outcome dataset contains 90 observed event records and 270 observed non-event records. The event records had inclusion probability 0.75, and non-event records had inclusion probability 0.45. Estimate the target-population prevalence using inverse-probability weights, and compare it with the unweighted observed prevalence.
Method. Weight each observed record by the reciprocal of its outcome group's inclusion probability, then divide the weighted event total by the total weighted count.
Step 1: Weighted event total. Each event record has weight \(1/0.75=4/3\). The estimated event total is \(90(4/3)=120\).
Step 2: Weighted non-event total. Each non-event record has weight \(1/0.45=20/9\). The estimated non-event total is \(270(20/9)=600\).
Step 3: Weighted prevalence. The weighted estimated total is \(120+600=720\). The estimated target prevalence is \(120/720=1/6=0.1667\) to four decimal places. The unweighted observed prevalence is \(90/(90+270)=90/360=0.25\).
Check and common mistake. The weighted event and non-event totals are consistent with undoing the unequal inclusion rates: the event count is expanded by \(1/0.75\), and the non-event count by \(1/0.45\). A common error is to multiply the observed counts by the inclusion probabilities again, which would shrink the groups that were already selected and move in the wrong direction. The weighted prevalence is an estimate; it is not a guarantee of the exact population prevalence in a finite dataset.
Limits and Exam Traps
Weighting is useful only when the inclusion mechanism is adequately understood. If inclusion probabilities are unknown, a weighted correction cannot be justified merely because the observed prevalence seems implausible. If inclusion depends on predictors as well as the outcome, outcome-only weights may also be insufficient. The analysis should state which population it aims to describe, how inclusion is modeled, and what assumptions support any correction.
Keep the target population and the analyzed records distinct. The population prevalence is \(P(Y=1)\); the prevalence among records with observed outcomes is \(P(Y=1\mid R=1)\). They coincide under equal outcome-specific inclusion probabilities, or at the boundary prevalences when the observed prevalence is defined. Otherwise, they can differ. This is a data-design issue, not evidence by itself that the outcome has changed or that a predictor causes the outcome.
Check Your Understanding
For each problem, distinguish the target-population quantity from the quantity calculated among included records.
- A hypothetical population has outcome prevalence 0.05. Outcomes are observed for 70% of event records and 70% of non-event records. Calculate the observed prevalence. (Answer: 0.0500)
- A population has prevalence 0.20, with \(q_1=0.90\) and \(q_0=0.60\). Calculate the observed prevalence to four decimal places. (Answer: 0.2727)
- In a hypothetical population of 1,000, there are 100 event records and 900 non-event records. Their inclusion probabilities are 0.80 and 0.40, respectively. Find the expected included counts and the resulting observed prevalence. (Answer: 80 event records, 360 non-event records, and prevalence 0.1818)
- A selected dataset has 48 event records with inclusion probability 0.60 and 140 non-event records with inclusion probability 0.70. Calculate the inverse-probability weighted prevalence to four decimal places. (Answer: 0.2857)
- A population prevalence is 0.30, with \(q_1=0.40\) and \(q_0=0.80\). Calculate the observed prevalence to four decimal places and state whether it is above or below the population prevalence. (Answer: 0.1765, below the population prevalence)