Why Coverage Matters
A prediction dataset can be internally consistent and still fail to represent the people or events for which its predictions are intended. For example, an insurer might want to predict the cost of all small-business policy renewals, but its historical records might include only renewals processed through one sales channel. The recorded data may be extensive; the key question is whether the source captured the intended population.
Coverage is about the match between a defined target and the units represented by a data source. It is not enough to state that a dataset contains many records, or that most records are from a particular group. First specify the population to which the model is intended to apply, then compare that population with the source records on the same unit, eligibility rules, and time period. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The unit must be clear. Depending on the task, it could be a customer, a policy at renewal, a claim, or a customer-month. If the intended population is policies at renewal but the source contains one row per customer, the counts cannot be compared until the row meaning is reconciled. The target and the source must also refer to the same period and use compatible eligibility rules. A unit excluded by a deliberate eligibility rule is not automatically a data-source coverage gap: the intended target should reflect the actual prediction task.
Measuring Overall and Subgroup Coverage
Suppose the intended target has \(N\) distinct units and the source represents \(C\) of them. Provided each target unit is counted at most once and the numerator is a subset of the denominator, the overall coverage rate is \(C/N\). The uncovered count is \(N-C\). These calculations describe how much of the defined target is represented; they do not show whether the covered units resemble the target in relevant ways.
The subgroup rates matter because an overall percentage is a weighted average of the group rates, with larger target groups contributing more. Here is why. In group \(g\), the covered count is \(C_g=N_gc_g\). Since the groups partition the target, adding their covered counts gives \(C=\sum_g N_gc_g\), while the total target count is \(N=\sum_gN_g\). Therefore,
This identity assumes that group membership is defined consistently for the target and source, each unit is counted once, and the groups cover the whole target. It explains why overall coverage can conceal a weak rate in a small group, or a zero rate in a group that is a substantial part of the target.
Worked Example: Finding Gaps Across Region and Application Channel
A hypothetical lender wants to build an application-risk model for all 12,000 applications received in a defined quarter. Its historical source records applications by region and channel as follows. Determine overall and subgroup coverage, then identify what an overall rate would obscure.
| Target group | Target applications | Covered applications |
|---|---|---|
| Region A, digital | 4,000 | 3,200 |
| Region A, paper | 2,000 | 1,600 |
| Region B, digital | 3,000 | 900 |
| Region B, paper | 3,000 | 0 |
Step 1: Check the target total. The four target cells sum to \(4{,}000+2{,}000+3{,}000+3{,}000=12{,}000\), as stated.
Step 2: Add the covered counts. The source covers \(3{,}200+1{,}600+900+0=5{,}700\) applications. Thus overall coverage is \(5{,}700/12{,}000=0.475\), or 47.5%.
Step 3: Calculate the uncovered counts. Subtract covered from target within each cell: Region A digital has \(4{,}000-3{,}200=800\) uncovered; Region A paper has \(2{,}000-1{,}600=400\); Region B digital has \(3{,}000-900=2{,}100\); and Region B paper has \(3{,}000-0=3{,}000\). These sum to \(800+400+2{,}100+3{,}000=6{,}300\). The check is \(5{,}700+6{,}300=12{,}000\).
Step 4: Compare group rates. Region A has \(4{,}800\) covered applications out of \(6{,}000\), so its rate is \(80\%\). Region B has \(900\) out of \(6{,}000\), so its rate is \(15\%\). Across channels, digital coverage is \(4{,}100/7{,}000\), approximately \(58.6\%\); paper coverage is \(1{,}600/5{,}000=32.0\%\).
Conclusion and trap avoided. Reporting only 47.5% overall misses the sharp regional difference and the complete absence of Region B paper applications. The method is to compute coverage by target group, not to infer group coverage from the overall rate. The common mistake is to report an incorrect uncovered cell count: each uncovered count is the target count minus the covered count in that same cell.
Coverage Is Not the Same as Representativeness
Coverage describes whether target units appear in the source. Representativeness asks whether the covered population is sufficiently similar to the intended target for the intended prediction use. A source can cover many units but disproportionately represent some groups. Conversely, the coverage rate can be modest while the observed population resembles the target on the characteristics relevant to a particular task. The assessment therefore depends on the target, the intended use, and the characteristics being compared.
A useful diagnostic is to compare the target and covered populations across factors that could matter to predictions, such as region, product, customer type, or channel. The relevant comparisons are not limited to counts: a group may have similar representation but different predictor or outcome patterns. Coverage information alone cannot establish that these patterns are the same, nor can it reveal values for units the source never captured.
Worked Example: A High Coverage Rate With a Different Population Mix
A hypothetical service provider wants to predict annual costs for 10,000 customers: 6,000 urban and 4,000 rural. Its source contains 5,400 urban and 1,600 rural customers. For illustration, assume the mean predicted annual cost within each group is $300 for urban customers and $600 for rural customers, and that these group means apply equally to covered and uncovered customers within the same group. Compare the source-mix average with the target-mix average.
Step 1: Calculate coverage. The source contains \(5{,}400+1{,}600=7{,}000\) customers, so overall coverage is \(7{,}000/10{,}000=70\%\). Urban coverage is \(5{,}400/6{,}000=90\%\), while rural coverage is \(1{,}600/4{,}000=40\%\).
Step 2: Compare the group shares. The target is 60% urban and 40% rural. The source is \(5{,}400/7{,}000\approx77.14\%\) urban and \(1{,}600/7{,}000\approx22.86\%\) rural. Thus, the source has a larger urban share than the target.
Step 3: Calculate the average using the source mix. The total predicted cost across source customers is \(5{,}400(300)+1{,}600(600)=1{,}620{,}000+960{,}000=\$2{,}580{,}000\). Dividing by 7,000 gives approximately $368.57 per customer, rounded to the nearest cent.
Step 4: Calculate the average using the target mix. Under the stated assumption, the target total is \(6{,}000(300)+4{,}000(600)=1{,}800{,}000+2{,}400{,}000=\$4{,}200{,}000\). Dividing by 10,000 gives $420.00 per customer. The difference between these two averages is \(420.00-368.57=\$51.43\) per customer, rounded to the nearest cent.
Conclusion and trap avoided. The calculation shows how a different group mix can change an aggregate even when the assumed within-group means are the same. It does not establish that the stated group means are valid for uncovered customers: that equality was an explicit assumption for this illustration. The common mistake is to treat a 70% overall coverage rate as proof that the source has the target's composition, or that uncovered customers behave like covered ones.
Define the Target Before Judging the Source
A coverage assessment is only as clear as its target definition. Consider whether the intended target includes all products, channels, locations, and time periods in which predictions will be used. The target should be stated in operational terms, not merely as “all customers” if some customers cannot receive the product or are outside the decision process. Earlier material on eligibility rules and exclusion rules helps distinguish the actual prediction scope from units intentionally outside it.
When a source cannot capture part of the intended target, there are two different actions to consider. The source may be broadened or supplemented to better match the intended use. Alternatively, the intended use may be narrowed to match the population that the available data can support. Narrowing the scope is not a technical fix for a gap: it changes the population to which predictions are meant to apply and should be stated and justified as such.
Worked Example: A Missing Sales Channel and a Revised Scope
A hypothetical insurer intends to develop a renewal model for 20,000 small-business policies renewing in a year: 8,000 were sold online, 6,000 through a call center, and 6,000 through partner brokers. The available administration source represents 7,200 online policies, 5,400 call-center policies, and no partner-broker policies. Calculate coverage for the stated target and assess the effect of a possible scope restriction.
Step 1: Verify the target and covered totals. The target total is \(8{,}000+6{,}000+6{,}000=20{,}000\). The covered total is \(7{,}200+5{,}400+0=12{,}600\).
Step 2: Calculate the original coverage rates. Overall coverage is \(12{,}600/20{,}000=63\%\). Online coverage is \(7{,}200/8{,}000=90\%\), call-center coverage is \(5{,}400/6{,}000=90\%\), and partner-broker coverage is \(0/6{,}000=0\%\).
Step 3: Check the missing count. The uncovered counts are \(8{,}000-7{,}200=800\) online, \(6{,}000-5{,}400=600\) call-center, and \(6{,}000-0=6{,}000\) partner-broker policies. They total \(800+600+6{,}000=7{,}400\); \(12{,}600+7{,}400=20{,}000\), confirming the count.
Step 4: Consider a narrower operational target. If the model is explicitly intended only for online and call-center renewals, the revised target is \(8{,}000+6{,}000=14{,}000\). The source then covers \(12{,}600/14{,}000=90\%\) of that narrower target. The model would not, on this evidence, be intended for partner-broker renewals.
Conclusion and trap avoided. The original 63% rate is the appropriate figure for the all-channel target; the 90% rate applies only after the scope is explicitly restricted. The common mistake is to report the higher rate while continuing to describe the target as all small-business renewals. A change in scope must be visible in the intended use, not hidden in the denominator.
Practical Interpretation and Common Traps
Coverage analysis is a diagnostic, not a guarantee that a model will work well in every group. It makes a limitation visible and helps define how broadly conclusions may be applied. If a group has little or no coverage, performance for that group cannot be established from the source alone. If coverage is uneven, an overall summary may describe the covered records more than it describes the intended population.
- Using the wrong denominator. Coverage is calculated against the intended target, not against the number of records already in the source. State any scope restriction before calculating the rate.
- Mixing units or periods. Do not compare a target count of policies with a source count of customers, or a current target with records from a different period, without reconciling the definitions.
- Equating coverage with representativeness. A high overall rate can hide subgroup gaps. Compare coverage and composition on relevant characteristics.
- Assuming uncovered units behave like covered units. That may be an assumption for an analysis, but it does not follow from the coverage rate. State the assumption and its implications.
- Treating an absent group as a small-rate group. A group with zero covered units has zero observed coverage; no within-group evidence can be obtained from that source.
Check Your Understanding
For each problem, use the stated target as the denominator and check that covered and uncovered counts reconcile.
- A hypothetical transit agency wants predictions for 3,000 weekday riders: 1,800 city riders and 1,200 suburban riders. Its source contains 1,620 city riders and 360 suburban riders. Find overall coverage, each subgroup rate, and the city share of the source. (Answer: overall coverage \(1{,}980/3{,}000=66\%\); city coverage \(90\%\); suburban coverage \(30\%\); city share of source \(1{,}620/1{,}980\approx81.8\%\).)
- A hypothetical clinic's target includes 4,000 adult appointments and 4,000 pediatric appointments. The source covers 3,200 adult and 1,200 pediatric appointments. Find overall coverage and total uncovered appointments. (Answer: \(4{,}400/8{,}000=55\%\) coverage; \(8{,}000-4{,}400=3{,}600\) uncovered.)
- A target has 7,000 units in Group A and 3,000 in Group B. Coverage is 80% in A and 20% in B. Calculate the total covered count and overall coverage. (Answer: A contributes \(7{,}000(0.80)=5{,}600\), B contributes \(3{,}000(0.20)=600\); total covered is 6,200 and overall coverage is \(6{,}200/10{,}000=62\%\).)
- A hypothetical prediction tool is intended for 5,000 urban and 5,000 rural accounts. Its source contains 4,500 urban accounts and no rural accounts. Find overall coverage and state the principal limitation. (Answer: \(4{,}500/10{,}000=45\%\) overall coverage; rural coverage is 0%, so this source provides no rural accounts on which to assess the model.)
- A model is described as applying to all 12,000 annual renewals: 5,000 direct, 4,000 agent, and 3,000 partner renewals. A source covers 4,000 direct, 3,600 agent, and 0 partner renewals. Calculate coverage for the all-channel target and for a separately stated direct-and-agent target. (Answer: all-channel coverage is \(7{,}600/12{,}000\approx63.3\%\); the direct-and-agent target is 9,000, with \(7{,}600/9{,}000\approx84.4\%\) coverage. The latter rate applies only if partner renewals are explicitly outside the intended use.)