Reporting Data and Population Limitations

Why the Scope of a Recommendation Matters

A model report can calculate a performance measure correctly and still make an unsupported recommendation. The issue may be that the records represent only a selected group, some outcome values are unreliable, the data come from a different time period, or the intended users fall outside the population represented in the analysis. A precise metric does not remove these limits.

The key reporting task is to connect each conclusion to the evidence that supports it. If validation records concern claims from one region and one period, for example, the report should not silently extend its conclusion to every region and future claims. State the population, data quality, and time period behind a result, then identify which uses remain untested.

Definition: The target population is the group of cases or decisions to which a report intends its conclusion to apply. The observed population is the group represented by the records used to build or evaluate the model. A recommendation is supported only to the extent that the observed data provide relevant evidence about the target population.

These populations can differ in more than their names. The observed data may exclude people who were denied coverage, policies from certain regions, claims without a completed investigation, or records from recent years. Such exclusions matter when the omitted cases have different outcomes, predictor values, or data collection processes. The report should describe the actual coverage rather than imply that the data represent a broader group.

Sampling and What a Performance Result Estimates

A performance measure is calculated on a particular set of observations. To interpret it as evidence about a larger population, ask how those observations entered the data. A probability sample drawn from the target population can support population-level estimates under its sampling design. A selected or convenience sample may not: inclusion can be related to the outcomes or predictors that affect performance.

Result: If $N$ cases form a finite target population with losses $L_1,\ldots,L_N$, and $n$ cases are selected by simple random sampling without replacement, the sample mean loss is an unbiased estimate of the population mean loss. This result does not automatically apply to a non-random sample or to a sample drawn from a different population.

The justification is direct. Under simple random sampling, each population case has inclusion probability $n/N$. Let $I_j$ equal 1 if case $j$ is sampled and 0 otherwise. The sample mean can be written as $\sum_{j=1}^N I_jL_j/n$. Taking its expectation gives:

E\left[\frac{1}{n}\sum_{j=1}^{N} I_jL_j\right] = \frac{1}{n}\sum_{j=1}^{N} E[I_j]L_j = \frac{1}{n}\sum_{j=1}^{N}\frac{n}{N}L_j = \frac{1}{N}\sum_{j=1}^{N}L_j.

The equality depends on the sampling design and on measuring the same loss consistently for sampled and target cases. If records are selected because they have complete outcomes, or only approved applicants are observed, the simple-random-sample argument does not apply. A report must not describe a sample metric as a target-population estimate without evidence that the selection process supports that interpretation.

Worked Example: Approved Applicants Do Not Represent All Applicants

A hypothetical insurer has records for 2,000 applicants. Outcomes are available for the 1,000 applicants whose policies were approved; 80 of those approved applicants had a claim during the observation period. No claim outcomes are available for the 1,000 applicants who were not approved. A report proposes using the observed claim rate to describe all applicants. Assess the claim.

Method. Calculate the observed rate for the group with outcomes, then calculate the most optimistic and pessimistic possible rates for the full applicant group. This shows how much the unknown outcomes alone can change the conclusion.

The observed rate among approved applicants is $80/1{,}000=0.08$, or 8%. For the full applicant group, the 80 known claims are fixed. If none of the 1,000 applicants without outcomes had a claim, the full-group rate would be $80/2{,}000=0.04$, or 4%.

If all 1,000 applicants without outcomes had a claim, the total would be $80+1{,}000=1{,}080$ claims, giving $1{,}080/2{,}000=0.54$, or 54%. Thus the available information permits a full-applicant rate anywhere from 4% to 54%.

Conclusion. The 8% rate describes the approved applicants with observed outcomes. It does not establish the claim rate for all applicants. The unknown outcomes make the full-applicant rate too uncertain to infer from these records alone; moreover, the observed group was selected through approval.

Common mistake avoided. Reporting $80/1{,}000=8\%$ as the rate for all 2,000 applicants uses the wrong denominator and ignores the missing outcomes. The known full-group denominator is 2,000, but its numerator is not known.

Data Quality Limits the Conclusions Too

Coverage is not only about who appears in a file. The report also needs to state whether key fields are complete, consistently defined, and measured accurately enough for the intended conclusion. Missing values, duplicate records, changed coding rules, and delayed outcome reporting can each alter a result. A large data set does not compensate for a systematic defect.

When an outcome is missing, a reported mean among complete records is a mean for those records. It equals the mean for all records only under additional assumptions about the missingness process. If the missing values could plausibly be systematically higher or lower, the observed mean alone does not determine the full-data mean.

Worked Example: Bounding a Mean With Missing Claim Costs

A hypothetical severity study contains 500 claims. Repair costs are recorded for 460 claims, whose total is $920,000. The other 40 costs are missing. Based on the study’s documented recording range, assume each missing cost lies between $1,000 and $5,000. Calculate the observed mean and the range of possible means across all 500 claims.

Method. First calculate the complete-record mean. Then add the smallest and largest possible totals for the missing costs to the known total and divide each result by all 500 claims.

The complete-record mean is $\$920{,}000/460=\$2{,}000$. The minimum possible total for the 40 missing costs is $40(\$1{,}000)=\$40{,}000$, so the minimum full-data mean is $(\$920{,}000+\$40{,}000)/500=\$1{,}920$.

The maximum possible total for the missing costs is $40(\$5{,}000)=\$200{,}000$, so the maximum full-data mean is $(\$920{,}000+\$200{,}000)/500=\$2{,}240$. The supported range is therefore $1,920 to $2,240 per claim, given the stated cost bounds.

Conclusion. The $2,000 complete-record mean is not automatically the mean for all 500 claims. Under the specified bounds, the all-claim mean could be as low as $1,920 or as high as $2,240. A recommendation that depends on differences smaller than this range would need better information about the missing costs.

Common mistake avoided. Multiplying the observed mean by 500 to infer a total of $1,000,000 implicitly assigns the observed mean to every missing claim. That assumption is not provided. The calculation above uses the available bounds instead of silently filling in missing values.

Time Period and Population Coverage

A model evaluated on historical data has evidence about those observations and their measurement conditions. It does not, by itself, establish performance in a later period. The population may change, the mix of cases may differ, or the process that records outcomes may be revised. When reporting a historical result, identify the dates represented and distinguish what was measured from what is expected in the future.

The same caution applies to geographic, product, or channel coverage. If records include only one region or product type, a report should name that boundary. An intended use outside the represented area is not necessarily wrong, but it is an extrapolation that needs additional evidence rather than a conclusion established by the current validation.

Worked Example: A Combined Historical Error Can Hide the Relevant Period

A hypothetical severity model is evaluated on two historical periods using the same definition of absolute error. In 2022, the evaluation contains 80 claims and the sum of absolute errors is $16,000. In 2023, it contains 120 claims and the sum of absolute errors is $36,000. A proposed report states that the model’s expected absolute error for 2024 is $260. Assess what the calculation supports.

Method. Calculate each period’s mean absolute error (MAE), then the pooled MAE. Compare the pooled result with the evidence from the more recent period, without treating either historical value as guaranteed future performance.

For 2022, the MAE is $\$16{,}000/80=\$200$ per claim. For 2023, it is $\$36{,}000/120=\$300$ per claim. Across both periods, the total absolute error is $\$16{,}000+\$36{,}000=\$52{,}000$, and the total number of claims is $80+120=200$. Thus the pooled MAE is $\$52{,}000/200=\$260$.

Conclusion. The $260 value is the observed pooled MAE across these 200 historical claims. It is not a direct estimate from 2023 alone, for which the observed MAE is $300, and it does not establish 2024 MAE. If 2024 is expected to resemble 2023 more closely than 2022, the pooled figure may be a poor summary for the intended use. The report should show the period-specific results and describe 2024 performance as unverified.

Common mistake avoided. Calling $260 the “2024 expected error” turns a historical weighted average into a future guarantee. The calculation is correct as a pooled historical metric; the unsupported step is extending it to a later period without evidence that the relevant conditions remain comparable.

Writing a Bounded Recommendation

A useful limitation is specific enough to affect a decision. “The model has limitations” does not tell a reader which evidence is missing or what use should be restricted. Instead, connect the issue to its consequence: which cases were observed, which were not, what data quality issue remains, and what conclusion or use is therefore unsupported.

Name the evidence.
State the observed population, the source and quality of the outcomes, and the dates represented.

Compare it with the intended use.
Identify any gap in population, geography, product, time period, or outcome availability.

State the consequence.
Explain which conclusion is supported by the available records and which proposed use lacks direct evidence.

Make the recommendation conditional.
Specify what additional evidence or validation would be needed before extending the use.

For example, a bounded statement might say: “The observed validation results describe approved policies with complete outcomes from the stated historical period. They do not establish performance for declined applicants or for later periods. Before applying the model to either group, evaluate it on records that represent that population and time period.” This wording reports the finding and its boundary without claiming that performance must be better or worse outside the evidence.

Avoid using a limitation as a substitute for analysis. If the report can calculate a range, a period-specific result, or a count of excluded records, provide it. At the same time, do not imply that a numerical bound resolves every concern: the calculation is only as useful as its stated assumptions and the quality of the underlying records.

Key takeaway: A model recommendation should not reach beyond the people, records, outcomes, and time period that the evidence can speak to. State the supported scope explicitly, identify what remains untested, and avoid presenting an extrapolation as an established result. This reporting discipline is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each situation, identify what the evidence supports and give the requested numerical result where applicable.

A study includes 600 completed applications, of which 42 have a recorded event. Outcomes are unavailable for 400 declined applications. Calculate the completed-application event rate and the possible full 1,000-application rate range if the unobserved outcomes could all be either zero or one. State which rate is directly observed. (Answer: Completed applications: $42/600=7\%$; full-application rate range: $42/1{,}000=4.2\%$ to $442/1{,}000=44.2\%$; only the 7% rate is directly observed.)
A data set has 300 records. The outcome is known for 270, with a total outcome value of $540,000. Each of the 30 missing outcomes is known to be between $500 and $2,500. Find the complete-record mean and the bounds on the all-record mean. (Answer: Complete-record mean: $2,000; all-record mean bounds: $(\$540{,}000+\$15{,}000)/300=\$1{,}850$ to $(\$540{,}000+\$75{,}000)/300=\$2{,}050$.)
A model’s absolute errors total $9,000 over 30 cases in one period and $28,000 over 70 cases in a later period. Calculate each period’s MAE and the pooled MAE. If a report concerns only the later-period population, identify the directly relevant historical metric. (Answer: First period MAE: $300; later period MAE: $400; pooled MAE: $(\$9{,}000+\$28{,}000)/(30+70)=\$370$; $400 is the directly relevant historical metric for the later-period population.)
A model was evaluated on 500 records from one province during 2021–2022. A recommendation proposes use across all provinces in 2025. Name two distinct evidence gaps and state what kind of evidence would address them. (Answer: Geographic coverage and time period are both gaps; validation records representing the other intended provinces and the 2025-relevant population or conditions would address them.)
A report gives a $1,500 mean outcome among complete records, but 15% of outcomes are missing and no bound or missingness assumption is supplied. Can the report claim that $1,500 is the mean for the full target population? State the key reason and one appropriate next step. (Answer: No; the complete-record mean need not equal the full-population mean when missing outcomes may be systematically different. Investigate the missingness and obtain additional outcomes or provide a justified sensitivity analysis or bounds.)

What You'll Learn