Why Prediction Scope Must Be Explicit
A report can describe a model’s performance accurately and still invite a mistaken decision if it does not say whose predictions the results describe. A model assessed for one kind of policyholder, at one decision point, over one outcome period does not automatically support predictions for every customer, decision date, or outcome. A clear scope tells readers what the report’s conclusions do—and do not—apply to.
Scope is not a claim that the model is suitable for every person or setting inside a broad label such as “customers.” It is a precise description of the cases, timing, outcome, and circumstances the report addresses. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
These elements answer different questions. The population answers “for whom or what?” The prediction time answers “when is the estimate available?” The outcome and horizon answer “what event or quantity, over what period?” The setting answers “under what operating conditions?” Omitting any one can make a sentence sound more general than the underlying analysis supports.
Build a Scope Statement
A useful report statement is specific enough that a reader can decide whether a case belongs within the report’s intended use. It should name the prediction unit, define eligibility in operational terms, and describe the timing and outcome. If the analysis covers only a restricted setting, say so directly rather than relying on readers to infer it.
State whether one prediction refers to a person, policy, claim, transaction, location, or another defined unit. Avoid an ambiguous label such as “account” if an account may contain several separately scored items.
Describe who or what is eligible, including material boundaries such as product, jurisdiction, age range, or customer status. Do not silently treat a subset used for analysis as the entire population of interest.
Say when the prediction is made and how far into the future its outcome refers. A forecast made at application is not interchangeable with one made after a policy has been active for a year.
Specify what is predicted and the operational circumstances covered. State important exclusions and avoid implying that unexamined conditions have the same performance.
A compact scope sentence can follow this pattern: “For [eligible units] in [defined setting], the model predicts [outcome] over [horizon], using information available at [prediction time]; the report does not address [material exclusions].” This is a drafting aid, not a substitute for explaining an important boundary in plain language.
Worked Examples
Worked Example: Defining Scope for a Renewal-Risk Prediction
A hypothetical insurer is considering a model that predicts whether a small-business policy will renew. Its analysis covers 1,200 policies that reached a scheduled renewal date: 900 commercial property policies and 300 commercial vehicle policies. The proposed action is to prioritize a service call before renewal. Write a scope statement and identify what the report must not imply.
Step 1: Identify the unit. One prediction concerns one policy at one scheduled renewal, not one business in general. A business holding two policies could therefore have two predictions.
Step 2: Describe the represented population. The analysis contains \(900+300=1{,}200\) policies. The two listed product groups account for all 1,200 cases because \(900+300=1{,}200\). The scope can name commercial property and commercial vehicle policies; it cannot silently expand to other products.
Step 3: Specify timing and outcome. The score is intended to be available before the scheduled renewal date, and the outcome is renewal of that policy at that date. It does not describe whether the customer will remain with the insurer indefinitely.
Step 4: State the boundary. A defensible statement is: “For commercial property and commercial vehicle policies approaching a scheduled renewal, the model estimates whether each policy will renew at that date, using information available before renewal, to help prioritize service calls. The report does not establish performance for other products, new-business applications, or decisions made after renewal.”
Method and trap avoided. The method is to define the unit, eligibility, timing, outcome, and intended action in order. The common mistake is to call this a model for “customer retention” and thereby imply that it predicts long-term customer relationships across every product. The actual outcome is narrower: renewal of an in-scope policy at a specified renewal point.
Worked Example: Prediction Time and Outcome Horizon
A hypothetical care network wants a report on 30-day unplanned readmission. The prediction is generated at discharge. The proposed population is adults discharged alive after an inpatient stay at the network’s three urban hospitals. Planned admissions and patients transferred to another acute-care hospital are outside the proposed scope. Write the scope and distinguish the stated outcome from two nearby outcomes.
Step 1: Define the unit and population. The unit is a patient discharge episode. The population is adults discharged alive after an inpatient stay at one of the three named urban hospitals, subject to the stated exclusions. A patient with two qualifying discharges can contribute two discharge episodes; the unit is not necessarily a unique patient over the entire period.
Step 2: Fix the time origin. The score is made at discharge. Thus, the prediction uses the discharge as its reference point; it is not a forecast made at admission or during the stay.
Step 3: Define the outcome and horizon. The outcome is an unplanned readmission within 30 days after discharge. A return visit to an emergency department without inpatient readmission is not the same outcome. Nor is readmission at any time over the next year.
Step 4: Write the scope statement. “For adults discharged alive after an inpatient stay at the network’s three urban hospitals, excluding planned admissions and transfers to another acute-care hospital, the model estimates the probability of an unplanned inpatient readmission within 30 days after discharge. The report concerns scores made at discharge and does not address emergency-department visits without admission, readmissions beyond 30 days, or other hospitals.”
Method and trap avoided. The method anchors the horizon to a clearly named prediction time and distinguishes the target event from similar events. A common error is to write “predicts return to hospital,” which could include emergency visits, planned admissions, or later readmissions. Those interpretations are broader than the defined outcome.
Worked Example: Avoiding an Unsupported Population Expansion
A hypothetical equipment insurer evaluates a model for estimating claim cost at the time a claim is first reported. The report covers 480 reported claims from refrigerated delivery equipment operating on paved routes in two regions. It does not include equipment used on unpaved routes. A manager proposes saying that the model estimates claim costs for “all delivery equipment.” Rewrite the claim and explain why the broader wording is unsupported.
Step 1: State the unit and prediction time. The unit is a reported claim, and the estimate is made when that claim is first reported. The model is not described as estimating the chance that any piece of equipment will have a claim.
Step 2: Name the covered conditions. The stated cases concern refrigerated delivery equipment on paved routes in two regions. Unpaved-route equipment is explicitly outside the evaluated scope.
Step 3: Name the outcome. The quantity is the cost of a reported claim. It is not total operating cost, the probability of a claim, or cost per equipment-year.
Step 4: Replace the overstatement. A suitable sentence is: “For reported claims involving refrigerated delivery equipment operating on paved routes in the two evaluated regions, the model estimates claim cost at first report. The report does not establish that the estimates apply to unpaved routes, other regions, other equipment types, or equipment with no reported claim.”
Method and trap avoided. The method is to retain the population and outcome actually described, then name the excluded conditions. The manager’s phrase “all delivery equipment” expands the equipment type and setting; it also risks changing a claim-cost prediction into a general equipment-risk prediction. Neither extension follows from the stated scope.
Why Scope Boundaries Matter
A report’s conclusions are conditional on the cases and circumstances it addresses. If performance or expected error is summarized for a target population \(T\), that result is a statement about \(T\), not automatically about a broader population \(U\). Even if \(T\) is contained within \(U\), the additional cases in \(U\) may differ in ways that change the result.
Here, \(L\) is a loss or prediction error, and \(U\setminus T\) denotes the cases in \(U\) that are not in \(T\). The identity follows by dividing \(U\) into the disjoint groups \(T\) and \(U\setminus T\). The overall conditional expectation is the probability-weighted average of the group-specific expectations. Knowing \(E[L\mid T]\) alone is not enough to determine \(E[L\mid U]\): the size and error behavior of the additional group also matter. Therefore, broader conclusions require support for the broader population; they cannot be obtained just by changing the wording of the report.
Common Scope Mistakes
- Using an audience label instead of eligibility rules. “Customers” may include applicants, active customers, former customers, and people with several policies. Name the cases that qualify.
- Leaving the prediction time implicit. “Predicts future claims” does not tell a reader whether the prediction is made at application, policy inception, or claim report.
- Using an ambiguous outcome. Distinguish a claim from claim cost, a readmission from an emergency visit, and renewal at a specific date from long-term retention.
- Assuming a broad setting from a narrow evaluation. A result for specified products or locations does not, by itself, establish the same conclusions elsewhere.
- Confusing the target population with the observed cases. The cases used to assess a model may be a subset of the population a decision-maker hopes to serve. State the intended population and any gap between it and the cases addressed.
Check Your Understanding
For each situation, draft or evaluate a scope statement. Keep the prediction unit, timing, outcome, and boundary distinct.
- A hypothetical lender scores small-business loan applications at submission to predict whether a loan will be 60 or more days past due within the first 12 months after funding. The analysis excludes applications for loans secured by real estate. What population, time, outcome, and exclusion should the report state? (Answer: One prediction per submitted small-business loan application; scored at submission; the outcome is reaching 60 or more days past due within 12 months after funding; real-estate-secured loan applications are outside scope.)
- A hypothetical insurer estimates repair cost for wind claims reported for residential roofs in three coastal counties. The estimate is made when a claim is opened. Does this establish scope for all homeowners’ claims in the state? State the supported scope and one unsupported extension. (Answer: It supports wording limited to reported wind claims for residential roofs in the three evaluated coastal counties, estimated at claim opening; it does not establish scope for all homeowners’ claims statewide or for other claim types.)
- A hypothetical clinic predicts whether an adult patient will miss the next scheduled appointment, using a score generated when that appointment is booked. The report excludes appointments canceled by the clinic. Identify the unit, time, outcome, and exclusion. (Answer: One scheduled appointment for an adult patient; scored when booked; outcome is the patient missing that appointment; clinic-canceled appointments are excluded.)
- A hypothetical fleet model predicts monthly maintenance cost per vehicle for electric delivery vans operating in one metropolitan area. A manager wants to use the report to support predictions for gasoline trucks in rural areas. What scope change is being proposed, and what must the report avoid claiming? (Answer: The proposed use changes both vehicle type and operating setting; the report must not claim that its conclusions establish performance for gasoline trucks in rural areas.)
- A hypothetical property model predicts whether an insured building will have a water-damage claim during the next six months, with scores issued on each policy anniversary. The analysis covers owner-occupied apartments but not rental buildings. Why is “predicts water claims for properties” too broad? (Answer: It omits the prediction time, six-month horizon, and owner-occupied-apartment restriction; it could incorrectly imply coverage of rental buildings and other property types.)