Why the Prediction Setting Must Be Stated
A model does not make predictions in the abstract. It predicts a particular outcome for particular cases, using information available at a particular time, under a particular operating process. A model that performs well in one setting may give unreliable predictions if it is used for a different population, with later or unavailable inputs, at a different decision point, or under changed operating conditions.
A useful assessment therefore describes the intended prediction setting before discussing model performance. This makes the assumptions testable and helps decision-makers understand whether a reported result applies to the cases they actually need to assess. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
For example, “the model predicts whether a customer will cancel within 60 days” is not a complete description. It should also say which customers are included, when their predictors are measured, and whether the model is used before or after a retention offer. Those details determine what the prediction means and whether the information used to build or evaluate the model could have been available in practice.
Five Questions to Make the Setting Concrete
The following questions organize an assumption review. They are not a substitute for checking the data and workflow; they identify what must be checked.
State which cases are in scope and whether the intended users will score the same kinds of cases as those represented in model development and evaluation.
Name the operational decision point. A predictor is usable only if its value is available by that time, not merely because it appears in a historical data table.
Specify the source and timing of each required input. If an input is missing, delayed, or created by a later action, the deployed model may not receive the same information as the evaluated model.
Define the event and the period after the prediction point during which it is counted. A model for a 30-day outcome is not automatically a model for a one-year outcome.
Describe relevant processes, rules, capacity, or technology that may affect how cases arise or how outcomes are generated. A model can become unreliable if those conditions change.
These questions are related but distinct. A case may belong to the intended population while lacking a predictor at the required decision time. All predictors may be available, while the outcome horizon differs from the one used in evaluation. The operating process may also change how predictors relate to outcomes. A clear review names each assumption separately rather than compressing them into “the model assumes future data will be similar.”
Worked Example: A Predictor Recorded After the Decision
A hypothetical lender plans to use a model to flag applications for additional review before making a credit decision. The development table contains 120 historical applications. It includes debt ratio, recorded when the application arrived, and “final review code,” entered after an application has completed the review process. The code is present for 84 of the 120 applications. Determine whether the code can be used in the intended prediction setting and state the implication.
Method. Compare the predictor’s recording time with the intended prediction time, then calculate how often it is present in the historical table. The common mistake to avoid is treating historical presence as proof of operational availability.
Step 1: Identify the prediction time. The model is intended to flag an application before a credit decision. Therefore, a usable predictor must be known by the time the application is flagged.
Step 2: Check the code’s timing. The final review code is entered after review is completed. That is later than the prediction time. It cannot be supplied to the model when the intended flag is produced, even for an application whose historical row contains the code.
Step 3: Calculate its historical presence. The observed proportion with a code is \(84/120=0.70\), or 70%. The number without a recorded code is \(120-84=36\). Neither calculation changes the timing fact: a post-review field is not an available input for a pre-decision prediction.
Conclusion. The final review code must not be used as a predictor for this use. Its inclusion would make the development or evaluation information inconsistent with the information available at deployment. The model should be assessed using predictors available before the flag is produced; if removing the code changes the model, performance must be evaluated for that usable version.
Timing, Outcome Horizons, and Actions
A prediction point, sometimes called an index time, anchors the rest of the setting. It determines which information is available and when the outcome window begins. A practical specification might say: “At the start of each day, predict whether a service call will occur in the next 30 days, using readings recorded by that start time.” The phrase “in the next 30 days” matters: readings collected during those 30 days cannot be treated as inputs available at the start.
The outcome horizon also needs to match the decision. If an organization needs a 30-day forecast but evaluates a model using a 90-day outcome, the reported performance answers a different question. Similarly, if an action is taken after the prediction, that action may affect the outcome. The historical relationship between predictors and outcomes may then not describe the relationship under the new workflow. This does not mean that predictions are necessarily useless; it means that the impact of the changed workflow should not be assumed away.
Worked Example: Checking a Prediction Horizon
A hypothetical equipment operator wants a prediction at the beginning of each shift about whether a unit will require an unplanned service visit within the next 14 days. The development file contains 300 shift records and a target indicating whether a visit occurred within 14 days after each recorded shift. One proposed predictor is the count of sensor alerts during the seven days after that shift. Decide whether the proposed predictor and target match the intended use.
Method. Set the prediction time to the beginning of the shift, then check every input and the target window relative to that time. The common mistake to avoid is assuming that a variable is valid just because it helps predict the target retrospectively.
Step 1: State the prediction time and target. The prediction is made at the start of a shift. The target is a service visit during the following 14 days. These define a coherent decision point and outcome horizon.
Step 2: Compare the proposed predictor’s time window. The alert count covers the seven days after the shift. Those alerts occur after the prediction is supposed to be issued. Therefore, that count is not available at the required time.
Step 3: Determine the implication. The post-shift alert count cannot be used for this prediction setting. The model must use information recorded by the start of the shift, such as an alert count over a specified period ending at that time, if such a measure is appropriate and available. Its performance must then be evaluated for the intended 14-day target using those eligible inputs.
Conclusion. The 14-day label window matches the stated target, but the proposed predictor violates the timing assumption. A high retrospective association would not resolve that mismatch: a future value cannot support a prediction that must be made before it exists.
Operating Conditions and Future Cases
Even when the cases, predictors, decision time, and outcome horizon are specified correctly, the conditions under which the model operates matter. Examples include a process rule that changes which cases receive an intervention, a capacity limit that changes service delays, or a technology change that alters how an outcome is recorded or produced. These are not merely background details if they affect the relationship between the available predictors and the outcome.
One way to express the key performance assumption is to let \(X\) denote the predictors, \(Y\) the outcome, and \(f(X)\) the model’s prediction. For squared-error performance, the expected loss in a future setting can be written as a weighted average of conditional errors:
This expression applies when \(X\) takes values in a discrete set; for continuous predictors, the sum is replaced by an integral. It follows from the law of total expectation: divide future cases into groups defined by \(X=x\), calculate the average squared error within each group, and weight that error by the future proportion of cases in the group. The formula shows why a historical overall error cannot be assumed to carry over automatically. The future group proportions or the error within a group may differ.
This is a reasoning tool for stating assumptions, not a guarantee that future conditional errors can be measured precisely. In practice, the setting must be described and evidence considered before carrying a performance estimate over to it.
Worked Example: Changed Mix Under the Same Conditional Errors
A hypothetical repair service uses a model to predict repair hours. For illustration, assume there are two operating groups, routine and intensive. Suppose the mean squared prediction error within the routine group is 4 square hours and within the intensive group is 16 square hours. In the evaluation setting, 75% of cases are routine and 25% intensive. A future operating plan is expected to have 40% routine and 60% intensive cases. Assuming the group-specific mean squared errors remain unchanged, calculate the overall mean squared error and root mean squared error in both settings.
Method. Weight each group’s conditional mean squared error by that setting’s group proportions. Then take the square root of the overall mean squared error to obtain root mean squared error. The common mistake to avoid is assuming the overall error stays constant merely because the error within each group is assumed unchanged.
Step 1: Calculate evaluation-setting mean squared error.
The evaluation-setting mean squared error is 7 square hours. Its root mean squared error is \(\sqrt{7}\approx2.6458\) hours.
Step 2: Calculate future-setting mean squared error.
The future-setting mean squared error is 11.2 square hours. Its root mean squared error is \(\sqrt{11.2}\approx3.3466\) hours.
Step 3: Check and interpret. The calculation uses proportions that each sum to 1: \(0.75+0.25=1\) and \(0.40+0.60=1\). The future root mean squared error is higher because the future setting gives more weight to the group with the larger assumed conditional error. Rounded to two decimals, the two root mean squared errors are 2.65 and 3.35 hours, respectively.
Conclusion. Even under the simplifying assumption that group-specific errors stay fixed, an overall performance figure can change when the future mix changes. If operating conditions also change the errors within groups, this calculation alone is not enough to support a future performance estimate.
Document the Assumption and Its Consequence
An assumption is useful in a report only if a reader can understand what would go wrong if it failed. Replace general statements such as “future conditions may differ” with a concrete statement that names the condition, its relevance, and the consequence. For example: “The model was evaluated using readings available at the start of a shift. If the deployed system receives those readings later, it may be unable to produce a prediction at the required decision time.”
A concise assessment can follow this sequence:
Name the cases, prediction time, available inputs, outcome, horizon, and important operating conditions.
Explain which parts of the development and evaluation data support the intended setting, and which are not represented or established.
Say whether a violation prevents scoring, changes the meaning of a predictor or outcome, or may make reported performance inapplicable.
Possible responses include restricting use to the supported setting, checking input availability before implementation, or evaluating the model under the changed conditions before relying on its predictions.
The response should match the assumption at issue. If a predictor is unavailable at decision time, documenting the limitation does not make it available; the model or workflow must be changed. If the future mix is different, an overall historical error should not be presented as a guaranteed future error. If the operating process changes, the report should state which conclusions still have support and which require further evidence.
Common Mistakes
- Confusing data presence with availability. A field in a historical table may have been recorded after the prediction point.
- Ignoring the time horizon. A model evaluated for one outcome window does not automatically answer a question about another window.
- Assuming a workflow change is irrelevant. New actions, rules, or capacity constraints may change how cases and outcomes are generated.
- Reporting a performance number without its setting. A metric is interpretable only with the cases, time point, inputs, target, and evaluation conditions that produced it.
- Using a vague caveat without a consequence. Explain what could fail and what that means for the decision, rather than simply saying that “results may vary.”
Check Your Understanding
For each situation, identify the assumption at issue and the main implication for use.
- A hypothetical clinic predicts whether a patient will return within 21 days, using information recorded at discharge. A proposed input is whether the patient attended a follow-up appointment two weeks later. Is that input available for the stated prediction? (Answer: No. The appointment occurs after discharge, so it cannot be used for a prediction made at discharge.)
- A model is evaluated on orders for delivery within five days. Operations wants to use its predictions for a 30-day delivery outcome. What setting detail differs? (Answer: The outcome horizon differs; performance for the five-day outcome does not by itself establish performance for the 30-day outcome.)
- In a hypothetical two-group setting, conditional mean squared errors are 9 and 25. The future proportions are 80% in the first group and 20% in the second. Calculate future mean squared error. (Answer: \(0.80(9)+0.20(25)=7.2+5=12.2\) square units.)
- A model predicts equipment failures from sensor summaries available at the start of a shift. A new operating rule changes how long equipment runs between inspections. What assumption should be reviewed, and why? (Answer: The operating-condition assumption; the changed inspection and operating process may alter the relationship between the available summaries and failures.)
- A risk score is intended to support a decision before a customer receives an offer. The evaluation data include whether the customer accepted an offer made after scoring. What should the reviewer establish before treating that field as a predictor? (Answer: Whether the field was genuinely available at the prediction time; if acceptance occurs after the score, it is not a usable input for that decision.)