Introduction
A prediction rule is useful only in relation to the cases for which it will be used. A model developed from records of one group may eventually be used for a different group, at a different time, or to support a decision that was not part of its original development. Those differences matter: the intended users need predictions for their cases, not merely a description of how well a rule fits the development data.
A prediction question therefore needs more than an outcome and a set of predictors. It should state who or what will receive predictions, when predictions will be made, and how the predictions will be used. These choices define the practical target of the prediction task and clarify which evidence would be relevant for judging its usefulness.
Development Cases and Prediction Cases
The development sample consists of the cases whose recorded data are used to build a prediction rule. The prediction population consists of the cases expected to receive predictions when the rule is put to use. These sets may overlap, but they are not the same idea. A sample describes where development data came from; the prediction population states where predictions are intended to go.
For example, a rule might be developed from patients treated at two hospitals in one year and intended for use at a third hospital in later years. It would be misleading to describe its purpose simply as “predicting patient outcomes.” The patient group, place, and time of intended use make the target more precise. They also raise a practical question: are the development cases informative about the cases that will receive predictions?
Specifying a population does not guarantee that a prediction rule will work well there. It makes the claim testable and the purpose interpretable. If a rule is intended for older adults, evidence from younger adults alone does not directly establish how useful it will be for older adults. If a rule will be used next year, performance on past cases does not by itself establish that conditions will remain similar.
Worked Example: A Prediction Rule for a Different Hospital
A hypothetical hospital group develops a rule using 800 adult admissions from Hospital A during 2024. It predicts whether a patient will be readmitted within 30 days. The group proposes using the rule for adult patients at Hospital B during 2026. Identify the development sample, prediction population, prediction occasion, and intended use.
Development sample. The 800 adult admissions recorded at Hospital A during 2024 are the development cases. These are the records used to construct the rule.
Prediction population. The proposed prediction population is adult patients admitted to Hospital B during 2026. It is not all patients in the hospital group, because the proposal specifies adults, one hospital, and a particular year.
Prediction occasion. The proposal must specify when during an admission the prediction is made. Suppose the intended timing is at discharge. The rule can then use only information available by discharge to predict readmission during the following 30 days. Information recorded after discharge would not be available at the stated prediction occasion.
Intended use. Suppose the prediction is meant to help the care team decide which patients should receive an additional follow-up call after discharge. The prediction informs a follow-up decision; it does not itself establish that a call will prevent readmission.
The development and prediction cases differ in both hospital and year. The rule’s performance among the 800 development cases alone cannot establish its performance for adult patients at Hospital B in 2026. That is a separate question about the cases and conditions of intended use.
Specify the Prediction Occasion
A prediction occasion is more than a calendar date. It identifies the point at which a prediction must be available and therefore helps determine which predictor information could actually be used. It also makes the time relationship between predictors and the outcome explicit. For a prediction made at a school-year start, for example, end-of-year grades cannot be used as predictors for that same year’s outcome.
A useful specification names three things: the event or time at which the prediction is issued, the information available by then, and the future period or event the outcome refers to. The prediction occasion can be a scheduled time, such as the start of each month, or a process point, such as a patient’s discharge. Without this detail, “predicting the outcome” may refer to tasks with very different practical requirements.
Worked Example: Predicting Demand Before a Clinic Session
A hypothetical clinic wants to predict the number of patients who will arrive for each morning session. A proposed rule uses a session’s scheduled appointment count and the number of cancellations known by 7:00 a.m. The clinic wants the prediction early enough to decide how many staff members to schedule.
Prediction population. The cases are morning clinic sessions at this clinic, not individual patients. Each session receives one prediction of its patient count.
Prediction occasion and available information. The prediction is issued at 7:00 a.m. on the day of the session. At that point, the scheduled appointment count and cancellations recorded by 7:00 a.m. are available. The number of patients who actually arrive later that morning is not yet available.
Outcome and timing. The outcome is the number of patients who arrive during that morning session. The prediction is made before those arrivals occur.
Intended use. The clinic uses the prediction to plan staffing for that session. A prediction issued after the session starts might still describe demand, but it could arrive too late to serve this staffing purpose.
This specification also makes an information error easier to detect. A proposed predictor such as “actual number of arrivals” would not be available at 7:00 a.m.; using it would answer a different, impractical question.
Intended Use Gives Predictions a Purpose
The same predicted outcome can support different uses. A predicted readmission risk might inform a follow-up call, help a service estimate its staffing needs, or be reported for planning. These are not interchangeable. The intended use affects which cases matter, when a prediction is needed, and what counts as useful information.
In particular, a prediction can be accurate on average yet poorly suited to a specific use. A clinic planning staffing may care about session-level patient counts, while a care team deciding whom to contact needs patient-level predictions. A population and unit of prediction should match the task: a group-level prediction does not automatically identify which individuals need an action.
Predictions also do not dictate decisions. A prediction is an estimate of an outcome; a decision is an action taken using that estimate along with other considerations. For example, a predicted risk can inform a follow-up decision, but it does not prove that follow-up will change the outcome. The prediction question and the question of whether an intervention causes a change are distinct.
Worked Example: How the Intended Use Changes the Target
A hypothetical school network has records from 1,200 students in three middle schools. It is considering predicting whether a student will need additional mathematics support during the next term. Compare two possible uses: (A) teachers receive predictions for students in their own classes before the term begins; (B) district staff estimate how many support places will be needed across all middle schools.
Use A: individual decisions. The prediction population is students enrolled in the relevant classes at the start of the next term. Each student is a prediction case, and the prediction must be available before teachers decide whom to contact or assess. The outcome is whether each student will need additional support during that term. The intended use is to help prioritize attention, not to label a student permanently.
Use B: resource planning. The prediction population is the set of middle schools or school terms for which district staff need estimates. The target is the total number of support places required across those schools, not an individual student’s outcome. The prediction must arrive before the district allocates places.
The same historical student records may help inform both tasks, but the prediction cases and uses differ. For Use A, an overall estimate of 90 students needing support would not identify which students might need attention. For Use B, a list of individual predictions is not yet the requested total; the district needs an aggregate estimate at the school or district level.
Why the Prediction Population Matters
A prediction rule’s errors occur among the cases to which it is applied. Therefore, a summary of predictive error is meaningful only in relation to a specified population and use. One simple measure is the mean squared prediction error: for each case, subtract the predicted outcome from the observed outcome, square the difference, and average across the cases of interest.
For \(m\) cases in a stated evaluation group, write the observed outcome for case \(i\) as \(Y_i\) and its prediction as \(\hat{Y}_i\). The mean squared prediction error is
This calculation is not a promise that the same error will occur in every future group. It summarizes errors for the evaluation cases. If those cases differ from the intended prediction population, the summary may not describe performance in the intended use.
A short justification shows why the population’s composition can matter. Suppose an evaluation group contains two types of cases. Type A has mean squared prediction error 4, and Type B has mean squared prediction error 16. If 80% of cases are Type A and 20% are Type B, the overall mean squared error is
If the intended prediction population instead has 30% Type A and 70% Type B, and the two types have the same respective errors, its overall mean squared error is
The calculation uses the same errors within each type, but the overall summaries differ because the population proportions differ. Thus an evaluation summary is not population-free: its relevance depends partly on whether the evaluation cases resemble the cases that will receive predictions. This is one reason to state the prediction population before describing a rule as useful.
Common Mistakes
- Treating the development sample as the intended population. The wrong reasoning is, “The rule was developed from these patients, so these patients define where it will be used.” The correction is to state separately which cases supplied development data and which cases are intended to receive predictions.
- Leaving timing vague. “Predict readmission” does not say whether the prediction is made at admission or discharge, or which information is available. State the prediction occasion and the outcome period.
- Describing an outcome without a use. A predicted quantity may be interesting but not actionable for the stated task. Name the decision or planning activity the prediction is intended to inform.
- Assuming good results transfer automatically. The wrong reasoning is, “The rule performed well in one setting, so it will perform equally well everywhere.” The correction is to identify the intended population and recognize that performance in that population requires relevant evidence.
- Confusing prediction with intervention effect. A prediction can inform an action, but it does not show that the action will cause a better outcome.
Check Your Understanding
For each situation, identify the prediction population, prediction occasion, and intended use where possible.
- A city develops a rule from past daily records to predict tomorrow’s water demand before operators set pumping plans. What cases receive predictions, and when must each prediction be available?
- A model is developed from patients at one clinic but proposed for use at another. Why is performance in the development sample not enough to establish performance for the proposed use?
- A school wants a prediction of total absences next month to arrange substitute coverage. Are the prediction cases students or school-months? Explain in relation to the requested use.
- Why might information recorded after a prediction occasion be unsuitable as a predictor, even if it is strongly related to the outcome?
- Explain why a prediction of which patients may need follow-up does not, by itself, show that follow-up will prevent the outcome.