Frame the Question Around a Decision
A report can describe a technically sound model and still leave its reader unsure what the model is for. A decision-maker needs to know what outcome is being predicted, who may use that prediction, which decision it may inform, and what action could follow. These are connected questions, but they are not interchangeable: a prediction is not itself a decision, and a decision is not automatically a causal conclusion.
A useful predictive question names a measurable target and connects it to a practical choice. For example, “Which accounts are likely to need a manual review next month, so a review team can allocate its limited capacity?” is more decision-ready than “Can we predict account outcomes?” It identifies a target event, a user, a decision, and a possible action. It does not claim that reviewing an account will change the event’s probability.
This framing is aligned with the SOA Exam PA (Predictive Analytics) syllabus. The goal here is not to choose an algorithm or report performance metrics. It is to establish what the analysis is intended to do and to keep the language of the report within what prediction can support.
Four Questions to Answer Clearly
Before presenting a model, write down four elements. They can be stated in a sentence or a short set of bullets, but each should be explicit.
Name the event, amount, or other outcome to be predicted, including its unit and relevant time horizon. “A claim” may be too vague; “whether a claim is reported during the next quarter” is more specific.
Name the role that will interpret the prediction, such as an underwriting manager or a service-planning team. Avoid implying that the model acts on its own if a person will make the decision.
State the choice the user faces, such as how to allocate a fixed number of reviews or how much support capacity to schedule.
Describe a plausible operational response, while making clear that the prediction informs rather than automatically determines that response.
The target should not be confused with the action. “Send a reminder” is an action; it is not a prediction target. A target might instead be “whether a payment is received by the due date.” Similarly, “reduce late payments” is an aspiration, not a precise prediction question. Stating the outcome and the time horizon helps the reader understand what the model’s output means.
A decision-ready statement also avoids overstating what is known. “The model identifies accounts with higher predicted late-payment rates” is a predictive statement. “The model identifies accounts whose late payments will be prevented by a reminder” is a causal statement. The second requires evidence about the effect of reminders, not merely a model that predicts late payment.
Worked Example: Matching a Prediction to a Review Decision
Method: Separate the target, user, decision, and action before describing the model. A hypothetical service operation has capacity to manually review 200 of its 1,000 accounts during the next month. A model assigns each account a predicted probability of a documentation delay during that month. In an illustrative evaluation set, 200 accounts are selected for review based on the model’s ranking; 40 have a delay. Among the other 800 accounts, 40 have a delay. What can the report say, and what must it avoid claiming?
Target. The outcome is whether an account experiences a documentation delay during the next month. It is a binary outcome over a stated period.
User and decision. A hypothetical operations manager is the intended user. The decision is how to allocate 200 available manual reviews among 1,000 accounts.
Calculate the observed rates. Among the selected 200 accounts, the observed delay rate is \(40/200=0.20\), or 20%. Among the remaining 800, it is \(40/800=0.05\), or 5%. Across all accounts, there are \(40+40=80\) delays among \(200+800=1{,}000\) accounts, for an overall rate of \(80/1{,}000=0.08\), or 8%.
Action and interpretation. The model may help the manager prioritize the limited review capacity. The selected accounts had a higher observed delay rate in this hypothetical evaluation. That comparison does not show that reviewing an account prevents a delay, nor that the model itself caused the difference. The review policy and any follow-up action would need to be evaluated separately to support such claims.
Common mistake avoided: Do not write, “Reviewing these 200 accounts will prevent 40 delays.” The data only show that 40 delays occurred among the selected accounts in the evaluation. They do not reveal how many delays would have occurred under a different review or intervention policy.
Prediction Does Not Establish Cause
A predictive model describes relationships between recorded information and outcomes. Even if a predictor is strongly associated with an outcome, the association does not by itself show what would happen if a decision-maker changed that predictor or applied an intervention. Other factors may influence both the predictor and the outcome, and the groups receiving different actions may differ before the actions are taken.
The distinction matters in report language. A prediction might estimate the chance of an outcome for a person with a particular recorded profile. A causal question instead asks how that person’s outcome would differ under two possible interventions. Predictive accuracy, even when good, does not answer that counterfactual question on its own.
Worked Example: A Training Program and an Aggregate Reversal
Method: Compare the overall association with the within-group rates, then distinguish that descriptive pattern from a causal conclusion. In a hypothetical workforce, employees are classified as higher-risk or lower-risk before a voluntary training program. The following counts summarize later claims.
| Risk group | Training status | Employees | Claims |
|---|---|---|---|
| Higher-risk | Trained | 800 | 160 |
| Higher-risk | Not trained | 200 | 80 |
| Lower-risk | Trained | 100 | 2 |
| Lower-risk | Not trained | 900 | 45 |
Within the higher-risk group. The trained claim rate is \(160/800=0.20\), or 20%. The untrained rate is \(80/200=0.40\), or 40%.
Within the lower-risk group. The trained claim rate is \(2/100=0.02\), or 2%. The untrained rate is \(45/900=0.05\), or 5%. Within each risk group, the observed rate is lower for trained employees.
Aggregate rates. There are \(800+100=900\) trained employees and \(160+2=162\) claims, giving \(162/900=0.18\), or 18%. There are \(200+900=1{,}100\) untrained employees and \(80+45=125\) claims, giving \(125/1{,}100\approx0.1136\), or 11.36% to two decimal places. The aggregate comparison goes in the opposite direction from both within-group comparisons.
Interpretation. In this hypothetical example, trained employees are more concentrated in the higher-risk group. The aggregate association therefore does not describe the within-group pattern. More importantly, neither comparison alone proves that training reduces claims: participation was voluntary, so trained and untrained employees may differ in other relevant ways.
Common mistake avoided: It would be wrong to conclude from the aggregate rates that training increases claims. It would also be wrong to conclude from the within-group rates that training causes a reduction. The rates are descriptive comparisons, not a controlled estimate of an intervention effect.
Why the Target Must Match the Decision
A model can predict a quantity accurately and still be poorly matched to the choice at hand. A monthly claim count may be useful for staffing a claims team, while the probability of at least one claim may be more relevant to deciding which cases receive a particular review. The report should explain which quantity is being forecast and why that quantity is relevant to the named decision.
The loss associated with a forecasting error can also depend on the decision. Under squared-error loss, the conditional mean is the forecast that minimizes expected squared error, subject to the usual finite-moment condition. This result explains why a mean forecast is suitable for that particular forecasting objective; it does not imply that the mean is best for every decision or loss function.
The condition on forecasts is important. Without finite second moments, an expression involving a cross-product in a squared-error decomposition may not have a defined expectation. The following derivation uses the stated conditions so that the relevant expectations exist.
Assume \(E[Y^2]<\infty\) and \(E[g(X)^2]<\infty\). Conditional Jensen’s inequality gives \(E[m(X)^2]\leq E[Y^2]<\infty\), so both \(m(X)\) and \(g(X)\) have finite second moments.
Write \(Y-g(X)=(Y-m(X))+(m(X)-g(X))\), then square both sides and take expectations.
By Cauchy–Schwarz, the product of the two square-integrable terms is integrable. Since \(m(X)-g(X)\) is determined by \(X\), \(E[(Y-m(X))(m(X)-g(X))]=E[E[Y-m(X)\mid X](m(X)-g(X))]=0\).
The expansion therefore gives \(E[(Y-g(X))^2]=E[(Y-m(X))^2]+E[(m(X)-g(X))^2]\). The last term is nonnegative, so the conditional mean has no greater expected squared error than any such forecast.
This result is about a specific prediction target and a specific error criterion. It does not say that a conditional mean is the right forecast for every decision, and it does not turn an association into a causal effect. In a report, the target and intended use should therefore be stated before presenting a forecast as useful.
Worked Example: Choosing a Forecast for Capacity Planning
Method: State the quantity being forecast, use conditional means for a squared-error objective, and connect the resulting forecasts to an operational use without claiming that the forecasts cause the outcomes. A hypothetical call center wants to forecast daily incoming calls for staffing. Based on information available before each day, there are two forecast situations: a routine day, occurring with probability 0.75, and a scheduled service-change day, occurring with probability 0.25. The conditional mean call counts are 100 and 300, respectively.
Identify the target and user. The target is the number of calls received on a day. The intended user is the staffing planner, and the decision is how much capacity to schedule. The forecast concerns call volume; it does not estimate the effect of changing staffing on call arrivals.
Calculate the overall conditional mean when the day type is not used. The mean forecast is the probability-weighted average: \(0.75(100)+0.25(300)=75+75=150\) calls per day.
Use the available day-type information. With the day type known, the conditional-mean forecast is 100 calls on a routine day and 300 calls on a scheduled service-change day. These are the forecasts that minimize expected squared error among forecasts based on the stated information, provided the call count has a finite second moment.
Check the weighted average. Averaging the two conditional forecasts over the stated day-type probabilities gives \(0.75(100)+0.25(300)=150\), matching the overall mean forecast. The arithmetic is internally consistent.
Interpretation and common mistake avoided. The day-type forecast can inform staffing capacity. It does not establish that scheduling 300 staff members, or any particular staffing action, would cause 300 calls or improve service outcomes. The forecast concerns demand; the effect of a staffing policy is a separate question.
Common Framing Traps
- Using an action as the target. “Predict who should receive a review” mixes a prediction with a decision. Name the outcome being forecast, then state that the forecast may inform review allocation.
- Leaving the user unstated. A model output has no decision meaning until the report identifies who is expected to interpret it and what choice that person faces.
- Using causal verbs for predictive results. “Causes,” “prevents,” and “reduces” require causal support. For a predictive association, use language such as “is associated with,” “has a higher predicted probability,” or “may help prioritize.”
- Assuming that a useful forecast prescribes an action. A prediction can inform a decision, but the decision may also depend on capacity, costs, rules, and human review.
- Choosing a target that does not match the decision. A forecast of average volume may not answer a decision about the chance of exceeding a capacity limit. State the quantity needed for the decision rather than relying on a generic claim that the model predicts well.
Check Your Understanding
For each situation, identify the target, user or decision, and the main interpretive caution.
- A hypothetical insurer predicts whether a submitted application will require additional documentation within 14 days. An operations lead has capacity to contact only a portion of applicants. State a suitable predictive question and one claim the report should avoid. (Answer: Predict whether an application will require additional documentation within 14 days so the operations lead can prioritize contacts; avoid claiming that contact itself prevents the documentation need.)
- A model predicts next-month repair cost for a hypothetical fleet, and a maintenance planner uses it to schedule inspections. Is “high predicted cost” evidence that an inspection will reduce cost? (Answer: No. It is a forecast associated with the information used by the model; the effect of inspection requires separate causal evidence.)
- For a squared-error forecast, assume \(E[Y^2]<\infty\), and compare the conditional-mean forecast \(m(X)\) with a forecast \(g(X)\) having finite second moment. What does the squared-error decomposition imply? (Answer: \(E[(Y-g(X))^2]=E[(Y-m(X))^2]+E[(m(X)-g(X))^2]\), so the conditional mean has no greater expected squared error.)
- In a hypothetical program, the aggregate claim rate is 18% for participants and 12% for nonparticipants, but participation rates differ substantially between higher-risk and lower-risk groups. What conclusion is justified from the aggregate rates alone? (Answer: Only that the observed aggregate rates differ in this data; the comparison alone establishes neither a within-group pattern nor a causal effect of participation.)
- A staffing report predicts a mean of 150 calls per day, while the decision is whether the call count will exceed a capacity limit of 180. What should the report clarify? (Answer: It should clarify that a mean forecast does not by itself state the probability of exceeding 180; the prediction target must match the capacity decision.)