Start With the Decision, Not the Algorithm
A method can be technically well fitted and still be a poor choice for the decision at hand. Predicting whether a claim will require further review is different from estimating its expected severity, and both are different from exploring whether a portfolio contains distinct groups. The first choice is therefore not “Which algorithm is most powerful?” but “What must the analysis deliver?”
This tutorial gives a decision framework for matching classification and regression trees, ensembles, principal components analysis, clustering, and k-nearest neighbors (kNN) to a task. The selection depends on whether an outcome is available, whether that outcome is categorical or quantitative, and whether the predictors are suited to rules, combinations, or distances. The framework is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
These labels describe the task, not the algorithm’s importance or sophistication. A complex method does not turn an exploratory task into a predictive one. Similarly, principal components may summarize predictors, but a component that explains much predictor variation is not automatically useful for predicting an outcome.
A Practical Matching Framework
Use the outcome and the structure of the predictors to narrow the choices. Then check whether the method’s output supports the decision: for example, a risk ranking, a numerical estimate, a compact representation, or a set of groups.
| Decision task | Useful candidate method | Why it may fit | Key limitation to check |
|---|---|---|---|
| Predict a categorical outcome | Classification tree or tree ensemble | Trees can represent nonlinear splits and interactions; ensembles can combine many trees | A single tree may be unstable; assess performance using a decision-relevant metric |
| Predict a quantitative outcome | Regression tree, tree ensemble, or kNN regression | These methods can represent nonlinear patterns without requiring one global straight-line relationship | Local averages or tree leaves can be unreliable when observations are sparse |
| Summarize correlated quantitative predictors | Principal components analysis (PCA) | Re-expresses predictors as orthogonal directions ordered by explained variation | It does not use an outcome and does not guarantee predictive relevance |
| Find groups without a known outcome | Clustering | Organizes observations by similarity in the chosen predictor representation | Groups depend on the representation and similarity measure; they are not outcome evidence |
| Predict from nearby examples | k-nearest neighbors | Uses outcomes among observations close to the new case | Distance can be uninformative if predictors are poorly represented or measured on incompatible scales |
A classification tree divides predictor space into regions and assigns a class or class probability to each region. A regression tree uses regions to predict a quantitative value, commonly a mean in that region. A tree ensemble combines predictions from multiple trees; random forests and boosting are examples. Ensembles can be useful when the relationship is complex or a single tree’s predictions are too sensitive to the particular training sample, though their combined predictions are generally less compact to explain as a small set of rules.
PCA and clustering answer different unsupervised questions. PCA finds directions that summarize variation in predictors; clustering groups observations according to similarity. Neither method, by itself, estimates a class label or a numerical outcome. kNN, by contrast, is supervised when it uses observed outcomes of nearby training cases to predict an outcome for a new case.
Worked Example: Choosing a Classifier for a Review Queue
Worked Example: Choosing a Classifier for a Review Queue
A hypothetical insurer wants to identify claims likely to require additional documentation. The outcome is binary: “requires additional documentation” or “does not.” On the same validation set of 100 claims, 20 actually required documentation. A classification tree predicts 15 of those 20 correctly and flags 10 of the 80 other claims. A random forest predicts 18 of the 20 correctly and flags 14 of the 80 other claims. If overall accuracy is the stated objective, which model is preferred? What changes if missing a claim that needs documentation is the main concern?
Method. This is a supervised classification task because the outcome is categorical and observed for the validation cases. Calculate the confusion-matrix quantities before choosing a model.
For the tree, true positives are 15, false negatives are \(20-15=5\), false positives are 10, and true negatives are \(80-10=70\). Its accuracy is \((15+70)/100=0.85\), or 85%. Its sensitivity is \(15/20=0.75\), and its specificity is \(70/80=0.875\).
For the random forest, true positives are 18, false negatives are \(20-18=2\), false positives are 14, and true negatives are \(80-14=66\). Its accuracy is \((18+66)/100=0.84\), or 84%. Its sensitivity is \(18/20=0.90\), and its specificity is \(66/80=0.825\).
If overall accuracy is the objective and the stated predictions are used, choose the tree: 85% is greater than 84%. If the main concern is missing claims that need documentation, the forest has the higher sensitivity, 90% versus 75%. Its lower accuracy reflects more false positives as well as fewer false negatives.
Check. The counts reconcile: for each model, true positives plus false negatives equal 20, and false positives plus true negatives equal 80. The common mistake is to call the forest the winner merely because it finds more positive cases. The preferred model depends on which error matters for the decision; accuracy alone does not capture every objective.
When Trees Fit Nonlinear Quantitative Outcomes
A regression tree is a natural candidate when the outcome is numerical and the expected relationship can change across regions of predictor space. It gives a piecewise-constant prediction: cases in the same terminal region receive the same prediction. This can represent a threshold or interaction without requiring one straight-line relationship across the entire predictor range. A tree is not automatically preferable to other regression methods; it is a candidate whose predictions should be assessed for the intended use.
For squared-error fitting, the reduction in within-region sum of squares provides a direct way to assess a proposed split. The reason is that deviations from the overall mean can be separated into within-group deviations and differences between group means.
Here \(g\) indexes the groups formed by a split, \(n_g\) is the number of observations in group \(g\), \(\bar{y}_g\) is that group’s mean, and \(\bar{y}\) is the overall mean. To see why the decomposition holds, write \(y_i-\bar{y}=(y_i-\bar{y}_g)+(\bar{y}_g-\bar{y})\) for each observation in group \(g\), then square and sum. The cross-product is zero because \(\sum_{i\in g}(y_i-\bar{y}_g)=0\). Thus a split’s reduction in within-group squared error equals the between-group term.
Worked Example: Assessing a Regression-Tree Split
In a hypothetical portfolio, eight claims have severity, in thousands of dollars, as follows. Four claims in a low-risk region have severities 2, 4, 4, and 6; four in a high-risk region have severities 8, 10, 12, and 14. A proposed tree split separates those regions. Using squared error, calculate the parent and child sums of squares and the reduction. What would the tree predict for a new claim in the high-risk region?
Method. This is regression because severity is quantitative. Under squared-error fitting, predict each group by its sample mean. No normal-distribution assumption is needed for this arithmetic; the criterion being used is squared error.
The overall mean is \((2+4+4+6+8+10+12+14)/8=60/8=7.5\). The parent sum of squares is \( (2-7.5)^2+(4-7.5)^2+(4-7.5)^2+(6-7.5)^2+(8-7.5)^2+(10-7.5)^2+(12-7.5)^2+(14-7.5)^2 \) \(=30.25+12.25+12.25+2.25+0.25+6.25+20.25+42.25=126\).
The low-risk mean is \((2+4+4+6)/4=16/4=4\), so its within-group sum of squares is \((2-4)^2+(4-4)^2+(4-4)^2+(6-4)^2=4+0+0+4=8\). The high-risk mean is \((8+10+12+14)/4=44/4=11\), so its sum of squares is \((8-11)^2+(10-11)^2+(12-11)^2+(14-11)^2=9+1+1+9=20\). The total after the split is \(8+20=28\), and the reduction is \(126-28=98\).
A new claim assigned to the high-risk terminal region receives the region mean, \(11\) thousand dollars, or $11,000. As a check on the reduction, the between-group term is \(4(4-7.5)^2+4(11-7.5)^2=4(12.25)+4(12.25)=98\), matching \(126-28\). The trap is to predict using the overall mean after a split; the tree’s prediction is the mean of the new case’s terminal region.
Distances, Components, and Groups Answer Different Questions
kNN predicts from the observed outcomes of the \(k\) training cases closest to a new case under a specified distance. For classification, it commonly uses the most frequent class among those neighbors; for regression, it commonly averages their outcomes. It is a plausible candidate when nearby cases are expected to have similar outcomes and the predictors provide a meaningful notion of closeness. The distance definition and predictor representation matter. The example below assumes the two predictor coordinates are already expressed on comparable, meaningful scales; it does not address how to choose or tune \(k\).
Worked Example: Making a k-Nearest-Neighbors Classification
A hypothetical classifier uses two quantitative predictors and Euclidean distance. For a new case at \((0,0)\), the four nearest candidates in the training data are: \((1,0)\), class A; \((0,2)\), class A; \((3,0)\), class B; and \((0,4)\), class B. With \(k=3\), determine the predicted class.
Method. Calculate each distance to \((0,0)\), select the three smallest, and use their majority class. The Euclidean distance from \((x_1,x_2)\) to \((0,0)\) is \(\sqrt{x_1^2+x_2^2}\).
The distances in the order listed are \(\sqrt{1^2+0^2}=1\), \(\sqrt{0^2+2^2}=2\), \(\sqrt{3^2+0^2}=3\), and \(\sqrt{0^2+4^2}=4\). The three nearest cases have classes A, A, and B. Class A receives two votes and class B receives one, so the predicted class is A.
The check is that exactly three neighbors were used and the vote is not tied. A common mistake is to use all four listed cases despite the instruction \(k=3\); doing so would produce a 2–2 tie rather than the specified three-neighbor prediction.
PCA is appropriate when the aim is to summarize predictor variation with a smaller number of directions, especially when quantitative predictors are correlated. It does not use the outcome. The first principal component is the direction along which the projected predictor values have the greatest variance; each later component is constrained to be orthogonal to the preceding ones and captures as much remaining variance as possible. The following calculation quantifies that summary for a two-predictor covariance matrix.
Worked Example: Deciding Whether PCA Summarizes Predictors
A hypothetical exploratory analysis has two centered quantitative predictors with covariance matrix \(\begin{pmatrix}4&2\\2&4\end{pmatrix}\). There is no outcome variable in this analysis. Find the principal directions, calculate the proportion of total variance explained by the first component, and state what decision PCA supports.
Method. The principal directions are eigenvectors of the covariance matrix, and their associated eigenvalues give the variances along those directions. To find the eigenvalues, solve \((4-\lambda)^2-2^2=0\). Thus \((4-\lambda)^2=4\), giving \(\lambda=6\) or \(\lambda=2\).
For \(\lambda=6\), the equations reduce to \(-2a+2b=0\), so \(a=b\); a unit direction is \((1,1)/\sqrt{2}\). For \(\lambda=2\), they reduce to \(2a+2b=0\), so \(a=-b\); a unit direction is \((1,-1)/\sqrt{2}\). Total variance is the sum of the eigenvalues, \(6+2=8\). The first component explains \(6/8=0.75\), or 75%, of the total variance.
PCA therefore supports representing these two predictors by their first component if a one-dimensional summary of predictor variation is the objective. It does not establish that this component predicts any outcome: none was supplied. The check is that the two explained-variance proportions sum to \((6+2)/8=1\). The trap is to treat “explains 75% of predictor variance” as “explains 75% of outcome variation.”
Choosing and Comparing Methods Carefully
Use this sequence to make a defensible method choice:
Specify whether the aim is a class, a numerical prediction, a lower-dimensional representation, or groups for exploration.
If an observed outcome is available and the goal is to predict it, use a supervised method. Without an outcome, PCA or clustering may address an exploratory objective.
Use classification methods for categorical outcomes and regression methods for quantitative outcomes. Consider trees for threshold-like regions, ensembles for a combination of trees, and kNN when local similarity is meaningful.
For example, accuracy and sensitivity answer different classification questions. A reported metric should correspond to the error or benefit that matters.
The same data can support different methods only when the task is stated clearly. If a portfolio’s purpose is to predict a binary event, clustering the predictors does not supply the required class prediction. If the purpose is instead to explore whether the portfolio contains distinct profiles, a supervised classifier requires an outcome that the task may not have. Methods should not be chosen simply because they return an attractive plot, a high explained-variance percentage, or a familiar score.
Common Exam Traps
- Confusing outcome types. A numeric code for a class does not automatically make the outcome quantitative. Confirm whether the values represent measurements or category labels.
- Using PCA as if it were supervised. PCA orders directions by predictor variation, not by predictive value for a response.
- Calling clusters predictions. A cluster label describes a grouping created from predictors; it is not evidence that members share a future outcome.
- Choosing by one metric without considering the decision. As the classification example shows, the model with greater accuracy need not have greater sensitivity.
- Assuming kNN works from any predictor representation. Its logic depends on the relevance of “near” under the chosen distance; a distance calculation is not meaningful merely because software can produce it.
Check Your Understanding
For each case, identify a suitable method family and the reason it fits the stated task.
- A hypothetical call center has a labeled binary outcome: whether a policyholder’s inquiry will require specialist escalation. The aim is to flag cases, and predictor relationships may involve thresholds. Name a suitable tree-based method family. (Answer: A classification tree or classification-tree ensemble, because the outcome is categorical and the task is supervised.)
- A hypothetical severity model has a continuous dollar outcome. A proposed split creates groups with means 5 and 9, each containing 3 observations. The parent mean is 7. Calculate the between-group contribution to the sum-of-squares reduction. (Answer: \(3(5-7)^2+3(9-7)^2=24\).)
- A hypothetical unlabelled data set has two centered predictors with covariance matrix \(\begin{pmatrix}3&0\\0&1\end{pmatrix}\). What proportion of total variance does the first principal component explain, and does this demonstrate outcome-prediction performance? (Answer: \(3/(3+1)=0.75\), or 75%; no, there is no outcome in the task.)
- A kNN classifier uses \(k=3\). The three closest labeled cases to a new observation have classes B, A, and B. What class is predicted by majority vote? (Answer: Class B.)
- An analyst has no target outcome and wants to form groups of customers according to similarity in several measured predictors. Should the analyst begin with a classifier, PCA, or clustering, and what does the output represent? (Answer: Clustering; its output represents predictor-based groups, not predicted outcome classes.)