From Claim Records to a Classification Decision
A claim classifier is useful only when its prediction supports a clearly defined decision. For example, a hypothetical insurer might want to identify claims likely to require additional review. The target could be \(Y=1\) for a claim exceeding a stated severity threshold and \(Y=0\) otherwise. The threshold defining the outcome must be fixed before modeling; changing it after examining validation results changes the problem being evaluated.
This case compares a pruned classification tree, a random forest and k-nearest neighbors (KNN). The aim is not to declare one algorithm universally best. It is to prepare predictors appropriately, compare the models on the same untouched validation cases, and select a decision rule that reflects the costs of the two types of classification error.
For a hypothetical claim-review task, the positive class might mean “the claim exceeds the review threshold,” not “the claim is fraudulent” or “the policyholder is at fault.” Defining the target precisely prevents a model score from being interpreted as evidence of something it was not trained to predict.
Prepare Predictors and Protect Validation
Separate the records into training and validation data before any data-dependent preparation. Estimate missing-value replacements, scaling constants, category groupings and any feature-selection decisions using the training data only. Apply those fitted choices to validation records without recalculating them there. Otherwise, validation information has influenced the fitted pipeline and the resulting performance estimate can be optimistic.
The preparation depends on the method. A tree can split on a numeric predictor without standardizing its units, although missing values and categorical encoding still need a consistent treatment. A random forest also generally does not require scaling for its tree splits. KNN, by contrast, compares distances, so a predictor measured in thousands can dominate another measured in fractions unless scales are made comparable. Categorical predictors need a representation that gives sensible distances; arbitrary integer codes can falsely imply an ordering and spacing.
For a numeric predictor \(X_j\), a common training-set standardization is \(Z_j=(X_j-\bar X_j)/s_j\), where \(\bar X_j\) and \(s_j\) are the training mean and standard deviation. The same training values are used to transform validation observations. If \(s_j=0\), the predictor is constant in training data and cannot be standardized by this formula; it should be removed or handled by a stated alternative.
If the positive class is uncommon, resampling may be used inside training folds, as in the earlier material on resampling imbalanced data within training folds. Do not resample the validation set: validation should preserve the class prevalence relevant to the intended population. If resampling changes the training class proportions, model probabilities may not represent population probabilities without calibration or an appropriate prior-probability correction. A predicted probability should therefore not automatically be treated as calibrated merely because the model outputs a number between zero and one.
Worked Example: Preparing a Claim for KNN
A hypothetical training set has claim age with mean 40 years and standard deviation 10 years, and reported amount in thousands of dollars with mean 20 and standard deviation 10. A validation claim has age 50 and amount 30. Two training claims have standardized predictor coordinates \(A=(0,0)\) and \(B=(1,0)\). Standardize the validation claim and find its two Euclidean distances.
Method. Compute standard scores from the training means and standard deviations, then use Euclidean distance in the standardized predictor space.
For age, the validation value is \(Z_{\text{age}}=(50-40)/10=1\). For amount, it is \(Z_{\text{amount}}=(30-20)/10=1\). Thus the validation point is \((1,1)\).
Its distance from \(A=(0,0)\) is \(d_A=\sqrt{(1-0)^2+(1-0)^2}=\sqrt{2}\approx1.4142\). Its distance from \(B=(1,0)\) is \(d_B=\sqrt{(1-1)^2+(1-0)^2}=1\). Therefore \(B\) is the nearer of these two training claims.
Check. The amount is expressed in thousands both when calculating its standard score and when reporting the original mean and standard deviation, so no factor-of-1,000 mismatch enters the calculation. The nearest point is \(B\), since \(1<\sqrt{2}\). This example avoids the mistake of comparing raw age and dollar amounts directly, which would make distance depend on arbitrary measurement units.
Fit and Compare the Three Classifiers
A classification tree repeatedly partitions predictor space into regions and assigns each terminal region a class prediction or estimated class probability. A pruned tree is restricted to avoid retaining branches that add little predictive value. The complexity or pruning choice should be made using training data, often by cross-validation, rather than by repeatedly inspecting the final validation results.
A random forest combines many trees fitted with randomized sampling and predictor selection. For classification, it can aggregate class votes or average tree-level class probabilities. KNN assigns a new case using the labels of its \(k\) nearest training observations under a specified distance and predictor representation. For an unweighted binary KNN classifier, the estimated positive-class proportion among the \(k\) neighbors is a natural score, but it is not guaranteed to be a calibrated probability.
Compare all three methods on the same validation cases and at stated thresholds. Otherwise, differences in class composition or case difficulty can be mistaken for differences in model quality. If model choice, tuning or threshold selection is adaptive, validation results used for those choices are no longer an independent final performance assessment. Nested cross-validation or a separate final test set can address that issue; the earlier material on nested cross-validation explains the general separation of model selection and evaluation.
For the positive class, sensitivity is \(TP/(TP+FN)\), the fraction of actual positives detected. Precision is \(TP/(TP+FP)\), the fraction of positive predictions that are correct. Specificity is \(TN/(TN+FP)\), the fraction of actual negatives correctly rejected. Accuracy is \((TP+TN)/(TP+FN+FP+TN)\). The F1 score is the harmonic mean of precision and sensitivity, \(2(\text{precision})(\text{sensitivity})/(\text{precision}+\text{sensitivity})\), when the denominator is positive. If a denominator is zero, that metric is undefined; a convention for reporting it must be stated rather than silently assuming a value.
Worked Example: Compare Validation Classifications
On the same 100 hypothetical validation claims, 20 are positive. At a common specified threshold, a pruned tree gives \(TP=12,FN=8,FP=10,TN=70\); a random forest gives \(TP=16,FN=4,FP=16,TN=64\); and KNN gives \(TP=14,FN=6,FP=8,TN=72\). Calculate sensitivity, precision, specificity, accuracy and F1 for each.
Method. Use the stated confusion-matrix definitions, keeping actual positives \(TP+FN=20\), actual negatives \(FP+TN=80\), and total \(100\) fixed across models.
Pruned tree. Sensitivity \(=12/20=0.6000\); precision \(=12/(12+10)=12/22=0.5455\); specificity \(=70/80=0.8750\); accuracy \(=(12+70)/100=0.8200\). Its F1 score is \(2(12)/(2(12)+10+8)=24/42=0.5714\).
Random forest. Sensitivity \(=16/20=0.8000\); precision \(=16/(16+16)=0.5000\); specificity \(=64/80=0.8000\); accuracy \(=(16+64)/100=0.8000\). Its F1 score is \(32/(32+16+4)=32/52=0.6154\).
KNN. Sensitivity \(=14/20=0.7000\); precision \(=14/(14+8)=14/22=0.6364\); specificity \(=72/80=0.9000\); accuracy \(=(14+72)/100=0.8600\). Its F1 score is \(28/(28+8+6)=28/42=0.6667\).
Check and interpretation. Each sensitivity denominator is 20, each specificity denominator is 80, and each accuracy denominator is 100. KNN has the highest accuracy, precision, specificity and F1 here; the forest has the highest sensitivity. Which is preferable depends on the relative consequences of missed positives and false alarms, not accuracy alone. The common mistake is to call the forest “best” solely because it detects the most positives, ignoring its larger number of false positives.
Choose the Decision Threshold From Costs
A model score is not itself a decision. Suppose the estimated probability of a positive claim is \(p\), the cost of a false positive is \(C_{FP}\), and the cost of a false negative is \(C_{FN}\). Assume correct classifications have zero incremental cost, costs are constant across cases, and \(p\) is a calibrated probability for the relevant population. Predicting positive has expected cost \((1-p)C_{FP}\), because that action incurs a false-positive cost only when the case is actually negative. Predicting negative has expected cost \(pC_{FN}\).
Thus the cost-minimizing threshold is \(c=C_{FP}/(C_{FP}+C_{FN})\) under these assumptions. It is not generally \(0.5\). If costs vary by claim, there are additional costs for correct actions, or probabilities are miscalibrated, this simple threshold may not be appropriate; compare the relevant expected costs for each action instead.
Worked Example: Convert Review Costs Into a Threshold
A hypothetical claim-review program assigns an incremental cost of 200 dollars to reviewing a claim that is actually negative, and an incremental cost of 1,200 dollars to failing to review a claim that is actually positive. Assume calibrated probabilities, zero incremental cost for correct decisions, and constant costs. Find the threshold and decide how to handle claims with estimated positive probabilities 0.10, 0.20 and 0.35.
Method. Here \(C_{FP}=200\) and \(C_{FN}=1{,}200\), so \(c=200/(200+1{,}200)=200/1{,}400=1/7\approx0.1429\). Review when \(p\ge1/7\).
At \(p=0.10\), reviewing has expected cost \((1-0.10)(200)=0.90(200)=180\) dollars. Not reviewing has expected cost \(0.10(1{,}200)=120\) dollars, so do not review.
At \(p=0.20\), reviewing costs \((1-0.20)(200)=160\) dollars in expectation, while not reviewing costs \(0.20(1{,}200)=240\) dollars. Review.
At \(p=0.35\), reviewing costs \(0.65(200)=130\) dollars, while not reviewing costs \(0.35(1{,}200)=420\) dollars. Review.
Check. The decisions agree with the threshold: 0.10 is below 0.1429, and 0.20 and 0.35 exceed it. The mistake to avoid is applying the default 0.5 threshold, which would reject both the 0.20 and 0.35 cases despite the much larger cost of a missed positive.
Validation Choices and Practical Traps
The comparison should be tied to the operational objective. If missing a positive is especially costly, sensitivity and the false-negative count deserve close attention. If unnecessary reviews consume substantial resources, precision, specificity or expected cost may be more relevant. A single summary metric can conceal these trade-offs, especially when positives are rare. Report the class counts and the threshold alongside metrics so that the results can be interpreted in context.
Use training folds for tuning the tree's pruning level, forest settings, KNN neighborhood size and any data preparation. If a threshold is selected to optimize a metric or estimated cost, that choice is also part of model development. Do not repeatedly change the threshold after seeing validation outcomes and then report that same validation performance as an unbiased final estimate. Keep a final test set untouched, or use an evaluation design that separates tuning from assessment.
- Leakage: fitting a scaler, imputation rule or resampling procedure using validation observations lets information cross the split.
- Misleading accuracy: a classifier can achieve high accuracy by mostly predicting the prevalent class while missing many positives.
- Unmatched comparisons: evaluate models on the same cases and report the same outcome definition and threshold convention.
- Uncalibrated scores: applying a cost-derived probability threshold requires probabilities with a meaningful population interpretation, not merely arbitrary ranking scores.
- Metric confusion: sensitivity uses actual positives as its denominator; precision uses predicted positives.
Check Your Understanding
For each hypothetical problem, calculate the requested quantity before checking the answer.
- A training set has mean payment 12 thousand dollars and standard deviation 4 thousand dollars. A validation claim has payment 20 thousand dollars. What is its standardized value using the training quantities? (Answer: \(z=2\).)
- A classifier has \(TP=18,FN=6,FP=12,TN=64\). Calculate sensitivity and precision. (Answer: Sensitivity \(=18/24=0.7500\); precision \(=18/30=0.6000\).)
- A hypothetical review action has false-positive cost 150 dollars and false-negative cost 450 dollars. Assuming calibrated probabilities and zero incremental cost for correct decisions, find the cost threshold and the decision for a claim with \(p=0.30\). (Answer: \(c=150/(150+450)=0.2500\); review the claim because \(0.30\ge0.25\).)
- On 80 validation claims, a model yields \(TP=10,FN=10,FP=5,TN=55\). Calculate accuracy and specificity. (Answer: Accuracy \(=(10+55)/80=0.8125\); specificity \(=55/(55+5)=0.9167\).)
- A forest has sensitivity 0.85 and precision 0.40, while a KNN classifier has sensitivity 0.70 and precision 0.65 on the same validation cases. If missed positives are more costly than false positives, can these metrics alone establish which model minimizes expected cost? State what else is needed. (Answer: No. The positive-class prevalence or confusion-matrix counts, calibrated probabilities or a fixed-threshold cost calculation, and the costs of the two error types are needed to compare expected cost.)