Worked KNN Classification Cases with Imbalance

Introduction

A KNN classifier assigns a class to a new observation by examining the labels of nearby training observations. The arithmetic is simple, but classification becomes less automatic when one class is rare, the neighbor votes tie, or the consequences of the two kinds of error differ. In those settings, the analyst must distinguish the estimated local class probability from the decision rule used to assign a label.

This tutorial works through those decisions using uniform neighbor votes and explicitly stated tie rules. It also shows how to adjust a local probability when the training sample has class proportions different from those in the population of interest. The calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: For a query observation, a KNN classifier selects the \(k\) training observations with the smallest distances to the query, using a specified distance metric and predictor scaling. With two classes, call the target class positive and the other class negative. If \(n_+\) of the \(k\) selected neighbors are positive, the uniform-vote estimate of the local positive-class probability is \(\hat p=n_+/k\). A majority-vote classifier predicts positive when \(\hat p>0.5\), and negative when \(\hat p<0.5\); a tie rule is needed when \(\hat p=0.5\).

The fraction \(\hat p\) is a local estimate, not a guarantee about the query observation. It is based on the selected neighborhood and can change with \(k\), the distance metric, predictor scaling, or the training data. In a worked calculation, use the specified scaling and metric, identify exactly which observations are among the \(k\) nearest, and only then count their labels.

Class Imbalance and Local Votes

Class imbalance means that one class occurs much more often than another in the data. A classifier that always predicts the common class can then have high overall accuracy while missing every observation in the rare class. For example, if 90 of 100 observations are negative, predicting negative for all 100 is 90% accurate, but it detects none of the 10 positive observations.

KNN makes its decision from a local neighborhood, not by simply assigning the global majority label. A rare class can make up a substantial fraction of the neighbors near a particular query even when it is uncommon overall. Conversely, a neighborhood may contain too few rare-class observations to support a stable estimate. Report the local vote fraction and the decision threshold separately: the fraction describes the neighborhood, while the threshold encodes how a classification decision is made.

Worked Example: An Imbalanced Neighborhood and Unequal Error Costs

A hypothetical claims-screening model has a rare positive class. For a new claim, the seven nearest training observations contain two positive labels and five negative labels. Assume uniform voting. A false positive costs 1 loss unit, and a false negative costs 4 loss units; correct classifications have zero loss. Find the local positive probability and the minimum-expected-loss classification.

Step 1: Calculate the neighborhood estimate. There are \(n_+=2\) positive labels among \(k=7\) neighbors, so

\hat p=\frac{2}{7}=0.2857

The negative vote fraction is \(5/7=0.7143\). Ordinary majority voting therefore predicts negative.

Step 2: Calculate the cost-sensitive threshold. Let \(C_{FP}=1\) be the false-positive cost and \(C_{FN}=4\) the false-negative cost. The threshold is \(C_{FP}/(C_{FP}+C_{FN})=1/5=0.2000\). Since \(2/7>1/5\), the minimum-cost decision is positive, not negative.

Step 3: Check expected losses directly. If the prediction is positive, an error occurs when the observation is negative. The estimated expected loss is \(1(5/7)=5/7=0.7143\). If the prediction is negative, an error occurs when the observation is positive, giving estimated expected loss \(4(2/7)=8/7=1.1429\). The positive decision has the smaller expected loss.

Check. The positive decision is consistent with both the threshold comparison and the direct loss comparison. This case avoids the mistake of treating a majority vote as mandatory when errors have unequal costs.

Why a Cost-Sensitive Threshold Works

Suppose \(\hat p\) estimates the probability that the query belongs to the positive class. Assume the only nonzero losses are \(C_{FP}>0\) for predicting positive when the true class is negative and \(C_{FN}>0\) for predicting negative when the true class is positive. The expected loss of predicting positive is \(C_{FP}(1-\hat p)\); the expected loss of predicting negative is \(C_{FN}\hat p\). Choose positive when the first expected loss is smaller:

C_{FP}(1-\hat p)<C_{FN}\hat p \quad\Longleftrightarrow\quad \hat p>\frac{C_{FP}}{C_{FP}+C_{FN}}.

At equality, the two decisions have equal expected loss, so a prespecified convention may be used. When the costs are equal, the threshold is \(0.5\), recovering the ordinary majority rule. When missing a positive case is more costly, \(C_{FN}\) is larger and the threshold is lower. A lower threshold can increase positive classifications, but it does not change the KNN estimate \(\hat p\) itself.

Decision rule: Under the stated two-class loss assumptions, predict positive if \(\hat p>C_{FP}/(C_{FP}+C_{FN})\), predict negative if \(\hat p<C_{FP}/(C_{FP}+C_{FN})\), and apply a stated tie convention if equality holds. The rule does not apply unchanged when correct classifications also have nonzero or class-dependent losses; in that case, compare the expected loss of each available decision using the specified loss matrix.

Worked Example: A Vote Tie and a Distance-Based Tie Rule

A hypothetical monitoring system uses the four nearest observations to classify an alert as positive or negative. Their distances and labels are: distance 0.14, positive; distance 0.19, negative; distance 0.27, positive; and distance 0.32, negative. The stipulated rule for equal vote counts is to choose the class whose neighbors have the smaller total distance. Find the vote probability and assigned class.

Step 1: Count the labels. There are two positive and two negative neighbors. Thus the uniform-vote positive probability is \(2/4=0.5000\), and neither class has a majority.

Step 2: Apply the stated tie rule. The total distance for the positive neighbors is \(0.14+0.27=0.41\). The total distance for the negative neighbors is \(0.19+0.32=0.51\). Since \(0.41<0.51\), the rule assigns the positive class.

Check. The assigned label is positive, but the vote-based probability remains \(0.5000\). The tie rule selects a label; it does not alter the count-based probability. This calculation avoids silently treating a tied vote as either a positive or negative majority.

There is no single universal rule for breaking a KNN vote tie. A model may use a stipulated distance-based rule, a fixed class ordering, or a different method stated in advance. The calculation must follow the rule given in the problem; inventing one after seeing the labels can make a result irreproducible.

Adjusting for a Changed Class Prevalence

A further distinction matters when the training data were sampled in a way that changes the class proportions. For example, a study may deliberately include equal numbers of positive and negative cases even though positive cases are uncommon in the target population. The unadjusted KNN vote fraction then estimates a probability under the sampled class proportions, not necessarily the target-population probability.

A prior-probability correction is appropriate if observations were sampled conditional on class, so that the sampling changes the class prevalence but preserves the predictor distribution within each class. Let \(\pi_s\) be the positive prevalence in the sample, \(\pi\) the positive prevalence in the target population, and \(\hat p_s\) the local positive probability estimated under the sample. Under that assumption, the target-population odds are the sample local odds multiplied by the target prior odds divided by the sample prior odds.

Find sample local odds.
Calculate \(\hat p_s/(1-\hat p_s)\) from the neighborhood probability estimated in the sampled data.

Adjust for the prevalence change.
Multiply by \(\{\pi/(1-\pi)\}/\{\pi_s/(1-\pi_s)\}\), the ratio of target prior odds to sample prior odds.

Convert odds back to probability.
If the adjusted odds are \(O\), the adjusted probability is \(O/(1+O)\).

The correction follows from Bayes' rule. For predictors \(x\), the odds in a population are the class-conditional density ratio \(f(x\mid +)/f(x\mid -)\) multiplied by the prior odds. Under class-conditional sampling, the density ratio is unchanged while the prior odds change. Therefore,

\frac{P(+\mid x)}{P(-\mid x)} = \frac{\hat p_s}{1-\hat p_s} \frac{\pi/(1-\pi)}{\pi_s/(1-\pi_s)}.

This adjustment should not be applied merely because the classes are imbalanced. If the training data are representative of the target population, the unadjusted neighborhood fraction already uses the relevant class prevalence. Nor is the correction justified if sampling changes the predictor distributions within classes; the class-conditional density ratio would then not be preserved.

Worked Example: Correcting a Balanced Sample to a Rare-Event Population

A hypothetical safety study has equal positive and negative class proportions by design, so \(\pi_s=0.50\). In the target population, positive prevalence is \(\pi=0.10\). Among the ten nearest observations, six are positive, giving \(\hat p_s=6/10\). Assume sampling was conditional on class and preserved predictor distributions within each class. Find the corrected target probability and classify using \(C_{FP}=1\) and \(C_{FN}=5\).

Step 1: Calculate the sample local odds. The sample probability is \(6/10=0.60\), so its odds are \(0.60/0.40=1.5\).

Step 2: Calculate the prior-odds adjustment. Target prior odds are \(0.10/0.90=1/9\). Sample prior odds are \(0.50/0.50=1\). Their ratio is \((1/9)/1=1/9\). The adjusted local odds are \(1.5(1/9)=1/6\).

Step 3: Convert adjusted odds to probability. Odds of \(1/6\) correspond to probability \((1/6)/(1+1/6)=1/7=0.1429\).

Step 4: Apply the cost threshold. The threshold is \(C_{FP}/(C_{FP}+C_{FN})=1/(1+5)=1/6=0.1667\). Since \(1/7<1/6\), predict negative. As a check, predicting positive has expected loss \(1(6/7)=0.8571\), while predicting negative has expected loss \(5(1/7)=0.7143\). The negative decision has smaller expected loss.

Check. The unadjusted sample estimate of \(0.60\) would exceed the cost threshold, but it is not the target-population probability under the stated sampling design. This example avoids using a deliberately balanced sample as if it represented population prevalence.

Common Mistakes and Practical Checks

Using the global majority as the local prediction. Identify the \(k\) nearest labels first. A rare class overall can still be common near a particular query.
Confusing a vote fraction with a decision. The fraction \(n_+/k\) is the local estimate; a threshold and, if needed, a tie convention turn it into a class label.
Using the wrong cost in the threshold. The numerator is the false-positive cost \(C_{FP}\), and the denominator is \(C_{FP}+C_{FN}\). If false negatives are more costly, the threshold should fall below 0.5.
Adjusting prevalence without a sampling reason. Correct only when the sample class proportions differ from the target proportions under an appropriate sampling assumption.
Reporting accuracy alone for an imbalanced problem. State what happens to the rare class as well. A high overall accuracy can coexist with failure to identify positive observations.

A reliable solution records the selected neighbors, their class counts, the vote fraction, the threshold or tie rule, and the resulting label. If prevalence correction is needed, show the odds adjustment before comparing the corrected probability with the decision threshold.

Check Your Understanding

For each problem, use the stated vote rule and costs. Probabilities in the answers are rounded to four decimal places.

A query has five nearest neighbors: two are in the minority class and three are in the majority class. False-positive cost is 2 and false-negative cost is 5. Find the minority-class vote probability and the minimum-cost label. (Answer: probability \(0.4000\); predict the minority class.)
For six neighbors, class A has distances 0.12, 0.24, and 0.35; class B has distances 0.15, 0.20, and 0.40. Votes are tied, and the rule is to choose the class with the smaller total neighbor distance. Give the A vote probability and assigned class. (Answer: probability \(0.5000\); class A.)
A sample has positive prevalence \(\pi_s=0.40\), while target prevalence is \(\pi=0.10\). A neighborhood gives sample local probability \(\hat p_s=0.50\). Assuming class-conditional sampling preserves predictor distributions, find the corrected target probability. (Answer: \(0.1429\).)
A neighborhood estimate is \(\hat p=0.40\). The false-positive cost is 3 and the false-negative cost is 7. Find the threshold and the minimum-cost label. (Answer: threshold \(0.3000\); predict positive.)
A balanced-by-design sample has \(\pi_s=0.50\), but target prevalence is \(\pi=0.20\). A local sample probability is \(0.50\). Find the corrected target probability and state whether it exceeds a cost threshold of \(0.25\). (Answer: probability \(0.2000\); it does not exceed the threshold.)

What You'll Learn