Worked KNN Regression Cases with Scaling and Tuning

Applying KNN Regression

In \(k\)-nearest-neighbor (KNN) regression, a prediction is formed from the observed responses of the training observations closest to a new predictor vector. The method is local: changing the predictors, the distance scale, or the number of neighbors can change which responses contribute to a prediction. In applied problems, the arithmetic is usually manageable; the important decisions are to scale sensibly, use the specified neighbor rule, and tune \(k\) using validation data rather than the training fit.

KNN regression does not require a parametric distribution for the response or a linear relationship between predictors and response. It does require a defined predictor representation and distance rule. Its predictions can be useful when nearby observations tend to have similar expected responses. If the predictors are on very different scales, a distance calculation in the original units can give one predictor disproportionate influence for purely numerical reasons.

Scaling Before Measuring Distance

For predictors measured in different units, a common approach is to scale each predictor using constants calculated from the training data. Two familiar choices are min-max scaling and standardization. For min-max scaling, predictor \(j\) is transformed as follows, where \(a_j\) and \(b_j\) are its training-set minimum and maximum:

For standardization, subtract the training mean and divide by the training standard deviation. After either transformation, Euclidean distance between a query and training observation is the square root of the sum of squared differences across the transformed predictors. The scaling rule is a modeling choice, not a guarantee that every predictor is equally relevant. The transformed features and the distance rule must be appropriate to the problem.

Unweighted and Distance-Weighted Predictions

The ordinary KNN prediction gives each selected neighbor equal influence. A weighted version gives closer neighbors more influence. One possible rule is inverse-distance weighting, with weight \(w_i=1/d_i\), where \(d_i\) is the distance to neighbor \(i\). The prediction is the weighted response average. Other weight rules are possible, so a problem's specified rule takes precedence.

Here \(N_k(\mathbf{x}_0)\) denotes the selected set of \(k\) neighbors. A distance-weighted prediction still averages responses, but it does not generally equal the unweighted average. If a selected neighbor has distance zero, inverse-distance weights are undefined; a standard practical rule is to predict from the zero-distance observations alone, or to use a specified alternative weighting rule. Do not apply the inverse-distance formula unchanged at zero distance.

Why the Neighbor Average Is the Least-Squares Prediction

The unweighted neighbor average has a simple justification. Within the selected neighborhood, treat the prediction as one constant \(c\). Choose \(c\) to minimize the sum of squared deviations between the neighbor responses and that constant:

Differentiate with respect to \(c\), set the derivative equal to zero, and solve. Since the second derivative is \(2k>0\), the resulting critical point is the minimum:

Thus the neighbor average is the constant that minimizes squared error among the selected responses. If squared deviations are instead weighted by positive weights, minimizing the weighted sum gives the weighted average in the preceding formula. This explains the calculation; it does not establish that the chosen neighbors will predict well for new data. That is why \(k\) must be assessed using data not used to fit the neighborhood predictions.

Tuning \(k\) with Validation Errors

A small \(k\) makes predictions depend on only a few observations and can be sensitive to individual responses. A larger \(k\) averages over a broader neighborhood and may smooth away local variation. The appropriate choice depends on predictive performance. Given validation responses \(y_j\) and predictions \(\widehat{y}_j(k)\), the validation mean squared error (MSE) is the average squared prediction error:

For a set of candidate values of \(k\), select the value with the smallest validation MSE, unless the problem specifies a different criterion or tie rule. The validation observations are prediction targets, not members of their own neighbor pools: each validation prediction is formed from the training observations. Training error alone is not a substitute for validation error because it measures fit on the observations used to construct the predictions.

A Reliable Calculation Sequence

A careful KNN regression solution keeps scaling, neighbor selection, averaging, and tuning as distinct steps. This makes it easier to identify where an incorrect result entered the calculation.

These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. Common errors include scaling the query differently from the training data, using distances from unscaled predictors after the problem specifies scaling, averaging only the weights rather than the weighted responses, and choosing \(k\) from training performance when validation errors are supplied. A final check should confirm the neighbor identities, the prediction units, and the direction of the validation comparison.