How k Controls Bias and Variance

Why the Number of Neighbors Matters

In K-nearest-neighbor (KNN) regression, a prediction is formed by averaging responses from the $k$ training observations closest to the predictor value being evaluated. The choice of $k$ controls how local that average is. With a small $k$, a prediction can respond sharply to nearby observations, including random noise. With a larger $k$, averaging tends to stabilize the result, but the neighborhood may include observations whose expected responses differ from the value at the target point.

This is the local version of the bias-variance tradeoff. It is a useful guide, not a guarantee that every increase in $k$ worsens bias or improves total error. The actual result depends on the response pattern, the locations of the predictors, and the noise. The calculations below make those dependencies explicit.

Definition: In KNN regression, let $x$ be a target predictor value, and let $N_k(x)$ be the indices of the $k$ training observations closest to $x$, using the specified distance. The ordinary local-average prediction is $$ \widehat m_k(x)=\frac{1}{k}\sum_{i\in N_k(x)}Y_i. $$ Here $Y_i$ is the response for training observation $i$. The regression function is $m(x)=E[Y\mid X=x]$. Bias and variance at $x$ describe the behavior of the prediction across repeated training responses, with the target $x$ held fixed.

The Local-Averaging Bias-Variance Tradeoff

To interpret the formulas, consider the regression model $Y_i=m(X_i)+\varepsilon_i$, where the errors have conditional mean zero. First suppose the predictor locations are fixed, and the errors for the observations in $N_k(x)$ are independent, each with variance $\sigma^2$. Under these assumptions, the expected prediction, conditional on those predictor locations, is the average of the corresponding regression-function values.

E[\widehat m_k(x)\mid X_1,\ldots,X_n] =\frac{1}{k}\sum_{i\in N_k(x)}m(X_i)

Consequently, its conditional bias for estimating $m(x)$ is the difference between that local average and the target regression value. If all the neighboring predictor values lie in a region where $m$ is nearly constant, this difference is small. If the neighborhood spans a region where $m$ changes substantially, the average can differ from $m(x)$.

\operatorname{Bias}\{\widehat m_k(x)\mid X_1,\ldots,X_n\} =\frac{1}{k}\sum_{i\in N_k(x)}m(X_i)-m(x)

The conditional variance follows from averaging independent errors. Each error has variance $\sigma^2$; multiplying each by $1/k$ multiplies its variance by $1/k^2$. Adding the $k$ independent variances therefore gives $\sigma^2/k$.

\operatorname{Var}\{\widehat m_k(x)\mid X_1,\ldots,X_n\} =\operatorname{Var}\left\{\frac{1}{k}\sum_{i\in N_k(x)}\varepsilon_i\right\} =\frac{1}{k^2}\,k\sigma^2 =\frac{\sigma^2}{k}

This variance calculation is for fixed predictor locations, independent errors, and a common conditional error variance. If errors are correlated, or have different variances, the variance of the average must instead include the relevant variances and covariances. If the predictor locations are themselves treated as random, the local average of $m(X_i)$ can also vary across training samples; $\sigma^2/k$ is then only the response-error component of the full variation.

Theorem: For a fixed target $x$, the conditional mean squared error for estimating $m(x)$ decomposes into squared conditional bias plus conditional variance: $$ E\left[(\widehat m_k(x)-m(x))^2\mid X_1,\ldots,X_n\right] =\operatorname{Bias}\{\widehat m_k(x)\mid X_1,\ldots,X_n\}^2 +\operatorname{Var}\{\widehat m_k(x)\mid X_1,\ldots,X_n\}. $$ For predicting a new response $Y_{\mathrm{new}}=m(x)+\varepsilon_{\mathrm{new}}$, add its conditional error variance when $\varepsilon_{\mathrm{new}}$ is independent of the training data. This is irreducible prediction noise, not variance of the fitted local average.

Why Neighborhood Width Affects Bias

A larger $k$ usually reaches farther from $x$, so its average includes responses from a broader region. When $m$ is curved, averaging across that region can move the expected prediction away from $m(x)$. A short approximation makes this effect visible.

Suppose predictor values are approximately uniform in a symmetric interval from $x-h$ to $x+h$, where $h$ is the neighborhood half-width, and suppose $m$ is smooth. A second-order Taylor expansion around $x$ is $m(x+u)\approx m(x)+m'(x)u+\frac{1}{2}m''(x)u^2$. Averaging over symmetric offsets removes the linear term because the average of $u$ is zero. The average of $u^2$ over this interval is $h^2/3$. Thus the approximate local-average bias is $\frac{1}{2}m''(x)h^2/3=m''(x)h^2/6$.

\operatorname{Bias}\{\widehat m_k(x)\} \approx \frac{m''(x)}{6}h^2

This approximation assumes a locally symmetric, roughly uniform neighborhood and a sufficiently smooth regression function. It is not a universal exact formula for any finite KNN sample. In particular, an asymmetric neighborhood can retain a first-order contribution, and irregular predictor density can change the effective averaging pattern. Subject to the stated conditions, increasing $h$ can increase the magnitude of curvature bias, while increasing $k$ reduces response-error variance under the independent, equal-variance assumptions.

Fix the target and the neighbors.
Identify the target $x$ and the $k$ training observations included in the local average.

Find the expected local average.
Average the values $m(X_i)$ for those neighbors, rather than their noisy observed responses.

Separate bias and variance.
Subtract $m(x)$ from the expected average for bias; use $\sigma^2/k$ for response-error variance only under the stated assumptions.

Compute the relevant error.
For estimating $m(x)$, add squared bias and estimator variance. For a new noisy response, also add its error variance.

Worked Calculations

Worked Example: Comparing Two Neighborhood Sizes

At a fixed target $x$, suppose the true regression value is $m(x)=10$. The first four neighbors, in order of distance, have conditional means $12, 11, 10,$ and $9$. Their independent response errors each have variance $9$. Compare the conditional estimation mean squared error using $k=1$ and $k=4$. Then calculate prediction mean squared error for a new response with independent error variance $9$.

Method. For each $k$, average the listed conditional means to find the expected local prediction. Its difference from 10 is bias. The response-error variance of the local average is $9/k$.

For $k=1$, the expected prediction is $12$, so the bias is $12-10=2$, and squared bias is $2^2=4$. The estimator variance is $9/1=9$. Therefore, estimation mean squared error is $4+9=13$. Adding new-response error variance $9$ gives prediction mean squared error $13+9=22$.

For $k=4$, the expected prediction is $(12+11+10+9)/4=42/4=10.5$. The bias is $10.5-10=0.5$, and squared bias is $0.5^2=0.25$. The estimator variance is $9/4=2.25$. Estimation mean squared error is $0.25+2.25=2.5$, and prediction mean squared error is $2.5+9=11.5$.

Check. Increasing $k$ lowered variance by $9-2.25=6.75$, while squared bias rose by $0.25-4=-3.75$, meaning it actually fell in this particular comparison. The computed estimation error fell from 13 to 2.5. This does not establish that larger $k$ always reduces bias; it reflects these particular neighboring means. A common mistake is to call the estimation mean squared error the prediction mean squared error and omit the new-response variance. Here the two prediction errors are 22 and 11.5, not 13 and 2.5.

Worked Example: Curvature Bias in a Symmetric Neighborhood

A hypothetical claim-cost response has conditional mean $m(x)=2x^2$. At target $x=0$, approximate the bias of a local average over a symmetric interval with half-width $h=0.5$, assuming predictor values are uniform within that interval. If the response-error variance is 1 and the neighborhood contains $k=12$ observations, calculate squared bias and the approximate estimation mean squared error.

Method. The function has second derivative $m''(x)=4$. Apply the symmetric-neighborhood approximation $m''(x)h^2/6$. Then use the conditional response-error variance $1/k$.

The approximate bias is $4(0.5)^2/6=4(0.25)/6=1/6=0.166666\ldots$. Its squared bias is $(1/6)^2=1/36=0.027777\ldots$, or 0.02778 to five decimal places.

The response-error variance of the average is $1/12=0.083333\ldots$. Thus approximate estimation mean squared error is $1/36+1/12=1/36+3/36=4/36=1/9=0.111111\ldots$.

Check. The squared bias, approximately 0.02778, is smaller than the variance, approximately 0.08333; their sum is approximately 0.11111. The approximation concerns the local average of the regression function under the stated symmetric uniform design, while $1/12$ accounts for response noise conditional on predictor locations. It does not include extra variation caused by random neighbor locations.

Worked Example: Keeping the Sign of Bias Straight

At a fixed target, suppose the true regression value is $m(x)=20$. The expected average of the four selected neighbors' responses is $20.0625$. Their independent response errors have variance $4$, and a new response at $x$ has independent error variance $1$. Calculate the conditional bias, squared bias, local-average variance, estimation mean squared error, and prediction mean squared error.

Method. Bias is expected prediction minus target regression value. The average's variance is $4/4$, and the new-response variance is added only for prediction error.

The signed bias is $20.0625-20=0.0625$. Its square is $0.0625^2=0.00390625$. The local-average variance is $4/k=4/4=1$. Therefore, estimation mean squared error is $0.00390625+1=1.00390625$.

For prediction of a new response, add its independent error variance of 1: $1.00390625+1=2.00390625$.

Check. The bias is positive because the expected local average, 20.0625, exceeds the target value, 20. The sign does not affect squared bias, but it does matter when describing the direction of error. A common arithmetic trap is to reverse the subtraction and report $-0.0625$; the defined order is expected prediction minus $m(x)$.

Interpreting k Without Overstating the Rule

The variance formula $\sigma^2/k$ gives a precise reason that averaging more independent noisy responses tends to stabilize a KNN regression estimate. The corresponding bias depends on the regression function values within the neighborhood. A small neighborhood can stay close to $m(x)$ when the function is smooth locally, but the estimate then averages fewer errors. A broad neighborhood averages more errors but may combine substantially different conditional means.

The direction of the tradeoff is therefore a useful expectation, not a theorem that bias increases with every increment in $k$. The selected neighbor set changes discretely as $k$ changes; a newly included observation may have a conditional mean that moves the average closer to $m(x)$. Also, predictor density and neighborhood shape affect which values are averaged. Total mean squared error balances both components, so neither the smallest nor the largest possible $k$ is automatically best.

Choice of k	Typical effect on local average	Important qualification
Small k	Uses a narrow set of observations; response-error variance is larger under independent equal-variance errors.	Can retain local detail, but may be sensitive to individual noisy responses.
Large k	Averages more responses; response-error variance $\sigma^2/k$ is smaller under the stated assumptions.	Can smooth over changes in $m(x)$; bias depends on the actual neighboring means.

For estimating the regression function, the relevant conditional error is squared bias plus variance. For predicting an individual future response, there is an additional irreducible error component from that response's own random deviation around $m(x)$. Keeping these targets separate prevents a common error: treating a stable estimate of the conditional mean as if it implied an equally stable individual outcome.

Key takeaway: Increasing $k$ reduces the response-error variance of an ordinary local average from $\sigma^2/k$, provided errors are independent with common variance. At the same time, a broader neighborhood can increase smoothing bias when the regression function changes across it. Compare both components for the target of interest; do not assume either component moves monotonically in every finite sample.

Check Your Understanding

For each problem, identify whether the requested error concerns estimating the regression function or predicting a new response.

At a fixed target, four selected responses have conditional means $5, 7, 6,$ and $6$, and $m(x)=6$. Find the conditional bias of their local average. (Answer: $0$)
Independent response errors have variance $18$. Under the fixed-predictor, equal-variance assumptions, find the response-error variance of an ordinary KNN average with $k=6$. (Answer: $3$)
A local average has bias $-0.4$ and variance $0.75$. Find its estimation mean squared error. (Answer: $0.16+0.75=0.91$)
A local average has squared bias $0.04$ and variance $0.5$. An independent new response has error variance $2$. Find prediction mean squared error. (Answer: $0.04+0.5+2=2.54$)
At a target with $m(x)=8$, the expected local prediction for $k=3$ is $7.5$. Independent response errors have variance $6$, and the new response has error variance $1.5$. Find signed bias, squared bias, estimator variance, and prediction mean squared error. (Answer: bias $=-0.5$; squared bias $=0.25$; estimator variance $=6/3=2$; prediction mean squared error $=0.25+2+1.5=3.75$)

What You'll Learn